Don’t make the consultants fool you!
Checklist for AI product managers to get the most from the design sprints
How to satisfy all stakeholders with a great product
Making buzzwords as “digitalization”, “innovation” and “big data” into the alive and profitable product is hard. And usually, it’s not the technology itself that fails, but the alignment between owners, managers, clients, employees, and sometimes the society. With AI products it’s even harder because this is a relatively new field where there are predominant:
- either generalist consultants who can tell about bright future and economic impact without concrete numbers and next steps;
- or deep experts, who know the potential and limitations of the technology, but they lack systematic business and social vision about it.
In this article, I want to share the approach that we have crafted with partners at Neurons Lab, that merges user-centric algorithms, economic growth, and social impact which aligns all the stakeholders involved in the product lifecycle. It is fundamentally based on the idea of design sprints, however, the main steps are re-built from scratch to match the specifics of the industry.
Design sprints for AI?
The main idea of original design sprints is to validate ideas rapidly: understanding market needs, brainstorm new ways to solve the problem, prototype first digital solutions, and test them with customers. Ideally, it even can be done within 5 days. The typical deliverables from such event are:
- Findings from the brainstorms (user stories, technology applications, prioritized novel use cases)
- An interactive prototype, that demonstrates the use case and is validated with the stakeholders
- A strategic roadmap for development with needed resources and risks
You can read more about the history and best practices of these sprints here. These results usually satisfy all important stakeholders as owners, managers, and customers of traditional digital products. In the next sections, I will describe activities and practices, that will create the above-mentioned deliverables that will align everyone involved in the development and use of the product that has artificial intelligence technology as a core.
What customers want
Quoting Daft Punk, your customers want to solve their problems:
Harder, better, faster, stronger
AI-related technologies indeed can make things more accurate, faster, remove routines or dangerous tasks from our lives. But how to formalize what exactly we can do for our customers here? I can recommend 2 frameworks that are used at Google:
- AI Canvas from “Prediction Machines” book. It will help you to turn customer ideas into user stories that use AI
- People + AI Guidebook. It will help to define requirements for data, metrics, explainability, and support for the above-mentioned user stories
You also want to show some demo and prepare it in hours, not weeks, right?
- Easy deployment tools (BentoML, Flask, Streamlit)
- No-code (as Lobe) / managed solutions (AWS, Azure, GCP, etc)
- Open-sourced ML models zoos as PapersWithCode, ModelZoo
As a result here, you will have a prioritized list of use cases with a couple of small demos demonstrating to the users how it’s gonna look like.
What business wants
Of course, a user-centric approach is a core, but the product also has to be profitable and be competitive in the market from the business model point of view. This is usually missed in the design sprints, which is a horrible overlook from my point of view. This part is extremely customized, but you want to narrow it down to the unit economics of a single prediction. Using AI Canvas from the “Prediction Machines” mentioned above, you can calculate the economic effect of using the AI feature:
- How much you’re going to save with each prediction thanks to the speedup/quality improvement/risk decrease?
- What is the accuracy you can expect and with what certainty?
and you will need to subtract the related costs, which can include but are not limited to:
- Development costs (in-house / consultants / outsource)
- Maintenance, human support, and cloud costs with the risks
This process is tricky and I recommend working on it with experts who have delivered 10+ projects in your domain area. As a result, you will have a business model draft and re-prioritized use cases based on the economics — maybe your favorite case gets profitable 5 years from now and you need to get a second one that gets faster to the market!
What manager wants
Okay, now we aligned users’ and business’ needs, but still, some people are going to be responsible for delivering this to the market. How should they monitor development? What metrics should they use? Which resource burn rate is acceptable, and which is not? How to manage R&D-related uncertainty? How to deliver in a flexible and Agile manner in such a way that the customer is happy and the business doesn’t burn additional capital?
Classic R&D and data science processes as CRISP-DM or TDSP consider isolated work on data, modeling, and deployment and the client has to wait for months while
“data science team is working with data and training the models, it’s highly unpredictable process ©”
Sounds a lot like an excuse, right? The alternative Agile Data Science approach treats every smallest R&D phase as a sub-product (see the pyramid on the picture above) that should deliver immediate value for the client. What kind of user stories and related metrics we can develop?
- Data acquisition: if the customer is not collecting any data yet, you need to define how data collection and organization immediately simplifies life and some processes (business metrics)
- Data exploration: primitive statistical insights should lead to primitive predictive decisions and rule-based systems that already can automate some work! Check already the client-based metrics from the first section
- Modeling: while making better and better models check both business and client-based metrics, you should arrive at the point of profitability somewhere at this stage with a deployed model
- Actionable insights: well, this is what everyone wants at the end :)
What colleagues want
We, managers, like transparent and predictable processes, but someone has to do actual work behind the master plan :) There are different motivations to do meaningful work but according to this structure, they are Autonomy, Mastery, and Purpose.
- Autonomy can be achieved via delegating more responsibility and decision making and switching from task-tracking to results-tracking systems. Decentralized organizational structures are something that you want to look for
- Mastery is coming from fundamental knowledge and state-of-the-art skills and technologies. The second one can be planned already using portals like PapersWithCode and AI Index, where you can find the latest developments in most areas
- Purpose is a bit more sensitive matter, but usually, it is related to giving the world something that it needs. The next section is about it in the context of AI, and your business strategy and culture has to be responsible as well :)
This is a non-traditional approach to think about the colleagues in the design sprint, however, for modern human-centric organizations, it is a must.
What the world wants
Artificial intelligence is a technology that comes with drawbacks — same as human intelligence, it can be biased, vague, and prone to external attacks. What can we do to make the adoption of AI right and purposeful? At the level of a design sprint, we need to make stakeholders aware of such problems, define sensitive moments in the products and offer potential solutions:
- Fair and unbiased AI: I recommend using “What-if-tool” from PAIR and a more technical Themis-ML library for fairness inspection
- Explainable AI: for interpreting ML predictions there are already multiple strategies and solutions as TCAV and SHAP
- Protected AI: you definitely need to protect your training data on the infrastructure level, and, potentially with federated learning (TF, PyTorch). Also, machine learning models can be attacked with adversarial attacks that push models to make wrong predictions. Check out these libraries as potential solutions.
Takeaways
In this article, we have re-evaluated the classic design sprint structure and deliverables for AI-based products. As the main idea, we have reviewed what each project stakeholder wants to be satisfied with the project during development and while exploiting it:
- End customers: a solution that delivers a tangible positive change with an interactive demo
- Business owners: a business model with a single prediction unit economics, burn rate, and associated risks
- Managers and employees: clear process and metrics for the first ones and fulfillment from the work for the latter
- Society: a great AI technology application that is fair, transparent, and protected from misuses
I didn’t mention marketing and competitor analysis that is sometimes done within design sprints, since I expect extensive competitor and market analysis to be done beforehand. Design sprints are great for brainstorming and testing different ideas of technology products, but defining the general vision and strategy of a company is a bit different topic. Let me know if you have any questions or if you want to share your experience of doing such design sprints. Good luck with your AI products!
P.S.
If you found this content useful and perspective, you can support me on Bitclout. Follow me also on Facebook for AI articles that are too short for Medium, Instagram for personal stuff, and Linkedin! Contact me if you want to collaborate on design sprints or other ML projects.
