Assessing the Business Value of New AI Initiatives: A Strategic Framework

Integrating artificial intelligence (AI) into all areas of business is critical to a company’s ability to gain—or maintain—a competitive edge. Organizations report increased revenue and decreased costs in the business functions where they’ve implemented AI, according to 2023 McKinsey & Company research. And two-thirds of company representatives surveyed expect to do more AI integration in the next few years.

AI can transform operations, streamline processes, enhance decision-making, and drive innovation. However, the success of AI initiatives hinges on an organization’s ability to select the right projects—ones that align with their strategic objectives and can deliver tangible value.

In my years as a management consultant at Bain & Company, I witnessed a significant number of businesses launch AI projects that either failed to be completed, failed to be deployed to production, or failed to deliver the expected outcomes. And now, as Toptal’s Artificial Intelligence and Data Analytics Practice Lead, I hear from enterprise and startup clients who say that even when a past AI project succeeded, it often took far longer to complete than they had initially imagined.

In my experience, there is generally a disconnect between internal data science teams and the rest of the business, which can lead to imprudent investments in AI. I recall one large insurance client telling me about an experience he’d had before coming to Toptal: The company was interested in leveraging AI to optimize their call center operations. And their in-house data scientists, excited by the potential cost savings of predictive staffing, built an extremely accurate model to forecast the number of calls by call type.

But the business was ultimately unable to act on the information. The forecast data was too granular and the forecast window too short for managers to enact realistic staffing changes to the call center. After the data team adjusted for practical business requirements (such as requiring enough time to have schedules released and reviewed by their staff; allotting enough time to recruit, hire, and train reps; and accounting for a lack of flexibility in continuous hours staffed), there was limited value available to optimize the call center’s operations.

The high failure rate and longer-than-expected timelines of many AI initiatives underscore the need for organizations to adopt a more strategic and systematic approach to evaluating these opportunities. By conducting thorough assessments and due diligence before embarking on an AI project, organizations can increase the likelihood of success and maximize their return on investment (ROI). The framework my team uses to evaluate AI opportunities addresses these issues directly, offering leaders a pragmatic method for classifying and prioritizing AI projects. Our analysis evaluates initiatives based on two main factors: the value they deliver and the ease of implementation.

Assessing the Potential Value of AI Projects

Identifying the value associated with a potential AI project involves aligning a company’s AI initiatives and strategic goals, estimating financial impact, and understanding the opportunity cost of not embracing AI in the given use case. Value must be assessed upfront, prior to kicking off an AI initiative, to avoid potentially wasting money on an unnecessary project simply because it seems cool or trendy. For example, with the rise of generative AI (Gen AI) tools and chatbots, I witnessed many companies jump into building their own versions—and then struggle to demonstrate value and drive adoption. On the other hand, companies I’ve seen succeed with building Gen AI tools took a systematic approach and first identified areas of their business where Gen AI could reduce costs and increase productivity.

Data scientists will always be eager to explore and build with cutting-edge technologies, but they need coaching from business leaders on exactly which problems need to be solved. The best place to start is with the financials.

What Is the Financial Impact?

Estimating the financial implications of AI projects is essential for evaluating their potential value and determining the ROI. To do this, organizations can use various methods, including cost-benefit analysis, ROI calculations, and scenario modeling. It is crucial to strike a balance between short-term gains (such as cost savings and efficiency improvements) and long-term potential (such as revenue growth and market expansion). For example, with Gen AI, many businesses focus on near-term cost savings by increasing automation. That ROI is easy to calculate, but thinking through the financials associated with the development of a new AI-powered product, while more complex, is key.

Does the Project Align With Organizational Strategy and Goals?

AI projects aligned with strategic goals offer opportunities for true cost reduction, innovation acceleration, and ROI enhancement, transforming business operations and positioning organizations for sustained growth and success in a rapidly evolving marketplace. Recently, my team worked with an AI executive at a chemical manufacturer where the CEO had mandated margin improvements. We narrowed our focus to identify AI projects in pricing and supply chain that would significantly impact the margins on deals—from finding areas where products were priced too low to reducing inefficiencies in the supply chain. Focusing on the strategic business goals allowed the executive to get buy-in and budget to implement several AI initiatives, even while other business units in the organization faced cuts.

What Are the Opportunity Costs?

Organizations that fail to embrace AI risk falling behind competitors who leverage AI technologies to drive efficiency, productivity, new products, and increased customer satisfaction. When evaluating a particular AI project, you should ask, “What would the impact be if our closest competitors were successful in doing this? Would they take market share? Would they be able to serve at a lower cost? Would they provide a more desirable offering?” This evaluation is more important than ever as Gen AI has opened the door for less AI-savvy companies to start developing powerful AI solutions quickly.

How Difficult Will the AI Project Be to Implement?

Ease of implementation is key when evaluating whether a particular AI initiative is worth pursuing for your organization. The more difficult a project is to execute, the more lengthy and expensive it will be in terms of time, resources, executive goodwill, and money. To evaluate difficulty, consider the availability of off-the-shelf tools, data availability and quality, technology feasibility, project complexity, and stakeholder involvement.

Off-the-Shelf Tools

Leveraging pre-built AI platforms and technologies can greatly expedite the implementation process, reduce development time, and lower costs associated with custom development. The decision to use an off-the-shelf foundation model like OpenAI’s GPT-4 versus building a proprietary large language model is fairly straightforward for most businesses; the vast amounts of data required to build a machine learning (ML) model of that kind are prohibitive, and the existing tools are very good. However, the choice between buying or building an AI tool to, say, automatically classify expenses may be more challenging.

Organizations should assess the suitability of off-the-shelf tools based on a buy versus build analysis. The key question to consider is whether the problem your company is solving is core to your competitive advantage. If it is, you will most likely want to build to ensure you are first-in-class in your domain. If it’s not, and there are affordable tools that fit your needs, then “buy” is probably the best option.

Data Availability and Quality

Adequate access to relevant and high-quality data is essential for training AI models, validating results, and deriving meaningful insights. Without quality data, much of the project’s resources and effort will be spent preparing and cleaning data, leading to lower project success rates.

Data readiness should be evaluated on the ability to provide an adequate signal to the problem at hand—and the ability to operate accurately in a production environment. Sometimes data scientists will clean data and build an ML model that can accurately predict a key outcome in a testing environment. However, once deployed to active usage in a production environment, ML engineers may discover issues such as data coming in different intervals, data not being available as quickly as it is needed, or data that is missing entirely.

Note that even if all the data you want or need isn’t immediately available, that doesn’t mean a project should be scrapped. Often, cleaning a single source of data can unlock several different AI opportunities. For example, I was advising a consumer product brand on building a customer data platform (CDP), but the client was struggling to get the necessary funding. By identifying the multiple high-value AI opportunities that would become available once there was a central and trusted customer data source (including personalized marketing, promotion optimizations, cross-selling, and more), we were able to get the project budget approved and fast-tracked.

Technological Feasibility and Project Complexity

Assessing the technological feasibility of an AI project involves understanding the technical requirements, limitations, and infrastructure needed to support it. Organizations should evaluate factors such as computational resources, data storage capabilities, software compatibility, and security considerations to ensure that AI initiatives can be implemented effectively and sustainably.

Complexity can be evaluated in multiple ways, from the kinds of AI models required to the organizational and business dimensions needed to capture value. The more complex an AI initiative is, the lower the likelihood of success. Not only do more complex projects require more resources, but they also deliver zero value if your business is unable to change or adapt to new processes, regardless of how amazing or groundbreaking the technology is.

My general rule is to start simple, if possible. In my experience, if simple data analysis or traditional ML approaches, like regression, are unable to find significant signals from the data, the probability of a more complex technology like deep learning being able to provide accurate and meaningful results is also diminished. One exception to this is when a task involves computer vision or natural language processing (NLP), as deep learning is required to capture the nuanced relationship of language and vision. And in those cases, your teams should start with an off-the-shelf model, such as OpenCV for vision or BERT for NLP.

Stakeholder Involvement

The ability to engage key stakeholders and develop organizational buy-in is critical to smooth AI implementation.. Most of the value associated with an AI project will come from organizational and process changes around AI modeling. For example, if data scientists can successfully create an accurate pricing model, business leaders must be able to take action based on the model. They will need to ensure the business can operationalize dynamic pricing, identify risk from model-based pricing, and train sales reps and managers to use the model in their negotiations. Organizational change can be difficult, and having supportive stakeholders makes it more likely that your initiative will fully capture value.

Toptal’s Framework for Evaluating AI Opportunities

After considering each of the above factors, the next step is to categorize AI initiatives into quadrants based on the potential value and ease of implementation. This exercise helps organizations prioritize projects that offer the highest ROI and have the highest likelihood of being successful and delivered on time. The four quadrants are:

• Low value: Easy projects that don’t have a substantial business upside.
• Easy value: Projects that can be quickly implemented and deliver immediate value to the business.
• Money pits: Long, challenging projects that don’t justify the investment.
• Moonshots: Projects that have the potential to deliver substantial value to the business, but will also require a more substantial investment.

There are both quantitative and qualitative approaches to using this framework. When working with clients, I conduct surveys of business leaders, product managers, and data scientists to rank and quantify opportunities across different dimensions. If I’m taking a more qualitative approach, I’ll lead workshops to quickly identify potential projects and ask the people in the room to use visual collaboration tools like Miro to place projects where they think they belong. If there are big gaps on value or feasibility, we’ll have an open discussion about why.

Moving Forward With Your High-value AI Initiatives

While it is easy to prioritize “easy value” projects, I argue that putting all your focus in that quadrant may be shortsighted. In addition to pursuing one or two of those initiatives, you should also look into your moonshots to understand what it is that makes these projects more challenging. In my experience, more often than not, data is the biggest roadblock. That’s when you have to ask yourself: If you can improve one or two data sources, will it unlock significant, long-lasting value—like a robust CDP did for our consumer product client?

The successful integration of AI into business operations is key for organizations seeking a competitive advantage today. However, the high failure rate of AI projects necessitates a strategic and systematic approach to selecting and prioritizing initiatives. The Toptal approach of balancing value with ease of implementation ensures that high-value AI initiatives are identified and successfully deployed, thereby maximizing ROI and securing sustained executive buy-in for ongoing AI endeavors. In forthcoming articles, we will build upon the output of this analysis to develop an AI roadmap that maximizes long-term value to the organization, ensuring that the strategic deployment of AI continues to drive innovation and growth.

Have a question for Chas or his team? Get in touch.