Choosing an AI Project: Define the Decision, Evidence, and Success Criteria Before the Model
Choosing an AI project requires leadership teams to step away from model architectures, parameters, and algorithms to first examine three foundational questions: what decision must the product make, what evidence exists to support that decision, and what baseline proves that machine learning is genuinely required. Framing the core objective in plain, non-machine learning terms and testing simple heuristics first ensures that technical investments target genuine operational needs rather than unvalidated assumptions.
Before writing a single line of training code, engineering leaders benefit from aligning product requirements with rigorous scoping protocols. Establishing analytical discipline early—such as through thinking like a data scientist as a strategic decision maker—helps cross-functional teams avoid mistaking algorithmic complexity for product progress.
Stating the Goal in Non-Machine Learning Terms
The first discipline in evaluating any prospective artificial intelligence or machine learning initiative is answering a deceptively simple question: “What am I trying to accomplish?” According to Google’s foundational guidance on ML problem framing, teams must define a clear, non-ML goal for their product or feature before considering any technical modeling solution. When business leaders define projects through algorithmic language—such as “build a multi-modal neural network” or “deploy an LLM agent”—they obscure the actual operational problem and make objective evaluation impossible.
A non-ML objective defines the intended real-world utility of the system rather than the technical pipeline that delivers it. Google illustrates this distinction through standard product objectives across multiple domains:
- Weather applications: Calculate precipitation in six-hour increments for a geographic region.
- Financial applications: Summarize financial information gathered from multiple news sources.
- Messaging platforms: Detect spam messages reliably.
- Mapping services: Calculate travel time between destinations.
- Banking systems: Identify fraudulent financial transactions.
- Commerce platforms: Reply to customer reviews with helpful answers.
- Media streaming: Recommend useful videos to viewers.
- Dining applications: Identify cuisine types directly from a restaurant’s menu.
- Fashion applications: Generate a variety of shirt designs.
By forcing the objective into everyday language, stakeholders can evaluate whether a proposed system solves a high-priority problem before allocating engineering and infrastructure capital. If an engineering team cannot state what the system accomplishes without using machine learning terminology, the initiative is not yet ready for technical scoping.
Deciding Between Predictive ML, Generative AI, and Non-ML Approaches
Machine learning is not a universal mechanism suitable for every software challenge; it is a specialized tool tailored to specific problem spaces where deterministic logic is inadequate. When an engineering team considers an advanced approach, they face a three-way architectural decision: implement a deterministic non-ML solution, build a predictive ML system, or deploy generative AI.
Predictive ML and generative AI operate on fundamentally different inputs, training methods, and evaluation criteria, as summarized below based on Google’s problem-framing taxonomy:
| System Category | Input Formats | System Output | Training Technique |
|---|---|---|---|
| Predictive ML | Text, Image, Audio, Video, Numerical data | Makes an explicit prediction (e.g., classifying email as spam, guessing tomorrow’s rainfall, predicting a stock price) that can typically be verified against ground-truth reality. | Uses datasets to train a supervised, unsupervised, or reinforcement learning model designed to perform a distinct, specialized task. |
| Generative AI | Text, Image, Audio, Video, Numerical data | Generates novel content based on user intent (e.g., summarizing an article, producing an audio clip, generating a short video). | Trains large foundational models (such as large language models or image generators) on extensive unlabeled data to fill in missing information, with optional fine-tuning on labeled datasets for specific tasks like classification. |
Predictive machine learning is best suited for scenarios where a model’s output must be directly validated against empirical real-world outcomes, such as fraud detection, equipment failure classification, or demand forecasting. Generative AI is suited for synthesis, creative variation, and natural language communication where the prompt expresses an intent and the system creates content to satisfy it. However, if the operational goal can be executed through rule engines, conditional logic, or database lookups, deploying either category introduces unnecessary operational fragility.
Establishing the Non-ML Heuristic Baseline
Before committing engineering resources to machine learning, teams must optimize their existing non-ML solution or construct a manual heuristic to serve as an empirical benchmark. Without a baseline, evaluating whether an ML model delivers meaningful incremental performance is impossible. A heuristic represents a rule of thumb or basic formula that approximates an answer without automated statistical learning.
Google outlines a direct scenario illustrating this dynamic: an engineering team managing incoming call center traffic aims to inform callers how long they will wait on hold given current call volume. Lacking an existing system, the team establishes a baseline heuristic: divide the number of callers currently on hold by the number of active call center agents, then multiply the result by ten minutes. However, because actual support calls fluctuate widely—some resolving in two minutes and others exceeding 45 minutes—the heuristic lacks precision. In this case, an ML model trained on historical hold times and call variables can be benchmarked directly against that simple multiplication rule to confirm whether its superior precision justifies implementation.
When comparing a non-ML baseline to a proposed ML model, teams must evaluate two critical dimensions:
- Quality Differential: How much performance improvement can an ML solution realistically provide over the optimized baseline? If the expected uplift is modest, the baseline heuristic remains the superior operational choice. Small percentage gains may only justify adoption in massive systems where incremental performance aggregates into substantial organizational value.
- Total Cost and Long-Term Maintenance: Implementing machine learning introduces significant ongoing compute, training, and operational costs. Beyond initial model development, machine learning systems require dedicated long-term maintenance, monitoring pipelines, and access to personnel with specialized ML expertise. If the expected operational gains cannot offset ongoing infrastructure and staffing costs, the project should not proceed.
Assessing Data Readiness and Feature Predictive Power
Predictive machine learning models cannot generate reliable outputs from inadequate inputs. Assessing data readiness requires systematically evaluating whether internal data assets satisfy six foundational characteristics identified by Google:
- Abundant: Models require a substantial volume of relevant and useful training examples to discern generalizable patterns.
- Consistent and Reliable: Data must be gathered systematically over time through stable instrumentation and verified operational processes, avoiding erratic shifts in logging or formatting. This level of consistency prevents the data collection problems that ruin machine learning before modeling begins across production telemetry pipelines.
- Trusted: Engineering teams must know the exact origin of their training data. Data sourced from first-party operational logs offers higher traceability than opaque inputs generated by external secondary systems. Similar governance rigor is required across internal monitoring infrastructure, reflecting principles found in why data science projects fail and how to build a high-ROI data culture to maintain auditability.
- Available at Prediction Time: All input variables used during training must be fully accessible in the correct schema at the exact moment a prediction is generated. If an attribute cannot be retrieved reliably in production, it must be excluded from training datasets entirely.
- Correct: Label accuracy is paramount. While minor noise is inevitable in massive datasets, having more than a small percentage of incorrect labels will degrade model predictions.
- Representative: The training dataset must mirror real-world operational distributions, user behaviors, and environmental variations. Training on unrepresentative data leads directly to performance failures when the system encounters production environments.
Beyond data cleanliness, the dataset must possess features that exhibit genuine predictive power relative to the target label. Predictive power measures the strength of the mathematical relationship between an input feature and the outcome being predicted. In a weather forecasting dataset, features such as cloud coverage, atmospheric temperature, and dew point demonstrate clear predictive power for precipitation, whereas attributes like moon phase or day of the week offer negligible correlation. Similarly, for a video recommendation system, attributes such as video description, content length, and past view counts offer plausible predictive signals for user engagement.
Determining predictive power can be conducted through manual feature engineering—adding and removing variables iteratively during model training—or through automated mathematical algorithms. Specifically, teams can compute the Pearson correlation coefficient for linear relationships, Adjusted Mutual Information (AMI) for non-linear feature interactions, and Shapley values to calculate the marginal contribution of individual features to the overall prediction output.
Bridging Predictions and Concrete Actions
A machine learning model has no intrinsic value if its predictions cannot trigger a concrete, beneficial action. Mathematical accuracy alone does not produce business value; the surrounding software system must be designed to consume the model’s output and execute a decision that assists the end user.
Google emphasizes this principle through direct product pairings: an algorithm that predicts whether a user will find a video engaging must feed directly into a recommendation engine that surfaces that video in the user interface. Similarly, a predictive weather model must integrate directly into a consumer application that alerts users to prepare for rain. If an organization constructs a high-performing classification model but lacks the operational integration, user interface hooks, or automated mechanisms to act on those classifications, the model remains an unutilized cost center.
Strategic Stopping Criteria: Editorial Recommendations for Governance
Based on the operational dependencies outlined in Google’s problem-framing principles, leadership teams should establish formal stopping criteria before allocating significant technical capital to model development. Rather than treating project continuation as the default state, decision-makers can protect organizational focus by establishing specific checkpoints where an initiative must be paused or discontinued:
- The Heuristic Parity Check: If an optimized rule-based heuristic or simple statistical formula achieves performance comparable to early ML prototypes, pause model development. Given the ongoing compute, pipeline monitoring, and maintenance demands of machine learning, a non-ML solution that meets business requirements represents a lower-risk, lower-cost production asset.
- The Data Latency Gate: If key features that provide predictive power during offline model exploration cannot be delivered reliably to the inference service within production latency requirements, halt predictive modeling until the data engineering pipelines are mature.
- The Predictive Power Threshold: If formal evaluations using Pearson correlation, Adjusted Mutual Information, or Shapley values show that available data attributes lack meaningful statistical correlation with the target label, suspend the project rather than attempting complex model architectures on noisy signals.
- The Workflow Actionability Review: If the downstream software architecture lacks an automated or human workflow capable of taking immediate operational action on model outputs, pause development until the business process is restructured to ingest predictions.
AI Project Feasibility Checklist
Before authorizing engineering resources for machine learning model development, ensure the prospective project satisfies each criterion derived from Google’s problem-framing framework:
- [ ] Non-ML Goal Defined: The primary objective is articulated in plain business terms that answer “What am I trying to accomplish?” without technical modeling jargon.
- [ ] Paradigm Verified: The problem has been evaluated against deterministic logic, predictive ML, and generative AI, selecting the simplest architecture that addresses the use case.
- [ ] Baseline Established: A non-ML baseline or heuristic exists, and the projected ML performance gain justifies long-term compute, operational, and maintenance expenses.
- [ ] Data Quality Confirmed: Training data has been validated across all six criteria: abundant, consistent, trusted, available at prediction time, correct, and representative of production environments.
- [ ] Predictive Power Quantified: Input features demonstrate verifiable predictive power for the target label, supported by correlation or feature importance metrics.
- [ ] Actionable Output Path: The product architecture contains concrete downstream mechanisms to translate model predictions into timely, valuable actions for the end user.
The Decision Maker's Handbook to Data Science
The definitive guide for non-technical executives and entrepreneurs on navigating AI, machine learning, and big data.