For most of the software era, companies operated with a simple belief: if a feature mattered enough, engineers could build it. AI changes that—now the question is not just how much effort but whether the feature is even possible at the required quality level. A model that fails as a decision-maker may succeed as a drafter, filter, ranker, or warning signal. The key insight: not every desired feature is a build problem; some are a fit problem. The same AI capability can be a brilliant product in one role and a dangerous liability in another.
Writing requirements for probabilistic products requires all three disciplines working together: AI PM defines the USID.O dimensions (Uncertainty, Scope, Intent, Degree, Outcome) that make requirements evaluable rather than ambiguous; Vibe-Coding simulates different requirement formulations to see which ones produce testable, measurable outcomes against real AI behavior; AI Engineering translates requirements into measurable thresholds, eval pipelines, and monitoring systems that track whether the product meets specifications in production.
Vibe-coding enables requirements simulation before writing formal specs. Quickly prototype different PRD formulations to see which ones produce testable outcomes. If an eval-first requirement seems too strict or too loose, vibe-code variations and test them against real AI behavior. Vibe-coding lets you stress-test your requirements against actual AI capabilities, revealing gaps and ambiguities before they become costly implementation errors.
Objective: Master the eval-first approach to writing requirements for AI products.
Chapter Overview
Split from current AI PM Toolkit. This chapter covers the USID.O framework, eval-first PRDs, LLM-as-Judge, and requirements that embrace probabilistic behavior.
Four Questions This Chapter Answers
- What are we trying to learn? What constitutes a real spec for a probabilistic product, including acceptable error rates and evaluation criteria alongside functional requirements.
- What is the fastest prototype that could teach it? Writing an eval-first PRD for one AI feature that defines success criteria, acceptable failure modes, and measurement approaches before writing any code.
- What would count as success or failure? A requirement document that enables objective evaluation of whether the AI feature meets user needs, not just whether it produces outputs.
- What engineering consequence follows from the result? Eval-first requirements prevent the common failure of shipping AI features with no measurable success criteria.
Learning Objectives
- Apply the USID.O framework to AI products
- Write eval-first PRDs
- Use LLM-as-Judge for requirements validation
- Define acceptance criteria for AI features