A principal developer in your back pocket.

NASA engineers measured what it costs to fix a mistake depending on when someone catches it. Three separate ways of estimating gave the same answer: the cost does not creep up, it climbs exponentially the later the mistake is found. Their conclusion was that early detection of errors is highly important to the success of any project (Stecklein et al., INCOSE, 2004).

That study looked at aerospace-class projects — spacecraft, military aircraft, satellites — so we cite the shape of the curve, not a number borrowed from a different kind of work.

It is also the reason Pogo does its questioning before any code exists. The cheapest moment to change your mind is while the plan is still a conversation.

Researchers took a standard set of programming problems and rewrote the descriptions to be ambiguous, inconsistent, or incomplete — 762 of them — then handed them to code-writing AI models. More than 60% of the responses generated code anyway instead of asking a single clarifying question. On those flawed descriptions, the pass rates of most models tested fell by 35–52% relative to their scores on the clean ones (Wu & Fard, ACM TOSEM, 2025). The models in that study were the 2024 generation.

A 2026 preprint ran its own version of the test across six models and found the same thing everywhere: ambiguity degraded all six. Worse, the models disagreed with each other — given the same vague sentence, different models built functionally different things, in up to 57% of cases. The models were fairly good at noticing that something was unclear, but only about half of what they flagged was genuinely ambiguous, and they could not reliably resolve it on their own (Yang et al., arXiv preprint, 2026).

So the gap is not that your coding tool is careless. It is that building is what it does, and nobody asked it to stop. Pogo refuses to generate anything until an interview has settled the plan.

The obvious shortcut is to ask the model to clean up its own vague description. Researchers tried exactly that, and reported that it “often produces irrelevant or inconsistent edits.” What worked instead was repairing the description in a structured way: pass rates rose about 31% on the problems that got repaired, and a description repaired for one model went on to help other models too (Jia et al., arXiv:2505.07270, 2025). Their repair was automated, not an interview — the finding we take from it is narrower and firmer: structured work on the description beats one-shot prompting.

A separate study looked at seven state-of-the-art systems that chain several AI agents together, and found failure rates ranging from 41% to 86.7%. The largest single category of failure was not the models being insufficiently clever — it was system design and specification, about 44% of the failures the authors annotated. Improving the specification alone, with the same prompt and the same model, improved the outcome (Cemri et al., arXiv:2503.13657, 2025). The authors are careful about the size of that gain, and so are we: it is a real improvement, not a dramatic one.

A clear plan is not a nicety you do first because it feels tidy. It is the measured difference between what you asked for and what gets built.

If a plan is what matters, the tempting fix is a template: one long checklist everybody fills in. That was tested. In a controlled experiment, the industry-standard checklist for reviewing requirements found no more faults than reviewers working with no method at all — and the result has been replicated at least three times since (Porter, Votta & Basili, IEEE Transactions on Software Engineering, 1995). Generic checklists do not work. Questions have to be specific to the situation in front of you.

That is also what separates experienced interviewers from inexperienced ones. Experts pick their approach from the situation — who is involved, how reachable they are, whether something already exists. Novices reuse the one technique they happen to know (Hickey & Davis, IEEE Requirements Engineering, 2003). That finding comes from interviews with nine expert practitioners, so we treat it as expert practice rather than a laboratory result.

There is also a map of what expert interviewers actually cover: the people involved, what they need, the processes, the data, the functions required, and the technology (Zaremba & Liaskos, IEEE Requirements Engineering, 2021). Those authors are candid that the field has little hard evidence about which question types produce better outcomes, so we use their map as coverage for ourselves — a way to check we have not left a whole dimension unasked — and not as proof that our questions are the best ones.

Pogo’s interview adapts every batch of questions to your idea and to what you have already answered, and only asks about a choice that would materially change the first build.

Carnegie Mellon’s Software Engineering Institute built a method for evaluating a design before anyone commits to building it. The reasoning: the decisions taken at that level are the hardest to change later, and they substantially determine how fast the finished thing feels, how reliably it stays up, and how secure it is. A structured look belongs before the build, when problems can still be solved cheaply. Those qualities also trade off against one another — you cannot maximise all of them at once — so the trade should be chosen deliberately rather than discovered later (Kazman, Klein et al., Carnegie Mellon Software Engineering Institute, 1998 and 2000).

The older idea underneath it is even simpler: organise a system around the decisions most likely to change, so that when one of them changes only one part has to move (Parnas, Communications of the ACM, 1972 — one of the most-cited papers in software engineering).

And write each significant decision down: what the situation was, what was decided, and all of the consequences — including the ones you would rather not advertise (Nygard, 2011, building on Tyree & Akerman, IEEE Software, 2005).

Your Pogo pack records what was decided and what is deliberately out of scope and why — before your coding tool writes a line.

The United States standards body puts it plainly in its secure software development guidance: “Most aspects of security can be addressed multiple times within an SDLC, but in general, the earlier in the SDLC that security is addressed, the less effort and cost is ultimately required to achieve the same level of security.” They call it “shifting left” (NIST Special Publication 800-218, 2022). That guidance is voluntary advice organisations tailor, not a rule anyone is obliged to follow — which is rather the point of deciding it up front.

The principles themselves are older than most software. Two of them — make the safe thing the default, and give every part only the access it genuinely needs — were named in 1975 (Saltzer & Schroeder, Proceedings of the IEEE, 1975). A 2012 retrospective found they were still standard practice decades later, alongside open design and separation of privilege (Smith, IEEE Security & Privacy, 2012).

And the questions to ask are settled too. Threat modelling asks four: “What are we working on? What can go wrong? What are we going to do about it? Did we do a good job?” (Shostack, Four Question Framework). Your interview answers the first two before anything is built; your pack ships a security checklist your builder runs before launch, which is where the other two live.

• Stecklein, Dabney, Dick, Haskins, Lovell & Moroney. “Error Cost Escalation Through the Project Life Cycle.” NASA Johnson Space Center, INCOSE International Symposium, 2004. NASA Technical Reports Server document 20100036670.
• Wu & Fard. “HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agents.” ACM Transactions on Software Engineering and Methodology, Vol. 34, Art. 189, 2025. doi:10.1145/3715109
• Yang et al. “Assessing the Impact of Requirement Ambiguity on LLM-based Function-Level Code Generation.” Preprint, 2026. arXiv:2604.21505
• Jia, Morris, Ye, Sarro & Mechtaev. “Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation.” 2025. arXiv:2505.07270
• Cemri et al. “Why Do Multi-Agent LLM Systems Fail?” 2025. arXiv:2503.13657
• Porter, Votta & Basili. “Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment.” IEEE Transactions on Software Engineering 21(6), 1995.
• Hickey & Davis. “Requirements Elicitation and Elicitation Technique Selection: A Model for Two Knowledge-Intensive Software Development Processes.” IEEE International Requirements Engineering Conference, 2003.
• Zaremba & Liaskos. IEEE International Requirements Engineering Conference, 2021.
• Kazman, Klein, Barbacci, Longstaff, Lipson & Carriere. “The Architecture Tradeoff Analysis Method.” IEEE International Conference on Engineering of Complex Computer Systems, 1998. Kazman, Klein & Clements. “ATAM: Method for Architecture Evaluation.” CMU/SEI-2000-TR-004, Carnegie Mellon Software Engineering Institute, 2000.
• Parnas. “On the Criteria To Be Used in Decomposing Systems into Modules.” Communications of the ACM 15(12), 1972.
• Nygard. “Documenting Architecture Decisions.” cognitect.com, 2011. Tyree & Akerman. “Architecture Decisions: Demystifying Architecture.” IEEE Software 22(2), 2005.
• Souppaya, Scarfone & Dodson. “Secure Software Development Framework (SSDF) Version 1.1.” NIST Special Publication 800-218, February 2022. nvlpubs.nist.gov
• Saltzer & Schroeder. “The Protection of Information in Computer Systems.” Proceedings of the IEEE 63(9), September 1975, pp. 1278–1308. cs.virginia.edu
• Smith. “A Contemporary Look at Saltzer and Schroeder’s 1975 Design Principles.” IEEE Security & Privacy 10(6), November–December 2012, pp. 20–25.
• Shostack. “The Four Question Framework: A Guide to Threat Modeling.” shostack.org
• Eveleens & Verhoef. “The Rise and Fall of the Chaos Report Figures.” IEEE Software 27(1), 2010.