Why Not Waterfall

Is waterfall development a perfectly reasonable choice, really?

Stage-gated delivery, PRINCE2 included, moves work through signed-off phases, with change control and tolerances as its brake. It suits work where requirements genuinely do not move: contractual, regulated, or tied to hardware.

The awkward part for agile purists: a detailed specification written up front is exactly what an AI agent works best from. The heavy documentation agile criticised is now the input that makes AI implementation reliable.

This was the incentive for me to write up a quick list of why waterfall (the method I practiced and even led for part of the 80s and 90s) actually didn’t work.

I was a practitioner, and a true believer, until we realized that trying the same mathod harder wasn’t going to make it work, and the standard complaint “we need to spend more time up front” actually made it worse.

Key concept of waterfall: Change is BAD

Let’s honor the above quote. “It suits work where requirements genuinely do not move.”

That is a perfectly reasonable assertion.

As a waterfall practitioner in the 80s and early 90s, I feel that I can provide some background.

In waterfall, the assumed “normal” situation is:

1. Business requirements and concept are gathered.
2. Technical requirements are composed
3. Design is devised
4. Coding is done
5. Testing checks that code matches design
6. Delivery happens

There was an optional step: 7. Maintenance contract

Often a contract is signed and committed to in or around step 1. The initial requirements are legally not allowed to change, under penalty.

The initial design is everything in waterfall.

For anything to flow backward is an aberration. Once done, a phase is supposed to be done.

In other words, the plan is fragile and brittle in the face of change.

The resulting product must be delivered as specified, regardless of what happens in the world.

A successful contract delivers exactly to the contract, even if that’s no longer useful or desirable when it’s delivered.

This supports the initial statement that it is for when requirements genuinely don’t change.

Of course we always have changes, and always had. There was never a project that wasn’t “disrupted” by changes of some sort. If it’s not a change the environment or technology, it was a disruptive change in the market.

But let’s not focus on that here (we did that elsewhere).

Documentation makes AI Reliable?

A large corpus of detailed documentation has not been proven to make AI reliable. Agents can still misunderstand intent, miss constraints, optimise the letter of the spec rather than its purpose, or fail during long multi-step work.

AI agents have limited context, and when there is too much documentation for them to manage, they lose context or else summarise to maintain context and lose detail. Sometimes they “guess” based on the existing code, and may or may not consider the tests.

In addition, everything that isn’t given in detail in the agentic instruction is a “dice roll” of sorts. I don’t mean that it’s random, but that it’s chosen by the agent.

If generating code from documents was deterministic then regenerating off the same specs would always result in the same system.

AI is (intentionally) not deterministic. You don’t get the same results, even when given the same input. That’s just not how it works.

There is also still a problem of interpretation with AI: if a spec says “the service must return results within 200 ms,” an AI could cache stale data, skip validation, or drop error handling to meet the timing requirement. The specification was met; the intended system was not.

Change is Inevitable

“if requirements legitimately don’t change” is a very difficult antecedent.

Both the world and our understanding of the problem change.

A specification can remain a correct description of yesterday’s intention while becoming an inadequate description of what is needed today.

Not every program must change. A bounded mathematical function with a fixed contract can remain unchanged indefinitely.

The arguments against unchanging requirements concern long-lived software serving real-world activities.

Meir Lehman — the Law of Continuing Change

“E-type systems must be continually adapted else they become progressively less satisfactory in use.”

“E-type” means software embedded in a real-world activity, whose adequacy depends on that changing context. His foundational paper is Programs, Life Cycles, and Laws of Software Evolution (1980).

The crucial point is that unchanged software can become less useful without developing any new internal defect. The mismatch grows between its assumptions and its environment. These “laws” are empirical generalisations about software evolution, rather than mathematical impossibility proofs applying to every program.

Frederick Brooks — change comes from successful use and changing surroundings

“All successful software gets changed.”

He identifies two pressures: useful software invites new applications and extensions, and successful software outlives the hardware and surrounding systems for which it was built. Users, institutions, laws, and technology keep moving.

Brooks also directly challenges once-for-all requirements: he argues that customers and engineers cannot completely and correctly specify a substantial product before trying versions of it.

Requirements definition therefore includes experimentation and iteration. Some requirements change because we learn, even when the external world stands still.

David Parnas and Paul Clements — design cannot be a purely forward derivation

In A Rational Design Process: How and Why to Fake It (1986) they explain why actual development departs from the ideal of deriving a design from complete requirements:

• Customers cannot fully articulate what they want.
• Implementation reveals facts that invalidate earlier decisions.
• People cannot comprehend all the interacting details at once.
• External changes invalidate assumptions.
• Errors and practical reuse constraints require revision. laputan.org

Importantly, they advocate careful documentation. Their proposal is to maintain a coherent explanation of the resulting design, including rejected alternatives and rationale, despite the untidy discovery process.

“Fake it” means presenting a rational explanation—not pretending that learning and backtracking never occurred. This supports evolving documentation, not static documentation.

This is the trick by which we made waterfall development (strictly gated phases) look like it was working when it really was not.

Peter Naur — maintenance requires understanding, not merely text transformation

In Programming as Theory Building (1985), Naur treats changing demands as a premise and argues that programming’s central product is the developers’ understanding of how the program addresses its subject matter.

This understanding enables sensible modification.

This is primarily an argument about what maintaining software requires, rather than independent proof that change is inevitable, but it is worth considering (see the quote above).

But Now It’s Fast

The hope is often that because it may take days or weeks to write code that may have taken months or years before, there isn’t time for anything to change.

It does seem more palatable if the period from specification to delivery is drastically shorter.

That’s a legit blessing, by the way. It’s great to get inside the OODA loop and quickly execute on a decision while it’s still valid and fresh. I won’t be a nay-sayer.

But that’s for the immediate change. That won’t necessarily hold for a large, comprehensive, and detailed set of specs. We might go obsolete generating a large specification (it happened in waterfall days just like that).

Some hope that because it was “easy” enough to generate the code in a few weeks, it would be just as easy to do it over with the changed specifications.

I suspect this is why people are keen on trying waterfall again, now that there is AI to speed some steps of the development process.

The hope people have is that “taking it back to requirements” will be essentially free now, that it’s a matter of updating a document and regenerating the code.

I have less faith that this is so easy; generating the code is not as safe, deterministic, and reliable as people may wish.

There are two issues:

1. Speed of coding doesn’t translate to speed of delivery. Sometimes it is slower.

2. Accepting new code is not driven by code generation speed. The customer will be faced with a new system that will work somewhat differently, or look different, or feel different. Even the invisible changes may be troublesome to the customer.

Green Fields Turn Brown

A waterfall ends when the code is delivered. This is a key notion from the start of the contract and all the way through.

A waterfall is a one-way flow from ideas to a single delivery, and then the developer is done.

The problem with this idea is that a useful program is never finished.

“It is invariably the case that a program, once in operation, will be felt to be only part of the answer to the problems at hand.” - Peter Naur

Once the program exists, you will not want to maintain it by editing the requirements and regenerating the whole application, for reasons given above.

But also, you run into the whole set of problems that you have with a large corpus of documentation.

The problem I’ve stated with document-driven specification and generation is:

In the grand scheme of things, greenfield development is a rather niche problem.

Nearly all developers, nearly all the time, are working in brownfield systems that may have been greenfield systems at one time.

The whole point of agile development was not tied to disdain for document writing, but rather a realisation that we need feedback loops in order to achieve market fit and satisfy customers.

If you could reliably generate the first version in minutes (which, by the way, is a “parlor trick” done by framework vendors and tool-makers for years), you still will have to deal with expanding needs and requirement connections for years.

In waterfall, teams deferred changes to the “follow-on maintenance contract” and ignored change as much as possible as they built the system that we knew would not really satisfy the customers.

But if no changes happen before or after delivery, and the initial requirements were exactly correct, then waterfall would be fine.

It’s been done before

This isn’t the first “generate the code from documents” trend. It’s been tried many times before, including CASE tools. I suppose it is possible that agentic development is enough of a game changer that it will work and sustain now.

I’m willing to be convinced, but it’s failed every time until now.

This time, for sure. - Bullwinkle J Moose