The Physics Beneath the Paradigm

Every major technology shift in software engineering arrives with the same promise: this changes everything.

The language varies. Sometimes it is revolutionary. Sometimes it is transformative. Occasionally someone reaches for unprecedented, which is a word that should require a licence to use in a technology context. But the structure of the promise is always the same. The old problems will disappear. The new tools will dissolve constraints that previous generations accepted as permanent. The boards that move fast will capture the advantage. The boards that hesitate will be punished by the market.

It is a compelling narrative. It has also been wrong, in the same way, every single time.

Three revolutions that changed everything except the problems

Agile

By the early 2000s, the industry had two decades of evidence that traditional project management did not work for software. Waterfall produced systems that were late, over budget, and misaligned with what users actually needed. Agile proposed to fix this — shorter cycles, working software over documentation, responding to change over following a plan. By 2008, every technology organisation of meaningful size was either adopting it or explaining why they had not yet.

The coordination cost did not shrink. It moved. Instead of being front-loaded into requirements documents and Gantt charts, it was distributed across standups, sprint planning, backlog refinement, retrospectives, three-amigos meetings, and an expanding vocabulary of ceremonies that consumed, in many organisations, more hours per week than the planning processes they replaced. The Standish Group's CHAOS reports showed only marginal improvement in project success rates — from roughly 16% in the mid-1990s to 29% by 2015, despite two decades of methodological transformation.

The failure mode changed. Instead of delivering the wrong system after two years, teams delivered the wrong increment every two weeks. The underlying problem persisted. Agile gave teams a better rhythm for execution. It did not solve the problem of decisions, because decisions were not what Agile was about.

Microservices

By 2013, the monolith was the enemy. Microservices proposed to fix the coupling problem: decompose the monolith into small, independently deployable services, each owned by a small team, communicating through well-defined APIs. Netflix, Amazon, and Spotify became the reference architectures. Conferences filled with talks about how to split things apart.

The coupling did not disappear. It migrated from code to infrastructure, from compile-time to runtime, from function calls to network calls, from visible coupling that the compiler caught to invisible coupling that production discovered. A service depending on three other services for a single user request now had three network boundaries where failure could occur, three APIs that could change, three teams whose deployment schedules could interfere with yours.

By 2018, the industry had a name for what happened next: distributed monolith. Systems with all the operational complexity of microservices and all the coupling of a monolith. Conway's Law had predicted this in 1967: the structure of the system mirrors the communication structure of the organisation that builds it. The architecture changed. The underlying problem did not.

Cloud migration

The public cloud, as pitched between 2010 and 2016, was going to eliminate an entire category of engineering work. No more capacity planning. No more hardware procurement. No more 3am calls because a disk filled up. Engineers could focus on building, not operating.

Hardware procurement did disappear. The operations problem did not. Instead of managing physical servers, teams now managed cloud resources — dozens of services, each with its own configuration surface, its own failure modes, its own pricing model, its own regional and zonal limits that only became visible under production load. The infrastructure-as-code movement emerged not because engineers wanted to write YAML for a living, but because the configuration complexity had grown beyond what any human could manage manually.

Gartner predicted in 2019 that 80% of organisations using cloud infrastructure would overspend their budgets by 20–40%, primarily due to the complexity of understanding what they were paying for. The prediction was accurate. The operations cost did not disappear. It migrated from headcount and hardware to cognitive overhead and configuration management.

The pattern beneath the pattern

Three shifts. Three genuine improvements in execution capability. Three cases where the structural problems — coordination, interfaces, decisions, accountability — transferred rather than resolved.

This is not a coincidence. It is not a failure of implementation. It is not because the technology was overhyped, although it usually was. It is because the problems that persist across paradigm shifts are not technology problems. They are structural properties of building complex systems with groups of people under uncertainty, and properties of that kind do not yield to better tools because better tools are not what is producing them.

They are constants. Not constants in the casual sense — properties that remain invariant when the surrounding variables change. The software engineering equivalent of physical laws. They emerge from constraints that are deeper than any technology choice, and every technology that has been tried has proved unable to reach them.

Brooks named this in 1986

Fred Brooks named the pattern. No Silver Bullet — Essence and Accidents of Software Engineering, his 1986 essay, distinguished the essential complexity of software — the conceptual structure of the problem itself, the specifications to be drawn out, the design to be reasoned through — from the accidental complexity of representing that conceptual structure in a particular language, on a particular machine, with a particular set of tools. Brooks argued that productivity gains in software had come almost entirely from attacking accidental complexity, and that no single development on the horizon would yield an order-of-magnitude improvement within a decade, because the essential complexity was untouched by tooling.

Forty years on, the framing is still correct. AI is the most aggressive attack on accidental complexity ever mounted, and it works. The execution gains are real. The essence is exactly where Brooks left it.

Seven constants

A property qualifies as a constant if it meets four conditions. It was true before the current paradigm. It is true during the current paradigm. It will be true after the next paradigm. And it emerges from structure, not tooling. The test distinguishes real constants from things that merely feel permanent because no current technology has addressed them.

These are the seven that pass the test:

Decisions still have costs. Making something cheaper to execute does not make it cheaper to get wrong. As execution costs collapse, the cost of decisions does not collapse with them. It concentrates. The decision about what to build, and in what order, and for whom, becomes the dominant cost, because the brake that high execution cost used to impose on bad decisions disappears.

The system mirrors the communication structure. Conway's Law is sixty years old and has never been wrong. It does not care whether the builders are humans, AI agents, or some hybrid. The system will reflect how its builders communicate. Change the communication structure and you change the system. Change the technology without changing the communication structure and you get the same system in new clothes.

Coordination cost is the real scaling limit. Not headcount. Not compute. Not funding. The ability to coordinate work across people, teams, and systems is what constrains an engineering organisation's throughput, and the mathematics of why — Brooks's channels, Amdahl's Law, the serial fraction — have been understood for half a century. Every paradigm shift promises to raise the ceiling. None has reduced the coordination cost per unit of complexity.

You cannot measure what actually matters. Goodhart's Law — when a measure becomes a target, it ceases to be a good measure — has survived every attempt to quantify software engineering productivity. Lines of code. Story points. Velocity. DORA metrics. Each captures something real and distorts behaviour the moment it is used for evaluation. The things that matter most — judgment, architectural coherence, the quality of decisions, the transfer of tacit understanding — remain stubbornly resistant to measurement.

Context is the bottleneck, not capability. The constraint is never the ability to write code. It is the understanding of why this code, here, now, and not something else. The bottleneck moved from the hands to the head a long time ago, and the head holds tacit knowledge — the kind that resists articulation, transfers only through shared experience, and remains permanently out of reach for any system that cannot participate in the experience.

Accountability does not distribute, it just hides. Shared ownership sounds democratic. In practice, it means that when something fails, no one is clearly responsible for fixing it. Every paradigm shift that distributes execution — microservices across teams, AI across workflows — creates the illusion that accountability is also distributed. It is not. It is diluted. And diluted accountability is indistinguishable from no accountability, until something breaks.

Interfaces are where everything breaks. Between modules. Between services. Between teams. Between humans and AI. The boundary between two components is always the most fragile point in the system. The technology on either side of the boundary changes constantly. The fragility of the boundary itself does not. Every paradigm shift introduces new interfaces and, with them, new failure modes — and the rate at which new interfaces appear has accelerated with every shift.

Why this matters now

AI is the most significant paradigm shift in the history of software engineering. The reduction in execution cost is real, large, and accelerating. Tasks that took days take hours. Tasks that took hours take minutes. Tasks that took an afternoon take a prompt.

Which makes it the most dangerous paradigm shift, too. Not because AI is harmful, but because the magnitude of the execution improvement makes it uniquely tempting to believe that the deeper problems will also be resolved. If AI can write the code, surely it can also coordinate the work. If AI can generate the tests, surely it can also decide what to test. If AI can scaffold the architecture, surely it can also own the outcome when the architecture is wrong.

It cannot. Not because the technology is immature, although it is, but because these are not technology problems. They are structural constants. They were present when engineers wrote assembly by hand. They are present now that AI writes Python on demand. They will be present when the next paradigm arrives.

The organisations that thrive through this shift will be the ones that distinguish between what AI changes — execution speed, capability per person, the economics of certain categories of work — and what it does not change: the cost of decisions, the physics of coordination, the fragility of interfaces, and the irreducible need for someone to own the outcome.

The distinction is not subtle, but it is easy to miss in a moment when the execution gains are loud and the structural properties are, as they always are, quiet.

The tools will keep changing. The physics will not.

Everything else is weather.


This essay is adapted from my book Engineering Constants: What Never Changes About Building Software. My first book, Engineering Economics: The Hidden Costs of Technical Decisions, is available now.

Where this is going. Engineering Constants: What Never Changes About Building Software — the seven constants, named, argued, and evidenced. The companion to Engineering Economics.

Get notified when it ships →