Skip to content
Back to Blog
AI & Governance

Was It Built as Decided? What We Found When We Checked Our Own Code

Shi YanNovember 16, 202610 min read

An architecture review ends with an approval. The design shows every call going through the API. The decision record says the team will use Kafka. Everyone signs, and the project moves on.

Six months later the UI is reading the database directly because someone needed a report in a hurry, and the service that was going to use Kafka is on RabbitMQ because that was what the platform team could offer that quarter. Nobody decided to abandon the architecture. It just was not what got built.

No review caught it, because reviews read documents, and the documents still say what was intended.

This is the oldest gap in architecture governance. It is the distance between what was decided and what exists. This post is about how cajeX now checks it, in beta, and what happened when we ran it on ourselves first.


Passing Every Check and Still Being Wrong

There is a kind of evidence that has become easy to collect. Is the default branch protected? Are reviews required? Are vulnerability alerts being closed on time? These are settings, and settings can be read.

They matter, and they are most of what an auditor asks about. But they are compliance hygiene. They are not architecture. A repository can pass every one of those checks while its structure contradicts every decision made about it.

What a settings check sees


  • The branch is protected
  • Reviews are required
  • Alerts are closed on time
  • Releases are signed
and still

What it cannot see


  • The UI calls the database directly
  • The broker is not the one that was decided
  • Three services exist that no design shows
  • A decision describes code that was replaced

Figure 1 — A well-governed repository and a well-built system are different claims.

When we classified the directives in the cajeX framework catalogue by the kind of evidence each needs, about a fifth turned out to expect evidence from the design: that something is structured a certain way, not merely configured. Until now, that kind of directive could be satisfied by a design that said the right thing. Nothing checked that the code did.


Five Layers, From Provable to Judged

"Does the code match the architecture" is not one question. Some of it can be established as fact. Some of it needs judgement. Treating the two the same way is how tools in this area lose people's trust, so we separated them.

The questionWhat reads it, and how sure the answer is
The team's own enforcementDoes the team already enforce this, with an architecture test or a policy gate?The check runs on the default branch. Proven: the named check exists, runs, and passes
ManifestsWhich libraries, frameworks and versions?Dependency files, parsed as data. Proven
Infrastructure as codeEncrypted, private, in the allowed regions?Terraform, Kubernetes and Docker files, parsed. Proven, as declared
StructureDoes every component in the design exist in the code? Is there code no design shows?The folder tree beside the selected design. Proven, with drift listed
Dependency rulesDoes one component use another it must not?A graph of imports between components. Proven
Judged reviewIs this decision actually implemented?A model reads the decision and the code it points to. Reviewed, never Proven, never recorded automatically

Figure 2 — Six rows, five layers of fact and one of judgement, each labelled with how much its answer can bear.

Start with what the team already enforces

The first row is the one we would point a new team at. Mature teams already encode their architecture as tests: an ArchUnit suite, a dependency-cruiser rule, a policy-as-code gate in the pipeline. Reading that the ArchitectureRules job exists, runs on the main branch, and passes is cheaper and more trustworthy than anything a tool could infer from outside.

What cajeX adds is the link. This directive is enforced by that job, and the job is still there and still green. If someone deletes the job, the directive stops being proven, and somebody is told.

Facts are read as data

The next four rows read files as structured data, not as text to be interpreted. A package.json is parsed, and a check asserts that a library is absent. A Terraform file is parsed, and a check asserts that every declared bucket is encrypted. The folder tree is compared with the components of the design that was selected, and the result is two lists: components the design shows that have no code, and folders no design explains.

None of that involves a model at run time. The steps are approved once and replayed exactly, which is the rule for all of Live Evidence.

One honest limit: an infrastructure check proves what is declared. It proves the Terraform says the storage is encrypted. Whether the deployed storage matches the Terraform is a different check, against a different system.

Judgement is labelled as judgement

Some decisions cannot be reduced to an assertion. "We use the outbox pattern for all cross-service writes" is true or false of a codebase, but no parser will tell you.

For those, a model reads the decision and the code the earlier layers point it to, and says whether the decision is implemented. It is held to three rules.

It cites, or it is refused
Every claim names a file and lines. Each citation is checked against what was actually read.
It trusts code over comments
A comment saying a rule is followed is not evidence that it is.
It never records itself
The result is marked Reviewed and waits for a person. It can never count as Proven.

Figure 3 — The three rules the judged review is held to.

The distinction between Proven and Reviewed is the point. A dashboard that mixes "a parser confirmed this" with "a model believes this" into one green number is not telling you what you need to know.


We Ran It on cajeX First

Before offering this to anyone, we pointed it at our own repository. cajeX is governed in cajeX: its architecture decisions are recorded, its design is drawn, its directives are approved.

The first run made 39 read-only calls and came back with three things we did not know.

A decision that had been overtaken

One record still described an agent framework the code had stopped using months earlier.

A decision the code had outgrown

Our record said uploads go straight to storage. The code had been sending them through the application since spring.

A component missing from the design

A whole directory, the catalogue of checks, appeared in no drawing.

Figure 4 — What the first run on our own code found.

The second finding is the instructive one. Our decision record on file uploads said the browser would upload directly to object storage using a signed link. That was the plan. In practice it did not fit: the uploader sent a different kind of request, the storage had no cross-origin rules, and workspaces with dedicated storage needed a lookup the browser could not do. So uploads had been routed through the application instead, sensibly, by people solving a real problem.

Nobody updated the record. For months our own governance tool said one thing about our own system and the system did another.

Neither side was wrong. The code was right and the record was stale. That is the usual case, and it is why the tool reports a disagreement rather than a violation.


Two Disagreements, Named Plainly

When the decision and the code differ, cajeX says which way.

Decided but not built

A decision is accepted, and the code shows the opposite

Designed but drifted

The selected design and the code disagree: something is missing, unexplained, or connected where it should not be

Figure 5 — The two disagreements, shown on the application beside the decisions they concern.

What happens next is a person's call, and there are two honest answers. Fix the code, because the decision still stands. Or fix the record, because the code found a better answer and the decision should say so.

For our upload flow we chose the second. We wrote a new decision recording what the code does and why, and marked the old one as superseded for that part. The chain stays visible: anyone reading it can see what was first intended, what was built, and when the record caught up.

That is a better outcome than either silently editing the old decision or forcing the code back to a plan that had already failed once.


Code Is the Most Guarded Thing You Have

Reading source code is a different order of trust from reading a branch setting, so the design is conservative about what it keeps.

What cajeX doesWhat it does not do
StorageKeeps facts: the parsed structure of a manifest, the edges between components with one example each, and quoted linesKeep copies of your files
The modelShows it only the excerpts a check points toHand it the repository to explore
AccessReads with a credential that can read contents and nothing elseHold anything that can push, merge or change a setting
RetentionEncrypts stored outputs under your workspace's key and removes them on your plan's scheduleKeep evidence after the workspace's key is destroyed

Figure 6 — What reading code means in practice.

The read-only claim is tested, not asserted. Each connector is run nightly against test accounts with calls that must be refused, to prove the credential cannot write.


Where This Sits

Implementation conformance is part of Live Evidence, which is in beta. It is turned on per workspace by cajeX and works against GitHub today. It builds on things that already exist in the product: applications with their decisions, the candidate designs a project selects, and conformance as the place the answer is shown.

If you take one thing from this, take the first layer. Before any tool reads your code, ask which of your architecture rules are already enforced by a test, and whether anyone would notice if that test were deleted. For most teams the answer to the second half is no.

We thought our own decisions were current. It took 39 read-only calls to find out two of them were not.

Ready to transform your architecture governance?

cajeX brings AI-powered reviews, knowledge management, and directive lifecycle management to your enterprise architecture team.