An auditor is shown a dashboard. Ninety-four percent of controls passing, a green tick beside each, the date of the last check.
A good auditor asks three questions. What did the check actually look at? How do I know this result was not edited afterwards? And if I wanted to confirm it myself, without your tool, could I?
Most governance platforms can answer the first. Very few can answer the second with anything better than "trust us". Almost none can answer the third.
That third question is the one that matters. Evidence that can only be verified by the system that produced it is a claim, and a claim from a vendor is the weakest kind there is. This post is about how cajeX records evidence so that it stands up without us in the room. It describes Live Evidence, which is in beta.
What Makes Evidence Acceptable
Strip away the terminology and an auditor needs four properties.
Reproducible
The same procedure ran this period as last period.Attributable
It is clear what was read, by which check, and who recorded the result.Tamper-evident
If anything was changed afterwards, that can be detected.Independently verifiable
Someone outside can confirm all of the above.Figure 1 — Four properties. A screenshot of a dashboard has none of them.
The first two are about how evidence is collected. The last two are about how it is kept. They are separate problems, and a system can be strong on one pair and empty on the other.
Reproducible: Why Collection Is Not Left to an Agent
The fashionable way to gather evidence from a system is to give an AI agent a set of tools and a goal, and let it investigate. It is flexible and it demonstrates well.
It is also unsuitable for audit evidence, for a reason that has nothing to do with how capable the model is. An agent decides what to look at as it goes. Next month it may look at something slightly different. You cannot say the same procedure ran in both periods, because there was no procedure, only a goal.
So cajeX separates three moments and treats each differently.
| Who decides the steps | Why | |
|---|---|---|
| Designing a check | A model may draft it; a person approves it | Drafting is where a model saves real time, and a person reads the result before it counts |
| Collecting evidence | Nobody. The approved steps are replayed exactly | This is what makes one period comparable with the next |
| Deciding pass or fail | Rules a person approved; where judgement is needed, a model with no tools, reading only quoted data | A rule gives the same answer twice. A judgement is labelled as one |
| Recording the result | A person, or rules a person approved | A fail always waits for a person |
Figure 2 — Assisted at design time, fixed at run time, a person at decision time.
In one line: agent-assisted when a check is designed, deterministic when it runs, human when it is decided.
There is a second reason for fixing the steps, which is safety. A check may use only operations classified as read, and every operation is treated as a write until shown otherwise. A model that chooses its own tools at run time is a model that could choose the wrong one. A list of approved calls cannot.
When a team wants passes to record themselves, they can allow it per application, for checks whose outcome is decided by approved rules. Even then, cajeX draws a sample each week for a person to confirm, so that "automatic" never quietly becomes "unexamined".
Attributable: What a Result Carries
Every recorded result holds enough to reconstruct what happened without asking anyone.
Changes to the default branch need an approving review. The rule set on the repository requires 0 approvals; the check expects at least 1.
Figure 3 — A result carries its source, the relevant excerpt, a fingerprint of the full answer, and an expiry.
Three details are deliberate.
The reason is in words. A result says what it found in a sentence, above the raw data, and a failure names the count or the value that failed. An auditor should not need to read JSON to learn why something is red.
Evidence expires. Each result has a validity window. When it closes, the check reads Stale. It does not go on reading Passing because nobody looked. A control that was verified in March and never since is not a verified control.
"Could not check" is its own state. If cajeX could not reach the repository, or the answer came back empty, that is neither a pass nor a fail, and it is shown as neither.
And one failing control raises one finding, listing every directive it affects, rather than fifty findings that are really the same problem.
Tamper-Evident: A Record That Is Only Added To
Now the harder half. How do you show a result was not changed after it was recorded?
The evidence record in cajeX is append-only. Results, attestations and corrections are each a new entry. Nothing is edited in place. If a result was wrong, the correction is another entry that says so, and both remain.
Each day's entries are then sealed: a fingerprint is computed over the day's record and kept. Change one character in any sealed entry afterwards, and that day's fingerprint no longer matches.
A result is recorded
A new entry, never an editThe day closes
Its entries are sealedLater, someone checks
Verify trailA mismatch is named
Which entry, which day, what changedFigure 4 — Daily seals turn "we did not change it" into something that can be tested.
We tested this the direct way. On our test environment, with the record sealed, one character was changed in one entry in the database. Verify trail named the entry, named its day, and said its content had changed. Then the character was put back, and the trail verified again.
A tamper-evident record does not prevent someone with database access from changing a row. Nothing does. What it guarantees is that the change cannot go unnoticed, which is the property an auditor is actually relying on.
Independently Verifiable: The Package
Here is the part we care about most.
For any application and period, cajeX exports an evidence package: the results, what was read, the attached files, and the seals. Inside it is a small verifier program and a plain document describing exactly how verification works.
The auditor runs the verifier on their own machine. It needs no account, no connection to cajeX, and no trust in us.
Step 1
Export
Step 2
Hand over
Step 3
Verify
Step 4
Or write your own
A period's evidence for one application, as files
No access to cajeX is given or needed
The included verifier confirms each entry against its day's seal
The method is documented, so the verifier itself need not be trusted
Figure 5 — The auditor checks the evidence. They do not check with us.
The fourth step is not decoration. A verifier we wrote is still our code. So the package documents the method fully enough that someone can write their own.
We checked that claim too. A second verifier was written from the document alone, using nothing but a standard programming language's built-in libraries, with no sight of ours. On a real package it reached the same result as the included one: every entry accounted for, each proved against its day's seal. Then a byte was changed in a copy of the package, and both verifiers named it.
If the instructions are good enough for a stranger to rebuild the check, the check does not depend on us.
When Only a Person Can Say
Not everything can be read from a system. Some directives are satisfied by a procedure being followed, a review having taken place, a contract existing.
For those, a person attests: they confirm the control holds and record what they looked at, with the files attached. Three rules keep an attestation from becoming a tick-box.
- Files are fixed once registered. A file cited by an attestation cannot be replaced, moved or removed afterwards.
- A second person can be required. Where a control calls for confirmation, the person confirming cannot be the person who attested.
- It is shown as what it is. An attested directive reads Attested, not Proven.
That last rule runs through the whole product. A directive can be shown to hold in four ways, and they are kept apart.
Decided
An accepted decision record says it complies
Reviewed
Documents, the design, or a review of the code show it
Proven
A check read a system and the result was recorded
Attested
A person confirmed it and said what they looked at
Figure 6 — Four kinds of support for a directive. They are never added into one percentage.
A single compliance score is appealing and hides exactly the distinction an auditor is trying to draw. Ninety percent "compliant", where a third of that is a person's word and another third is a document saying the right thing, is not the same as ninety percent proven. Showing the four separately is less flattering and more useful.
How Long It Is Kept
The full outputs and attached files are kept for a period that depends on the plan. The record of what was checked, with its fingerprints, outlives that, so that the fact of a check and its result can still be shown after the bulky data has gone.
A legal hold stops deletion for the evidence it covers, for as long as it stands.
And credentials for the systems being read are kept in a vault belonging to the workspace, encrypted under a key the workspace holds. Destroying that key makes the stored credentials unreadable.
The Test We Hold It To
When we built this, the bar was not "does it produce a convincing report". It was a harder question: if a sceptical auditor took the output away and tried to break it, what would they find?
They would find a procedure that ran the same way each period. Results that say what was read and why they failed. A record where a changed character is detectable. And a package they can verify with a program they wrote themselves.
None of that requires them to believe anything we say. That is what we think evidence should mean.
Live Evidence is in beta, turned on per workspace by cajeX, and reads GitHub, Jira and Confluence today. The Live Evidence page has the details. For what it finds when it reads code rather than settings, see was it built as decided?, and for how the rules it checks come to exist, what are architecture directives.