Labs — internal agent systems

Problem

Auditing a project end to end before launch is the easiest thing to skip in a one-person team. "I had a look" is not reliable, because the checklist lives in your head and different items fall off it on every project. The second issue is subtler: telling a language model to "audit this site" does not work. A finding that is not grounded in measurement comes out confident and wrong, and its wrongness only becomes visible when you try to fix it. The value of an audit is not the number of problems found but the fact that every problem found is real.

Architecture

The pipeline runs in order. A scout profiles the project and decides whether the conditional auditors are needed. Then the Playwright harness runs in parallel: screenshots, a three-run Lighthouse pass and an axe accessibility scan. For projects at production maturity, a real build output is taken as well. Once those measurements exist, 13 fixed and 1 conditional auditor run in parallel — visual design, interaction, conversion, copy, design system, technical and content SEO, frontend and backend performance, accessibility, security, legal compliance, data integrity, and data visualisation if it is detected. Each auditor's output goes into its own QA red team; findings that survive go to a single cross-agent reviewer. The result is a markdown report with an approval checkbox next to every finding. A single write-capable agent applies the approved fixes, and a verification pass follows.

Key Decisions

1. The audit starts after measurement; it does not replace it

   Before the auditors say anything, the Playwright screenshots, the median of a three-run Lighthouse pass and the axe output are already in hand. A performance finding is no longer "the page feels slow" but is written against a measured metric. The same applies to accessibility: a violation caught by axe and a model's guess do not carry the same weight.

   Trade-offSetting up the harness is an extra step on every project and the route configuration is written by hand. On a project where measurement is not possible — an application that does not come up locally, or needs a database — the pipeline narrows and part of the audit simply cannot run.

2. Every auditor's output passes through a separate red team

   Having one agent review its own finding does not work; it defends its own text. So each auditor has a separate QA pass whose job is to find fault: hallucination, severity inflation, blind spots. A finding that does not pass is sent back at most twice.

   Trade-offThe agent count, and therefore the cost, nearly doubles and the audit takes longer. On a small project that cost can exceed the value of the problem found — which is why I run the kit on projects that are about to launch, not on every project.

3. Write access belongs to one agent, and its scope is fixed in advance

   None of the agents that produce findings can modify a file. Only the fix-applier applies changes, and it cannot step outside the file list of the approved fix. That way, "something broke during the audit" never turns into a hunt for which agent did what.

   Trade-offAn obvious problem outside the scope is left unfixed and needs a separate pass. Because bulk fixes all go through a single agent, the pipeline bottlenecks here.

4. No fix is applied without approval

   The report arrives with markdown checkboxes: apply, defer, won't fix. The decision stays with me, because some findings are correct but wrong for that project — a legal-compliance finding on an early prototype is technically right and practically not the next task.

   Trade-offOn a hundred-item report the approval step can take as long as the work itself. Bulk approval shortcuts soften this but do not remove it; it is not a fully automatic pipeline, and I do not want it to be.

Performance

Lighthouse runs three times and the median per metric is taken — a single run, especially on a cold start, does not represent reality. The auditors run in parallel, so the pipeline's total duration depends on the longest chain rather than the slowest single auditor.

Results

Agents

18 agents · 5 slash commands · 5 TypeScript harness scripts

Audit coverage

13 fixed auditors + 1 conditional (when a dashboard or chart library is detected)

Verification layer

A red team per agent plus a single cross-agent reviewer

Architectural guarantee

Write access belongs to one agent, and it cannot step outside the approved file list

What I Would Do Differently

I should have added the cross-agent reviewer from the start. In the first version every auditor wrote its own report, and the same problem arrived from three different agents with three different severities; whoever read the report did not know which one to trust. The same pattern shows up on the code side of this portfolio: in Bergaz Operations, leaving the guarantee in the application layer meant every new path had to re-establish the same rule. Here too, until deduplication moved into one place, every auditor was doing its own.

Problem

I support researchers with reproducible statistical analysis. Statistics is not my field; code is. Those two facts together determined how the system had to be built. Making a language model compute a p-value produces a number that looks right and is not, and the error usually surfaces at the latest and most expensive moment — when the research is being assessed. So the pipeline makes one promise: every number in the report can be traced back to the code and the input that produced it. Interpreting what a result means and writing the research text is the researcher's work; the pipeline does not replace it.

Architecture

Eight agents run in sequence: scale librarian, data profiler, score builder, design interpreter, test selector, analysis runner, reporter and reviewer. Alongside them sit task agents — for example a chart builder and a quality gate that checks each stage's output against what the researcher asked for. All numeric work is on the Python side: data handling, hypothesis tests, normality, reliability coefficients, regression diagnostics and effect sizes. Every result is written to a structured record that carries which test and which input it came from; the report takes its numbers from that record exactly as they are. The model does not compute: it runs the flow, writes the rationale for the chosen test, and moves results into tables that follow the reporting rules. The pipeline stops at three points and waits for a human: confirming the scale mapping, confirming the hypothesis, and accepting the results together with the review notes.

Key Decisions

1. The model never computes

   Every p-value, F, t, reliability coefficient, correlation and regression output is produced by real statistical libraries. A number produced by a model cannot be defended because its source cannot be shown; handing the arithmetic to a library makes every figure in the output reproducible.

   Trade-offEvery crossing between model and computation needs a schema: what the agent will ask for and what Python will return must be defined in advance. That means adding a new test creates work on both sides — the pipeline moves noticeably slower than one that operates on free text.

2. Three approval points, at fixed positions

   The approval points were placed immediately before the pipeline's most expensive mistakes. If the scale mapping is wrong, every subsequent computation is wrong; if the hypothesis is wrong, a correct computation answers the wrong question; if the report is built before the results are accepted, correction costs the most. Domain decisions belong to the researcher and statistical test selection to the system — and the approval points make that split visible.

   Trade-offThe pipeline is not, and cannot be, fully automatic: it waits for an outside answer at three points. If the answer is late the work stops, and if the researcher is not in a position to make those three decisions the pipeline does not move at all.

3. Every number is traceable to the code that produced it

   Every value in the report comes from a result record that carries which test ran on which input; the report takes the number from that record as it is and never retypes it. Approvals are recorded too, with who gave them and which file they were based on. So the answer to "where did this number come from" is not a guess but a file and a step. The researcher writes the interpretation and the text; what the pipeline delivers is the numbers, the tables and their source.

   Trade-offA result that does not fit the record cannot move to the next stage, and even a small change can mean updating the record format. For anyone expecting a ready-made interpretation from the pipeline, that boundary has to be discussed openly at the start.

4. Reporting conventions apply from the first draft

   In one delivery the computations were correct but the report went back for revision over formatting conventions. Those rules are now defined inside the pipeline and applied from the first draft, rather than treated as polish to be added later.

   Trade-offThe rules came out of feedback from a single assessment process and may not be identical at every institution. Working with a different institution requires reviewing the rule set, and today that is done by hand.

Performance

The pipeline's duration comes from the approval points, not the computation: an analysis run is measured in minutes, waiting for approval in days. The pipeline is therefore split so that it can prepare the next step while waiting at an approval point.

Results

Agents

8 sequential pipeline agents + 12 task agents

Active development

22 commits

Architectural guarantee

Every numeric result comes from Python libraries; the model does no arithmetic

Architectural guarantee

No report is delivered before all three approval points are passed

What I Would Do Differently

I did not write the reporting conventions up front; they were added after a delivery went wrong. Today, when building a pipeline, I would ask "where does the output get rejected" before "how is the output produced" — rejection criteria shape the pipeline. The same lesson came out of Launch Audit Kit: there too the verification layer was added later, and until it was, some of the findings produced were unusable.