ATP360 Auditing a test run

When an AI drives a test, passing or failing is rarely the whole question. Anyone reviewing the result — an internal AI governance board, an auditor, a change advisory board, or a customer asking why a build was signed off — needs to see what the agent was instructed to do, what it actually did to the machine, and why it reached the conclusion it did.

The action log is that record.

The Action Log

Every entry is retained with the run and labelled with one of four types, so a reviewer can follow any one thread through a run without reading the rest. The log is the evidence a human draws on when approving or overriding a verdict, which is the point at which an AI result becomes an accepted one.

Keeping these apart matters for review. Stated intent and recorded fact are different classes of evidence, and an audit that conflates them can only report what the agent claimed. Where narration and an action result disagree, the action result is what happened.

The four entry types

CUA Instructions — what the agent was authorised to do

The instructions the orchestrator sends the agent, telling it what to do next. One instruction covers a whole authored test step, so a single entry is often followed by a long run of machine actions.

For example: "Open the Start menu, search for Notepad, and launch it."

These prompts are generated from the test case as authored. Nobody typed them during the run. For audit purposes they are the scope of what the agent was asked to do, and the point of comparison for anything it did beyond that scope.

Expect roughly one per authored step. The orchestrator can also send a fresh prompt mid-run when it decides to move things along, and each of those is itself recorded as a planner decision, so the reason for it is auditable too.

Assistant — what the agent said it was doing

The agent’s own account of what it sees and what it is about to do, in plain language, as it works through each step.

For example: "The Start menu is open with the search box focused. Let me type Notepad."

This is the most readable thread in the record and the fastest way to understand a run in narrative form — usually a dozen or so lines even for a run with dozens of machine actions. Treated as evidence, though, it is a statement of belief and intent rather than proof: it records what the agent thought was in front of it, which is exactly what you need when the question is how a mistake began.

CUA Actions — what was actually done

Every physical thing the agent did to the machine, each paired with its result and a screenshot. This is the forensic layer of the record and the only type carrying visual evidence.

Entries come in pairs: a command is issued, then a result comes back saying whether it succeeded and which screenshot captured the screen at that moment. Each action records:

• Action type — a key press, a click, typed text, a wait, and so on
• Text — keys pressed or characters typed
• Coordinates — where on screen a click landed
• Result — whether the command completed
• Screenshot — the screen capture taken with the action
• Gate decisions — the cheap check, made by a second and smaller model, that the actions taken match the step that was asked for. It looks for signs of a loop or a blocker and decides whether the planner needs to be consulted at all; most of the time it concludes that progress is normal.
• Planner decisions — the considered calls, each carrying its rationale: send another prompt, continue, interrupt the run, mark the workflow done, or mark it failed.
• The closing result — the point at which the engine considers the workflow finished.
• Attaching evidence to a change record, release sign-off or audit response
• Retaining a run alongside your own governance documentation
• Independent analysis, or querying many runs together for patterns in AI decisions
• Operator messages. When a run pauses to ask a person for help — which account to use, for example — the reply that person sends back is stored with its own time and step. These are the only entries in a run that a human typed, which makes them the record of where human judgement entered an otherwise autonomous run. Fully unattended runs have none.
• Errors and the run verdict. The run’s phase, its pass or fail result, and the type and reason for any failure are top-level facts about the run rather than moments within it, so the status bar and result carry them. Reasoning tells you why that verdict was reached, and the review record tells you whether it stood.

This is the stream that answers "what exactly did it click?", and the one that demonstrates the agent stayed within the bounds of what it was asked to do. It dominates the log by volume, so reviewers commonly hide it while establishing the shape of a run, then bring it back to substantiate a particular step.

Reasoning — why the engine decided as it did

The engine’s own decision record. Where the other three types describe a run, Reasoning explains it, and it is the type most governance reviews depend on.

The same stream records the run’s lifecycle: created and started, prompts sent, pauses and resumes, credentials injected and sessions restarted. Every event is time-stamped, and because the events are of different kinds their detail varies from entry to entry.

Two questions a reviewer can only answer from this thread: whether the agent was stopped by a deliberate decision or simply ran out of steps, and what evidence the engine believed it had when it declared the work complete.

Reconstructing a run

The types interleave in a repeating shape, which makes a long record straightforward to follow in order:

1. The orchestrator sends a prompt for the step (CUA Instructions) — one entry, phrased as an objective.
2. The agent describes what it is about to do (Assistant) — narration reappears between clusters of actions as the agent reorients.
3. Commands land on the machine, each with a result (CUA Actions) — repeats many times per prompt.
4. The engine decides what happens next (Reasoning) — a gate check first, then a planner decision if one is needed: another prompt, an interrupt, or a verdict.

Working backwards is usually quicker when reviewing an outcome you want to challenge. Read Reasoning alone first to find the closing decision and its rationale. Add Assistant to see what the agent believed at that moment. Then add CUA Actions and open the screenshots to establish what was really on screen, and compare that against the instruction for the step.

Live runs and past runs

The action log behaves the same way during a test and afterwards. A run in progress shows entries as they are produced; a completed run reopened from Test Results shows the same four types, in the same order, filtered by the same control. Nothing is summarised away when a run finishes, so a review carried out months later reads exactly as it would have done live. Reopening past runs is covered in ATP360 Managing and Reviewing Tests.

Downloading the raw record

Download Results, at the top of the action log, exports the underlying record as raw JSON. Use it when the review needs to leave the product:

The export is the same data the log displays, in full. Filters applied in the interface affect only what is shown on screen; they never affect what was recorded or what the download contains.

Sign-off and overriding a verdict

A verdict produced by the engine is not the end of the record. No result is final until a person approves it or overrides it, and the action log is what that person reviews before deciding.

Runs awaiting a decision are listed under Test Runs to Review, where each row offers Approve, Override, and a view icon to open the run and read its action log. Where a verdict was wrong, overriding it records a reason and your notes, and both the original and the replacement verdict are kept — so the disagreement itself becomes part of the audit record rather than replacing it. See Reviewing runs and the override record for how to approve and override, and for the fields the override record keeps.

Read together, the two records answer the question an auditor is most likely to ask. The action log shows what the agent did and why; the override record shows whether a human agreed. Override and review rates over time are tracked on the AI Efficacy tab, also described in ATP360 Managing and Reviewing Tests.

Human involvement and the run verdict

Two further parts of the record sit outside the four types but travel with the run:

• Operator messages. When a run pauses to ask a person for help — which account to use, for example — the reply that person sends back is stored with its own time and step. These are the only entries in a run that a human typed, which makes them the record of where human judgement entered an otherwise autonomous run. Fully unattended runs have none.
• Errors and the run verdict. The run’s phase, its pass or fail result, and the type and reason for any failure are top-level facts about the run rather than moments within it, so the status bar and result carry them. Reasoning tells you why that verdict was reached, and the review record tells you whether it stood.