Log every decision your agent makes

An agent makes thousands of small decisions a day: which tool to call, which team gets a ticket, whether to close an alert or wake someone up. Most teams log the conversation and the errors. Very few log the decisions.

That gap is invisible until the day someone asks, "why did the system close this?" Without a record of what the agent saw, what it scored and which rule was in force, the honest answer is "we don't know". A decision you can't replay is a decision you can't check, tune or defend.

Seven fields, written at the moment of decision

1. What came in. The input, or a pointer to it if it's large or sensitive. Enough to see the case again exactly as the model saw it.

2. Which model. Its name and version. When a model is updated, its scores change meaning, and you need to know which decisions came from which version.

3. The score. The probability or confidence the model returned, before any rule was applied.

4. The threshold in force. The exact lines used for this decision, with a version number. Thresholds change; the log has to say which one applied.

5. The action. Act, review or escalate, and what that meant in practice: closed automatically, sent to a queue, someone paged.

6. Who reviewed it. If a person looked, who it was and what they decided. Agreements and overrides are the most useful signal you will ever get.

7. The outcome. What turned out to be true. It usually arrives later, from a customer reply, an investigation or a refund, so leave room for it and fill it in when it comes.

The outcome is the field everyone forgets, and the one everything else depends on.

Why the outcome matters most

Calibration means that when a model says 0.8, it is right about 80% of the time. The only way to check that is to group past decisions by score and count how often each group was right. Without outcomes joined back to decisions, that count is impossible, and every threshold you set rests on trust instead of evidence.

The same join tells you when the world has moved. If cases scored around 0.3 used to be real one time in ten and are now real one time in four, your lines are in the wrong place, and the log shows it weeks before an incident does.

Version your thresholds like code

Lines drawn from costs and capacity will change: a new team member, a costlier kind of miss, a quieter season. Keep each set of thresholds in version control, give it a number, and write that number into every decision. Then "why was this closed?" has a precise answer: score 0.012, threshold set 14, act below 0.02.

This is the same idea as the receipts some enterprise agent platforms now produce for every run: the authority applied, the evidence used, the decision made. You don't need a platform to do it. You need the seven fields, and the habit of writing them.

Read the log every week

A log nobody reads is just storage. Three numbers, once a week, are enough to keep a decision system honest:

How often the machine acted alone, and whether that share is drifting. A sudden rise usually means a model change or a shift in the inputs, not a better system.

How often people overruled the model in review. A high rate means the middle band is too wide or the model is miscalibrated there; a rate near zero means you may be paying people to approve what the machine could close.

How full the review queue was. If it is always full, your thresholds are set by hope rather than by capacity. Draw them so the middle fits the team you actually have.

None of this is glamorous. It is a table with seven columns and a weekly look at three numbers. But it is the difference between an agent you hope is right and one you can show is right, and when something goes wrong at 2:14 a.m., it is the record that tells you why.