> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sovara-labs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Why annotate?

> Why annotation matters for improving agent behavior.

Annotation is where a run becomes training signal for the team. It records
whether the agent succeeded, why the run mattered, and what a better answer or
fix should preserve.

## The problem

Manual annotation is expensive. It takes domain expertise, context switching,
and careful reading of traces. A naive queue fills up quickly with noise:
duplicates, obvious mistakes, easy successes, and runs that are already covered
by earlier examples.

The hard failures are different. They are often hidden in domain assumptions,
financial conventions, internal policy, or multi-step tool use. These are the
cases where annotation is most valuable, because the missing knowledge is not
visible from the final answer alone.

## What's wrong with random sampling?

Random sampling gives you a rough sense of the agent's failure rate. It is a
reasonable starting point when the system is new and failures are everywhere.

It becomes inefficient once the agent is already reasonably good. Reviewers
spend too much time on obvious successes, repeated mistakes, and behavior
already covered by earlier annotations. The more capable the agent gets, the
more valuable it is to spend review time on the few runs that reveal something
new.

## Why not just use LLM-as-a-judge?

LLM judges are useful for straightforward checks. They can catch clear format
errors, direct mismatches, or failures that are obvious from the local context.

They are weaker when the failure depends on domain knowledge the agent also
missed. If a general judge can reliably infer the issue from the same context,
the agent often could have avoided the mistake in the first place. The most
valuable failures are the ones that require a human to notice the hidden
assumption and turn it into reusable guidance.

That is where annotation gives teams an edge. It captures the domain judgment
behind the failure, not just the fact that the output was wrong.