Contents · 6 min read
An agent can produce a convincing final answer without producing the thing it was asked to make. It can also stop after useful work while leaving an essential check unfinished. A benchmark needs a way to tell those situations apart.
The quantitative-finance agent benchmark on my GitHub is useful to examine through that lens. Its most interesting design choices concern the boundaries between a task, its execution, and the evidence used to judge it. This note follows one concrete task through those boundaries.
Scope of this note. This is a source-based walkthrough of revision
ad28072, not a new benchmark run or a model ranking. The repository is a curated edition of collaborative work originating at Varsity Tech, with original team contributions retained. Its provenance notice records that context.
Start with an inspectable task
“Fix this strategy” leaves too much room for interpretation. What must change? Which files count as the submission? What would convince an evaluator that the task was completed?
The look-ahead bias task makes those expectations explicit. The supplied moving-average strategy assigns a position from the current signal without the intended one-bar shift. The task asks for three separate artifacts:
| Artifact | What it contributes |
|---|---|
diagnosis.md | An explanation of the timing problem and the correction. |
lookahead_fixed.py | The actual code change that an evaluator can inspect. |
comparison.json | Structured before-and-after measurements with required keys. |
This separation matters. An explanation cannot stand in for a code change; a code change cannot stand in for evidence that it was exercised. The task also specifies the available data, tools, environment, and time limit. Together, these form a contract between the agent and the evaluation system.
There is a subtle limitation worth preserving. The constructed fixture expects the corrected return to fall below the original return. That is a property of this task’s check, not a general theorem that every correction of temporal leakage must lower every performance metric. The causal alignment is the underlying issue; a score rule is only one way of checking evidence for it.
Separate execution from judgment
The agent chooses actions. The runtime controls what can execute and records what happened. The evaluator then consumes saved evidence. Keeping these responsibilities distinct makes the evaluation easier to inspect and revisit.
The stages describe responsibilities, not a compulsory order of individual tool calls. Judgment operates on saved evidence rather than directing the agent’s next action.
The runtime architecture distinguishes a Run, a Session, and a Job. A Run represents an assigned attempt. A Session holds the active interaction and workspace state. A Job tracks an asynchronous operation when the relevant interface supports it.
Those distinctions are practical. A long computation needs monitoring and cancellation. A command can return a non-zero exit code even when its surrounding wrapper completes normally. A session can reach a time limit without completing the task. None of those states should disappear behind a single “finished” label.
The engineering challenge is therefore partly one of preserving meaning: keep the termination reason, tool results, output files, and evaluation records distinguishable. A transcript alone is not a substitute for all of them.
Ask two different questions
The evaluation separates result quality, called QR, from process quality, called QP.
| Track | Question | Evidence to inspect |
|---|---|---|
| Result quality | Was the required outcome produced? | Artifacts, numerical checks, code checks, and relevant semantic judgments. |
| Process quality | How did the agent work toward it? | Tool use, execution outcomes, code changes, planning, and problem solving. |
This does not make either track objective by definition. A heuristic can reward an expected tool path and miss a valid alternative. A semantic judge can produce well-formed output while making a poor judgment. The evaluation documentation describes these constraints explicitly.
The retained full-evaluation configuration combines the two tracks with weights of 0.60 and 0.40. Those numbers encode a particular evaluation design. They should travel with the reported result and its version, rather than being presented as universal constants for agent quality.
Missing evidence is not a zero
The aggregation code makes a useful distinction between a low score and a score that could not be computed. In full mode, if either required track is absent, the overall result remains missing. It does not silently substitute zero or renormalize the remaining track.
The following calls use the repository’s scoring function, with deliberately synthetic inputs:
compute_overall(qr={"score": 0.8}, qp={"score": 0.6}, eval_mode="full")
# 0.72
compute_overall(qr={"score": 0.8}, qp=None, eval_mode="full")
# None
compute_overall(qr={"score": 0.8}, qp=None, eval_mode="qr")
# 0.8
The second and third calls differ in what was requested. A result-only evaluation can be complete with QR alone. A full evaluation cannot. These are examples of aggregation behavior, not measured agent performance.
At the benchmark level, this distinction makes coverage essential. An average calculated from computable attempts should be accompanied by the number of attempts that could not be scored. Otherwise, an apparent improvement can conceal a change in which runs entered the denominator.
Read the verifier as critically as the agent
A task contract is only as convincing as its checks. The look-ahead task has both structural artifact requirements and a task-specific verifier.
Some checks look for code patterns such as shift(1). Others use keywords in execution records or the conversation. These are inspectable and useful, but they do not amount to a complete semantic proof of the implementation. A non-empty Python file is not evidence that the intended program ran correctly; a mention of lower returns is not, by itself, a validated measurement.
This suggests a concrete direction for stronger evaluation: connect the submitted code, the actual execution, and the resulting measurements as tightly as possible. Test causal behavior where feasible, make heuristic checks visible, and use human review to examine the cases the automated checks cannot settle. That is a direction for improvement, not a claim that all of it is already implemented.
Preserve the ability to ask again
Saved evidence makes it possible to revisit a judgment when a rubric or evaluator changes. That is different from changing the acting model or its prompt: new behavior requires a new run and new artifacts.
For me, that is the central lesson of this design. A benchmark should leave enough of a trail to explain what a result means, what evidence supports it, and what remains uncertain. The score is a useful summary only after those questions have answers.
The repository includes implementation and retained tests. This article has not reproduced an end-to-end agent run, audited every task, or established leaderboard-wide validity. Those are separate validation tasks. The source snapshot is linked so the argument can be checked against the code rather than taken on trust.