One Call, a Whole Job: Testing a Model With Agents Inside

Sentinel is a model with agents inside: a caller sends one request and gets back finished work, not a plan. That is a claim about behaviour, so we tested it the way a customer would use it. Three jobs that each take several steps, each sent as a single OpenAI-style request to /v1/chat/completions with the model id sentinel, 3 times each. No tools were defined by the caller and no follow-up messages were sent. Afterwards a separate harness checked the work on disk with checks the model never saw.

The jobs

• Fix a failing test suite. A small statistics module whose median is wrong for even-length lists. Pass means: the tests now pass, the test file is byte-for-byte unchanged, four hidden median cases pass, and the reply says what was wrong.
• Build and test a CSV totals module. Ten invoices with quoted amounts and thousands separators. Pass means: the model's totals() returns the exact per-customer totals when we call it ourselves, its own tests pass, and the reply states the grand total (6,140.34).
• Analyse a log into a report. 318 lines with five error codes at fixed counts. Pass means: report.md has a table with every code and the exact count, sorted most frequent first, and the reply names E101 with 41.

What came back

9 of 9 calls passed every check (fix a failing test suite 3/3; build and test a csv totals module 3/3; analyse a log into a report 3/3). The median call took 3.9 min end to end (fastest 1.5 min, slowest 8.6 min) and made 4 tool calls inside the one request. No self-heal was needed in this run. Every response named its base: Huihui-Qwen3.8-27B-abliterated:UD-Q4_K_XL.

One bar per call, grouped by job. Length is wall-clock time from request to response; solid bars passed every check, outlined bars failed one. Times are from evals/model_proof.py on 2026-10-14: every request went to the Sentinel model endpoint on our Mac mini, which orchestrates the agents and runs the tools; inference for the base model ran on a MacBook Pro (Apple M4 Max, 48 GB) reached over our private tailnet.

Job	Run	Result	Time	Steps	Tools called
Fix a failing test suite	1	pass	7.3 min	5	list_dir → read_file ×2 → edit_file → shell
Build and test a CSV totals module	1	pass (re-checked)	8.6 min	7	shell ×3 → write_file ×2 → shell
Analyse a log into a report	1	pass	3.5 min	4	read_file → shell → write_file
Fix a failing test suite	2	pass	1.5 min	5	list_dir → read_file ×2 → edit_file → shell
Build and test a CSV totals module	2	pass	3.8 min	4	read_file → write_file ×2 → shell
Analyse a log into a report	2	pass	3.9 min	5	read_file → shell ×2 → write_file
Fix a failing test suite	3	pass	3.7 min	6	shell → read_file ×2 → shell → edit_file → shell
Build and test a CSV totals module	3	pass	4.2 min	5	read_file → write_file ×2 → shell
Analyse a log into a report	3	pass	3.9 min	5	read_file → shell ×2 → write_file

Every call passed every check.

One check we got wrong

Our check for the totals job serialised the model's totals() result as JSON, which crashed when a model returned Decimal values, a correct type for money. The check now converts each total to a number before comparing. The affected calls were re-evaluated from the files the model left on disk, without rerunning the model or rewriting anything; the data file keeps each original verdict next to the new one.

• Build and test a CSV totals module, run 1: first recorded as fail (totals() did not run: when serializing dict item 'Acme'), now pass.

The base matters

The same three jobs, sent the same way to Sentinel running on a smaller base (huihui_ai/qwen3-abliterated:8b), passed 0 of 3. The reasons are in the file: fix a failing test suite: tests still fail, hidden median cases fail; build and test a csv totals module: grand total 6,140.34 not in the reply; analyse a log into a report: table counts wrong (E101 14, E204 11, E330 9, E412 5, E505 2), reply does not name E101 with 41. The loop, the tools and the checks were identical. A model with agents inside is only as good as the base those agents think with, which is why every response names it.

How far this is from a general result

• Three jobs, small and self-contained. They exercise reading, editing, running code and checking it, not long projects, the web, or devices.
• 9 calls on one setup and one base model, with inference on a MacBook Pro (Apple M4 Max, 48 GB) reached over our private tailnet. A pass rate from 9 calls has wide error bars; treat it as an observation, not a benchmark.
• The checks are strict but narrow: they verify the outcome we asked for, not the quality of everything the model wrote along the way.
• Sentinel is built on open-weight base models; we did not pretrain one. The base is named in every response and in the data file.

What it means

From the caller's side there is one request and one answer. Inside, the model planned, used its own tools, ran the code it wrote and checked it before replying. The harness is published as model_proof.py and the raw results, including every prompt, answer and tool sequence, are published as model-proof-20261014-1315-c940.json and model-proof-20261014-1311-1e22.json. Run it again and the page is rebuilt from the new file.