Test doubles that can say no

Here's a test that passes whether or not your code works:

const mock = createMockAgentRunner({
  responses: { researcher: { output: 'done', delay: 5000 } },
});

const controller = new AbortController();
const run = orchestrator.run(agent, 'go', { signal: controller.signal });
controller.abort();

await run;   // ...and?

If the double ignores the signal and runs to completion, the call returns normally. Your assertion — whatever it is — is measuring a mock that never had the option of stopping.

In 1.28.1, createMockAgentRunner honours AbortSignal.

What changed

The double's configured delay is now interruptible. Abort during it and the call rejects, the way a real client does when you abort a request in flight:

const mock = createMockAgentRunner({
  responses: { slow: { output: 'never arrives', delay: 5000 } },
});

const controller = new AbortController();
const call = mock.run(agent, 'go', { signal: controller.signal });
setTimeout(() => controller.abort(), 10);

await expect(call).rejects.toThrow();   // and now it does

The rule it follows is the one a real HTTP client follows: aborting before the call starts or while it's in flight rejects, but an answer already in hand isn't thrown away because the signal fired afterwards.

That last clause matters more than it sounds. It's the difference between a double that models cancellation and one that just fails aggressively.

What it exposed

Two behaviours in the orchestrator that had never actually been exercised.

DAG timeouts now cut work off. A node with timeout: 50 sitting in front of a 500ms operation reaches "error" instead of "completed". Previously the timeout fired, the stand-in slept through it, and the node finished anyway — so the timeout was configuration with no observable effect in test.

race tells a cancelled loser from a failed one. When a race picks a winner, it cuts off the others. An agent that honours that signal stops with an error — and stopping-because-we-told-you-to is not the same event as failing:

const result = await orchestrator.runRace(pattern, { input });
// race_cancelled now fires for agents the race actually stopped

Before, the race_cancelled event fired only when a loser had ignored the signal and run to completion — which is to say, precisely when nothing had been cancelled. Now it fires when something was.

If you build dashboards on orchestrator events, this is the one to know about: cancellations that used to look like failures will start looking like cancellations.

Why we're telling you rather than just shipping it

Because the shape is worth borrowing.

A test double is a claim about how the real thing behaves. When the double can only succeed, every test written against it inherits that limit — and those tests look identical to real ones. Green, named for the guarantee, asserting nothing that could fail.

The tell, in our case, was a comment. Two of our own tests said, in so many words, this passes because the mock ignores abort signals — and then asserted the passing behaviour as though it were the contract. The comment was honest. The assertion was the problem.

Worth asking of your own doubles: can this thing fail in the way the real thing fails? If not, the tests around it are measuring the double.

Upgrading

npm install @directive-run/ai@latest

If you have tests asserting that a timed-out DAG node completes, or that a race loser appears as an error, those assertions were describing the double and will now describe the runtime. That's the upgrade working.

Cancellation is one of those things every async API claims to support and few can demonstrate. It's testable here now — which is a smaller sentence than it deserves.

Related

• The PII screen sees more of your data now – The personal-data guardrail keeps up with modules registered at runtime, and no longer skips a whole value because one member is awkward.
• Pricing tables that know their own age – Every rate table in @directive-run/ai now ships the date a human last checked it against the provider — and a check that fails when that date gets old.
• Swap a derivation while the system is running – Replace a computed value's definition at runtime and settle() comes back like it should. Plus system.meta.revision(), an O(1) way to know when your metadata cache is stale.