Skip to main content

4 min read

A/B Testing Example

A complete experiment engine. Register experiments, assign variants deterministically, track exposures automatically – with two constraints and two resolvers.

Overview

This example builds a self-contained A/B testing system:

  • Experiment registry – register experiments with weighted variants at runtime
  • Deterministic assignment – hash-based variant selection (same user always gets same variant)
  • Automatic exposure tracking – constraint chain records exposures without manual instrumentation
  • Pause/resume – flip one fact to halt all constraint evaluation
  • Reset – clear assignments and exposures, let the engine re-assign

The Module

import { createModule, createSystem, t, type ModuleSchema } from "@directive-run/core";

interface Variant { id: string; weight: number; label: string }
interface Experiment { id: string; name: string; variants: Variant[]; active: boolean }

const schema = {
  facts: {
    experiments: t.object<Experiment[]>(),
    assignments: t.object<Record<string, string>>(),
    exposures: t.object<Record<string, number>>(),
    userId: t.string(),
    paused: t.boolean(),
  },
  derivations: {
    activeExperiments: t.object<Experiment[]>(),
    assignedCount: t.number(),
    exposedCount: t.number(),
  },
  events: {
    registerExperiment: { id: t.string(), name: t.string(), variants: t.object<Variant[]>() },
    assignVariant: { experimentId: t.string(), variantId: t.string() },
    pauseAll: {},
    resumeAll: {},
    reset: {},
  },
  requirements: {
    ASSIGN_VARIANT: { experimentId: t.string() },
    TRACK_EXPOSURE: { experimentId: t.string(), variantId: t.string() },
  },
} satisfies ModuleSchema;

const abTesting = createModule("ab-testing", {
  schema,

  init: (facts) => {
    facts.experiments = [];
    facts.assignments = {};
    facts.exposures = {};
    facts.userId = "user-abc123";
    facts.paused = false;
  },

  derive: {
    activeExperiments: (facts) =>
      facts.experiments.filter((e) => e.active && !facts.paused),
    assignedCount: (facts) => Object.keys(facts.assignments).length,
    exposedCount: (facts) => Object.keys(facts.exposures).length,
  },

  events: {
    registerExperiment: (facts, { id, name, variants }) => {
      facts.experiments = [
        ...facts.experiments,
        { id, name, variants, active: true },
      ];
    },
    assignVariant: (facts, { experimentId, variantId }) => {
      facts.assignments = { ...facts.assignments, [experimentId]: variantId };
    },
    pauseAll: (facts) => {
      facts.paused = true;
    },
    resumeAll: (facts) => {
      facts.paused = false;
    },
    reset: (facts) => {
      facts.assignments = {};
      facts.exposures = {};
    },
  },

  constraints: {
    needsAssignment: {
      priority: 100,
      when: (facts) => {
        if (facts.paused) {
          return false;
        }

        return facts.experiments.some((e) => e.active && !facts.assignments[e.id]);
      },
      require: (facts) => {
        const unassigned = facts.experiments.find(
          (e) => e.active && !facts.assignments[e.id],
        );

        return { type: "ASSIGN_VARIANT", experimentId: unassigned!.id };
      },
    },

    needsExposure: {
      priority: 50,
      when: (facts) => {
        if (facts.paused) {
          return false;
        }

        return Object.keys(facts.assignments).some((id) => !facts.exposures[id]);
      },
      require: (facts) => {
        const experimentId = Object.keys(facts.assignments).find(
          (id) => !facts.exposures[id],
        );

        return {
          type: "TRACK_EXPOSURE",
          experimentId: experimentId!,
          variantId: facts.assignments[experimentId!],
        };
      },
    },
  },

  resolvers: {
    assignVariant: {
      requirement: "ASSIGN_VARIANT",
      resolve: async (req, context) => {
        const experiment = context.facts.experiments.find((e) => e.id === req.experimentId);
        const variantId = pickVariant(context.facts.userId, req.experimentId, experiment!.variants);
        context.facts.assignments = { ...context.facts.assignments, [req.experimentId]: variantId };
      },
    },

    trackExposure: {
      requirement: "TRACK_EXPOSURE",
      resolve: async (req, context) => {
        context.facts.exposures = {
          ...context.facts.exposures,
          [req.experimentId]: Date.now(),
        };
      },
    },
  },

  effects: {
    logAssignment: {
      deps: ["assignments"],
      run: (facts, prev) => {
        if (!prev) {
          return;
        }

        for (const [id, variant] of Object.entries(facts.assignments)) {
          if (!prev.assignments[id]) {
            console.log(`[ab-testing] Assigned ${id} → ${variant}`);
          }
        }
      },
    },
  },
});

How It Works

The engine runs a two-step constraint chain:

  1. Register – events.registerExperiment() adds to the experiments array
  2. Assign – needsAssignment constraint fires: active experiment + no assignment → ASSIGN_VARIANT
  3. Resolve – assignVariant resolver hashes userId + experimentId → weighted variant pick
  4. Expose – needsExposure constraint fires: assigned + no exposure → TRACK_EXPOSURE
  5. Record – trackExposure resolver stores timestamp in exposures

The engine settles automatically. No manual orchestration needed.

Key Patterns

Deterministic hashing

The same userId + experimentId always produces the same variant:

function hashCode(str: string): number {
  let hash = 0;
  for (let i = 0; i < str.length; i++) {
    const char = str.charCodeAt(i);
    hash = ((hash << 5) - hash + char) | 0;
  }

  return Math.abs(hash);
}

function pickVariant(userId: string, experimentId: string, variants: Variant[]): string {
  const hash = hashCode(`${userId}:${experimentId}`);
  const totalWeight = variants.reduce((sum, v) => sum + v.weight, 0);
  let roll = hash % totalWeight;

  for (const variant of variants) {
    roll -= variant.weight;
    if (roll < 0) {
      return variant.id;
    }
  }

  return variants[variants.length - 1].id;
}

Automatic exposure tracking

No manual trackExposure() calls. The constraint chain fires automatically after assignment:

needsAssignment → ASSIGN_VARIANT → needsExposure → TRACK_EXPOSURE → settled

Pause guard

Both constraints check facts.paused first. Flipping one boolean halts the entire experiment engine without clearing assignments.

Try It

cd examples/ab-testing
pnpm install
pnpm dev

Register experiments, watch the constraint chain assign variants and track exposures. Use "Pause All" to halt evaluation. Use "Reset" to clear assignments and watch re-assignment happen automatically.

Was this helpful? Star us on GitHub or support the project.

We care about your data. We'll never share your email.

Powered by Directive. This signup uses a Directive module with facts, derivations, constraints, and resolvers – zero useState, zero useEffect. Read how it works →

Directive - Constraint-Driven State Management for TypeScript