The Loop, Verified: Evidence That Elis AI Saves, Contests, and Selects Decision Chains

Status: Internal draft — listed in the whitepapers index, which is
restricted to sales reps and platform admins. Not a public document.
Version: 0.3-draft
Study dates: 2026-08-05 → 2026-08-06
Evidence policy (owner directive): every finding below rests exclusively
on runs executed 2026-08-05 → 2026-08-06. No earlier telemetry is used
anywhere in this paper.

Terminology

Terms below are used precisely throughout this paper and are grounded in the
current implementation, not simplified for narrative convenience.

• The platform / "the system" means the Elis AI orchestration service — the
  agent → template → task → step engine that receives a prompt, resolves an
  agent and template, dispatches to a miner, and persists the run
  (.claude/docs/orchestration-and-observability.md §0). When this paper says
  a prompt was "never seen by the system," it means no prior chain_runs,
  agent_chain_variants, or EMA state existed for that exact prompt/agent
  pairing before this study's Round 0 run (§2.2) — not that the prompt or
  its subject matter was absent from any model's training data. The two are
  unrelated: a model can have encountered similar text during pretraining while
  the platform's own routing and quality state for that specific prompt is
  still cold.
• EMA (exponential moving average). The running average of grader scores
  (and related per-run signals) maintained per layer — agent, template,
  tool-bundle, model-profile, variant, miner-model — on a 0–1 scale,
  updated with alpha = 0.2 after each attributed run (SpeedModeMonitor uses
  alpha = 0.10). It is the learned quality signal that selection reads back
  at rank time (e.g. quality_fit in the model scorer;
  grader_score_ema/success_ema in template and agent ranking).
• Canary / traffic-gated canary. A newly minted decision-chain variant — an
  agent_template_versions row with status='canary' — produced by rewriting
  an existing template. In this paper every canary is triggered by a
  negative-feedback rewrite (FeedbackTemplateRewriter). It is never switched
  fully live: it is exposed to only a fraction of traffic (traffic_fraction)
  while the existing incumbent (the latest active version) keeps serving
  the rest, so both versions accumulate independent grade samples under
  identical conditions before any decision is made.
• Traffic gate. The deterministic admission mechanism that decides, per
  run, whether the canary or the incumbent serves. Implemented as
  _admit_canary in backend/agents/template_retriever.py: it takes the SHA1
  hash of the UTF-8 bytes of "{run_id}|{template_id}", reads the first 8
  bytes of the digest as a big-endian integer, divides by 2^64 to map it into
  [0, 1), and admits the canary only if that value is less than the
  configured traffic_fraction. Because the hash is a pure function of
  (run_id, template_id), the same run always gets the same answer on a retry
  (idempotent repair doesn't flap), while different runs spread smoothly across
  the fraction. Production defaults traffic_fraction to 0.10; this study set
  it to 0.5 (§2.3) to reach a verdict in fewer runs.
• Promotion gate. The adjudication that decides whether a canary graduates,
  implemented by CanaryPromoter (backend/agents/canary_promoter.py). Once a
  canary has accumulated at least CANARY_MIN_SAMPLES (default 20) attributed
  graded runs, the promoter compares the canary's grade statistics against its
  incumbent parent's: it promotes (archives the parent, flips the canary to
  active at full traffic) when the canary is ahead, and holds — archives
  the canary, keeps the incumbent — when the canary underperforms by more than
  CANARY_DEMOTE_MARGIN (default 0.05), a deliberately conservative margin
  chosen to stop flapping at the boundary rather than promote/demote on noise.
  This study tightened the sample threshold to 10–20 samples (§2.3) to reach a
  verdict sooner; the promote/hold comparison logic itself was not changed.
• Grader. The platform's own model-based grader. After a run completes, it
  scores the answer and writes the result to chain_runs.grader_score on a
  0–1 scale; that score, not any human rating, is what feeds every EMA above.
  Operationally this means the paper measures the loop's ability to improve
  its own quality signal — the one selection actually reads back — not
  independent human-judged answer quality (§2.4, §4).

1. Executive summary

1.1 Abstract

This paper reports what was directly observed when the self-learning loop was
exercised end-to-end on live runs across three campaigns run 2026-08-05 →
2026-08-06 on the local development stack: a real downvote minting a rewritten
decision chain, that chain being served under a deterministic traffic gate
alongside its incumbent, grades from both being attributed and accumulated as
per-version EMAs, and a promotion gate adjudicating the result. The headline
result is a matched pair of adjudications — the gate promoted a
feedback-born chain that scored higher than its incumbent (0.7678 vs 0.7312,
grader-score EMA on a 0–1 scale, bakery campaign, 20 attributed samples) and
held one that scored lower (0.8838 vs 0.9252, trains campaign, 10 samples)
— plus a separate finding that reusing an already-built chain cut average
generation tokens per run 15.4% (6,045 → 5,111, n=29, trains campaign). So
what: the mechanism the product claims will make routing better over time is
not aspirational — in this window it ran end to end, produced a verifiable
decision in both directions (it can say yes and it can say no to a rewrite),
and left a reproducible trail; what remains unproven is whether that mechanism
moves real-tenant answer quality at scale, which is the gap §5 registers.

1.2 Introduction

The pitch for Elis AI's self-learning loop is that agent, template, and model
selection improve themselves from graded outcomes — a downvote should be able
to change what gets served next, and a rewritten chain should only stick if it
actually performs better. Before this study, that claim rested on aggregate
observational data that could not support a causal reading: code and policy
changed continuously across the observation window, the agent mix shifted
month to month, and the large majority of historical runs were personal-scope
with no real tenant corpus (full confound list:
.claude/docs/self-learning-proof-experiment.md §0). This paper does not
attempt to remove those confounds — that is the job of the pre-registered
three-arm experiment in that document, which has not yet run. What this paper
sets out to prove instead is narrower and mechanical: that every stage of the
loop — mint, serve, attribute, accumulate, adjudicate — actually executes on
live runs, that its outputs are reproducible from logged state, and that the
promotion gate's decision tracks the grade rather than always promoting. That
is a claim about the mechanism firing correctly, not about how much it
improves quality at scale.

Three claims, in decreasing order of evidential strength:

1. The self-learning loop is operational end-to-end, verified live
   (campaigns: bakery, trains, and email; window 2026-08-05 → 2026-08-06). A
   user's 👎 on a real answer mints a rewritten decision chain as a
   traffic-gated canary; the canary serves deterministically alongside the
   incumbent; every run's grade is attributed to the exact chain version that
   served it; samples accumulate as a per-version grader-score EMA (0–1 scale,
   alpha = 0.2); and the promotion gate adjudicates. Every one of those
   transitions was observed on live runs in this window and is reproducible
   from logged state (§6).

2. The selection gate works in both directions, measured as a
   grader-score EMA on a 0–1 scale and evaluated by the promotion gate at its
   sample threshold. In the first two adjudicated contests in the system's
   history, the gate promoted a feedback-born chain on the bakery campaign
   (arithmetic reasoning) that overtook its incumbent (0.7678 vs 0.7312 over 20
   attributed samples, 2026-08-05→06) and refused to promote one on the
   trains campaign (multi-step word problem) that stayed worse (0.8838 vs
   0.9252 at 10 samples, same window). Promote-when-better and hold-when-worse,
   both demonstrated.

3. Chain reuse is measurably cheaper, measured in average generation
   tokens per run. On the trains campaign (n=29 repeat runs vs n=2 first-ask
   runs, 2026-08-05→06) — the one campaign with a clean first-ask baseline —
   repeat runs of the same question cost 15.4% fewer generation tokens
   than the first ask (6,045 → 5,111 avg, n=29), consistent with the
   built-chain / cached-context hypothesis.

What this paper does not claim: a statistically powered accuracy
improvement from learning (n=2 contests, one in each direction, self-graded, in
a development environment), or any result on real tenant workload. §5 names the
registered experiment that will settle the former and the pilot prerequisite
for the latter.

2. Method

2.1 Environment and campaigns

All runs executed on the local development stack — a single 32GB host running
the full agent → template → task → step orchestration funnel
(.claude/docs/orchestration-and-observability.md §0) against a local
Postgres, never production. Three campaigns supplied the evidence in this
paper: bakery (an arithmetic-reasoning prompt; the promoted contest, §3.2),
trains (a multi-step word problem; the held contest and the
token-efficiency comparison, §3.2–3.3), and email (a code-generation
prompt that surfaced a path-coverage gap, §3.5). Every prompt is synthetic —
authored for this study, not drawn from customer traffic — and every run is
first-party: no data from any real tenant appears anywhere in this paper
(owner directive, see header).

2.2 Protocol

Per campaign, the protocol ran in four stages:

• Round 0 — the fresh prompt's very first run. "Fresh" means no prior
  chain_runs, agent_chain_variants, or EMA state existed for that exact
  prompt before this run (see Terminology, "never seen by the system") — this
  run is what mints the agent and the chain in the first place, and it records
  the first-ask token cost used as the baseline in §3.3.
• Phase A — additional baseline runs of the same prompt on the newly
  minted, unchanged chain. These extend the first-ask/repeat cost comparison
  in §3.3 beyond the single Round-0 run.
• Phase B — a 👎 is submitted against a Phase-A answer through the real,
  production feedback endpoint (POST /api/messages/<id>/feedback), carrying
  facets plus a written note: free text describing concretely what was
  wrong with that answer, authored by the campaign operator standing in for
  the end user. There was no real end user in this study — the operator plays
  that role deliberately, submitting through the same endpoint and payload
  shape a real user's browser would send, so the rewrite path being tested is
  the real one, not a stand-in. That feedback is what mints the canary rewrite
  (FeedbackTemplateRewriter).
• Phase C — the same prompt is repeated in chunks of 5 runs (a fresh
  conversation_id every run, per §2.3), and for every run the deterministic
  traffic gate (Terminology) decides whether the canary or the incumbent
  serves. Each run's grade is attributed to whichever version served it and
  accumulated into that version's EMA; once the canary has enough attributed
  samples, the promotion gate (Terminology) adjudicates.

This sequencing is what §3.1 calls "the loop, segment by segment": mint
(Phase B) → serve (Phase C's traffic gate) → attribute → accumulate (EMA) →
adjudicate (promotion gate).

2.3 Controls and hygiene

• Resource ceiling. All chunked execution ran under the project's hard
  90%-of-host-memory ceiling (.github/instructions/resource-usage.instructions.md)
  — concretely, 90% of this study's 32GB host is ≈28.8GB; runs were
  chunked and watched against that number rather than run as one unbounded
  batch.
• Log files, two jobs at once. Every phase streamed its state to the raw
  chronological logs in .claude/logs/learning-evidence/
  (promotion-campaign.log, campaign-trains.log, campaign-email.log).
  Those files serve two roles simultaneously: they are the
  interruption-resumability cursor (a killed or interrupted campaign resumes
  from the last logged chunk instead of re-running from scratch), and they are
  the raw evidence record this paper cites in §6 — the same file is both the
  operational safety net and the audit trail.
• Exactly-once propagation. Grade propagation into per-version EMAs ran
  via a cursor so a retried or re-read chunk cannot double-count a grade.
• Mint throttle. A throttle prevented additional canary versions from
  being minted mid-contest, so a single contest always compares exactly one
  canary against exactly one incumbent.
• Test knobs, disclosed. Canary traffic fraction was set to 0.5 for this
  study (prod default 0.10, per _admit_canary's caller in
  template_retriever.py) and the promotion sample threshold was tightened to
  10–20 samples (prod default CANARY_MIN_SAMPLES=20, per
  backend/agents/canary_promoter.py) — both changes only shorten how many
  runs it takes to reach a verdict; neither changes the gate's decision
  logic (the deterministic hash in the traffic gate, or the demote-margin
  comparison in the promotion gate).

2.4 Grading

The platform, in this paper, always means the Elis AI orchestration
service — the agent → template → task → step engine that receives a prompt,
resolves an agent and template, dispatches to a miner, and persists the run
(Terminology; .claude/docs/orchestration-and-observability.md §0). The
grader is that platform's own model-based grader: after a run completes, it
scores the answer and writes the result to chain_runs.grader_score on a 0–1
scale. That score — not any human rating — is what feeds every EMA in this
paper (Terminology). Operationally this means the paper measures the loop's
ability to improve its own quality signal, the one selection actually reads
back at rank time; it is not a measurement of human-judged answer quality, and
§4 says so explicitly.

2.5 Reproducing this protocol

This is not the three-arm blinded design (control-repeat /
treatment-rewrite / sham-mutation) that removes regression-to-the-mean and
version-churn as explanations — that design is pre-registered but not yet run
(.claude/docs/self-learning-proof-experiment.md). What follows reproduces
this paper's simpler, non-blinded protocol:

1. Pick a prompt with no prior chain_runs / agent_chain_variants / EMA
   state for the agent it will match (confirm by query — this is what "never
   seen by the system" means; see Terminology).
2. Run it once (Round 0) in a fresh conversation. This mints the agent and
   chain and records the first-ask token cost.
3. Run several more repeats of the same prompt on the unchanged chain
   (Phase A) to build the first-ask-vs-repeat cost comparison.
4. Submit a 👎 with facets and a written note through
   POST /api/messages/<id>/feedback (Phase B) against one of those
   answers. Confirm a new agent_template_versions row with
   status='canary' appears, parented to the incumbent.
5. Repeat the same prompt in chunks of 5 fresh conversations (Phase C)
   until the canary accumulates enough attributed samples for the promotion
   threshold (CANARY_MIN_SAMPLES, default 20, or a tightened test value).
   Watch host memory/CPU against the 90% ceiling (§2.3) while chunking.
6. Read chain_runs (grades + which version served each run),
   agent_template_versions (canary vs incumbent status), and
   variant_quality_comparisons (the promotion verdict) to see the same
   trace this paper reports in §3.1.

Reproducing the causal claim — that the rewrite, not chance or version
churn, produced any grade difference — requires the three-arm design, not this
protocol; that is precisely the gap §5 registers.

3. Results

3.1 The loop, segment by segment (all observed live)

• 👎 → canary mint through the production path: user_feedback_rewrite
  versions created at 0.5 traffic fraction in both completed campaigns.
• Deterministic serving: canary vs incumbent split reproduces from
  (run_id, template_id) hashing; serving and attribution agree independently.
• Attribution: 100% of orchestrator runs carry the exact chain version that
  served them (13/13, then sustained across ~60 campaign runs).
• Accumulation: per-version EMAs track grades in hand-checkable steps
  (e.g. 0.5 → 0.574 → 0.629 → 0.680 → 0.708 → 0.731 across five 0.8-grade runs).
• Self-repair: the cohesion sweep in quarantine mode archived 8 incoherent
  catch-all agents; a subsequent prompt in the archived agent's territory was
  re-served by a freshly minted specialist with no user-visible failure
  (gate-b verification). The evolution arm also minted an unprompted template
  upgrade from a high-quality run.

3.2 The contests

The bakery trajectory is the learning story in miniature: the canary trailed
at 5 samples (0.6881), led at 10 (0.7475), and won at 20 (0.7678) — evidence
accumulated, the estimate converged, and the gate acted only at threshold. The
trains hold is equally load-bearing: a plausible rewrite that graded worse was
not promoted. A gate that only ever promotes proves nothing.

3.3 Token efficiency (first ask vs repeat)

Every dispatch is stamped with its run identity, so cost joins to the exact
run that incurred it. On the trains campaign — the one with a clean first-ask baseline:

−15.4% on reuse in tokens. Priced per dispatch at the serving model's
catalog rate — tokens are not interchangeable across models, so each dispatch
is costed at the model that actually ran it — the saving is larger in dollars:

Dollars fall faster than tokens because reuse also shifts the mix: repeat
runs lean harder on cheaper output-light dispatches and the small-tier model
(§3.4). Repeat cost was also stable (bakery repeats: ~3.9k tokens ± small
across chunks). Embedding-kind dispatches are reported separately by
construction; indexing cost is never blended into inference cost. Catalog list
rates used throughout (per 1M tokens in/out: haiku $1/$4, gpt-4.1 $1/$4,
o3-class $2/$8) — the same source the platform's savings feature prices from,
never live billing.

3.4 Right-sized model selection (the binder's contribution)

Chain reuse (§3.3) is one savings mechanism; the runtime binder choosing the
smallest adequate model is a second, and the campaign corpus measures it
directly. Across all campaign runs (generation dispatches, run-joined):

96% of calls landed on a small-tier model while grader quality held at
0.73–0.93 — the definition of right-sizing: the binder did not buy frontier
capacity these tasks didn't need.

Counterfactual: one frontier model doing every task in the corpus (same
token volumes — 269,683 in / 53,965 out — priced at each model's catalog list
rate) versus what the binder actually chose:

Sending this workload to an Opus-class frontier model would have cost 2.7×
what the binder's mix cost, for tasks a small model already answered at
0.73–0.93 grader quality. Honest bounds: the counterfactual holds token counts
fixed, uses list prices only, and these campaign prompts are exactly the kind
a right-sizer should route small — the claim is "small where small suffices,
without quality loss," not a universal 63% saving. (gpt-4.1's list price sits
at the small tier, which is why that comparison is flat — the saving comes
from avoiding the models priced 2–4× higher.) A workload-weighted version over
real tenant traffic belongs to the §5 milestones.

3.5 A finding the protocol produced for free

The third campaign (a code-generation prompt) minted its canary and then
accumulated zero samples: code-shaped prompts route down an execution path
not yet wired to the execution-record writer. The campaign protocol surfaces
path-coverage gaps by construction — a run that produces no record is a run
the loop cannot learn from, and the protocol makes that visible in one chunk.

4. What may be claimed, and what may not

Claimable now (each backed by logged, reproducible post-fix runs):
the loop runs end-to-end; feedback causes chain rewrites; rewritten chains are
contested under traffic gating; the gate promotes better chains and refuses
worse ones; chain reuse reduced generation tokens 15.4% in the measured
campaign; the matching layer self-repairs via cohesion quarantine without
stranding demand.

Not claimable: that answers are N% more accurate because of learning
(two contests, one per direction, is a demonstration, not a distribution);
anything about real tenant outcomes (all evidence is first-party, dev
environment, synthetic prompts); durability of the token saving beyond the
measured campaign; human-judged quality (grader is model-based).

Estimated-vs-realized savings is measured separately, and it is not this
paper's evidence base. A 2026-08-27 local analysis (token_saver_plans
joined to saved_token_ledger, 281 plans / 186 ledger-joined) found modest
overprediction of token savings (mean realized/estimated ratio 0.846 across
the 12 plans with both counts) and no usable signal on dollar savings in
either direction: the only non-zero realized dollar figures are priced from
the estimate itself (estimated_unsplit_execution_usage, circular by
construction), every independently priced basis recorded $0.00, and 59% of
joined rows carry a blank measurement basis — a measurement-pipeline gap
rather than proof of zero savings. Token and dollar savings are distinct
quantities (more tokens on a cheaper model can still cost less). That gap
feeds recalibration of the estimator as a training signal; it is not a
billing input (settlement always charges the reserved planning estimate) and
it is not evidence for or against the self-learning claims made above.

5. Registered next milestones

1. Three-arm controlled experiment (design pre-registered in the repo:
   control-repeat, treatment-rewrite, sham-mutation arms; paired within-question;
   blind grading; regression-to-mean floor reported). Its Phase 0 blocker —
   attribution — is now cleared. This is what converts §3.2 from demonstration
   to measurement.
2. Wire the remaining early-exit paths (expert, task-orchestrated, and the
   code path from §3.5) so every run enters the loop; then re-run the paused
   campaign.
3. Real tenant corpus — ≥2 external organizations, ≥1,000 org-scoped runs,
   ≥8 weeks, stable agent set. The pilot ask in the deck is the vehicle. No
   tenant-level claim before this exists.

6. Reproducibility

Raw chronological logs: .claude/logs/learning-evidence/ —
promotion-campaign.log (bakery contest, chunk by chunk, plus combined
summaries), campaign-trains.log, campaign-email.log (path-gap finding),
plus the numbered query outputs from the 0.1 study retained for the §2 defect
record. Deploy-facing summary: .claude/docs/dev-17-deploy-notes.md. Defect
catalogue with fixes and verification: .claude/docs/template-attribution-bug.md.
Experiment design: .claude/docs/self-learning-proof-experiment.md.

All measurements ran read-write against the development stack only; production
was touched read-only and contributes nothing to the post-fix evidence base.

© 2026 Elis AI. All rights reserved. tryelisai.com