Pam Dev-Team Review, 2026-08-27

pam@4e68027 + pam-launcher@877b6c9, sandbox review, live Pam untouched.

Findings digest (all 24, ranked)

Findings digest: what is actually stopping Pam from hitting 95 percent

The one-line answer. Pam's WISMO failures and her four low scores have two independent
causes, and neither is "the model needs a better prompt": the deterministic lookup
enforcement that already exists is wired into the offline lanes and not into the live send
path, and the grading pipeline is wrong in both directions, so the numbers she is being
measured against are not measuring what they claim.

The reproduction, verbatim from checks/chk03_tracking_skip_repro.py. These three lines
are the check's own output and are reproduced here so the digest cannot drift from what the
code actually does:

CLASSIFICATION primary=caught-post-hoc-only

CLASSIFICATION unworded=not-caught

CLASSIFICATION primary_with_unrelated_tool=not-caught

Read plainly: a WISMO reply claiming a live tracking check that never happened is caught
only after the fact, only in the nightly backtest, only if it uses the word "live", and not
at all if any unrelated tool fired on the ticket. On the live customer lane it is not caught
at any point.

What round 4 added. Rounds 1 to 3 audited the backtest grader, the golden checker, the
live send path, the policy grader and the Pam-versus-Ross referee. This round finished the
brief's deliverable 3 by auditing the last uncited link in the chain: the code that turns
graded rows into the four published percentages, repo/brain/eval/metrics_core.py (the
append-only snapshot ledger) and the metric functions it calls in
repo/brain/dashboard/patch_generated_pages.py. Findings F-21 to F-24 come from there, and
they are a different class from everything above them. F-01 to F-20 are reasons Pam's answers
are wrong or are graded wrongly. F-21 to F-24 are reasons the four numbers themselves cannot
currently be trusted to move when a fix lands: the Backtest headline has two definitions that
disagree, its window freezes instead of going stale, the Autopilot rate rises as Pam answers
less, and the ledger's one alarm cannot see a ruler stop reporting. None of them raises a
score. All four have to be true before "95 percent" means anything.

What round 3 added. The first two candidates audited the backtest grader, the golden
checker and the live send path. This round extended the same audit to the two scoring files
nothing had cited yet: repo/brain/eval/policy_grade.py, the second grading lane, and
repo/brain/eval/grading_referee.py, the referee that produces the Pam-versus-Ross agreement
number and owns the only grader-drift alarm. Findings F-17 through F-20 come from there, and
they sharpen rather than change the bottom line: the largest single defect in the pipeline is
that one unrelated tool call, or one grader-side re-fetch, tells the grading model to treat a
fabricated delivery status as verified.

How to read a finding. severity: real-defect means it is demonstrated by a check under
checks/defects/ that currently exits non-zero against the unfixed review copy.
severity: coverage-gap means the review copy cannot answer the question at all; those carry
a written exception in docs/exceptions.md. expected_gain_pct is percentage points toward
the 95 percent bar, my estimate, ordered highest first. Every estimate is a judgment call and
is labelled as one; the evidence under it is not.

F-01 - The live send path enforces no lookup and no fabricated-claim check

severity: real-defect

tag: RESTRUCTURE

expected_gain_pct: 12

scores_moved: Team Members, Ross's Claude, Autopilot

evidence: repo/launcher/pam_pipeline.py:1257, repo/brain/guardrail.py:223, repo/brain/tools/draft_gates.py:346, repo/brain/eval/draft_morning.py:420

expected_impact: Should move Team Members and Ross's Claude most, because both grade live-lane drafts; roughly 10 to 15 points if the WISMO class is as large in live traffic as it is in the golden set.

gating_check: checks/defects/defect_f01_live_lane_unenforced.py

The enforcement already exists. repo/brain/tools/draft_gates.py:346 is GATE 1
(lookup_gate), a three-state deterministic check on whether the facts a reply depends on
were actually fetched, and it is called from repo/brain/eval/draft_morning.py:420 and from
the backtest. The live lane never calls it. repo/launcher/pam_pipeline.py:1257 runs
guardrail.check and nothing else, and repo/brain/guardrail.py:223 has no rule about
lookups, tracking or tools at all: it checks completion claims, address and ship promises,
approval lines, the dollar ceiling, refund promises, preamble leaks and internal URLs.

So the rule "you MUST call your read-only lookup tools" (repo/brain/RULES.md:52) is
enforced by the model remembering it, on the one lane where a customer is waiting. That is
the determinism objection from Hard Rule 14 in its purest form: a step with a fixed
input-to-output test (did a lookup run for this category before this reply?) is being left
to judgment.

The fix is not new code, it is moving existing code. Call lookup_gate from
pam_pipeline.finalize before the guardrail, on customer-reply channels, with the same
three states the offline lane uses. That is why this is tagged RESTRUCTURE rather than FIX:
the change is where enforcement lives, not what it does.

F-02 - Any tool call disarms the live-tracking tripwire

severity: real-defect

tag: FIX

expected_gain_pct: 9

scores_moved: Backtest, Ross's Claude

evidence: repo/brain/eval/backtest_distill.py:972, repo/brain/eval/backtest_distill.py:957

expected_impact: Backtest most directly, since the tripwire is what flips a fabricated-claim row to fail there; expect the measured fabrication rate to rise before it falls, which is the point.

gating_check: checks/defects/defect_f02_live_claim_tool_whitewash.py

_live_claim_mismatch returns False as soon as tools is non-empty
(repo/brain/eval/backtest_distill.py:972), on the stated reasoning that she "really did
fire a live tool". But tools is every tool fired on the ticket, not every TRACKING tool. A
prior-ticket search, a store-credit lookup or a Gorgias read is enough to make a fabricated
live-tracking claim invisible. The demonstration runs the identical reply twice and gets
True with no tools and False with gorgias_search_tickets.

Fix: compare against the tracking-capable tool names (shipstation_tracking,
shopify_order_lookup, the tracking-snapshot path), not against the truthiness of the list.

F-17 - One unrelated tool call tells the grader every specific is verified

severity: real-defect

tag: FIX

expected_gain_pct: 9

scores_moved: Backtest, Ross's Claude

evidence: repo/brain/eval/backtest_distill.py:1902, repo/brain/eval/backtest_distill.py:1450, repo/brain/eval/backtest_distill.py:1453

expected_impact: Backtest first and Ross's Claude second. Not additive with F-02: both come from the same input, so fixing the input once collects both. Listed separately because the consumer, the gating check and the fix site are different.

gating_check: checks/defects/defect_f17_grader_tool_truthiness.py

F-02 is the regex tripwire. This is the grader itself, and it is the larger of the two.
repo/brain/eval/backtest_distill.py:1902 computes the grader's fabrication context as
looked_up=(bool(tools) or injected), and :1450 turns a True there into a flat
instruction: the order, tracking, carrier and status specifics she states "come from a real
lookup, so treat them as VERIFIED". The else branch at :1453 is the opposite
instruction. There is no third state.

So one gorgias_search_tickets call on a ticket switches the grader from "any specific she
states is FABRICATED" to "treat them as VERIFIED", for the whole draft. The demonstration
lifts the keyword expression out of the sealed source and evaluates it: tools=[] gives
False, tools=['gorgias_search_tickets'] gives True.

This is why the Backtest number can be high on exactly the class Anthony described. The
fabricated WISMO reply is not being missed by a weak regex; it is being actively vouched for
to the grading model.

Fix: build looked_up from the tracking-capable tool names, and pass the real per-claim
provenance rather than one boolean for the whole draft. Same one-line fix as F-02, applied at
the other call site.

F-03 - A fabricated delivery status is invisible unless it uses the word "live"

severity: real-defect

tag: FIX

expected_gain_pct: 8

scores_moved: Backtest, Team Members, Ross's Claude

evidence: repo/brain/eval/backtest_distill.py:942, repo/brain/AGENT.md:186, fixtures/wismo_tracking_fixture.json:1

expected_impact: Backtest and Team Members; this is the defect class Anthony described, and the golden set's C1 cluster (g-10, g-19, g-20) is the same shape.

gating_check: checks/defects/defect_f03_status_claim_undetected.py

Every pattern in _LIVE_CLAIM_PATTERNS (repo/brain/eval/backtest_distill.py:942) requires
the word "live" or the word "carrier" beside a verb of having done the check. The failure
Anthony described does not need either word. "I checked your tracking and your package is
out for delivery today" asserts a live delivery state, is forbidden in prose by
repo/brain/AGENT.md:186, and trips nothing: not the pre-send guardrail, not the tripwire,
not the fabrication flags. That is the CLASSIFICATION unworded=not-caught line above.

Fix: detect the CLAIM (a delivery-state assertion: delivered, out for delivery, in transit,
arriving today) rather than the wording of the check, and require a live snapshot or a
tracking tool call to back it. The delivery-state vocabulary is small and closed, which is
exactly what makes this a script rather than a judgment.

F-04 - The golden referee cannot adjudicate nine of its thirty rows and never says so

severity: real-defect

tag: FIX

expected_gain_pct: 7

scores_moved: Team Members

evidence: repo/brain/eval/golden/golden_check.py:41, repo/brain/eval/golden/golden_check.py:43, repo/brain/eval/golden/golden_check.py:100

expected_impact: Team Members directly: the 23.8 percent figure is 5 over 21, and 21 is what 30 becomes once the nine unadjudicable rows drop out. Reporting them makes the number honest rather than higher.

gating_check: checks/defects/defect_f04_golden_unadjudicable_verdicts.py

A row is only ever compared when its scorer_verdict is in SCORER_FAIL_VERDICTS
(repo/brain/eval/golden/golden_check.py:41) or SCORER_PASS_VERDICTS (:43). Nine of the
thirty labeled rows carry stale or pending, so they can never be a mismatch no matter
what Ross wrote. Three of those nine (g-24, g-25, g-27) are labeled pam_wrong: real
failures the mismatch metric structurally cannot see.

The unknown-value guard at repo/brain/eval/golden/golden_check.py:100 validates
ross_label only. An unrecognized scorer_verdict is never reported, so status.json shows
unknown_labels: [] and looks clean while a third of the set sits outside the comparison.
Meanwhile n_labeled counts all thirty, so any rate computed from it is over a denominator
that includes rows the referee refuses to judge.

Fix: validate scorer_verdict against a known set the same way labels are validated, report
the unadjudicable population in status.json explicitly, and compute the rate over the
adjudicable rows only.

F-05 - "pending" means three different failures and all of them leave the denominator

severity: real-defect

tag: FIX

expected_gain_pct: 6

scores_moved: Team Members, Autopilot

expected_impact: Team Members and Autopilot; both are computed from the same denominator, and today an infrastructure failure raises them instead of lowering them.

evidence: repo/brain/eval/draft_morning.py:992, repo/brain/eval/draft_morning.py:1016, repo/brain/eval/draft_morning.py:1189, repo/brain/eval/draft_morning.py:1235

gating_check: checks/defects/defect_f05_pending_conflation.py

_check_one records pending when the Gorgias ticket fetch does not return 200
(repo/brain/eval/draft_morning.py:992) and when no agent reply has appeared yet (:1016).
_verdict_for defaults any ticket with no verdict row at all to pending (:1189). The
aggregate is gate_denom = p + pe + f (:1235), so all three drop out silently.

A transient API failure and a genuinely-not-yet-answered ticket are not the same event, and
neither is a row the pipeline lost. Today a bad auth token, a rate limit or a pull bug
SHRINKS the denominator, which raises the pass rate. "5 of 21, 7 pending" is 28 drafts of
which a quarter were never graded, and nothing in the reported number says so.

Fix: give each case its own verdict token (fetch_error, awaiting_reply, missing_row),
report them next to the rate, and alert when the ungraded share crosses a threshold.

F-18 - The policy grader decides Pam looked the order up by looking it up itself

severity: real-defect

tag: FIX

expected_gain_pct: 6

scores_moved: Team Members, Backtest

evidence: repo/brain/eval/policy_grade.py:242, repo/brain/eval/policy_grade.py:250, repo/brain/eval/policy_grade.py:251, repo/brain/eval/policy_grade.py:264, repo/brain/eval/draft_morning.py:473

expected_impact: Team Members most, because the morning-draft lane is the lane those 21 tickets are drawn from, and Backtest through the shared grader contract. Expect the measured fabrication rate on order categories to rise once the real signal is used.

gating_check: checks/defects/defect_f18_policy_grade_lookup_proxy.py

repo/brain/eval/policy_grade.py grades every posted morning draft a second time, for policy
correctness, through the same grade_compare the backtest uses. That grader needs the one
input F-17 is about: did Pam have verified order data? The real answer exists at draft time
and is thrown away. repo/brain/eval/draft_morning.py:473 is
pam, _tools = bd.run_pam(...), and _tools is never persisted anywhere the grading path
can read it.

So repo/brain/eval/policy_grade.py:242 substitutes a proxy, and the proxy answers a
different question. :250 re-fetches the order itself, days later, and :251 returns True
whenever that fetch finds a real order. Whether the ORDER exists has nothing to do with
whether PAM looked it up. On every order-based category (WISMO, ORDER ISSUE,
RETURN ISSUE, PAYMENT) with a resolvable order, :264 hands the grader
looked_up=True, which is the :1450 "treat them as VERIFIED" instruction from F-17.

The module's own docstring names the discarded _tools return and calls the proxy
fail-closed. It is fail-closed on a fetch failure, which is the case it was written for, and
fail-OPEN on the case that matters: the order exists and Pam never looked at it. That is the
whole fabrication class.

Fix: persist run_pam's tool list into draft-run.jsonl at draft time and read it here. The
signal is already produced; one variable name is throwing it away.

F-06 - A human "pam-fail" tag overrides every comparison, including a verbatim send

severity: real-defect

tag: FIX

expected_gain_pct: 5

scores_moved: Team Members

expected_impact: Team Members; two of the five recorded mismatches are this mechanism, so it is roughly 40 percent of the disagreement this metric is currently reporting.

evidence: repo/brain/eval/draft_morning.py:994, repo/brain/eval/draft_morning.py:996, repo/brain/eval/draft_morning.py:17

gating_check: checks/defects/defect_f06_tag_override.py

repo/brain/eval/draft_morning.py:994 returns a final fail the moment the ticket carries a
pam-fail tag, before any comparison runs, and the module's own header calls the tag "an
absolute override" (:17). Golden rows g-01 and g-12 are both verdict_method: tag, both
scored fail, and Ross labeled both pam_right. On g-01 the agent sent Pam's draft
verbatim: the recorded human_reply is Pam's text plus the auto-signature and the quoted
inbound email. A draft used word for word is on the books as a failure.

Fix: keep the tag as a strong signal, not as a terminal verdict. Record it, and when the tag
disagrees with the text comparison, route the row to the golden set for adjudication instead
of closing it.

F-07 - The grader scores resolution shape, so a wrong fact or an added promise passes

severity: real-defect

tag: RESTRUCTURE

expected_gain_pct: 5

scores_moved: Team Members, Autopilot, Ross's Claude

expected_impact: Team Members and Autopilot; the other three of the five recorded mismatches are this, and it is the reason a 0.042-similarity draft can be scored a pass.

evidence: repo/brain/eval/draft_morning.py:794, repo/brain/eval/draft_morning.py:1026, repo/brain/eval/golden/candidates.jsonl:14

gating_check: checks/defects/defect_f07_grader_blind_to_added_claims.py

Three recorded mismatches, three angles on the same hole:

* g-14, similarity 0.365, method intent: the haiku judge
(repo/brain/eval/draft_morning.py:794) called the resolution equivalent while Pam had
said two items were sold out when one was.
* g-29, similarity 0.042, method intent: a draft sharing 4 percent of its wording with what
the human sent still passed, because the judge only asks whether the resolution matches.
Pam told the customer to mail a defect back with no prepaid label, off the returns process.
* g-30, similarity 0.817, method similarity: above the 0.75 auto-pass threshold
(repo/brain/eval/draft_morning.py:1026), so no judge ran at all. The sentence Pam ADDED,
a promise to bundle the replacement with a subscription shipment, is a commitment
operations cannot keep, and a similarity gate cannot see an addition.

Correctness is graded as "same resolution as the human", never as "consistent with the
verified facts and the documented process". Tagged RESTRUCTURE because the fix is a third
grading dimension (claim conformance against the injected facts), not a tweak to the
existing two.

F-21 - The Backtest headline has two definitions, and the permanent ledger stores the one nobody reads

severity: real-defect

tag: RESTRUCTURE

expected_gain_pct: 5

scores_moved: Backtest

expected_impact: Backtest, and indirectly all four. No fix in this digest can be shown to move the Backtest score while two different numbers wear that name, so this is the finding that makes the other twenty measurable rather than the one that raises a score by itself.

evidence: repo/brain/dashboard/patch_generated_pages.py:406, repo/brain/dashboard/patch_generated_pages.py:417, repo/brain/dashboard/patch_generated_pages.py:852, repo/brain/eval/metrics_core.py:86

gating_check: checks/defects/defect_f21_backtest_two_definitions.py

Two functions answer "what is Pam's 7-day Backtest score", and they disagree by design.

repo/brain/dashboard/patch_generated_pages.py:406 is the official one, the number on the
page. It computes nothing. It runs a regex over the HTML the same script just rendered and
reads the percentage back out of its own tile.
repo/brain/dashboard/patch_generated_pages.py:417 is the other one, recomputed from the raw
per-ticket log, and its own docstring says it is a fallback and that it disagrees with the
official figure ("93.3 vs 95 on 2026-07-13").

The dashboard uses the official one and drops to the fallback only when the scrape fails
(repo/brain/dashboard/patch_generated_pages.py:852). repo/brain/eval/metrics_core.py:86,
the append-only snapshot ledger that exists to be the audit trail of record, calls the
fallback directly and never the official one, then files the result under
backtest_rolling_7d.pct. So the page and the permanent record carry different Backtest
numbers for the same night, under the same name, with nothing anywhere saying so. The
demonstration scores one night both ways and gets 95.0 and 90.0.

The scrape is also markup-coupled. One extra space in the tile and the regex returns None,
at which point :854 silently substitutes the other definition and the page's headline
changes meaning with no visible difference.

Tagged RESTRUCTURE because the fix is not a better regex. One function should compute the
Backtest score from the records, every consumer (page, ledger, Slack post) should call that
one function, and the page should render what it computed instead of the computation reading
the page.

F-08 - The standing prompt and the injected context contradict each other on live status

severity: real-defect

tag: FIX

expected_gain_pct: 4

scores_moved: Backtest, Team Members, Ross's Claude

expected_impact: Backtest, Team Members and Ross's Claude alike, since all three grade WISMO drafts; a contradiction resolved by the model is a coin flip on the exact class Pam is failing.

evidence: repo/brain/AGENT.md:186, repo/brain/tools/cst_inject.py:110, repo/brain/tools/cst_inject.py:719, repo/launcher/run_pam.sh:75

gating_check: checks/defects/defect_f08_prompt_injection_contradiction.py

repo/brain/AGENT.md:186 tells Pam she can see the carrier, tracking number and ship date
but NOT the live delivery state, and must never claim delivered or out for delivery. On the
same turn, repo/brain/tools/cst_inject.py:110 renders a LIVE CARRIER CHECK line with scan
history and instructs her to base every delivery-status statement on it, and
repo/launcher/run_pam.sh:75 has that injection switched on in production.

Both instructions are correct in their own context and the model is left to reconcile them.
Worse, the injection is conditional: repo/brain/tools/cst_inject.py:719 requires a brand
AND an email or order hint, so on a ticket with no identifier nothing is injected and nothing
tells Pam that the live block she was taught to rely on is absent this time.

Fix: one rule, conditional on the data. "State the delivery status ONLY from a LIVE CARRIER
CHECK block present in this turn's context; if there is no such block, say tracking is the
source of truth and share the number." Then the prose and the injection agree, and the
absence of the block is meaningful rather than silent.

F-19 - The grader-drift tripwire is switched off by a free-text note

severity: real-defect

tag: FIX

expected_gain_pct: 4

scores_moved: Ross's Claude, Backtest

evidence: repo/brain/eval/grading_referee.py:78, repo/brain/eval/grading_referee.py:81, repo/brain/eval/grading_referee.py:82

expected_impact: Ross's Claude directly, since agreement with Ross's grade IS that score's subject; Backtest indirectly, because this is the only alarm that would say the primary grader is drifting. No score moves until it is fixed, which is the point.

gating_check: checks/defects/defect_f19_referee_exclusion_freetext.py

repo/brain/eval/grading_referee.py is the only thing watching for grader bugs. It compares
the primary grader against Ross's blind grade and fires a Slack alert when 7-day agreement
drops below 90 percent. :78 defines the predicate that decides whether a reconciled row
counts as a measurement, and :81 is its third condition: the hand-typed Ross Notes cell
contains the substring harness bug.

Two problems, and they compound.

First, it is a substring match on prose. The demonstration runs the sealed predicate on three
notes: confirmed harness bug excludes the row, this was NOT a harness bug, the grader was
wrong also excludes it, and unrelated to the harness bug of 2026-07-11 excludes it too.

Second, it is one-directional by construction. The predicate requires the ruling to be
primary, so a disagreement the primary grader WON leaves the denominator and one the
auditor won stays in it. The same row with ruling='audit' is not excluded. The only
available bias is upward, on the one number that would report the grader is wrong, and
:82 applies it with no record that a measurement was dropped.

The exclusion itself is defensible: the 2026-07-11 harness bug really did feed the blind
auditor a wrong input, and those rows really are not grader drift. What is not defensible is
the mechanism. Fix: tag the affected rows by ticket id in a committed list at the time the
harness bug is found, count the exclusions in the Slack readout, and drop the substring
match.

F-22 - The "rolling 7-day" Backtest window is the last seven dates in the file, not the last seven days

severity: real-defect

tag: FIX

expected_gain_pct: 4

scores_moved: Backtest

expected_impact: Backtest. Does not raise the number; it stops the number from freezing. A backtest that silently stops running currently republishes its last good week forever, which would let Pam appear to hold 95 percent through a total grading outage.

evidence: repo/brain/dashboard/patch_generated_pages.py:417, repo/brain/dashboard/patch_generated_pages.py:447, repo/brain/eval/metrics_core.py:86

gating_check: checks/defects/defect_f22_rolling_window_unbounded.py

repo/brain/dashboard/patch_generated_pages.py:447 builds the window as
sorted(dates_seen)[-cap_days:]: every distinct date string in the log, sorted as text,
last seven kept. There is no comparison to today, no maximum age and no minimum row count.

Three consequences, all reproduced by the demonstration:

* It freezes rather than going stale. A history holding only January rows publishes as a
current 7-day 100 percent over n=7, with no flag and no None. Paired with F-21, that frozen
figure is written into a permanent ledger with tonight's timestamp on it.
* Seven dates is not seven days. One 40-row night plus six 1-row nights publishes as a
"7-day" 87 percent in which 87 percent of the denominator is a single night.
* Text sorting, not date sorting. A row dated 8/26/2026 sorts after every ISO date and
evicts the oldest real night out of the window.

repo/brain/eval/metrics_core.py:86 is also the one call in build_metrics that does not
receive the pinned anchor_date, because this function has no parameter to receive it. That
contradicts the module's own docstring, which states the snapshot "describes exactly one
point in time".

Fix: give the function an anchor_date, parse dates as dates, bound the window to
anchor_date - 6 days, and return None with a stated reason when the window is empty or the
newest record is older than the window.

F-09 - The fabrication backstop is switched off on exactly the tickets that have data

severity: real-defect

tag: FIX

expected_gain_pct: 3

scores_moved: Backtest

expected_impact: Backtest only; it restores a deterministic check on the majority of rows, which will surface fabrications the current number cannot see.

evidence: repo/brain/eval/backtest_distill.py:1903, repo/brain/eval/backtest_distill.py:1892, repo/brain/eval/backtest_distill.py:1939

gating_check: checks/defects/defect_f09_fab_flags_disabled_when_injected.py

fab = [] if (injected or tools) else _fabrication_flags(...)
(repo/brain/eval/backtest_distill.py:1903), where injected is simply "the order was
found" (:1892). Since v3 injection runs whenever an order resolves, the deterministic
fabricated-identifier check never runs on the tickets that matter most. The reasoning in the
comment is sound (injected specifics would false-flag), but the remedy throws out the check
instead of narrowing it. The code's own note at :1939 records that fab_flags "was
recorded and never consumed" anyway.

Fix: run the check with the injected facts in the haystack, which is what its signature
already supports, rather than skipping it.

F-20 - The referee scores two non-verdicts as agreement and an ungraded row as a Pam failure

severity: real-defect

tag: FIX

expected_gain_pct: 3

scores_moved: Ross's Claude, Backtest

evidence: repo/brain/eval/grading_referee.py:61, repo/brain/eval/grading_referee.py:123, repo/brain/eval/golden/candidates.jsonl:1

expected_impact: Ross's Claude, whose subject is grader agreement, and Backtest through the Policy-Correct column published off the same rows. Both numbers move on re-baseline rather than on a Pam change.

gating_check: checks/defects/defect_f20_referee_verdict_vocabulary.py

Neither of the referee's two scoring lines validates that the cell it reads holds a verdict.
repo/brain/eval/grading_referee.py:61 is a bare uppercased string equality, so two graders
that both failed to produce a verdict are scored as agreeing: the demonstration gets
agree='YES' for pending against PENDING, and 'YES' for two blanks. The
grader-agreement number counts a double no-op as a match, which is the exact opposite of what
that number is for.

:123 reports Policy-Correct percent as PASS over every primary row in the window. On
PASS/FAIL/pending it publishes 33.3 percent where the figure over adjudicated rows is 50.0.
An ungraded row is published as a Pam failure.

This is the "7 pending grades" shape from the brief, in the referee rather than the backtest,
and the vocabulary is not hypothetical: 9 of the 30 rows in
repo/brain/eval/golden/candidates.jsonl:1, the only graded record set in the review copy,
carry pending or stale in the same field these lines read (F-04 is the same population
seen from the golden checker's side).

Fix: one named set of adjudicable verdicts, shared by the referee, the golden checker and the
backtest summary. Anything outside it is unmeasured, reported as its own count, and never
silently folded into agreement or into a pass rate.

F-10 - A failed Pam generation is persisted and reviewed as if it were a draft

severity: real-defect

tag: FIX

expected_gain_pct: 3

scores_moved: Ross's Claude, Team Members

expected_impact: Ross's Claude most, since the Ardie review page is the artifact she grades; removing failed generations from the population removes a class of guaranteed fails.

evidence: repo/brain/eval/ardie_review_build.py:598, repo/brain/eval/ardie_review_build.py:612, repo/brain/eval/ardie_review_build.py:617

gating_check: checks/defects/defect_f10_ardie_failed_draft_persisted.py

repo/brain/eval/ardie_review_build.py:598 writes r["reply"] = reply or "(timed out)", so
an empty generation becomes a row whose customer reply is the literal string "(timed out)".
If the worker raises instead, the error is printed and swallowed (:612) and the row is
merged into the persistent store anyway (:617) with no reply at all. Rows are version-gated
so they are never re-drafted.

Fix: mark a failed generation and exclude it from the reviewable population, or retry it on
the next run. A lane whose infrastructure failures look like content cannot produce a
trustworthy grade.

F-23 - The metrics ledger's only alarm cannot see a ruler go dark

severity: real-defect

tag: FIX

expected_gain_pct: 3

scores_moved: Ross's Claude, Autopilot, Backtest

expected_impact: Ross's Claude and Autopilot first, since those are the two lanes that vanish under ordinary conditions. It moves no score upward; it stops a lane that stopped reporting from reading as a lane that is fine.

evidence: repo/brain/eval/metrics_core.py:124, repo/brain/eval/metrics_core.py:133, repo/brain/eval/metrics_core.py:102, repo/brain/eval/metrics_core.py:68

gating_check: checks/defects/defect_f23_drift_blind_to_dark_ruler.py

repo/brain/eval/metrics_core.py:124 is the whole safety mechanism on the permanent snapshot
ledger: quarantine the row if a headline percentage moved more than 30 points overnight. The
loop walks the new row's percentages and skips any key the previous row lacked
(repo/brain/eval/metrics_core.py:133). Nothing walks the other direction, so a percentage
that was there yesterday and is gone tonight is not a move, is not drift, and does not
quarantine.

Two ordinary conditions make a ruler vanish rather than move:

* Ross's local grade cache is missing or empty, and repo/brain/eval/metrics_core.py:102
stores {} for all three of his windows.
* Every autopilot draft escalated, so _policy_last_night returns pct: None (F-24), and
repo/brain/eval/metrics_core.py:68 only harvests int and float values.

The demonstration takes Anthony's four real headline figures, blanks two of them the way the
code actually blanks them, and gets quarantined=False, reason=None. The row still appends,
still carries a fresh built_at, still exits 0. The negative control moving the same two
lanes 46 and 66 points does quarantine, so this is a one-directional blind spot, not a dead
check: a wrong number is caught and a missing number is not.

Fix: compare the union of the two key sets, and quarantine on a key present yesterday and
absent tonight with the reason "ruler went dark". Record per-lane coverage counts in the row
so a percentage can never be read without the denominator behind it.

F-11 - A missing prompt source boots Pam live on a stale prompt

severity: real-defect

tag: FIX

expected_gain_pct: 2

scores_moved: Backtest, Team Members

expected_impact: Backtest and Team Members indirectly and unpredictably: a rule fix believed to be shipped may not be shipped, so a regression can appear with no code change behind it.

evidence: repo/launcher/run_pam.sh:133, repo/launcher/run_pam.sh:5, repo/launcher/run_pam.sh:141

gating_check: checks/defects/defect_f11_stale_prompt_fallback.py

repo/launcher/run_pam.sh:121 rebuilds the system prompt from the three brain files on every
launch (inside the existence guard that opens at :96), which is right. The else branch (:133) prints a warning and keeps whatever
system_prompt.md is already on disk, and the script continues to the exec at :141 because
it runs set -uo pipefail without -e (:5). A renamed file or a partially synced checkout
therefore puts Pam live on yesterday's rules with one line in a cron log as the only signal,
and the nightly drift gate catches it up to a day later.

Fix: exit non-zero in that branch. A prompt-build failure should be a failed start, not a
silent behaviour change on the customer lane.

F-12 - The grading lane is pinned to one machine, so nobody can review or test it

severity: real-defect

tag: RESTRUCTURE

expected_gain_pct: 2

scores_moved: Backtest, Team Members, Autopilot, Ross's Claude

expected_impact: Indirect but broad across Backtest, Team Members, Autopilot and Ross's Claude: every fix above lands faster and safer when the lane can be run and tested outside production.

evidence: repo/brain/eval/draft_morning.py:61, repo/brain/eval/ardie_review_build.py:27, repo/brain/eval/ross_checker_pull.py:41

gating_check: checks/defects/defect_f12_hardcoded_live_paths.py

Sixty-two files under brain/ and launcher/ resolve absolute paths on one Mac.
repo/brain/eval/draft_morning.py:61 and repo/brain/eval/ardie_review_build.py:27 do not
just reference such a path, they point sys.path at it and import their dependencies
through it, so on any other machine those modules import nothing at all.
repo/brain/eval/ross_checker_pull.py:41 hardcodes a service-account key path.

repo/brain/eval/backtest_distill.py:37 already shows the fix: derive the root from
__file__ with an environment override. Applying that pattern to the rest of the lane is
mechanical.

This is why the review had to reason about the grading code statically in several places
instead of running it. That limitation is stated wherever it applies.

The 2026-08-26 repair round produced direct evidence that this defect is worse than an
inconvenience. repo/brain/eval/bucket_classifier.py:33 pins sys.path at the same absolute
directory and imports guardrail out of it, and backtest_distill.py imports that module at
module level. So importing backtest_distill to test its detectors did one of two things
depending on the machine: it failed outright inside the controller's sandbox (which is how
this was found, a check that exited non-zero with no diagnosis), or it silently succeeded on
the operator's own Mac BY READING THE LIVE PRODUCTION TREE. A review whose result depends on
whose machine ran it is not a review, and a review that reads production to grade production
is the thing this job was told never to do. The three checks that used those detectors now
extract them from the sealed source instead of importing the module (checks/_lib.py
load_symbols), which is a workaround in the harness, not a fix in Pam. The fix in Pam is
this finding.

F-24 - The Autopilot night's pass rate counts only the drafts Pam chose to answer

severity: real-defect

tag: FIX

expected_gain_pct: 2

scores_moved: Autopilot

expected_impact: Autopilot. Expect the reported 66.7 percent to fall once escalations enter the denominator, which is the correct direction: today the metric rises as Pam answers less, so it cannot be used as a go-live gate.

evidence: repo/brain/dashboard/patch_generated_pages.py:482, repo/brain/dashboard/patch_generated_pages.py:495, repo/brain/dashboard/patch_generated_pages.py:497

gating_check: checks/defects/defect_f24_autopilot_escalations_leave_denominator.py

repo/brain/dashboard/patch_generated_pages.py:495 splits the night into escalated and
covered, and :497 computes the headline over covered alone. Escalating is Pam declining
to answer, so declining raises the score.

The demonstration runs three nights through the sealed function. Nine of ten escalated with
the tenth correct publishes 100.0 percent on covered=1. Ten of ten answered with one
correct publishes 10.0 percent. The first night is indistinguishable in the headline from a
perfect night. A night where she escalated everything returns pct: None, so the Autopilot
ruler disappears rather than reading zero, which F-23 shows the ledger's drift alarm cannot
detect.

This is F-05's shape ("pending" leaving the denominator) in a different file and a different
lane, and it is the lane behind the 66.7 percent Autopilot number. Listed separately because
the file, the mechanism and the fix site are all different; not additive with F-05.

The raw dict does return covered and escalated alongside, so coverage is reconstructible
by a reader of the JSON. The headline is not, and the headline is what gets compared to 95.

Fix: divide by the whole night, count an escalation as not-passed, and publish coverage next
to the percentage the way _coverage_txt
(repo/brain/dashboard/patch_generated_pages.py:464) already does for the other lanes.

F-13 - The preamble-leak rule misses any leaked line shorter than 40 characters

severity: real-defect

tag: FIX

expected_gain_pct: 1.5

scores_moved: Team Members, Ross's Claude

expected_impact: Small but certain on Team Members and Ross's Claude; it removes a class of visibly broken customer replies that currently reach the rep unflagged.

evidence: repo/brain/guardrail.py:216, repo/brain/eval/golden/candidates.jsonl:25

gating_check: checks/defects/defect_f13_preamble_leak_gap.py

repo/brain/guardrail.py:216 flags a leaked preamble only when the greeting starts more than
40 characters into the customer body. Golden row g-25 opens with a 23-character internal
marker line above the greeting, and guardrail.check returns clean on the recorded draft.
Ross's note on that row names the leak explicitly.

Fix: flag ANY non-empty content above the greeting when the greeting is on a later line,
rather than gating on a character count.

F-14 - The engine that makes the LLM call is not in the review copy

severity: coverage-gap

tag: RESTRUCTURE

expected_gain_pct: 1

scores_moved: Backtest, Team Members, Autopilot, Ross's Claude

expected_impact: Unknown by construction for Backtest, Team Members, Autopilot and Ross's Claude alike: the single step where the model answers a customer, and the tool loop around it, cannot be reviewed from what was seeded.

evidence: repo/launcher/run_pam.sh:141, repo/launcher/pam_pipeline.py:1243, repo/launcher/pam_pipeline.py:950

gating_check: none

repo/launcher/run_pam.sh:141 execs slack_orchestrator.py and
repo/launcher/pam_pipeline.py:1243 and :950 import it. That file was not seeded. So the
generation call, the tool-call loop, the retry and timeout behaviour, and the catch-up replay
are all outside this review. Anything this digest says about how tools get called is inferred
from the config, the prompt and the pipeline's inputs and outputs, never read.

This is a bigger blind spot than the missing system_prompt.md the spec anticipated, and it
is the first thing to seed if this review is ever repeated. Excepted in
docs/exceptions.md.

F-15 - The per-ticket records behind the four headline numbers are not in the review copy

severity: coverage-gap

tag: FIX

expected_gain_pct: 1

scores_moved: Backtest, Team Members, Autopilot, Ross's Claude

expected_impact: Unknown by construction for all four (Backtest, Team Members, Autopilot, Ross's Claude); without the records, cluster frequency cannot be measured and no fix above can be prioritised by real volume rather than by judgment.

evidence: repo/brain/eval/backtest_distill.py:46, repo/brain/eval/backtest_distill.py:1759, repo/brain/eval/ross_checker_pull.py:38

gating_check: none

Every record file the four scores are computed from lives under data/logs/
(repo/brain/eval/backtest_distill.py:46, :1759;
repo/brain/eval/ross_checker_pull.py:38), and data/logs/ was stripped when this copy was
sealed: repo/brain/data/ contains kb/ and nothing else. The headline numbers themselves
appear nowhere in the tree.

The consequence is stated in docs/failure-clusters.md: the clustering is grounded in the 30
golden rows and is blind to the populations behind 71 percent, 66.7 percent and 56.2 percent.
Every expected_gain_pct in this digest is therefore a judgment call, not a measurement.
Excepted in docs/exceptions.md.

F-16 - The generated system prompt is absent, and derivable

severity: coverage-gap

tag: FIX

expected_gain_pct: 0.5

scores_moved: Backtest

expected_impact: Low, on Backtest. The inputs are all present, so the artifact can be regenerated; what is unverifiable is which version actually reached live Pam on a given day.

evidence: repo/launcher/run_pam.sh:121, repo/brain/AGENT.md:1, repo/brain/SOUL.md:1, repo/brain/RULES.md:1

gating_check: none

launcher/agents/pam/system_prompt.md is not present in the seeded repo. It is a build
artifact, not missing source: repo/launcher/run_pam.sh:121 regenerates it from AGENT.md,
SOUL.md and RULES.md on every launch, and all three are in the review copy. The gap is
low severity and is recorded rather than reconstructed, per the cut lines. It stops being
harmless in combination with F-11, where a stale copy can be what actually shipped.
Excepted in docs/exceptions.md.

What I would do first, in order

1. F-01. Move lookup_gate onto the live lane. It is the only change here that stops a
fabricated status reaching a customer rather than merely counting it afterwards.
2. F-04 through F-07 together. Fix the grader before chasing the score. Right now all
five recorded scorer/Ross disagreements are grader defects, so a Pam fix and a grader fix
are indistinguishable in the numbers. Fixing the measurement first is what makes every
later change legible.
3. F-02, F-17, F-18 together. They are one bug at three call sites: looked_up is a
single boolean derived from tool truthiness, injection, or the grader's own re-fetch, and
it silences the fabrication check for the whole draft. Fix the input, not the detectors.
4. F-03, F-08. Widen the claim detector to the delivery-state vocabulary, and make the
standing prompt and the injected context say the same thing about live status.
5. F-19, F-20. Fix the referee before trusting any agreement number. Right now the alarm
that would tell you the grader is drifting can be silenced by a spreadsheet cell, and it
counts two ungraded rows as a match.
6. F-15. Seed the per-ticket records into the next review copy. Until then every priority
above is ordered by argument, not by frequency.

A word on the 95 percent bar. Three of the four scores are produced by the pipeline that
findings F-04 to F-07 and F-17 to F-20 show to be defective in both directions. I would not treat any of them
as a go-live gate until the grader is fixed and re-baselined. That is not an argument for
going live sooner; it is an argument that the current 71 / 23.8 / 66.7 / 56.2 are not yet
measuring what the bar assumes they measure.

Structure map (deterministic vs LLM)

Structure map: one customer email, end to end

Scope. repo/launcher (pam-launcher, seeded from 877b6c9) plus repo/brain (Pam's
brain, seeded from 4e68027), read-only. Every row below is anchored to a file and line in
that sealed copy. Nothing here was inferred from the live machine.

How to read the table. Determinism is exactly one of deterministic script or
LLM call (model: <name>). Enforced by names the code that makes the step's outcome
mandatory. Flag carries LLM-TRUSTED-UNENFORCED when a step's correctness depends on the
model remembering to do something and no deterministic code checks that it did.

Coverage gaps, stated before the map so nothing below reads as complete.

1. launcher/agents/pam/system_prompt.md is not present in the seeded repo. It is not
missing source: repo/launcher/run_pam.sh:121 rebuilds it on every launch by
concatenating AGENT.md + SOUL.md + RULES.md (+ a rendered GUIDELINES section) and
substituting the per-order cap. All three inputs ARE in the review copy, so the artifact
is derivable and this gap is low severity. It is still a gap: the file that actually
reached live Pam on any given day is not in evidence, and run_pam.sh:133 can keep a
stale one (finding F-10).
2. The bigger gap: the engine is not in the review copy. run_pam.sh:141 execs
slack_orchestrator.py, and repo/launcher/pam_pipeline.py:1243 imports it. That file
lives in a separate directory that was not seeded. So the step where the model actually
answers a customer, and the tool-call loop around it, cannot be read here. Every LLM row
below is therefore labelled with the model the committed config selects
(repo/launcher/config.yaml:14), and the loop's behaviour is inferred from its inputs
and outputs, not read.
3. The graded per-ticket records behind the four headline scores are not in the repo
either. See docs/failure-clusters.md.

The live lane (the one a customer's words actually travel down)

| Step | File:line | Determinism | Enforced by | Flag |
|---|---|---|---|---|
| 1. launchd starts Pam | repo/launcher/com.claude.pam-agent.plist:10 | deterministic script | launchd KeepAlive + repo/launcher/com.claude.pam-watchdog.plist:5 | |
| 2. Boot env: channels, roles, injection switch | repo/launcher/run_pam.sh:75 | deterministic script | run_pam.sh, fails fast on a missing Slack token (repo/launcher/run_pam.sh:35) | |
| 3. System prompt rebuilt from the three brain files | repo/launcher/run_pam.sh:121 | deterministic script | shell concat in fixed order AGENT, SOUL, RULES, guarded by the existence test at repo/launcher/run_pam.sh:96 | fail-open: repo/launcher/run_pam.sh:133 keeps a stale prompt and boots anyway (F-10) |
| 3b. Non-negotiable rules concatenated last, closest to the customer's question | repo/brain/RULES.md:52 | deterministic script | the concat order in repo/launcher/run_pam.sh:124; the rule text itself is instruction to the model, not code | LLM-TRUSTED-UNENFORCED: "you MUST call your read-only lookup tools" is prose with no deterministic counterpart on the send path |
| 4. Engine selected and exec'd | repo/launcher/run_pam.sh:141 | deterministic script | nothing in the review copy: slack_orchestrator.py was not seeded | COVERAGE-GAP |
| 5. Agent config: model, prompt file, pipeline hook | repo/launcher/config.yaml:14 | deterministic script | repo/launcher/config.yaml:53 binds pam_pipeline as the reply pipeline | |
| 6. Rep pastes the customer email into a Slack channel and mentions Pam | repo/launcher/pam_pipeline.py:3 | deterministic script | the engine's event router (not in the review copy) | COVERAGE-GAP |
| 7. Channel mode chosen (customer-reply vs analyst vs manager) | repo/launcher/pam_pipeline.py:197 | deterministic script | repo/launcher/pam_pipeline.py:292 renders the per-mode instructions | |
| 8. Trivial-message filter | repo/brain/guardrail.py:48 | deterministic script | repo/launcher/pam_pipeline.py:379 (currently disabled by decision) | |
| 9. Deterministic pre-draft injection assembled | repo/launcher/pam_pipeline.py:358 | deterministic script | repo/brain/tools/cst_inject.py:707 | fail-open and silent: repo/brain/tools/cst_inject.py:719 needs a brand AND an email or order hint, else nothing is injected and nobody is told |
| 10. Live carrier snapshot attached to the facts | repo/brain/tools/cst_inject.py:729 | deterministic script | repo/brain/tools/tracking_snapshot.py:1, fail-soft by contract | if the snapshot does not resolve, the draft proceeds with no live status and no marker |
| 11. Facts block rendered, including the LIVE CARRIER CHECK line | repo/brain/tools/cst_inject.py:110 | deterministic script | none: it is text handed to the model | LLM-TRUSTED-UNENFORCED: the injected block says to base every delivery-status statement on it, repo/brain/AGENT.md:186 says the opposite (F-08) |
| 12. Pam drafts the reply | repo/launcher/run_pam.sh:141 | LLM call (model: claude-sonnet-5) | model selection at repo/launcher/config.yaml:14; the call itself is in the unseeded engine | COVERAGE-GAP |
| 13. Order / tracking lookup during drafting | repo/brain/AGENT.md:183 | LLM call (model: claude-sonnet-5) | nothing. repo/brain/RULES.md:52 and repo/brain/AGENT.md:123 instruct it in prose; no code checks that a tracking or order lookup ran before a WISMO reply is allowed to send | LLM-TRUSTED-UNENFORCED: the tracking / order-lookup call is trusted to the model (F-01, F-03) |
| 14. Pre-send guardrail | repo/launcher/pam_pipeline.py:1257 | deterministic script | repo/brain/guardrail.py:223: completion claims, address/ship promises, approval lines, the $100 ceiling, refund promises, preamble leaks, internal URLs | LLM-TRUSTED-UNENFORCED: guardrail.check has no lookup-or-tracking rule at all, so step 13 is unchecked here too (F-01) |
| 15. One corrective regeneration when the guardrail fires | repo/launcher/pam_pipeline.py:1277 | LLM call (model: claude-sonnet-5) | repo/brain/guardrail.py:288 builds the correction text deterministically | |
| 16. Re-check after the rewrite | repo/launcher/pam_pipeline.py:1287 | deterministic script | still-violating drafts get a visible rep-review banner and a dead-letter row | |
| 17. __APPROVAL__ lines stripped and audited | repo/launcher/pam_pipeline.py:398 | deterministic script | repo/brain/guardrail.py:150 parses them, repo/launcher/pam_pipeline.py:526 applies daily caps | |
| 18. Over-ceiling concession held for a CS lead | repo/launcher/pam_pipeline.py:557 | deterministic script | repo/brain/guardrail.py:115 computes the dollar figure, repo/launcher/authority_blocker.py:1 writes the authority line | |
| 19. Reply posted, feedback buttons attached, audit written | repo/launcher/pam_pipeline.py:433 | deterministic script | repo/launcher/pam_pipeline.py:443 mirrors it to the audit sheet | |

The grading lanes (where the four scores come from)

| Step | File:line | Determinism | Enforced by | Flag |
|---|---|---|---|---|
| 20. Morning draft cron drafts on real tickets | repo/brain/eval/draft_morning.py:69 | LLM call (model: claude-sonnet-5) | GATE 1 lookup_gate at repo/brain/eval/draft_morning.py:420, GATE 2 qc_lint at repo/brain/eval/draft_morning.py:496 | the gates that DO enforce a lookup run here and only here, never on the live lane (F-01) |
| 21. Team-member comparison: did the human send Pam's draft | repo/brain/eval/draft_morning.py:751 | deterministic script | similarity threshold at repo/brain/eval/draft_morning.py:1026 | |
| 22. Intent judge when similarity is inconclusive | repo/brain/eval/draft_morning.py:794 | LLM call (model: claude-haiku-4-5-20251001) | budget cap at repo/brain/eval/draft_morning.py:824 | |
| 23. Verdict recorded, denominators computed | repo/brain/eval/draft_morning.py:1235 | deterministic script | gate_denom = pass + pass_edited + fail | three different failures all record pending and leave the denominator (F-05) |
| 24. Nightly backtest drafts and grades | repo/brain/eval/backtest_distill.py:1894 | LLM call (model: claude-sonnet-5 drafting, claude-opus-5 grading) | repo/brain/eval/backtest_distill.py:1433 pins the grader model | |
| 25. Fabricated-identifier backstop | repo/brain/eval/backtest_distill.py:1903 | deterministic script | itself, when it runs | skipped on every ticket where facts were found (F-07) |
| 26. Live-tracking claim tripwire | repo/brain/eval/backtest_distill.py:1953 | deterministic script | repo/brain/eval/backtest_distill.py:957, post-hoc only | disarmed by any tool call (F-02); blind to claims that avoid the word "live" (F-03) |
| 27. Ross's Claude grades pulled from the sheet | repo/brain/eval/ross_checker_pull.py:139 | deterministic script | offline-safe cache, fails open to the previous file | |
| 28. Ardie review page drafts one Pam reply per ticket | repo/brain/eval/ardie_review_build.py:315 | LLM call (model: claude-sonnet-5) | one retry at repo/brain/eval/ardie_review_build.py:325 | a timed-out or raising draft is persisted as content (F-06) |
| 29. Golden-set referee: scorer verdict vs Ross's label | repo/brain/eval/golden/golden_check.py:71 | deterministic script | repo/brain/eval/golden/golden_check.py:41 and :43 define what counts | nine of thirty labeled rows carry a verdict outside both sets and can never be adjudicated (F-04) |

The one-sentence answer AC1 asks for

**No, a tracking or order-lookup tool call is not deterministically enforced before a WISMO
reply can send.** The enforcement exists (repo/brain/tools/draft_gates.py:346) and is wired
into the two offline lanes only. On the live lane the only pre-send check is
repo/brain/guardrail.py:223, which has no lookup rule, so a reply asserting a delivery
status with no lookup behind it passes and posts. Reproduced in checks/chk03_tracking_skip_repro.py.

Failure clusters

Failure clusters, pinned to the structure map

What this is grounded in, and what it is not. Every row below traces to a real record in
repo/brain/eval/golden/candidates.jsonl (30 hand-labeled rows) or
repo/brain/eval/golden/status.json (5 recorded scorer/Ross mismatches). Nothing is
invented and no ticket is cited that is not in that file.

The full nightly backtest run's per-ticket records are not present in this review copy.
repo/brain/eval/backtest_distill.py:46 points every record file at data/logs/
(backtest-records-history.jsonl at repo/brain/eval/backtest_distill.py:1759,
failure-modes.json at repo/brain/eval/backtest_distill.py:2348), and data/logs/ was
stripped when this review copy was sealed: repo/brain/data/ contains kb/ and nothing
else. The four headline numbers (Backtest 71 percent, Team
Members 23.8 percent, Autopilot 66.7 percent, Ross's Claude 56.2 percent) appear nowhere in
the seeded tree: grepping the whole repo for them returns nothing. So the clustering here is
necessarily PARTIAL. It covers the 30 golden rows, which is the only graded evidence a
reviewer actually has, and it is blind to the roughly 43-ticket backtest population and the
32 Ross's-Claude rows behind those percentages. Closing that gap is finding F-15.

Pinned step refers to the numbered steps in docs/structure-map.md.

Clusters

| Cluster | What goes wrong | Pinned step | Rows |
|---|---|---|---|
| C1 live status asserted without a lookup | Pam states or implies a delivery state she never verified, or her state is stale by the time the human replies | 13, 10 | g-10, g-19, g-20 |
| C2 stock and availability claimed without a lookup | Pam tells the customer what is out of stock without a verified stock read | 9, 13 | g-06, g-14 |
| C3 money figures asserted without verification | A refund amount or discount code that no lookup returned | 13, 17 | g-02, g-07 |
| C4 off-playbook process commitments | The resolution is plausible but is not the process operations runs, or promises something operations cannot do | 14 | g-03, g-05, g-11, g-29, g-30 |
| C5 should not have replied at all | Marketing blasts, B2B newsletters and vendor pitches reaching the drafting lane | 7 | g-23, g-24, g-26, g-27 |
| C6 internal text leaking into the customer reply | Reasoning, markers or narration above the greeting | 14 | g-09, g-25 |
| C7 grader-side disagreement, Pam was fine | The draft was right (or equally right) and the scoring lane called it wrong, or the label is cosmetic | 21, 22 | g-01, g-04, g-08, g-12, g-13, g-15, g-16, g-17, g-18, g-21, g-22, g-28 |

Every row, with its root cause

| gid | cluster | root cause | pinned step |
|---|---|---|---|
| g-01 | C7 | a human pam-fail tag short-circuits every comparison (draft_morning.py:994); the agent sent this draft verbatim and it still scored a failure | 21 |
| g-02 | C3 | promised a 20 percent code that does not exist; nothing checks a promised code against a real one, and the guardrail has no rule for invented offers | 13 |
| g-03 | C4 | asked the customer to confirm an address that needed no confirming; no playbook-conformance check exists on the send path | 14 |
| g-04 | C7 | a reasonable clarifying question scored a fail; the agent's different path was also fine | 22 |
| g-05 | C4 | drafted a full status plus a fee-credit promise onto a thread the customer had already closed; nothing checks thread state before drafting | 7 |
| g-06 | C2 | said several styles were unavailable when one was; the stock block is injected only when the order resolves (cst_inject.py:719) and nothing enforces its use | 9 |
| g-07 | C3 | quoted a 1.79 refund where the team issued 20.52; the order total was never read back before the figure went out | 13 |
| g-08 | C7 | correct on the card-details refusal; the agent's invoice path was also fine | 22 |
| g-09 | C6 | internal reasoning leaked above the customer text; guardrail._preamble_leak fired here but the underlying rule is narrow (F-13) | 14 |
| g-10 | C1 | the agent had tracking showing the return delivered to a different address; Pam had no live tracking and no rule forced her to say so | 13 |
| g-11 | C4 | both Pam and the agent should have confirmed the order number first; no identifier-confirmation gate on the send path | 13 |
| g-12 | C7 | pam-fail tag override again; Ross judged Pam's reship/refund/credit better than the agent's spam deflection | 21 |
| g-13 | C7 | same in-transit read as the agent, scored pass_edited | 21 |
| g-14 | C2 | said both items were out of stock when one was; the intent judge called the resolution equivalent and passed it (F-07) | 22 |
| g-15 | C7 | correct carrier hand-off and shipping-fee credit per policy | 21 |
| g-16 | C7 | correct MDNR handling, credit or reship, explicitly no card refund | 21 |
| g-17 | C7 | correct, which is why the agent used the draft | 21 |
| g-18 | C7 | correctly says shipped, gives tracking and the exchange path | 21 |
| g-19 | C1 | correct at draft time, tracking moved afterwards; staleness is a lane property, not a Pam error, and the lane has no re-read before send | 20 |
| g-20 | C1 | the agent had a delivered scan, Pam said the package was moving; the live snapshot is fail-soft (cst_inject.py:729) and its absence is silent | 10 |
| g-21 | C7 | both confirm the address update; the agent added the billing limit | 21 |
| g-22 | C7 | near-verbatim, the agent trimmed one sentence; labeled style_only | 21 |
| g-23 | C5 | vendor pitch declined cleanly (correct), but the row is scorer_verdict pending and can never be adjudicated (F-04) | 29 |
| g-24 | C5 | a B2B newsletter was answered at all; no should-not-reply gate runs before drafting | 7 |
| g-25 | C6 | the draft opens with a literal internal marker line; the preamble-leak rule needs the greeting to start past 40 characters, so a 23-character leak passes (F-13) | 14 |
| g-26 | C5 | correctly identified a marketing blast and said no reply; also stuck on pending | 29 |
| g-27 | C5 | same marketing-blast class as g-26 and Pam answered it; the classifier's decision is never enforced | 7 |
| g-28 | C7 | sent verbatim, scored pass | 21 |
| g-29 | C4 | told the customer to mail a defect back with no prepaid label, off the returns-portal process; the intent judge passed a draft with 0.042 similarity (F-07) | 22 |
| g-30 | C4 | added a promise to bundle the replacement with the subscription shipment, which operations cannot do; similarity 0.817 auto-passed above the 0.75 threshold, so no judge ever looked (F-07) | 21 |

The five recorded mismatches, individually

| gid | scorer | Ross | root-cause tag | pinned step |
|---|---|---|---|---|
| g-01 | fail | pam_right | grader: human tag overrides a verbatim send (F-06) | 21 |
| g-12 | fail | pam_right | grader: human tag overrides a better draft (F-06) | 21 |
| g-14 | pass_edited | pam_wrong | grader: intent judge blind to an unverified stock claim (F-07) | 22 |
| g-29 | pass | pam_wrong | grader: intent judge blind to an off-process instruction (F-07) | 22 |
| g-30 | pass | pam_wrong | grader: similarity threshold blind to an added commitment (F-07) | 21 |

Read that table again, because it is the single most useful thing in this document. All
five recorded disagreements between the scorer and Ross are SCORER defects, not Pam defects.
Two are a tag that beats every comparison, three are a grader that scores resolution shape
and never checks a claim against the facts. Pam's own worst cluster (C1, C2, C3: asserting
facts nothing looked up) is present in the data but is NOT what the mismatch metric is
measuring. Fixing the four scores therefore has two independent halves: enforce the lookup
on the live lane (F-01 to F-03), and stop the grader from being wrong in both directions
(F-04 to F-07).

What the clustering cannot tell you

* Frequency. Thirty hand-labeled rows cannot tell you whether C1 is 5 percent or 30 percent
of live WISMO traffic. The nightly records that could are not in the repo (F-15).
* The Autopilot and Ross's Claude populations are not represented here at all. Neither
draft-policy-grades.jsonl nor ross-checker-grades.jsonl is in the seeded tree; only the
code that writes them is.
* Nine of the thirty rows carry a scorer_verdict the referee cannot adjudicate
(stale or pending), so they contribute a label and a tally entry but can never be a
mismatch. That is finding F-04, and it means even this small denominator is softer than it
looks.

Coverage exceptions

Exceptions

A finding lands here when it is real and cannot be demonstrated by a check that fails
against the review copy. Every row says why, in plain terms. checks/chk06_defects_block_release.py
refuses to pass if a real-defect finding has neither a failing check nor a row here.

All three rows below are coverage-gap findings: the review copy does not contain the thing
that would have to be examined. A check cannot demonstrate the absence of evidence as a
defect in the code, and writing one that fails just because a file is missing would be a
harness that reports red on a sealing decision rather than on Pam.

| finding | reason |
|---|---|
| F-14 | The engine (slack_orchestrator.py) was not seeded. A check could assert the file is absent, but that would gate the review on how the repo was sealed rather than on a defect in Pam. The right remedy is to seed the engine into the next review copy, which is a handoff action, not a code fix. Recorded here so the blind spot is on the record: every claim in this review about the tool-call loop is inference from inputs and outputs, never a read of the code. |
| F-15 | The per-ticket graded records behind the four headline numbers live under data/logs/, which was stripped at seal. Same reasoning: the absence is a sealing decision, not a defect a check should block on. The consequence is stated wherever it bites, chiefly that every expected_gain_pct in the digest is a judgment call rather than a frequency measurement. |
| F-16 | launcher/agents/pam/system_prompt.md is a build artifact regenerated on every launch from three files that ARE in the review copy, so there is nothing broken to demonstrate. The cut lines forbid reconstructing its content, and this review does not. It is recorded because in combination with F-11 (a stale prompt can boot) the artifact's absence stops being purely cosmetic. |

Not excepted, deliberately

Every other finding in docs/findings-digest.md marked real-defect carries a check under
checks/defects/ that exits non-zero right now against the unfixed review copy. Those are
the ones that block. If one of them is fixed upstream, its check flips to exit 0 and CHK-06
will start failing, which is the correct signal to move the finding out of the blocking set
rather than to edit the check.