中文版

Decision Model Providers — Per-Feature Switch and Local Replay

Lets each rental AI feature (listing, chat, duplicate) choose its own System One decision model, TypeSafe Jev or the self-hosted open-weight Kev, through one Provider interface, a registry and per-provider threshold tables. Before any switch, real production calls are replayed against Kev on the developer's own machine through a loopback-only client, and a per-unit report compares the two models. Updated after the duplicate check shipped (#146) to adapt its judge, sweep gate, threshold and eval tool.

  • Predecessor Builds on the 2026-09-23 TypeSafe Jev listing check (shared client, J13 naming)
  • ADRs 0064 · 0066 · 0067
  • New ADR 0068
  • Status Approved

View source markdown ↗ generated by claude-opus-5-5 · diagrams mermaid

Summary

Today every Jev feature is locked to one hard-coded model. This design lets each rental feature pick its own decision model — TypeSafe Jev or the self-hosted Kev — through one interface, a registry and per-provider threshold tables. Before any switch, real production calls are replayed against Kev on the developer's own machine, and a report compares the two.

  • Area backend/go
  • Providers jev (TypeSafe) and kev (self-hosted, same API)
  • Features Listing check, chat message check, duplicate check
  • Default jev for every feature (P1)
  • Switching Env var + redeploy (P2)
  • Comparison Local replay, not live shadow calls (P6)
  • Privacy Loopback-only replay — ADR-0068
  • Status Approved; grill closed 2026-09-26 13:44 +08 (Q1–Q9, F1–F4); updated after the duplicate check shipped in #146 (U1–U5)

Related: Jev listing check (shared client, J13 naming); Jev chat message check; Jev duplicate check; ADR-0068 (replay privacy); ADR-0064, ADR-0066, ADR-0067; Kev: github.com/jaredpalmer/kev.

Why

The shared client sends "model": "jev-latest" to https://api.typesafe.ai, hard-coded. Other decision models now exist. Kev is a family of open-weight models that copies TypeSafe's System One API exactly.

Same as Jev
  • POST /v1/systemone, same request and answer JSON
  • Noul, Choice, Score questions
  • Returns probabilities
Different from Jev
  • We host it ourselves (GPU)
  • Probabilities are calibrated differently
  • Ranks answers less well: 45–57% vs 70% automated at a 5% error budget

We want:

  1. Each feature picks its own provider: listing, chat, duplicate.
  2. Compare before switching: replay real production calls against Kev on a local GPU machine, then decide.
  3. Easy to add providers later: one interface. A new adapter is written only when a provider with a different API is known.

Deliverables

#Deliverable
1Provider interface in the shared typesafe package
2Client WithModel option; model name no longer hard-coded
3Provider registry (jev, kev) built at startup from config
4Per-feature provider env vars
5Per-provider threshold tables in each judge; startup refuses an untested pair
6cmd/replay-typesafe: local replay of production calls with a report, plus a shared loopback-only HTTP client (F1) used by replay and real-data eval
7--base-url and --model flags on cmd/eval-listing-check, cmd/eval-message-check and cmd/eval-duplicate-check; --threshold scoring in cmd/eval-message-check; --threshold override in cmd/eval-duplicate-check (U3)
8ADR-0068

Decisions

#Decision
P1Production default stays jev for every feature.
P2Switching is by env var and a redeploy. An admin page comes later, only if switches become frequent.
P3Each feature has its own provider env var (TYPESAFE_LISTING_PROVIDER, TYPESAFE_CHAT_PROVIDER, TYPESAFE_DUPLICATE_PROVIDER).
P4Thresholds live in code, one table per judge, keyed by provider name. Each entry keeps today's shape: one shared Noul and (listing) one shared Score threshold. A new entry needs an eval or replay run first. Per-question thresholds only in a later PR with evidence (Q8).
P5The server refuses to start when a feature switch is on and its provider is not registered (including a missing Jev key) or has no threshold entry. Today a missing key silently turns the feature off (Q6).
P6Comparison is by replay on a local machine, not a live shadow call. Production only ever calls its chosen provider.
P7The replay tool reads production with the developer's normal DB login. It runs SELECTs only. No read-only role.
P8Replay output stays in local files. Nothing is written back to production. A typesafe_replays table comes later, only if others need the results.
P9The shared interface is the only abstraction now. A second adapter is written only when a provider with a different API is chosen.
P10Call logging stays inside the HTTP client. Jev and Kev share that client, so both are logged. Logging moves into a wrapper when a second adapter arrives.
P11The package keeps the name typesafe. "System One" is the API shape TypeSafe defined and Kev copies.
P12The registry list lives in code. Env holds only URLs and keys. A new provider needs a threshold table anyway, which is a code change.
P13Kev production activation rule (Q7): Kev's first threshold entry ships only with an immutable model id pinned in code, plus the replay evidence. Changing the weights or the id needs a fresh evaluation and threshold review. kev-latest is for local evaluation only. Jev stays jev-latest.

Grill decisions (2026-09-26)

#DecisionWhyWhere
Q1Listing is on only when its judge exists. Env reads and SetRentalJevEnabledFunc deleted. Sweep nil guard.Two sources of truth could disagree and call a nil judge§5.4
Q2Replay selects model LIKE 'jev-%', optional --jev-model, counts per versionThe log stores the returned model (jev-1.13.0), not jev-latest§5.6, §5.7
Q3One unit = latest 200-row per ref_id, tie-break id DESC, --limit after dedupeRetries and form checks log many rows per unit§5.6, §5.7
Q4run.json freezes window, filters, target and unit list. No keys. Rerun rejects conflicting flags.New rows and a moving --since would change a resumed run§5.6
Q5Local labels.jsonl, hand label wins, Parts A/B separate, --report-only, no recall claimAdmins only see what Jev found. A missing label is not a negative.§5.7
Q6Switch on + provider not ready = startup error, including a missing Jev keySilent off hides lost keys and bad provider names§5.4, §8
Q7Kev needs a pinned immutable model id before its first threshold. kev-latest local only.Weights swapped under one name make a tuned threshold drift§5.3, P13
Q8One shared Noul/Score threshold per judge per provider. Sweeps per unit.Labels are per message, and the table cannot hold per-question numbers§5.5, §5.7
Q9Message eval scored against ExpectedFindings at --threshold. Listing eval descriptive. --model required with --base-url.Only message cases have right answers. Avoid sending jev-latest to Kev.§5.8
F1Replay destination loopback only: flag check, dial check, no redirects, no proxy, no overrideADR-0068 allows only the developer's own machine§5.6
F2Same guard for eval with db or file source and a custom --base-urlThose sources may hold real data§5.8
F3Injected fake judge skips only its own feature's lookupTests inject fakes without a registry§5.4
F4Replay per-call timeout, 60s default, recorded, resumablehttp.Client{} has no timeout§5.6

Duplicate check shipped (2026-09-26, #146)

The duplicate check was built before this change. It follows the old pattern, so this change adapts it. What shipped:

ThingWhere
Judge built whenever a client or a key exists, switch not readrental/module.go:348-355
Sweep reads global config every tick (config.Get()), plus a SetEnabledFunc test hookusecase/duplicate_check_sweep.go:54-65,110
IsRentalDuplicateEnabled() = key && rental switch && duplicate switchconfig.go:61-63
DuplicateHintThreshold = 0.7 defined twice. The sweep uses its own copy. The eval uses the adapter's.duplicate_judge.go:14, duplicate_check_sweep.go:19,222
Port returns raw probabilities. The sweep applies the cut and saves p on the hint.port/ports.go:1221
Eval already scores against ExpectedDuplicate at the thresholdcmd/eval-duplicate-check/main.go:220
Logged state holds addresses by position, no ids. Hints have no call id.duplicate_judge.go:44-54, rental_duplicate_hints
#DecisionWhyWhere
U1Duplicate is on only when its judge exists, same as listing (Q1). Sweep isEnabled, its config.Get() read and SetEnabledFunc deleted. Judge built only when the switch is on and the provider resolves.Two sources of truth again. A global config read breaks "no TYPESAFE_* read outside config".§5.4
U2Threshold table {"jev": 0.7} in the adapter. NewDuplicateJudge(p, name) errors on a missing entry. The port gains Threshold() float64. The sweep uses it and its own constant is deleted.Two copies of 0.7 can drift. The sweep needs raw p for the hint row, so the judge cannot hide p.§5.5
U3cmd/eval-duplicate-check gets --base-url and --model (F2 guard too), plus --threshold. Default is the jev entry. Required with --base-url.Kev must not be scored silently on Jev's number§5.8
U4Replay supports duplicate_check. One unit = one subject call. Finding = any candidate >= threshold. Agreement pooled over all candidate questions. No labels.The log has no candidate ids and hints have no call id, so a hint cannot be tied to a replayed answer§5.6, §5.7
U5ADR-0068 covers cmd/eval-duplicate-check tooSame real-data rule for all three eval toolsADR-0068
Rejected alternativeWhy rejected
Pass the threshold into the sweep constructor (U2)Module and judge must then agree on one number. The judge already knows its provider.
Port returns hit flags instead of p (U2)The hint row stores p. Admins see it.
Add call_id to rental_duplicate_hints now for replay labels (U4)A schema change for a report no one has run yet. Add it if duplicate labels are needed.
Match hints to replayed candidates by address (U4)Addresses change after the call. Wrong joins would score wrong.

Assumptions

#AssumptionIf wrong
A1Kev never returns a model name starting with jev-.Kev rows would enter replay. Tighten the filter.
A2A 200 row with a NULL model is not a usable Jev row. LIKE drops it.Such rows are silently skipped, and the report counts do not show them.
A3Admin reviewed means the finding was real. false_alarm means it was not (admin_message.go:84).Part A precision is wrong. Revisit the label map.
A4typesafe_calls keeps 90 days (retention job).The default window and retention-loss counts change.
A5Gremlin confirms real-data use (ADR-0066/0067/0068) before any replay or real-data eval.Do not run the replay.
A6Prod and UAT secrets are checked outside this repo before deploy (§8 step 1).A switched-on feature stops startup.

Rejected alternatives

AlternativeWhy rejected
Live shadow calls in productionDoubles calls and needs Kev hosted. Replay on a local GPU answers the same question.
Fallback chain (Jev down → Kev)Not the goal. Mixing two models' probabilities in one feature breaks thresholds.
One global model switchFeatures must choose separately (P3).
Adapter per provider nowKev speaks the same API. A second adapter would copy the client.
Thresholds in envAbout 10 more env vars, and easy to set an untested number.
kind field on registry entriesIt would only ever have one value today. Add it with the second adapter.
Read-only DB role for replayRejected by owner for simplicity (P7).
isRentalJevEnabled() keeps reading env (Q1)A second source of truth. It can disagree with the registry and panic on a nil judge.
provider column on typesafe_calls (Q2)The returned model name already tells Jev from Kev.
Resume keyed on ref_id, no frozen window (Q4)The window and the latest row still move between runs.
Missing Jev key stays silently off (Q6)Hides a lost key. One rule for all providers is simpler.
Choose a Kev version now (Q7)No evidence yet. Pinning belongs with the first threshold.
Per-question thresholds now (Q8)Labels are per message. No evidence a question needs its own number.
Score listing eval (Q9)Its cases have no right answers. Writing them is new labeling work.
Override flag for a non-loopback replay target (F1)Breaks ADR-0068. An SSH tunnel covers a LAN GPU.

Scope

In: the shared interface, client model option, registry, per-feature config, threshold tables, startup checks, the replay tool, eval tool flags, ADR-0068.

Out: see Out of scope.

Interface and client

In internal/shared/infrastructure/typesafe:

// Provider answers System One questions (noul, choice, score) about a state.
// *Client implements it for every provider that speaks the System One API.
type Provider interface {
    Judge(ctx context.Context, caller Caller, state any,
        questions map[string]Question) (*Result, error)
}

It has the same signature as rental's current rentaltypesafe.Client interface (listing_judge.go:20). Rental deletes its own interface and uses sharedtypesafe.Provider. Test fakes keep working, because the method is unchanged.

Client changes

  • New option WithModel(name string). DefaultModel = "jev-latest" stays the default.
  • judgeRequest.Model uses c.model, not the constant.
  • WithBaseURL already exists. Production now sets it for kev.
  • An empty API key sends no Authorization header. A local Kev may run without a key.
  • Logging is unchanged (P10). The typesafe_calls.model column already records the model name returned by the server.

Registry

type ProviderSpec struct {
    Name    string // "jev", "kev"
    BaseURL string
    APIKey  string
    Model   string
}

// BuildProviders returns one *Client per spec that is configured.
func BuildProviders(specs []ProviderSpec, opts ...Option) map[string]Provider

The list is built in cmd/server/main.go from config. All clients share the call-log repo and logger options.

NameRegistered whenBaseURLKeyModel
jevTYPESAFE_API_KEY is sethttps://api.typesafe.aiTYPESAFE_API_KEYjev-latest
kevKEV_BASE_URL is setKEV_BASE_URLKEV_API_KEY (optional)KevModel constant. kev-latest today; pinned before first threshold (P13).
Startup wiring: config builds the provider registry; the rental module picks a provider and threshold per feature
Startup wiring: config builds the provider registry; the rental module picks a provider and threshold per feature.

Model pinning (Q7)

Kev's first threshold ships with an immutable model id pinned in KevModel, plus the replay report.

A threshold is tuned on one model's numbers. kev-latest lets whoever runs the Kev server swap weights under the same name, so a tuned threshold would drift silently.

Rejected: choose a Kev version now — no evidence yet.

  • No Kev version is chosen today. KevModel = "kev-latest" and there is no kev threshold entry, so P5 already stops Kev from running in production.
  • The PR that adds Kev's first threshold also sets KevModel to an immutable id and attaches the replay report.
  • A unit test fails if any judge has a kev threshold entry while KevModel ends in -latest.
  • Local replay and eval tools pass --model, so kev-latest keeps working there.
  • Jev stays jev-latest to keep today's behaviour. Jev's returned version can change without notice, and our thresholds are ours to review. The replay report records the Jev versions seen. How a hosted Kev artifact stays immutable is for the hosting ADR.

app.Container.TypeSafeClient *typesafe.Client becomes DecisionProviders map[string]typesafe.Provider. newTypeSafeClient in rental/module.go is removed. The module looks up providers by name.

Config

TypeSafeConfig gains:

KevBaseURL
KEV_BASE_URL, default ""
KevAPIKey
KEV_API_KEY, default ""
ListingProvider
TYPESAFE_LISTING_PROVIDER, default jev
ChatProvider
TYPESAFE_CHAT_PROVIDER, default jev
DuplicateProvider
TYPESAFE_DUPLICATE_PROVIDER, default jev

The on/off switches stay (TYPESAFE_RENTAL_ENABLED, TYPESAFE_RENTAL_CHAT_ENABLED, TYPESAFE_RENTAL_DUPLICATE_ENABLED). What "on" means changes (Q6):

SituationTodayAfter
Switch offFeature offFeature off. No provider or threshold lookup.
Switch on, provider registered, threshold entry existsFeature onFeature on
Switch on, TYPESAFE_API_KEY empty, provider jevSilently off (config.go:49-63)Startup error
Switch on, provider kev, no KEV_BASE_URLn/aStartup error
Switch on, provider has no threshold entryn/aStartup error
Per-feature startup check. Switch and readiness are separate; not ready means the server does not start
Per-feature startup check. Switch and readiness are separate; not ready means the server does not start.
  • The switch and readiness are separate. IsRentalEnabled(), IsRentalChatEnabled() and IsRentalDuplicateEnabled() report the switches only. They no longer look at the key.
  • At startup, for each switched-on feature, the module resolves the selected provider from the registry and the threshold entry for that name. Both must succeed.
  • Only then is the judge built and injected. Any failure returns an error naming the feature and provider, and the server does not start.
  • Chat keeps needing both the rental and the chat switch, as today. Duplicate keeps needing both the rental and the duplicate switch.
  • A fake judge injected through module options (o.listingJudge, o.messageJudge, o.duplicateJudge) skips the provider and threshold lookup for its own feature only (F3). The other features still run their startup checks.

Listing "on" = the listing judge exists (Q1)

Today the module always builds the listing judge, and isRentalJevEnabled() (listing_check_queue.go:20-43) reads env directly. After this change:

  • The module builds the listing judge only when the switch is on and the provider and threshold resolve. Switch off: the judge is nil, like chat. Switch on but not ready: startup error, not nil (Q6).
  • isRentalJevEnabled(), its env reads, and the global SetRentalJevEnabledFunc are deleted. No code reads TYPESAFE_* env outside config.
  • Per-use-case test hooks (SetJevEnabledFunc) stay only where a test still needs them. Most tests inject a fake judge or nil instead.
PathTodayAfter
Queue on save (queueListingCheck, called from listing.go:332)Global env checkSkip when uc.judge == nil
Form check (listing_check_form.go:43-78)isRentalJevActive() && judge != niljudge != nil
Sweep (listing_check_sweep.go:61-103)isEnabled(), then calls judge with no nil checkReturns at once when judge == nil (nil guard)

Duplicate "on" = the duplicate judge exists (U1)

The same rule as listing:

PathTodayAfter
Module build (module.go:348-355)Built whenever a client or key existsBuilt only when IsRentalDuplicateEnabled() is true and provider and threshold resolve
Sweep gate (duplicate_check_sweep.go:54-65,110)config.Get() switch read and judge != niljudge != nil only. isEnabled, the config import and SetEnabledFunc deleted.

Queueing (dup_check_due_at) is not gated today and stays that way. It costs nothing.

Threshold tables

Each judge holds a table keyed by provider name. Today only jev has an entry:

var listingThresholds = map[string]ListingThresholds{
    "jev": {Noul: 0.7, Score: 1.5},
}
var chatThresholds = map[string]float64{"jev": 0.7}
var duplicateThresholds = map[string]float64{"jev": 0.7}
  • listing / jev / Noul any Noul >= threshold → finding (6 Noul questions) 0.7
  • listing / jev / Score addr_completeness < threshold → finding; desc_quality tip only 1.5
  • chat / jev / Noul any asked Noul >= threshold → finding (3 Noul questions) 0.7
  • duplicate / jev / Noul candidate Noul >= threshold → hint (1 Noul per candidate, up to 10) 0.7
  • kev No entry yet needs replay + pinned model —
  • Constructors take the provider name: NewListingJudge(p Provider, name string) (*ListingJudge, error).
  • A missing entry returns an error. The module returns that error, so the server refuses to start (P5).
  • This check runs only when the feature is on. A feature that is off builds no judge. This is new for listing (Q1). Chat already works this way.
  • The existing constants (NoulThreshold, ScoreThreshold, ChatNoulThreshold, DuplicateHintThreshold) become the jev entries. Tests that reference them are updated.

Duplicate threshold owned by the judge (U2)

port.DuplicateJudge gains Threshold() float64. The sweep compares p >= uc.judge.Threshold().

The duplicate judge returns raw p per candidate. The sweep saves p on the hint row, so it needs raw p and the cut. Today 0.7 is defined in both duplicate_judge.go:14 and duplicate_check_sweep.go:19; the sweep's copy is deleted. cmd/eval-duplicate-check reads the jev entry as its default threshold.

Rejected: threshold passed into the sweep constructor; port returns hit flags instead of p.

Replay tool cmd/replay-typesafe

go run ./cmd/replay-typesafe --dsn "$PROD_DSN" --feature message_check \
  --since 2026-09-01 --base-url http://127.0.0.1:8009 --model kev-latest --out ~/replay/
--dsn
Postgres DSN. Required. No fallback to .env.
--feature
listing_check, message_check or duplicate_check (U4)
--since
Earliest created_at. Default: 90 days ago.
--limit
Max calls. 0 means all.
--jev-model
Optional. One Jev version only, e.g. jev-1.13.0. Must start with jev-, or the tool exits. Default: all Jev versions.
--base-url, --model, --api-key
The provider to replay against. --base-url must be loopback (F1).
--timeout
Per-call deadline. Default 60s (F4).
--report-only
Skip all Kev calls. Rebuild report.md from answers.jsonl, labels.jsonl and fresh admin labels.
--out
Output directory. Required. One directory per evaluation. If it already holds run.json, the run resumes (Q4).
Replay flow. Production is read with SELECTs only; inputs reach only a loopback Kev; outputs are local files of ids and probabilities
Replay flow. Production is read with SELECTs only; inputs reach only a loopback Kev; outputs are local files of ids and probabilities.

Flow

  1. Select the latest Jev call per unit (Q3)

    One message or one listing (ref_id) is one unit, however many times it was checked.

  2. Decode

    Decode input into state and questions, using UnmarshalQuestion.

  3. Re-send

    Send them through a *Client built with WithBaseURL and WithModel, and no call-log repo. So the tool never writes to typesafe_calls.

  4. Record

    Append one line per call to answers.jsonl: {call_id, ref_id, jev: <output>, kev: <answers>, error}.

  5. Resume

    On restart, read run.json and answers.jsonl, then replay only units not yet answered.

  6. Report

    Load labels, then write report.md.

Calls run one at a time. A failed call is recorded with its error and not retried in the same run.

SELECT * FROM (
  SELECT DISTINCT ON (ref_id) id, ref_id, model, input, output, created_at
  FROM typesafe_calls
  WHERE app = 'rental' AND feature = $1
    AND created_at >= $2 AND created_at < $5   -- $5 = frozen until
    AND http_status = 200 AND model LIKE 'jev-%'
    AND ($3 = '' OR model = $3)   -- $3 = --jev-model
    AND ref_id IS NOT NULL
  ORDER BY ref_id, created_at DESC, id DESC
) latest
ORDER BY created_at, id
LIMIT NULLIF($4, 0)               -- $4 = --limit, applied after dedupe

A message is sent once, then retried by the sweep if the first try fails. Each try logs a row with the same ref_id. A listing logs a row on every form check and every sweep check, and it can be edited between them.

Per-call timeout (F4)

The client's http.Client{} has no timeout (client.go:97). The judges add their own deadline, but replay has none. So each replay call runs under --timeout (default 60s). A timeout is recorded as that unit's error, like any other failure, and a rerun picks the unit up again.

Loopback-only destination (F1, required by ADR-0068)

Real production inputs may only reach a model on the developer's own machine. The replay tool enforces this in three layers:

  1. Flag check

    --base-url host must be 127.0.0.1, [::1] or localhost. Anything else exits before any DB query.

  2. Dial check

    The HTTP transport's dialer checks the actual IP it connects to (a net.Dialer Control hook). A non-loopback IP fails the call. So a DNS change for localhost cannot get around the flag check.

  3. No redirects, no proxy

    CheckRedirect returns an error, because Go re-sends a POST body on 307/308. The transport's Proxy is nil, so HTTPS_PROXY in the environment cannot route calls away.

Frozen run (Q4)

Production keeps logging while a replay runs. A listing may get a newer row, so its "latest" call changes. --since also moves each day. So the first run freezes its inputs in --out/run.json:

feature, jev_model, limit
From flags
since
From --since, or run-1 time minus 90 days
until
Run-1 start time
base_url, model
The target provider
units
The selected {call_id, ref_id} list, ids only
  • API keys are never written to run.json.
  • A rerun in the same --out uses run.json, not the clock. Any flag given that differs from run.json makes the tool exit with an error naming the field.
  • The rerun fetches only unanswered units, by call_id. A unit whose row is gone (purged by the 90-day retention) is counted as a retention loss, not an error.
  • The report counts at most one line per ref_id, even if a partial answers.jsonl holds duplicates. The first line for a ref_id wins.
  • A new evaluation (new window or new target) needs a new --out directory.

Labels and report

FeatureLabelUnit
message_checkrental_messages.review_status: reviewed = real, false_alarm = not; NULL = unlabeledPer message (ref_id = message id). A message has a finding if any Noul crosses the threshold.
duplicate_checkNone (U4)Per subject property (ref_id = subject id). The latest logged sweep call, with the candidates of that call. A unit has a finding if any candidate crosses the threshold. Agreement with Jev only.
listing_checkNonePer listing (ref_id = listing id). The latest logged snapshot — not necessarily what an admin saw, because form checks also log and the listing may have changed since. Agreement with Jev only.

Label queries are SELECTs on the id list only.

Hand labels (Q5)

Admin labels exist only where Jev found something. The admin messages list returns only rows with findings (admin_message_handler.go:26-50, admin_message.go:66-79). So a message where only Kev finds something has no admin label, and the admin panel cannot show it.

  • The developer inspects a disagreement by reading that message by id, with their existing production DB access (P7). This is gated on the same Gremlin confirmation as the replay itself (ADR-0068).
  • The developer writes the result in --out/labels.jsonl: {"ref_id": "...", "label": "real" | "not_real"}. Ids only. No message text.
  • The replay tool never prints or writes message text, including in its disagreement list.
  • A hand label wins over an admin label for the same ref_id.
  • If inspection is not done or not allowed, the unit stays unlabeled. A missing label is never read as "not real".

Report contents

The report follows the judges' real shape (Q8). Each judge has one shared Noul threshold and one shared Score threshold per provider (listing: 6 Noul + 2 Score questions; chat: 3 Noul; duplicate: 1 Noul per candidate, up to 10). Admin labels are per message, not per question. report.md has, per feature:

  1. Agreement, per question

    How often Kev and Jev give the same answer on each question, with Jev at its threshold and Kev at the same number. This shows where Kev differs. It is not scored against labels. Duplicate question keys (candidate_0…) are positions, not fixed questions, so duplicate agreement is one pooled number over all candidate questions (U4).

  2. Threshold sweep, per unit

    One shared Kev Noul threshold, 0.50 to 0.90 in 0.05 steps. One shared Kev Score threshold, 0.5 to 2.5 in 0.25 steps. A unit has a finding by the judge's exact rule — chat: any asked Noul >= threshold (message_judge.go:96-107); listing: any Noul >= threshold, or addr_completeness Score < threshold, with desc_quality below threshold a tip only (listing_judge.go:232-260); duplicate: any candidate Noul >= threshold (duplicate_check_sweep.go:221-226). For each step: unit agreement with Jev, and unit-level precision where labels exist. A message-level label is never copied onto its questions.

  3. Disagreements

    Call ids and ref ids where Kev finds something and Jev does not, and the reverse. This is the developer's hand-label worklist.

  4. Label results, in two separate parts

    Part A (Jev found): units with an admin label → precision for both models. Part B (disagreements): units with a hand label → which model was right when they differ. Unlabeled disagreements are counted, not scored.

  5. Counts

    Eligible rows (before dedupe), rows with no ref_id (dropped), distinct units, units replayed, failed, lost to retention, admin-labeled, hand-labeled, unlabeled disagreements. Units are also counted per Jev version, so a version change inside the window is visible.

Output files hold ids, probabilities and run settings only. No state text and no API key is written.

Eval tool flags

cmd/eval-listing-check, cmd/eval-message-check and cmd/eval-duplicate-check gain --base-url and --model.

  • When --base-url is set, the API key is optional and --model is required. Otherwise the tool would send jev-latest to Kev. Missing --model exits with an error.
  • Without --base-url, both flags keep today's behaviour (TypeSafe, jev-latest).
  • Real data stays local (F2). Only the built-in --source synthetic counts as safe. --source db and any file path are treated as possibly real. With such a source and a custom --base-url, the tool uses the loopback-only client (flag check, dial check, no redirects, no proxy). Synthetic cases may go to any URL.

Today the listing and message tools print probabilities only. The duplicate tool already scores (Q9, U3):

eval-duplicate-check

scored, as today
  • Right answers exist: ExpectedDuplicate (cases.go:34), already scored at DuplicateHintThreshold (main.go:220).
  • New --threshold flag. Default is the jev table entry.
  • Required when --base-url is set, so Kev is never scored on Jev's number without saying so.

eval-message-check

scored
  • Right answers exist: ExpectedFindings (cases.go:29), today used only by --dry-run (main.go:170).
  • New --threshold flag, required when scoring, no default.
  • Finding codes = asked Nouls >= threshold (Q8 rule).
  • Exact match when finding codes equal ExpectedFindings as a set.
  • Report: exact-match count, plus per case the missed and extra codes.

eval-listing-check

descriptive
  • EvalCase has no expected field (cases.go:21-29).
  • Prints probabilities as today.
  • No score, because the cases have no labels.

The synthetic message and duplicate sets are the only data with known negatives. Replay has none (Q5), so the eval tools and replay answer different questions.

Duplicate check

The duplicate check shipped first (#146). This change adapts it (see Duplicate check shipped):

  • The judge takes a Provider and a provider name. It reads TYPESAFE_DUPLICATE_PROVIDER through config.
  • On = judge exists. The sweep's global config read is deleted (U1).
  • Its 0.7 threshold becomes the jev entry of a table, reached through Threshold() (U2).
  • The eval tool gets the provider flags and a threshold override (U3).
  • Replay supports it without labels (U4).

Edge cases

CaseBehaviour
Feature on, provider not registered (e.g. kev with no KEV_BASE_URL)Startup error naming the feature and provider
Feature on, provider jev, TYPESAFE_API_KEY emptyStartup error. Today this is silently off.
Feature on, provider has no threshold entryStartup error
Feature off, provider value invalidIgnored. No judge is built.
Unknown provider name in envStartup error if that feature is on
Kev server down during replayError recorded per call. Rerun resumes.
Rerun with a flag that differs from run.jsonTool exits, naming the field. Nothing is replayed.
New rows logged between runsNot seen. The query is frozen at until.
Unit's row purged between runsCounted as a retention loss
Partial answers.jsonl has two lines for one ref_idReport keeps the first one
Replay --base-url is not loopbackTool exits before any DB query
localhost resolves to a non-loopback IPDial fails. Recorded as that unit's error.
Kev server answers with a redirectCall fails. Not followed.
HTTPS_PROXY set in the shellIgnored by the guarded client
Eval with --source db or a file, plus a non-loopback --base-urlTool exits
Kev call hangsFails at --timeout (60s). Rerun retries it.
Test injects a fake listing judge, chat switch on with no providerListing lookup skipped. Chat still fails startup.
Kev-only finding, no inspection doneUnlabeled. Counted, never scored as "not real".
Hand label and admin label differHand label wins
typesafe_calls row older than 90 daysAlready purged. Not replayable.
input row fails to decodeRecorded as an error line. The run continues.
Jev version changed inside the windowAll versions replayed by default. Report counts per version. --jev-model picks one.
--jev-model does not start with jev-Tool exits before any query
Same ref_id logged several timesLatest 200-row wins. Tie on created_at broken by id DESC.
Row with no ref_idNot replayed. Counted in the report.
Choice option orderGo marshals map keys in sorted order, so both providers see the same order
Duplicate switch on, sweep has a nil judgeCannot happen: switch on without a ready provider is a startup error. Switch off: nil judge, sweep returns at once.
Duplicate judge on Kev with no kev entryStartup error, like the other judges
Duplicate replay, subject re-checked after an address changeLatest call per subject wins, with that call's candidates
Duplicate eval with --base-url and no --thresholdTool exits

Tests

Rental Go tests run with -p 1.

AreaTest
ClientWithModel sends that model in the body. The default is jev-latest. An empty key sends no Authorization header.
RegistryA kev threshold entry with KevModel ending in -latest fails the test (P13). jev is registered only with a key. kev only with a URL. Missing specs are absent from the map.
ConfigProvider env defaults to jev. IsRentalEnabled() / IsRentalChatEnabled() follow the switches only, with or without a key.
JudgesThreshold lookup by provider. A missing entry fails the constructor. Existing judge tests pass on jev.
ModuleInjected fake judge skips only its own feature's lookup; another switched-on feature with no provider still errors. Feature on with an unregistered or untested provider → error. Switch on with jev and an empty TYPESAFE_API_KEY → startup error naming the feature (missing-key test). Feature off → no error, no judge.
Listing on-stateNil judge: save queues nothing, form check returns checked: false, sweep returns without claiming rows. No test uses SetRentalJevEnabledFunc (deleted).
Duplicate on-stateSwitch off → no judge built. Nil judge → sweep makes no query. Sweep reads no config. No test uses the sweep's SetEnabledFunc (deleted). Sweep hits follow Threshold(): a fake judge at 0.8 makes a 0.75 candidate a miss.
Destination guardFlag check rejects http://10.0.0.5, https://api.typesafe.ai and accepts 127.0.0.1, [::1], localhost. Dial check fails a connect to a non-loopback IP even when the URL host was allowed (fake resolver). A 307 from an httptest server is not followed and the redirect target gets no request. HTTPS_PROXY is not used.
ReplayPer-call timeout: a handler that sleeps past --timeout is recorded as an error, and the next run retries it. Decode → re-send round trip against an httptest fake. Resume skips done ids. The client has no log repo. Report sweep math on a fixed answers file: unit finding follows each judge's rule (chat asked-Noul only; listing desc_quality never yields a finding); labels are never split per question. A bad input row is recorded, not fatal. Selection keeps jev-1.13.0 rows and drops kev-* rows. --jev-model narrows to one version and rejects a non-jev- value. Report counts per version. Dedupe keeps one latest row per ref_id, with the id tie-break; a null ref_id is counted, not replayed; --limit counts units, not rows. Rerun uses run.json window, rejects a conflicting flag, fetches only unanswered units, counts a purged unit as retention loss. run.json holds no API key. Report keeps one line per ref_id. Duplicate: unit finding = any candidate >= threshold, agreement pooled, no label query, report says unlabeled. Hand label beats admin label; a missing label is never scored as negative; Parts A and B are separate; --report-only makes no Kev call; no output file holds message text.
Eval flagsReal source (db or a file) with a non-loopback --base-url exits; synthetic with the same URL runs. --base-url and --model reach the client. The key is optional when --base-url is set. --base-url without --model exits (both tools). Message eval scoring: exact set match, missed and extra codes per case, only asked questions count, --threshold required. Listing eval prints no score. Duplicate eval: default threshold is the jev entry; --threshold changes the cut; --base-url without --threshold exits.

Rollout

  1. Ship on jev

    Every feature on jev. One behaviour change: a switch that is on (TYPESAFE_RENTAL_ENABLED, TYPESAFE_RENTAL_CHAT_ENABLED, TYPESAFE_RENTAL_DUPLICATE_ENABLED) with an empty TYPESAFE_API_KEY now stops startup. The operator checks each environment's secrets first; this repo holds none, so it cannot be done here.

  2. Local Kev eval

    Run Kev locally. Run the eval tools with --base-url and --model. After Gremlin's confirmation (ADR-0068), run the replay against production with a loopback --base-url.

  3. Pin + threshold

    If Kev wins for a feature: one PR pins KevModel to an immutable id, adds the threshold entry, and puts the replay numbers (including Jev versions seen) in the PR body (P13). A later weights or id change repeats this step.

  4. Hosting ADR

    Hosting Kev for production is a new third party and needs its own ADR. It also says how the pinned model artifact stays immutable.

  5. Switch

    Set KEV_BASE_URL and TYPESAFE_<FEATURE>_PROVIDER=kev, then redeploy.

Privacy — ADR-0068

Each typesafe_calls row stores the full input: listing addresses and descriptions, chat text with up to 5 earlier messages, and duplicate-check property addresses (a subject and up to 10 candidates from other landlords). ADR-0068 allows the replay tool to read these from production and send them only to a Kev on the developer's own machine.

  • Loopback enforced with no override; the same rule for all three eval tools with a non-synthetic source and a custom --base-url.
  • Output files hold call ids, ref ids, probabilities and run settings only. No state text and no API key.
  • SELECT-only; nothing is written to production. The developer's DB login can write; a read-only role was declined for simplicity.
  • Hand labels: read one message by id with existing DB access; the local file holds ids and a label only.
  • Gated on the same Gremlin confirmation as ADR-0066 and ADR-0067 calibration. Gremlin is the accountable owner (as in ADR-0064).
  • Only the last 90 days of calls can be replayed. Hosting Kev for production needs its own ADR.

Out of scope

  • Admin page for switching providers (later, if switches become frequent).
  • Live shadow calls.
  • Storing replay results in production.
  • A second adapter, and the logging wrapper that goes with it.
  • Hosting Kev for production.
  • Kev threshold numbers. They come from the first replay.
  • Per-question thresholds. A later PR may add them with replay evidence.
  • Fallback between providers.