Decision Model Providers — Per-Feature Switch and Local Replay
Lets each rental AI feature (listing, chat, duplicate) choose its own System One decision model, TypeSafe Jev or the self-hosted open-weight Kev, through one Provider interface, a registry and per-provider threshold tables. Before any switch, real production calls are replayed against Kev on the developer's own machine through a loopback-only client, and a per-unit report compares the two models. Updated after the duplicate check shipped (#146) to adapt its judge, sweep gate, threshold and eval tool.
Summary
Today every Jev feature is locked to one hard-coded model. This design lets each rental feature pick its own decision model — TypeSafe Jev or the self-hosted Kev — through one interface, a registry and per-provider threshold tables. Before any switch, real production calls are replayed against Kev on the developer's own machine, and a report compares the two.
Related: Jev listing check (shared client, J13 naming); Jev chat message check; Jev duplicate check; ADR-0068 (replay privacy); ADR-0064, ADR-0066, ADR-0067; Kev: github.com/jaredpalmer/kev.
Why
The shared client sends "model": "jev-latest" to https://api.typesafe.ai, hard-coded.
Other decision models now exist. Kev is a family of open-weight models that copies TypeSafe's System One API exactly.
POST /v1/systemone, same request and answer JSON- Noul, Choice, Score questions
- Returns probabilities
- We host it ourselves (GPU)
- Probabilities are calibrated differently
- Ranks answers less well: 45–57% vs 70% automated at a 5% error budget
We want:
- Each feature picks its own provider: listing, chat, duplicate.
- Compare before switching: replay real production calls against Kev on a local GPU machine, then decide.
- Easy to add providers later: one interface. A new adapter is written only when a provider with a different API is known.
Deliverables
| # | Deliverable |
|---|---|
| 1 | Provider interface in the shared typesafe package |
| 2 | Client WithModel option; model name no longer hard-coded |
| 3 | Provider registry (jev, kev) built at startup from config |
| 4 | Per-feature provider env vars |
| 5 | Per-provider threshold tables in each judge; startup refuses an untested pair |
| 6 | cmd/replay-typesafe: local replay of production calls with a report, plus a shared loopback-only HTTP client (F1) used by replay and real-data eval |
| 7 | --base-url and --model flags on cmd/eval-listing-check, cmd/eval-message-check and cmd/eval-duplicate-check; --threshold scoring in cmd/eval-message-check; --threshold override in cmd/eval-duplicate-check (U3) |
| 8 | ADR-0068 |
Decisions
| # | Decision |
|---|---|
| P1 | Production default stays jev for every feature. |
| P2 | Switching is by env var and a redeploy. An admin page comes later, only if switches become frequent. |
| P3 | Each feature has its own provider env var (TYPESAFE_LISTING_PROVIDER, TYPESAFE_CHAT_PROVIDER, TYPESAFE_DUPLICATE_PROVIDER). |
| P4 | Thresholds live in code, one table per judge, keyed by provider name. Each entry keeps today's shape: one shared Noul and (listing) one shared Score threshold. A new entry needs an eval or replay run first. Per-question thresholds only in a later PR with evidence (Q8). |
| P5 | The server refuses to start when a feature switch is on and its provider is not registered (including a missing Jev key) or has no threshold entry. Today a missing key silently turns the feature off (Q6). |
| P6 | Comparison is by replay on a local machine, not a live shadow call. Production only ever calls its chosen provider. |
| P7 | The replay tool reads production with the developer's normal DB login. It runs SELECTs only. No read-only role. |
| P8 | Replay output stays in local files. Nothing is written back to production. A typesafe_replays table comes later, only if others need the results. |
| P9 | The shared interface is the only abstraction now. A second adapter is written only when a provider with a different API is chosen. |
| P10 | Call logging stays inside the HTTP client. Jev and Kev share that client, so both are logged. Logging moves into a wrapper when a second adapter arrives. |
| P11 | The package keeps the name typesafe. "System One" is the API shape TypeSafe defined and Kev copies. |
| P12 | The registry list lives in code. Env holds only URLs and keys. A new provider needs a threshold table anyway, which is a code change. |
| P13 | Kev production activation rule (Q7): Kev's first threshold entry ships only with an immutable model id pinned in code, plus the replay evidence. Changing the weights or the id needs a fresh evaluation and threshold review. kev-latest is for local evaluation only. Jev stays jev-latest. |
Grill decisions (2026-09-26)
| # | Decision | Why | Where |
|---|---|---|---|
| Q1 | Listing is on only when its judge exists. Env reads and SetRentalJevEnabledFunc deleted. Sweep nil guard. | Two sources of truth could disagree and call a nil judge | §5.4 |
| Q2 | Replay selects model LIKE 'jev-%', optional --jev-model, counts per version | The log stores the returned model (jev-1.13.0), not jev-latest | §5.6, §5.7 |
| Q3 | One unit = latest 200-row per ref_id, tie-break id DESC, --limit after dedupe | Retries and form checks log many rows per unit | §5.6, §5.7 |
| Q4 | run.json freezes window, filters, target and unit list. No keys. Rerun rejects conflicting flags. | New rows and a moving --since would change a resumed run | §5.6 |
| Q5 | Local labels.jsonl, hand label wins, Parts A/B separate, --report-only, no recall claim | Admins only see what Jev found. A missing label is not a negative. | §5.7 |
| Q6 | Switch on + provider not ready = startup error, including a missing Jev key | Silent off hides lost keys and bad provider names | §5.4, §8 |
| Q7 | Kev needs a pinned immutable model id before its first threshold. kev-latest local only. | Weights swapped under one name make a tuned threshold drift | §5.3, P13 |
| Q8 | One shared Noul/Score threshold per judge per provider. Sweeps per unit. | Labels are per message, and the table cannot hold per-question numbers | §5.5, §5.7 |
| Q9 | Message eval scored against ExpectedFindings at --threshold. Listing eval descriptive. --model required with --base-url. | Only message cases have right answers. Avoid sending jev-latest to Kev. | §5.8 |
| F1 | Replay destination loopback only: flag check, dial check, no redirects, no proxy, no override | ADR-0068 allows only the developer's own machine | §5.6 |
| F2 | Same guard for eval with db or file source and a custom --base-url | Those sources may hold real data | §5.8 |
| F3 | Injected fake judge skips only its own feature's lookup | Tests inject fakes without a registry | §5.4 |
| F4 | Replay per-call timeout, 60s default, recorded, resumable | http.Client{} has no timeout | §5.6 |
Duplicate check shipped (2026-09-26, #146)
The duplicate check was built before this change. It follows the old pattern, so this change adapts it. What shipped:
| Thing | Where |
|---|---|
| Judge built whenever a client or a key exists, switch not read | rental/module.go:348-355 |
Sweep reads global config every tick (config.Get()), plus a SetEnabledFunc test hook | usecase/duplicate_check_sweep.go:54-65,110 |
IsRentalDuplicateEnabled() = key && rental switch && duplicate switch | config.go:61-63 |
DuplicateHintThreshold = 0.7 defined twice. The sweep uses its own copy. The eval uses the adapter's. | duplicate_judge.go:14, duplicate_check_sweep.go:19,222 |
| Port returns raw probabilities. The sweep applies the cut and saves p on the hint. | port/ports.go:1221 |
Eval already scores against ExpectedDuplicate at the threshold | cmd/eval-duplicate-check/main.go:220 |
| Logged state holds addresses by position, no ids. Hints have no call id. | duplicate_judge.go:44-54, rental_duplicate_hints |
| # | Decision | Why | Where |
|---|---|---|---|
| U1 | Duplicate is on only when its judge exists, same as listing (Q1). Sweep isEnabled, its config.Get() read and SetEnabledFunc deleted. Judge built only when the switch is on and the provider resolves. | Two sources of truth again. A global config read breaks "no TYPESAFE_* read outside config". | §5.4 |
| U2 | Threshold table {"jev": 0.7} in the adapter. NewDuplicateJudge(p, name) errors on a missing entry. The port gains Threshold() float64. The sweep uses it and its own constant is deleted. | Two copies of 0.7 can drift. The sweep needs raw p for the hint row, so the judge cannot hide p. | §5.5 |
| U3 | cmd/eval-duplicate-check gets --base-url and --model (F2 guard too), plus --threshold. Default is the jev entry. Required with --base-url. | Kev must not be scored silently on Jev's number | §5.8 |
| U4 | Replay supports duplicate_check. One unit = one subject call. Finding = any candidate >= threshold. Agreement pooled over all candidate questions. No labels. | The log has no candidate ids and hints have no call id, so a hint cannot be tied to a replayed answer | §5.6, §5.7 |
| U5 | ADR-0068 covers cmd/eval-duplicate-check too | Same real-data rule for all three eval tools | ADR-0068 |
| Rejected alternative | Why rejected |
|---|---|
| Pass the threshold into the sweep constructor (U2) | Module and judge must then agree on one number. The judge already knows its provider. |
| Port returns hit flags instead of p (U2) | The hint row stores p. Admins see it. |
Add call_id to rental_duplicate_hints now for replay labels (U4) | A schema change for a report no one has run yet. Add it if duplicate labels are needed. |
| Match hints to replayed candidates by address (U4) | Addresses change after the call. Wrong joins would score wrong. |
Assumptions
| # | Assumption | If wrong |
|---|---|---|
| A1 | Kev never returns a model name starting with jev-. | Kev rows would enter replay. Tighten the filter. |
| A2 | A 200 row with a NULL model is not a usable Jev row. LIKE drops it. | Such rows are silently skipped, and the report counts do not show them. |
| A3 | Admin reviewed means the finding was real. false_alarm means it was not (admin_message.go:84). | Part A precision is wrong. Revisit the label map. |
| A4 | typesafe_calls keeps 90 days (retention job). | The default window and retention-loss counts change. |
| A5 | Gremlin confirms real-data use (ADR-0066/0067/0068) before any replay or real-data eval. | Do not run the replay. |
| A6 | Prod and UAT secrets are checked outside this repo before deploy (§8 step 1). | A switched-on feature stops startup. |
Rejected alternatives
| Alternative | Why rejected |
|---|---|
| Live shadow calls in production | Doubles calls and needs Kev hosted. Replay on a local GPU answers the same question. |
| Fallback chain (Jev down → Kev) | Not the goal. Mixing two models' probabilities in one feature breaks thresholds. |
| One global model switch | Features must choose separately (P3). |
| Adapter per provider now | Kev speaks the same API. A second adapter would copy the client. |
| Thresholds in env | About 10 more env vars, and easy to set an untested number. |
kind field on registry entries | It would only ever have one value today. Add it with the second adapter. |
| Read-only DB role for replay | Rejected by owner for simplicity (P7). |
isRentalJevEnabled() keeps reading env (Q1) | A second source of truth. It can disagree with the registry and panic on a nil judge. |
provider column on typesafe_calls (Q2) | The returned model name already tells Jev from Kev. |
Resume keyed on ref_id, no frozen window (Q4) | The window and the latest row still move between runs. |
| Missing Jev key stays silently off (Q6) | Hides a lost key. One rule for all providers is simpler. |
| Choose a Kev version now (Q7) | No evidence yet. Pinning belongs with the first threshold. |
| Per-question thresholds now (Q8) | Labels are per message. No evidence a question needs its own number. |
| Score listing eval (Q9) | Its cases have no right answers. Writing them is new labeling work. |
| Override flag for a non-loopback replay target (F1) | Breaks ADR-0068. An SSH tunnel covers a LAN GPU. |
Scope
In: the shared interface, client model option, registry, per-feature config, threshold tables, startup checks, the replay tool, eval tool flags, ADR-0068.
Out: see Out of scope.
Interface and client
In internal/shared/infrastructure/typesafe:
// Provider answers System One questions (noul, choice, score) about a state.
// *Client implements it for every provider that speaks the System One API.
type Provider interface {
Judge(ctx context.Context, caller Caller, state any,
questions map[string]Question) (*Result, error)
}
It has the same signature as rental's current rentaltypesafe.Client interface (listing_judge.go:20).
Rental deletes its own interface and uses sharedtypesafe.Provider. Test fakes keep working, because the method is unchanged.
Client changes
- New option
WithModel(name string).DefaultModel = "jev-latest"stays the default. judgeRequest.Modelusesc.model, not the constant.WithBaseURLalready exists. Production now sets it forkev.- An empty API key sends no
Authorizationheader. A local Kev may run without a key. - Logging is unchanged (P10). The
typesafe_calls.modelcolumn already records the model name returned by the server.
Registry
type ProviderSpec struct {
Name string // "jev", "kev"
BaseURL string
APIKey string
Model string
}
// BuildProviders returns one *Client per spec that is configured.
func BuildProviders(specs []ProviderSpec, opts ...Option) map[string]Provider
The list is built in cmd/server/main.go from config. All clients share the call-log repo and logger options.
| Name | Registered when | BaseURL | Key | Model |
|---|---|---|---|---|
jev | TYPESAFE_API_KEY is set | https://api.typesafe.ai | TYPESAFE_API_KEY | jev-latest |
kev | KEV_BASE_URL is set | KEV_BASE_URL | KEV_API_KEY (optional) | KevModel constant. kev-latest today; pinned before first threshold (P13). |
Model pinning (Q7)
Kev's first threshold ships with an immutable model id pinned in KevModel, plus the replay report.
A threshold is tuned on one model's numbers. kev-latest lets whoever runs the Kev server swap weights under
the same name, so a tuned threshold would drift silently.
Rejected: choose a Kev version now — no evidence yet.
- No Kev version is chosen today.
KevModel = "kev-latest"and there is nokevthreshold entry, so P5 already stops Kev from running in production. - The PR that adds Kev's first threshold also sets
KevModelto an immutable id and attaches the replay report. - A unit test fails if any judge has a
kevthreshold entry whileKevModelends in-latest. - Local replay and eval tools pass
--model, sokev-latestkeeps working there. - Jev stays
jev-latestto keep today's behaviour. Jev's returned version can change without notice, and our thresholds are ours to review. The replay report records the Jev versions seen. How a hosted Kev artifact stays immutable is for the hosting ADR.
app.Container.TypeSafeClient *typesafe.Client becomes DecisionProviders map[string]typesafe.Provider.
newTypeSafeClient in rental/module.go is removed. The module looks up providers by name.
Config
TypeSafeConfig gains:
KevBaseURLKEV_BASE_URL, default""KevAPIKeyKEV_API_KEY, default""ListingProviderTYPESAFE_LISTING_PROVIDER, defaultjevChatProviderTYPESAFE_CHAT_PROVIDER, defaultjevDuplicateProviderTYPESAFE_DUPLICATE_PROVIDER, defaultjev
The on/off switches stay (TYPESAFE_RENTAL_ENABLED, TYPESAFE_RENTAL_CHAT_ENABLED,
TYPESAFE_RENTAL_DUPLICATE_ENABLED). What "on" means changes (Q6):
| Situation | Today | After |
|---|---|---|
| Switch off | Feature off | Feature off. No provider or threshold lookup. |
| Switch on, provider registered, threshold entry exists | Feature on | Feature on |
Switch on, TYPESAFE_API_KEY empty, provider jev | Silently off (config.go:49-63) | Startup error |
Switch on, provider kev, no KEV_BASE_URL | n/a | Startup error |
| Switch on, provider has no threshold entry | n/a | Startup error |
- The switch and readiness are separate.
IsRentalEnabled(),IsRentalChatEnabled()andIsRentalDuplicateEnabled()report the switches only. They no longer look at the key. - At startup, for each switched-on feature, the module resolves the selected provider from the registry and the threshold entry for that name. Both must succeed.
- Only then is the judge built and injected. Any failure returns an error naming the feature and provider, and the server does not start.
- Chat keeps needing both the rental and the chat switch, as today. Duplicate keeps needing both the rental and the duplicate switch.
- A fake judge injected through module options (
o.listingJudge,o.messageJudge,o.duplicateJudge) skips the provider and threshold lookup for its own feature only (F3). The other features still run their startup checks.
Listing "on" = the listing judge exists (Q1)
Today the module always builds the listing judge, and isRentalJevEnabled()
(listing_check_queue.go:20-43) reads env directly. After this change:
- The module builds the listing judge only when the switch is on and the provider and threshold resolve. Switch off: the judge is
nil, like chat. Switch on but not ready: startup error, notnil(Q6). isRentalJevEnabled(), its env reads, and the globalSetRentalJevEnabledFuncare deleted. No code readsTYPESAFE_*env outside config.- Per-use-case test hooks (
SetJevEnabledFunc) stay only where a test still needs them. Most tests inject a fake judge ornilinstead.
| Path | Today | After |
|---|---|---|
Queue on save (queueListingCheck, called from listing.go:332) | Global env check | Skip when uc.judge == nil |
Form check (listing_check_form.go:43-78) | isRentalJevActive() && judge != nil | judge != nil |
Sweep (listing_check_sweep.go:61-103) | isEnabled(), then calls judge with no nil check | Returns at once when judge == nil (nil guard) |
Duplicate "on" = the duplicate judge exists (U1)
The same rule as listing:
| Path | Today | After |
|---|---|---|
Module build (module.go:348-355) | Built whenever a client or key exists | Built only when IsRentalDuplicateEnabled() is true and provider and threshold resolve |
Sweep gate (duplicate_check_sweep.go:54-65,110) | config.Get() switch read and judge != nil | judge != nil only. isEnabled, the config import and SetEnabledFunc deleted. |
Queueing (dup_check_due_at) is not gated today and stays that way. It costs nothing.
Threshold tables
Each judge holds a table keyed by provider name. Today only jev has an entry:
var listingThresholds = map[string]ListingThresholds{
"jev": {Noul: 0.7, Score: 1.5},
}
var chatThresholds = map[string]float64{"jev": 0.7}
var duplicateThresholds = map[string]float64{"jev": 0.7}
-
listing / jev / Noulany Noul >= threshold→ finding (6 Noul questions) 0.7 -
listing / jev / Scoreaddr_completeness < threshold→ finding;desc_qualitytip only 1.5 -
chat / jev / Noulany asked Noul >= threshold→ finding (3 Noul questions) 0.7 -
duplicate / jev / Noulcandidate Noul >= threshold→ hint (1 Noul per candidate, up to 10) 0.7 -
kevNo entry yet needs replay + pinned model —
- Constructors take the provider name:
NewListingJudge(p Provider, name string) (*ListingJudge, error). - A missing entry returns an error. The module returns that error, so the server refuses to start (P5).
- This check runs only when the feature is on. A feature that is off builds no judge. This is new for listing (Q1). Chat already works this way.
- The existing constants (
NoulThreshold,ScoreThreshold,ChatNoulThreshold,DuplicateHintThreshold) become thejeventries. Tests that reference them are updated.
Duplicate threshold owned by the judge (U2)
port.DuplicateJudge gains Threshold() float64. The sweep compares p >= uc.judge.Threshold().
The duplicate judge returns raw p per candidate. The sweep saves p on the hint row, so it needs raw p and the cut.
Today 0.7 is defined in both duplicate_judge.go:14 and duplicate_check_sweep.go:19; the sweep's copy is deleted.
cmd/eval-duplicate-check reads the jev entry as its default threshold.
Rejected: threshold passed into the sweep constructor; port returns hit flags instead of p.
Replay tool cmd/replay-typesafe
go run ./cmd/replay-typesafe --dsn "$PROD_DSN" --feature message_check \
--since 2026-09-01 --base-url http://127.0.0.1:8009 --model kev-latest --out ~/replay/
--dsn- Postgres DSN. Required. No fallback to
.env. --featurelisting_check,message_checkorduplicate_check(U4)--since- Earliest
created_at. Default: 90 days ago. --limit- Max calls.
0means all. --jev-model- Optional. One Jev version only, e.g.
jev-1.13.0. Must start withjev-, or the tool exits. Default: all Jev versions. --base-url,--model,--api-key- The provider to replay against.
--base-urlmust be loopback (F1). --timeout- Per-call deadline. Default
60s(F4). --report-only- Skip all Kev calls. Rebuild
report.mdfromanswers.jsonl,labels.jsonland fresh admin labels. --out- Output directory. Required. One directory per evaluation. If it already holds
run.json, the run resumes (Q4).
Flow
-
Select the latest Jev call per unit (Q3)
One message or one listing (
ref_id) is one unit, however many times it was checked. -
Decode
Decode
inputintostateandquestions, usingUnmarshalQuestion. -
Re-send
Send them through a
*Clientbuilt withWithBaseURLandWithModel, and no call-log repo. So the tool never writes totypesafe_calls. -
Record
Append one line per call to
answers.jsonl:{call_id, ref_id, jev: <output>, kev: <answers>, error}. -
Resume
On restart, read
run.jsonandanswers.jsonl, then replay only units not yet answered. -
Report
Load labels, then write
report.md.
Calls run one at a time. A failed call is recorded with its error and not retried in the same run.
SELECT * FROM (
SELECT DISTINCT ON (ref_id) id, ref_id, model, input, output, created_at
FROM typesafe_calls
WHERE app = 'rental' AND feature = $1
AND created_at >= $2 AND created_at < $5 -- $5 = frozen until
AND http_status = 200 AND model LIKE 'jev-%'
AND ($3 = '' OR model = $3) -- $3 = --jev-model
AND ref_id IS NOT NULL
ORDER BY ref_id, created_at DESC, id DESC
) latest
ORDER BY created_at, id
LIMIT NULLIF($4, 0) -- $4 = --limit, applied after dedupe
A message is sent once, then retried by the sweep if the first try fails. Each try logs a row with the same ref_id.
A listing logs a row on every form check and every sweep check, and it can be edited between them.
Per-call timeout (F4)
The client's http.Client{} has no timeout (client.go:97). The judges add their own deadline, but replay has none.
So each replay call runs under --timeout (default 60s). A timeout is recorded as that unit's error, like any other failure,
and a rerun picks the unit up again.
Loopback-only destination (F1, required by ADR-0068)
Real production inputs may only reach a model on the developer's own machine. The replay tool enforces this in three layers:
-
Flag check
--base-urlhost must be127.0.0.1,[::1]orlocalhost. Anything else exits before any DB query. -
Dial check
The HTTP transport's dialer checks the actual IP it connects to (a
net.DialerControlhook). A non-loopback IP fails the call. So a DNS change forlocalhostcannot get around the flag check. -
No redirects, no proxy
CheckRedirectreturns an error, because Go re-sends a POST body on 307/308. The transport'sProxyisnil, soHTTPS_PROXYin the environment cannot route calls away.
Frozen run (Q4)
Production keeps logging while a replay runs. A listing may get a newer row, so its "latest" call changes. --since also moves
each day. So the first run freezes its inputs in --out/run.json:
feature,jev_model,limit- From flags
since- From
--since, or run-1 time minus 90 days until- Run-1 start time
base_url,model- The target provider
units- The selected
{call_id, ref_id}list, ids only
- API keys are never written to
run.json. - A rerun in the same
--outusesrun.json, not the clock. Any flag given that differs fromrun.jsonmakes the tool exit with an error naming the field. - The rerun fetches only unanswered units, by
call_id. A unit whose row is gone (purged by the 90-day retention) is counted as a retention loss, not an error. - The report counts at most one line per
ref_id, even if a partialanswers.jsonlholds duplicates. The first line for aref_idwins. - A new evaluation (new window or new target) needs a new
--outdirectory.
Labels and report
| Feature | Label | Unit |
|---|---|---|
message_check | rental_messages.review_status: reviewed = real, false_alarm = not; NULL = unlabeled | Per message (ref_id = message id). A message has a finding if any Noul crosses the threshold. |
duplicate_check | None (U4) | Per subject property (ref_id = subject id). The latest logged sweep call, with the candidates of that call. A unit has a finding if any candidate crosses the threshold. Agreement with Jev only. |
listing_check | None | Per listing (ref_id = listing id). The latest logged snapshot — not necessarily what an admin saw, because form checks also log and the listing may have changed since. Agreement with Jev only. |
Label queries are SELECTs on the id list only.
Hand labels (Q5)
Admin labels exist only where Jev found something. The admin messages list returns only rows with findings
(admin_message_handler.go:26-50, admin_message.go:66-79). So a message where only Kev finds something has
no admin label, and the admin panel cannot show it.
- The developer inspects a disagreement by reading that message by id, with their existing production DB access (P7). This is gated on the same Gremlin confirmation as the replay itself (ADR-0068).
- The developer writes the result in
--out/labels.jsonl:{"ref_id": "...", "label": "real" | "not_real"}. Ids only. No message text. - The replay tool never prints or writes message text, including in its disagreement list.
- A hand label wins over an admin label for the same
ref_id. - If inspection is not done or not allowed, the unit stays unlabeled. A missing label is never read as "not real".
Report contents
The report follows the judges' real shape (Q8). Each judge has one shared Noul threshold and one shared Score
threshold per provider (listing: 6 Noul + 2 Score questions; chat: 3 Noul; duplicate: 1 Noul per candidate, up to 10). Admin labels are per message, not per question.
report.md has, per feature:
-
Agreement, per question
How often Kev and Jev give the same answer on each question, with Jev at its threshold and Kev at the same number. This shows where Kev differs. It is not scored against labels. Duplicate question keys (
candidate_0…) are positions, not fixed questions, so duplicate agreement is one pooled number over all candidate questions (U4). -
Threshold sweep, per unit
One shared Kev Noul threshold, 0.50 to 0.90 in 0.05 steps. One shared Kev Score threshold, 0.5 to 2.5 in 0.25 steps. A unit has a finding by the judge's exact rule — chat: any asked Noul
>=threshold (message_judge.go:96-107); listing: any Noul>=threshold, oraddr_completenessScore<threshold, withdesc_qualitybelow threshold a tip only (listing_judge.go:232-260); duplicate: any candidate Noul>=threshold (duplicate_check_sweep.go:221-226). For each step: unit agreement with Jev, and unit-level precision where labels exist. A message-level label is never copied onto its questions. -
Disagreements
Call ids and ref ids where Kev finds something and Jev does not, and the reverse. This is the developer's hand-label worklist.
-
Label results, in two separate parts
Part A (Jev found): units with an admin label → precision for both models. Part B (disagreements): units with a hand label → which model was right when they differ. Unlabeled disagreements are counted, not scored.
-
Counts
Eligible rows (before dedupe), rows with no
ref_id(dropped), distinct units, units replayed, failed, lost to retention, admin-labeled, hand-labeled, unlabeled disagreements. Units are also counted per Jev version, so a version change inside the window is visible.
Output files hold ids, probabilities and run settings only. No state text and no API key is written.
Eval tool flags
cmd/eval-listing-check, cmd/eval-message-check and cmd/eval-duplicate-check gain --base-url and --model.
- When
--base-urlis set, the API key is optional and--modelis required. Otherwise the tool would sendjev-latestto Kev. Missing--modelexits with an error. - Without
--base-url, both flags keep today's behaviour (TypeSafe,jev-latest). - Real data stays local (F2). Only the built-in
--source syntheticcounts as safe.--source dband any file path are treated as possibly real. With such a source and a custom--base-url, the tool uses the loopback-only client (flag check, dial check, no redirects, no proxy). Synthetic cases may go to any URL.
Today the listing and message tools print probabilities only. The duplicate tool already scores (Q9, U3):
eval-duplicate-check
scored, as today
- Right answers exist:
ExpectedDuplicate(cases.go:34), already scored atDuplicateHintThreshold(main.go:220). - New
--thresholdflag. Default is thejevtable entry. - Required when
--base-urlis set, so Kev is never scored on Jev's number without saying so.
eval-message-check
scored
- Right answers exist:
ExpectedFindings(cases.go:29), today used only by--dry-run(main.go:170). - New
--thresholdflag, required when scoring, no default. - Finding codes = asked Nouls
>=threshold (Q8 rule). - Exact match when finding codes equal
ExpectedFindingsas a set. - Report: exact-match count, plus per case the missed and extra codes.
eval-listing-check
descriptive
EvalCasehas no expected field (cases.go:21-29).- Prints probabilities as today.
- No score, because the cases have no labels.
The synthetic message and duplicate sets are the only data with known negatives. Replay has none (Q5), so the eval tools and replay answer different questions.
Duplicate check
The duplicate check shipped first (#146). This change adapts it (see Duplicate check shipped):
- The judge takes a
Providerand a provider name. It readsTYPESAFE_DUPLICATE_PROVIDERthrough config. - On = judge exists. The sweep's global config read is deleted (U1).
- Its 0.7 threshold becomes the
jeventry of a table, reached throughThreshold()(U2). - The eval tool gets the provider flags and a threshold override (U3).
- Replay supports it without labels (U4).
Edge cases
| Case | Behaviour |
|---|---|
Feature on, provider not registered (e.g. kev with no KEV_BASE_URL) | Startup error naming the feature and provider |
Feature on, provider jev, TYPESAFE_API_KEY empty | Startup error. Today this is silently off. |
| Feature on, provider has no threshold entry | Startup error |
| Feature off, provider value invalid | Ignored. No judge is built. |
| Unknown provider name in env | Startup error if that feature is on |
| Kev server down during replay | Error recorded per call. Rerun resumes. |
Rerun with a flag that differs from run.json | Tool exits, naming the field. Nothing is replayed. |
| New rows logged between runs | Not seen. The query is frozen at until. |
| Unit's row purged between runs | Counted as a retention loss |
Partial answers.jsonl has two lines for one ref_id | Report keeps the first one |
Replay --base-url is not loopback | Tool exits before any DB query |
localhost resolves to a non-loopback IP | Dial fails. Recorded as that unit's error. |
| Kev server answers with a redirect | Call fails. Not followed. |
HTTPS_PROXY set in the shell | Ignored by the guarded client |
Eval with --source db or a file, plus a non-loopback --base-url | Tool exits |
| Kev call hangs | Fails at --timeout (60s). Rerun retries it. |
| Test injects a fake listing judge, chat switch on with no provider | Listing lookup skipped. Chat still fails startup. |
| Kev-only finding, no inspection done | Unlabeled. Counted, never scored as "not real". |
| Hand label and admin label differ | Hand label wins |
typesafe_calls row older than 90 days | Already purged. Not replayable. |
input row fails to decode | Recorded as an error line. The run continues. |
| Jev version changed inside the window | All versions replayed by default. Report counts per version. --jev-model picks one. |
--jev-model does not start with jev- | Tool exits before any query |
Same ref_id logged several times | Latest 200-row wins. Tie on created_at broken by id DESC. |
Row with no ref_id | Not replayed. Counted in the report. |
| Choice option order | Go marshals map keys in sorted order, so both providers see the same order |
| Duplicate switch on, sweep has a nil judge | Cannot happen: switch on without a ready provider is a startup error. Switch off: nil judge, sweep returns at once. |
Duplicate judge on Kev with no kev entry | Startup error, like the other judges |
| Duplicate replay, subject re-checked after an address change | Latest call per subject wins, with that call's candidates |
Duplicate eval with --base-url and no --threshold | Tool exits |
Tests
Rental Go tests run with -p 1.
| Area | Test |
|---|---|
| Client | WithModel sends that model in the body. The default is jev-latest. An empty key sends no Authorization header. |
| Registry | A kev threshold entry with KevModel ending in -latest fails the test (P13). jev is registered only with a key. kev only with a URL. Missing specs are absent from the map. |
| Config | Provider env defaults to jev. IsRentalEnabled() / IsRentalChatEnabled() follow the switches only, with or without a key. |
| Judges | Threshold lookup by provider. A missing entry fails the constructor. Existing judge tests pass on jev. |
| Module | Injected fake judge skips only its own feature's lookup; another switched-on feature with no provider still errors. Feature on with an unregistered or untested provider → error. Switch on with jev and an empty TYPESAFE_API_KEY → startup error naming the feature (missing-key test). Feature off → no error, no judge. |
| Listing on-state | Nil judge: save queues nothing, form check returns checked: false, sweep returns without claiming rows. No test uses SetRentalJevEnabledFunc (deleted). |
| Duplicate on-state | Switch off → no judge built. Nil judge → sweep makes no query. Sweep reads no config. No test uses the sweep's SetEnabledFunc (deleted). Sweep hits follow Threshold(): a fake judge at 0.8 makes a 0.75 candidate a miss. |
| Destination guard | Flag check rejects http://10.0.0.5, https://api.typesafe.ai and accepts 127.0.0.1, [::1], localhost. Dial check fails a connect to a non-loopback IP even when the URL host was allowed (fake resolver). A 307 from an httptest server is not followed and the redirect target gets no request. HTTPS_PROXY is not used. |
| Replay | Per-call timeout: a handler that sleeps past --timeout is recorded as an error, and the next run retries it. Decode → re-send round trip against an httptest fake. Resume skips done ids. The client has no log repo. Report sweep math on a fixed answers file: unit finding follows each judge's rule (chat asked-Noul only; listing desc_quality never yields a finding); labels are never split per question. A bad input row is recorded, not fatal. Selection keeps jev-1.13.0 rows and drops kev-* rows. --jev-model narrows to one version and rejects a non-jev- value. Report counts per version. Dedupe keeps one latest row per ref_id, with the id tie-break; a null ref_id is counted, not replayed; --limit counts units, not rows. Rerun uses run.json window, rejects a conflicting flag, fetches only unanswered units, counts a purged unit as retention loss. run.json holds no API key. Report keeps one line per ref_id. Duplicate: unit finding = any candidate >= threshold, agreement pooled, no label query, report says unlabeled. Hand label beats admin label; a missing label is never scored as negative; Parts A and B are separate; --report-only makes no Kev call; no output file holds message text. |
| Eval flags | Real source (db or a file) with a non-loopback --base-url exits; synthetic with the same URL runs. --base-url and --model reach the client. The key is optional when --base-url is set. --base-url without --model exits (both tools). Message eval scoring: exact set match, missed and extra codes per case, only asked questions count, --threshold required. Listing eval prints no score. Duplicate eval: default threshold is the jev entry; --threshold changes the cut; --base-url without --threshold exits. |
Rollout
-
Ship on
jevEvery feature on
jev. One behaviour change: a switch that is on (TYPESAFE_RENTAL_ENABLED,TYPESAFE_RENTAL_CHAT_ENABLED,TYPESAFE_RENTAL_DUPLICATE_ENABLED) with an emptyTYPESAFE_API_KEYnow stops startup. The operator checks each environment's secrets first; this repo holds none, so it cannot be done here. -
Local Kev eval
Run Kev locally. Run the eval tools with
--base-urland--model. After Gremlin's confirmation (ADR-0068), run the replay against production with a loopback--base-url. -
Pin + threshold
If Kev wins for a feature: one PR pins
KevModelto an immutable id, adds the threshold entry, and puts the replay numbers (including Jev versions seen) in the PR body (P13). A later weights or id change repeats this step. -
Hosting ADR
Hosting Kev for production is a new third party and needs its own ADR. It also says how the pinned model artifact stays immutable.
-
Switch
Set
KEV_BASE_URLandTYPESAFE_<FEATURE>_PROVIDER=kev, then redeploy.
Privacy — ADR-0068
Each typesafe_calls row stores the full input: listing addresses and descriptions, chat text with up to 5 earlier messages,
and duplicate-check property addresses (a subject and up to 10 candidates from other landlords). ADR-0068 allows the replay tool to read these from production and send them only to a Kev on the
developer's own machine.
- Loopback enforced with no override; the same rule for all three eval tools with a non-synthetic source and a custom
--base-url. - Output files hold call ids, ref ids, probabilities and run settings only. No state text and no API key.
- SELECT-only; nothing is written to production. The developer's DB login can write; a read-only role was declined for simplicity.
- Hand labels: read one message by id with existing DB access; the local file holds ids and a label only.
- Gated on the same Gremlin confirmation as ADR-0066 and ADR-0067 calibration. Gremlin is the accountable owner (as in ADR-0064).
- Only the last 90 days of calls can be replayed. Hosting Kev for production needs its own ADR.
Out of scope
- Admin page for switching providers (later, if switches become frequent).
- Live shadow calls.
- Storing replay results in production.
- A second adapter, and the logging wrapper that goes with it.
- Hosting Kev for production.
- Kev threshold numbers. They come from the first replay.
- Per-question thresholds. A later PR may add them with replay evidence.
- Fallback between providers.