July 22, 2026 中文版

DW Semantic History Search — Design

Adds local, pgvector-based semantic search over Digital Worker chat history. The current Postgres FTS ('simple' config) cannot segment Chinese or code-switched rojak, leaving 40% Malay + 10-20% Chinese chat effectively unsearchable; local bge-m3 embeddings fix this with no API cost and enable cross-lingual retrieval. Approved after the 2026-07-22 grill session resolved all open questions.

ADRs 0028 · 0034
New ADR 0050 (accepted)
Status Approved

View source markdown ↗ generated by claude-opus-4-8 · diagrams mermaid

Digital Worker chat-history search runs on Postgres full-text search that cannot read Chinese or code-switched "rojak" — so the 40% Malay and 10–20% Chinese portions of the chat are effectively unsearchable today. This design adds local, pgvector-based semantic search using bge-m3 embeddings, produced by a single periodic watermark sweep and merged into the existing search path. Approved after the 2026-07-22 grill session resolved every open question.

Module backend/go/internal/modules/digitalworker
Model bge-m3 (1024-dim), local via Ollama
Storage halfvec(1024), exact KNN, no ANN index
Write path One periodic watermark sweep over closed windows

Problem

DW chat history search today uses Postgres full-text search (HistorySearchRepository.Search, learning_repositories.go:533):

search_tsv @@ plainto_tsquery('simple', ?)

Semantic search is the only approach that works for zh/ms/rojak recall, and it enables cross-lingual retrieval (跨语言检索): a query in one language finds relevant messages written in another.

Goals & Non-Goals

Goals

in scope

Semantic search for English, Malay, Chinese, and code-switched messages.
Local, free, private — no chat content sent to third-party APIs.
Feature-flagged so it can be turned off entirely.
Embedding model configurable, swappable later without code changes.

Non-Goals (deferred)

out of scope

Semantic memory injection — stays ADR-0034 recency-ranked until the over-budget signal.
Reranker — phase 2.
Cross-domain search / platform embedding standard — per-domain models instead.
Paid models (OpenAI/Voyage) — local is enough; privacy + zero cost.
Inline embedding on ingest — the sweep handles it.
MRL / truncation and an ANN index — corpus too small.

Key Decisions

#	Decision	Rationale
1	Per-domain models; no cross-domain search	Keeps DW isolated from `ai`/`crm`
2	bge-m3 (1024-dim), local via Ollama	Handles rojak natively, strong on Malay (low-resource)
3	Embed raw — no LLM/translation normalization	One deterministic step; normalization adds cost + non-determinism
4	No MRL / no truncation — full 1024 dims	Small corpus; MRL solves a scale problem we don't have
5	`halfvec(1024)` + `halfvec_cosine_ops`	Half the storage of `vector`; cosine is standard
6	Own table `dw_message_embeddings`	Different model/dim from the `ai` module's `vector(1536)`
7	Store `model` column; always filter by it	Two models never share a vector space, even at equal dims
8	One periodic watermark sweep (no enqueue, no separate backfill)	A window can only embed after it closes; one path gives async, retry, backfill, model-swap
9	Substance filter on windows, not messages	A trivial "ok" inside a window is often the answer; skip only tiny windows
10	Chunk as small conversation windows	Biggest accuracy lever; chat meaning spans several short messages
11	Feature flag master switch	Ship dark, roll out per environment
12	Embedding model configurable, default bge-m3	Swap models later (a re-embed job) without code changes

How are embeddings produced?

One periodic watermark sweep over closed conversation windows.

A window can only be embedded after its session closes, so per-message jobs are the wrong shape. A single sweep, keyed per (org, provider, channel, model) watermark, gives async embedding, retry (watermark stalls on transient failure), backfill (empty watermark), and model-swap rebuild in one code path. Ingest is untouched.

Rejected: enqueue-on-ingest + worker + separate backfill — three moving parts for a job that can't run until a session closes anyway.

Which vector index?

None — btree on (org_id, model) + exact KNN.

Retention coupling caps the table at ~30 days of windows (hundreds to low-thousands of vectors per org), where exact scan is single-digit ms with perfect recall.

Rejected: HNSW/IVFFlat — filtered ANN post-filters on org_id/model (can under-return or miss hits) and IVFFlat needs training data. Revisit HNSW only near ~50–100k live vectors per org.

Architecture

Ingest untouched; the sweep embeds closed windows (top); hybrid search merges FTS + KNN and degrades to FTS on failure (bottom).

Regenerable cache — but only within retention: dw_chat_events.message_text is the source of truth; every vector is derived and rebuildable. The retention purge (PurgeEvents, event_repository.go:160) also deletes dw_message_embeddings rows at the same cutoff — embedded chunk text must not outlive the org's retention promise. Semantic depth therefore equals FTS depth; a model swap re-embeds live rows only.

One sweep, four behaviors: async (ingest untouched), retry (transient failure → watermark doesn't advance), backfill (an empty watermark is the backfill), model swap (watermarks are per model; a new model starts empty and rebuilds).

Data Model

dw_message_embeddings
  id            uuid pk
  org_id        varchar   -- tenancy scope; every query filters on it
  model         varchar   -- e.g. 'bge-m3'; every query filters on it (#7)
  embedding     halfvec(1024)          -- (#5)
  content       text      -- the exact chunk text that was embedded
  source_message_ids  ...  -- which dw_chat_events rows this chunk covers (#10)
  channel / provider / time-range metadata  -- to map a hit back to messages
  created_at    timestamptz

No ANN index in phase 1. Plain btree on (org_id, model) + exact KNN (ORDER BY embedding <=> ? LIMIT n). Retention caps the corpus at hundreds–low-thousands of vectors/org, where exact scan has perfect recall. Sweep state (per (org, provider, channel, model) watermark + poison-window records) is persisted; exact table shape is a plan-level detail.

Substance Filter — window-level

The substance filter applies to the completed window, never to individual messages. Every message inside a qualifying window is retained — a trivial "ok" or "BetterAuth" is often the answer to the preceding question; dropping it per-message would delete meaning from the chunk.

Chunking — time-gap sessions

Embed small conversation windows, not single messages. Chat meaning spreads across short fragments ("migrate 那个" / "Clerk 还是 BetterAuth?" / "BetterAuth"), so per-message embedding fragments the meaning and misses it.

Scope: strictly (org_id, provider, external_channel_id) — never crosses channels/providers.
Eligibility: the existing WindowAfter predicate (event_type='message', decision IN ('received','processed')) — denied/verification traffic never embedded.
Session split: a new window starts after a 30-minute gap.
Cap: max 15 messages per window + the model input limit; a capped session continues in the next window.
No overlap in phase 1; no threads (ExternalThreadID is dropped by chatEventRow — a schema change, not phase 1).
The 30-min gap and 15-message cap are named, tested constants.

Configuration & Feature Flag

Master switch (#11): Two-tier flag digital_worker_semantic_search (admin_feature_flags seeded OFF + org_feature_flags), checked in addition to the module digital_worker flag. Off → the sweep is a no-op and search falls back to FTS; ingest unchanged either way.
Configurable model (#12): Deployment-level env config DigitalWorker.EmbedModel (default bge-m3), reusing OllamaURL. Not per-org — the model must match what the Ollama host serves, and model identity defines the vector space. Changing it starts the new model's watermark empty; search filters by the active model so old/new vectors never mix.
Heuristics: The 20-char substance threshold, 30-min gap, and 15-message cap are named, tested Go constants — nothing new in org_module_configs.

Phasing

Phase 1 — this design
Watermark embedding sweep over closed windows, dw_message_embeddings table, hybrid search inside the existing SearchAgentHistory path (no new endpoint), feature flag, configurable model.
Phase 2 — local reranker
bge-reranker-v2-m3: retrieve top ~50 by vector search → rerank → return top ~5–10. Add when precision complaints appear. Runs as a separate local inference service (TEI/Infinity), not Ollama.

Testing Approach — two tiers

Tier 1 — fake embedder (CI-blocking)

Deterministic vectors keyed by input text, against the test Postgres (verify pgvector in test-DB migrations). Covers: windowing (gap split, 15-msg cap, continuation), window substance filter, watermark (advance, transient stall, poison skip), hybrid merge/dedup/limit, model-filter, query-time degradation to FTS, flag-off no-op, retention purge.

Tier 2 — real-model smoke (env-gated, not CI-blocking)

t.Skip unless the Ollama env var is set. Seed rojak/zh/ms/en messages; assert an English query retrieves the semantically-matching non-English message (cross-lingual recall). Validates the model, not the code — a required rollout-checklist item before enabling an environment's flag.

Resolved Questions

All five original open questions were closed in the 2026-07-22 grill session:

Question	Resolution
Substance threshold	Window-level length heuristic, ~20 chars of meaningful text; no triage reuse, no AI check.
Windowing rule	Time-gap sessions (30 min) capped at 15 messages, per (org, provider, channel); no overlap; no threads.
Index type	Neither — exact KNN + btree on `(org_id, model)`; HNSW only near ~50–100k live vectors.
Config surface	Two-tier `digital_worker_semantic_search` flag; `EmbedModel` env default `bge-m3`; heuristics as Go constants.
Backfill scope	Moot — retention coupling means backfill = all live rows; nothing older exists.