Two correctness bugs in multi-process cluster mode (m8p10), both found live on a
real 3-pod k3s cluster while validating the deployment, both invisible to the
existing in-process / unauthenticated test suites:
1. Forwarded item/embedding writes dropped on the floor. create_item /
write_embedding gated the leader's peer broadcast on `if internal { return }`.
A write to a NON-leader gateway is forwarded to the leader with the internal
marker set (loop-prevention), so it hit that branch and terminated WITHOUT
broadcasting — the item landed only on the leader. Signals were unaffected
(the WAL relay ships regardless of the marker), which masked it. Fix: gate on
leadership, not the marker — a follower applying a marked broadcast/heal
terminates; the leader (external OR forwarded) always fans out. Forwarded
writes now return the {replicated_to,failed} report instead of a bodyless null.
2. Heal item/embedding backfill 401'd whenever TIDAL_API_KEY is set.
post_marked_blocking sent the internal marker but no Authorization header. The
marker is a trust signal, not an auth bypass (the bearer middleware runs
first), so every backfill POST was rejected 401 — a region that missed an item
while down stayed permanently inconsistent at lag 0. Unauthenticated tests
never caught it. Fix: thread TIDAL_API_KEY into RegionClusterState and attach
it (same key on every region) to the backfill POSTs.
Verified live: forwarded writes via follower gateways converge to all 3 regions;
a region scaled to 0 during an item write backfills on heal (item_failures=0).
Follow-up: add multiproc regression tests with auth for both paths.
Splits monolithic cluster.rs into tidal-server/src/cluster/ modules. Adds redeliver-missed
relay, bounded HLC drift, lag tracking, and reconcile idempotence. Five new tier-3 test suites
(chaos, lifecycle, multiproc, region, routes, runbook) all green. Docs, CHANGELOG, and ROADMAP
updated with G4/G5/G6 known gaps.
- Eliminate the tidal/ self-contained doc mirror; docs now have two canonical
homes (root *.md and docs/), with planning/specs/research/reviews moved up
- Remove stale .agents/skills and .ai mirrors; canonicalize skills under .claude/
- Add pre-commit hook + scripts/check-docs.sh doc-guard + scripts/install-hooks.sh
- Implement M0-M10 seven-dimension review findings across engine, net, server,
and tidalctl (durability, replication, query, WAL, storage, CLI hardening)
Add support for defining ranking profiles in schema YAML, allowing
deployments to override builtin profiles with deployment-specific
signal names and tuning parameters.
- Add override_register() to ProfileRegistry for clean builtin replacement
- Add with_profiles() to TidalDbBuilder to thread schema profiles
- Parse profiles section in config.rs with full sort/strategy/agg support
- Validate that profile signal references exist in schema at startup
- Change load_schema() to return (Schema, Vec<RankingProfile>)
This closes the gap where the builtin for_you profile referenced
signals (view, like, share) that don't exist in deployment schemas,
causing all feed scores to normalize to 1.0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clap now reads PORT from the environment, accepting either a bare port
number (e.g. 8080 -> 0.0.0.0:8080) or a full host:port. CLI --listen
flag still takes precedence. Deploy Dockerfile defaults PORT=9500 and
removes the hardcoded --listen argument.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add /health/startup and /health/live probes, flip readiness to 503 on
SIGTERM so k8s stops routing before drain. Update standalone Dockerfile
for internal deployment: port 9500, schema mounted at runtime (not baked
in), persistent data dir, non-root user with fixed UID 10001.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix 9 compilation errors across tidal-server and testing/cluster.rs so
that `cargo run -p tidal-server -- standalone` works end-to-end.
Bugs fixed:
- cluster.rs: wrong return types `RetrieveResult`→`Results` and
`SearchResult`→`SearchResults` on retrieve/search helpers
- state.rs: `RegionId` imported from private path; now uses
`tidaldb::replication::RegionId`
- state.rs: missing `Ok()` wrapper on `ServerState::cluster()` return
- state.rs: cluster match arms returned `TidalError` where `ServerError`
required; added `.map_err(ServerError::from)` on write_item,
write_embedding, retrieve, search
- error.rs: `Result<T>` alias lacked default E param; callers in router
used two-arg form `Result<T, AppError>` — changed to
`Result<T, E = ServerError>`
- router.rs: `with_state()` called before cluster routes were added,
making `app` `Router<()>`; restructured to call `with_state` once at end
- router.rs: `TidalErrorWrapper(TidalError)` used to map `QueryError`;
fixed with `|e| TidalErrorWrapper(e.into())`
- router.rs: `Search::limit()` takes `u32` but code cast to `usize`
- router.rs: `bm25_score`/`semantic_score` are `f32` in SearchResultItem
but `f64` in response struct; added `.map(f64::from)` conversion
Also split cluster.rs into cluster.rs + cluster_transport.rs to stay
under the 600-line limit required by CODING_GUIDELINES §9.
Verified all README curl examples work:
POST /items, POST /embeddings, POST /signals, GET /feed, GET /search,
GET /health all return correct HTTP status codes and JSON responses.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
