jordan/tidaldb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
142 Commits 1 Branch 0 Tags 14 MiB
936da3c520
Commit Graph

5 Commits

Author SHA1 Message Date
jordan
a588f01f63 ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details 
Three parallel reviews of 6385425 found two BLOCKERs I introduced, one CRITICAL,
and a CHANGELOG that named profiles that do not exist. All verified before fixing.

BLOCKER 1 -- an undated row ranked #1 instead of last. `Sort::New` mapped a
missing age to 0.0, but every DATED candidate scores `-age_hours`, i.e. <= 0.0.
So 0.0 was the MAXIMUM of the scale, not a neutral value: a row with no readable
`created_at` outranked the genuinely newest item and `normalize` reported its
score as 1.0.

The codebase had already ruled on this and gone the other way. `score_shortest`,
`score_longest` and `score_date_saved` all use the NEG_INFINITY sentinel, and
`helpers.rs` carries a regression test named for the exact anomaly --
`normalize_neg_inf_sentinel_on_negated_scale_folds_to_bottom` -- because
`Shortest` hit it first: "a pre-clamped 0.0 floated above them and the
last-ranked (missing) item reported the highest score 1.0". I reintroduced it on
a new negated scale.

Three reachable data classes, none hypothetical: a legacy row written before
`created_at` was materialized (`state_rebuild.rs` documents the class), an empty
map from `deserialize_metadata` on a short or corrupt row, and an entity dropped
by the `.ok().flatten()` in the metadata map builder -- which means a TRANSIENT
STORAGE READ ERROR could promote an item to the top of the feed.

Now the two `None` causes are distinguished. Map absent entirely -> 0.0, the whole
set ties, unchanged. Map present but this entity has no usable `created_at` ->
NEG_INFINITY, sorts last. A future-dated timestamp still clamps to age 0, because
that is a real value that legitimately means "newest"; an absent one carries no
recency claim at all.

Same conflation in `score_hot`, and worse there because `Sort::Hot` now reports
`needs_item_metadata`, so the map is loaded on essentially every Hot query and the
per-entity branch is the LIVE one. `unwrap_or(DEFAULT_HOT_AGE_HOURS)` handed a
single undated row the freshest divisor in the set: against a year-old cohort at
the builtin gravity 1.8 the ratio `(8762/26)^1.8` is ~3e4, so a corrupt row with
two views outranked correctly-dated items with tens of thousands. Across all four
Hot builtins. Now scored at Hot's floor.

BLOCKER 2 -- my own load-shedding "fix" INVERTED the survivor set. `truncate_to_
newest` ranked candidates against the newest ids of the WHOLE universe, but
`scan_candidates` iterates the universe bitmap ASCENDING and breaks at
`(limit * multiplier).max(200)`, so `candidates` is the LOW-id prefix. On a
catalog whose ids are assigned in creation order -- which is exactly what
`metadata_with_created_at`'s `Timestamp::now()` default produces -- the global
newest are the HIGH ids. Above roughly `max_candidates + 4*cap` items the two sets
stop intersecting, every candidate tied at `usize::MAX`, the stable sort became a
NO-OP, and `truncate(cap)` kept the OLDEST candidates. The `select_nth_unstable_by`
I replaced kept the newest available. Strictly worse than the bug I set out to fix,
and only visible when degraded.

The key is now composite -- recency rank, then DESCENDING id for anything the
oversample did not cover -- so where index and candidate window overlap the
survivors are genuinely newest, and where they do not it degrades to the documented
pre-existing approximation instead of inverting. Back to `select_nth_unstable_by_
key`: this path runs ONLY when the load shedder has already decided the node cannot
afford the work, so it must stay a linear partition, and only membership of
`[0, cap)` matters because Stage 3 re-orders the survivors anyway.

Mutation-proven, and the numbers show why the old test was blind: rank-only key
returns `EntityId(100)` where the composite key returns `EntityId(200)`, and the
pre-existing 150-item test PASSES under that same mutation -- its fixture had the
oversample covering the whole universe AND the newest items at the low ids, the one
configuration where the defect cannot appear. New test uses a 3000-item
time-ordered catalog; limit=50 does NOT exercise it (max_candidates == cap == 200
so the guard skips), limit=25 does.

Three of my ordering fixtures pinned nothing. `finalize`'s tie-break 2 is ASCENDING
entity id, and I had made newest-first coincide with it, so any mutation that
merely TIED the set still produced the asserted vector. The gate mutation proves
it: reverting `needs_metadata_for_sort` to its old four-variant form yielded
`left: [1, 2, 3, 4, 5]` -- literally the old fixture's expectation. No test in the
suite failed if the retrieve executor stopped consulting `Sort::needs_item_metadata`,
which is the exact drift this work exists to prevent. Every ordering fixture is now
non-monotonic in id, so it disagrees with BOTH descending id (the old proxy) and
ascending id (the tie collapse).

CHANGELOG named profiles that do not exist. `recent_uploads` -- zero hits
repo-wide; I invented it. `following` is Sort::New, not Hot. `brief` is Hot, not
New. `related` (Hot{1.2}) and `chronological` (New) were omitted entirely. The real
blast radius is SEVEN profiles, not the four-plus-three I claimed, and five of them
carry a diversity block so they already loaded item metadata and change behaviour
the instant this image rolls. Replaced with a verified table. Also cited the spec
conformance this brings: docs/specs/09-ranking-scoring.md:1214 already specified
`Sort::New` as "created_at DESC".

created_at units are now trusted with a warning instead of silently. A
seconds-unit value parses as u64 and was stored verbatim; `read_age_hours` divides
by nanos-per-hour, so `1700000000` reads as 56 years old. `Sort::New` then scores
-496731 against boost sums in single digits -- re-entering through DATA the exact
"recency annihilates every boost" defect this work removed -- and Hot buries the
item by a factor of 5.1e7. Pre-change both sorts ignored the value, so it was
inert; this work made it live. The repo's OWN fixtures made that mistake in three
places, which is the proof it is the natural one.

`metadata_with_created_at` now warns when the value is too small to be nanoseconds,
logging entity id, value, age_hours and age_years so an operator can act. It does
NOT rewrite the value -- guessing the unit would corrupt what the `created_at`
range index already reads as nanoseconds -- and does NOT reject the write, which
would break an API that currently accepts it. Threshold 6e17 ns (1989): nanosecond
timestamps after 1990 exceed 6.31e17 while seconds/millis/micros for any plausible
date stay under 1e16, so the ranges cannot overlap. VERIFIED on a live server:
seconds, millis and micros each warn with age_years=56; a real nanosecond value is
silent. The nanosecond contract is now documented on the public DTO and propagates
to the OpenAPI schema, where it was invisible before.

Stale docs corrected: two published blog posts and two claim-verification ledgers
were certifying "entity recency (higher ID = newer)"; the e2e fixture contract
justified an interleaving decision with reasoning that is now false (its conclusion
still holds, for a different reason); and k8s/discover/schema.yaml's "NO sort:"
rationale cited behaviour this work removed, so it is now marked PENDING
RE-MEASUREMENT with the three specific measurements named rather than left reading
as justified.

Full lib suite 2133 passed. Clippy 66 vs 66 at baseline, zero added, zero errors.
Fast integration suites all green. Real-server e2e re-verified: `new` and `hot`
both return newest-first, `new`'s scores now evenly spaced across evenly spaced
ages.
 2026-08-31 21:31:31 -06:00
jordan
4766f566de feat(observability): HTTP metrics, structured logs, dashboard, live tidalctl 
There was no metric anywhere that could answer "how much traffic are we serving"
or "what is our error rate". The engine published a rich DOMAIN surface (search
latency, WAL fsync, quorum timeouts, replication lag) and nothing about HTTP, so
a cluster could serve 401s or 503s indefinitely with every existing gauge looking
healthy. Logs were collected but unusable. There was no way to ask a RUNNING node
anything.

1. HTTP metrics. tidaldb_http_requests_total{route,method,status} plus a
   per-route duration histogram, recorded by one layer placed OUTSIDE the auth,
   timeout and rate-limit layers so it sees the status actually returned to the
   client. Cardinality is the whole design: the route label is axum's MatchedPath
   TEMPLATE, not the path, and unmatched requests collapse into one <unmatched>
   bucket so a 404 flood cannot mint series. A hard cap folds anything past it
   into an overflow bucket while established series keep counting.

   The engine owns the /metrics listener but must not learn what a route or a
   status code is, so it gained one registration hook
   (MetricsState::set_extra_renderer) and tidal-server publishes through it. One
   scrape target per node, not two.

2. Structured logs. The previous init was a bare tracing_subscriber::fmt(), which
   produced two real defects: ANSI escapes leaked into collected logs, and every
   line failed the collector's JSON parse and was stamped level=info — so
   `level:error` matched NOTHING and errors were invisible to the log platform
   while being collected. JSON_LOGS=1 emits the collector's exact wire format
   (ts/level/service/env/msg), span fields are lifted so request_id lands on every
   line of a request, and ANSI is off unconditionally in both formats.

   Verified against the running binary, which caught a defect no unit test would
   have: dependencies logging through the `log` crate arrived with target="log"
   and four log.* metadata fields (absolute cargo registry paths, indexed
   forever). The real module is now lifted into target and the bridge metadata
   pruned.

3. Dashboard. docs/ops/grafana-tidaldb.json, 13 panels, mirrored into the fleet
   as a grafana-database-dashboards key. Every metric name was checked against a
   live endpoint and all 26 PromQL expressions were executed against the live
   TSDB before commit, because a dashboard full of "No data" is worse than none.
   Confirmed loaded in Grafana (uid tidaldb-overview, Databases folder).

4. tidalctl live mode. Every other subcommand reads a data dir AT REST, some
   requiring a stopped node. `search`, `feed`, `cluster-status` and `watch` take
   --url and talk to a running server, with --ca/--insecure because a cluster's
   client port is served with the INTERNAL cluster CA. Exit codes follow the crate
   contract, so `tidalctl cluster-status && deploy` gates on convergence.

   Its first real run immediately found a reporting defect: the aggregated
   /cluster/status reported two HEALTHY peers as UNREACHABLE PARTITIONED at 13.3M
   lag, having derived lag against an uninitialised applied=0, while every node's
   own status reported lag=0, reseed=false and identical frontiers, with
   pod-to-pod connectivity open and nothing logged. cluster-status now names that
   signature "NO REPORT (aggregated view; query the node directly)" instead of
   repeating it as replication lag; a genuine non-zero-applied lag still reports
   BEHIND. The underlying gap is documented as open work in
   docs/ops/observability.md.

Verified: 2101 + 175 engine/server unit tests, 8 standalone integration (3 new,
including the cardinality proof and the cross-crate metrics seam), 23 tidalctl
(10 new), reseed + catchup + admin-gate e2e green, clippy clean, and both the
metrics and the log format exercised against a real running binary.
 2026-08-23 10:31:57 -06:00
jx12n
6651c14adc feat(m11): cluster security (m11p7) + perf instrumentation floor 
m11p7 — secure the cluster, all opt-in (pre-m11p7 byte-for-byte):
- gRPC replication mTLS by default via a custom tokio-rustls acceptor +
  DynamicCertResolver; zero-drop content-hash cert rotation (k8s ..data swap,
  no pod restart, no inotify)
- inter-node HTTP TLS sharing the same resolver (one rotation, both planes) +
  per-node keyed-BLAKE3 signed x-tidal-node-token; marker-without-token -> 403
- admin audit log (operator-leg only) + per-principal rate limit (engine
  RateLimiter; sibling nodes exempt)
- k8s cert-manager manifest (certs.yaml) + scripts/gen-cluster-certs.sh fallback;
  secret.example.yaml gains TIDAL_CLUSTER_KEY (file-mounted, hot-rotatable)
- exit gate verified real: mtls.rs (gRPC foreign-pod), cluster_security.rs
  (HTTP foreign + zero-drop rotation under load), 7 security unit tests

perf — instrument floor (sweep Wave 1):
- new tidal/benches/wal.rs + tidal-server/benches/scatter.rs
- p99->mean honesty relabel; sweep manifest at docs/reviews/perf-sweep-2026-06-13.md
- add @tidal-performance agent (Martin Thompson)

new: cluster/{audit,http_tls,security}.rs, tests/cluster_security.rs,
docs/planning/milestone-11/phase-7.md
 2026-06-13 01:25:35 -06:00
jx12n
1092d34c39 feat: kubernetes deployment, OpenAPI spec, guides, and docker consolidation 
- Add k8s/ manifests (StatefulSet, kustomize, PDB, ServiceMonitor) + docs/runbooks/kubernetes.md
- Add tidal-server/src/openapi.rs (utoipa OpenAPI spec) and wire into router
- Add docs/guides/ (build-a-feed-app, embeddings, server-deployment) + foryou_feed example
- Consolidate tidal/docker/ into root docker/ (single canonical home)
- Update API.md, QUICKSTART.md, README.md, CLAUDE.md, check-docs.sh accordingly
 2026-06-09 17:06:34 -06:00
jx12n
b55ad70141 fix: M0-M10 third-pass remediation — durability, replication, and CLI hardening 
Resolves the 142 findings from tidal/docs/reviews/CODE_REVIEW_m0-m10.md across
the engine, server, net, and CLI surfaces:

- WAL/session-journal durability, checkpoint format, and crash-recovery hardening
- Replication shipper/receiver, tenant isolation, and migration paths
- Cluster scatter-gather, router, standalone server + health/offload endpoints
- tidalctl refactored into command modules with JSON output and WAL-state tooling
- Cohort, governance, signal-ledger, and vector-registry correctness fixes
- Expanded UAT/integration/durability test coverage across all milestones
 2026-06-08 10:28:34 -06:00