3 Commits

Author	SHA1	Message	Date
jordan	2dc00538e8	discover: repin to m12-agesort-20260901 ... The discover profile is the reason the age-aware ranking work exists, but its kustomization still pinned m12-poisonfix-20260831 (sha256:93a2929d) -- the image on which score_hot reads no created_at at all and Sort::New scores by raw entity id. Deploying it against that digest would reproduce exactly the limitation its own NO sort: block documents. Necessary but not sufficient: cold-start still ties because hot's log10(max(views,1)) numerator is exactly 0.0 at 0 or 1 views, so the NO sort: decision stands and its STATUS: PENDING RE-MEASUREMENT is unchanged. No deployment performed. The StatefulSet stays replicas: 0 and targets ns thepeach-staging, which does not exist on orchard9-k3sf (zero *peach* namespaces); its base image comment says tidaldb:deploy is kind-loaded, so this is a local target, not a fleet one.	2026-08-31 23:12:08 -06:00
jordan	a588f01f63	ranking: fix two BLOCKERs in the age-aware sorts, and stop trusting created_at units ... Three parallel reviews of 6385425 found two BLOCKERs I introduced, one CRITICAL, and a CHANGELOG that named profiles that do not exist. All verified before fixing. BLOCKER 1 -- an undated row ranked #1 instead of last. `Sort::New` mapped a missing age to 0.0, but every DATED candidate scores `-age_hours`, i.e. <= 0.0. So 0.0 was the MAXIMUM of the scale, not a neutral value: a row with no readable `created_at` outranked the genuinely newest item and `normalize` reported its score as 1.0. The codebase had already ruled on this and gone the other way. `score_shortest`, `score_longest` and `score_date_saved` all use the NEG_INFINITY sentinel, and `helpers.rs` carries a regression test named for the exact anomaly -- `normalize_neg_inf_sentinel_on_negated_scale_folds_to_bottom` -- because `Shortest` hit it first: "a pre-clamped 0.0 floated above them and the last-ranked (missing) item reported the highest score 1.0". I reintroduced it on a new negated scale. Three reachable data classes, none hypothetical: a legacy row written before `created_at` was materialized (`state_rebuild.rs` documents the class), an empty map from `deserialize_metadata` on a short or corrupt row, and an entity dropped by the `.ok().flatten()` in the metadata map builder -- which means a TRANSIENT STORAGE READ ERROR could promote an item to the top of the feed. Now the two `None` causes are distinguished. Map absent entirely -> 0.0, the whole set ties, unchanged. Map present but this entity has no usable `created_at` -> NEG_INFINITY, sorts last. A future-dated timestamp still clamps to age 0, because that is a real value that legitimately means "newest"; an absent one carries no recency claim at all. Same conflation in `score_hot`, and worse there because `Sort::Hot` now reports `needs_item_metadata`, so the map is loaded on essentially every Hot query and the per-entity branch is the LIVE one. `unwrap_or(DEFAULT_HOT_AGE_HOURS)` handed a single undated row the freshest divisor in the set: against a year-old cohort at the builtin gravity 1.8 the ratio `(8762/26)^1.8` is ~3e4, so a corrupt row with two views outranked correctly-dated items with tens of thousands. Across all four Hot builtins. Now scored at Hot's floor. BLOCKER 2 -- my own load-shedding "fix" INVERTED the survivor set. `truncate_to_ newest` ranked candidates against the newest ids of the WHOLE universe, but `scan_candidates` iterates the universe bitmap ASCENDING and breaks at `(limit * multiplier).max(200)`, so `candidates` is the LOW-id prefix. On a catalog whose ids are assigned in creation order -- which is exactly what `metadata_with_created_at`'s `Timestamp::now()` default produces -- the global newest are the HIGH ids. Above roughly `max_candidates + 4*cap` items the two sets stop intersecting, every candidate tied at `usize::MAX`, the stable sort became a NO-OP, and `truncate(cap)` kept the OLDEST candidates. The `select_nth_unstable_by` I replaced kept the newest available. Strictly worse than the bug I set out to fix, and only visible when degraded. The key is now composite -- recency rank, then DESCENDING id for anything the oversample did not cover -- so where index and candidate window overlap the survivors are genuinely newest, and where they do not it degrades to the documented pre-existing approximation instead of inverting. Back to `select_nth_unstable_by_ key`: this path runs ONLY when the load shedder has already decided the node cannot afford the work, so it must stay a linear partition, and only membership of `[0, cap)` matters because Stage 3 re-orders the survivors anyway. Mutation-proven, and the numbers show why the old test was blind: rank-only key returns `EntityId(100)` where the composite key returns `EntityId(200)`, and the pre-existing 150-item test PASSES under that same mutation -- its fixture had the oversample covering the whole universe AND the newest items at the low ids, the one configuration where the defect cannot appear. New test uses a 3000-item time-ordered catalog; limit=50 does NOT exercise it (max_candidates == cap == 200 so the guard skips), limit=25 does. Three of my ordering fixtures pinned nothing. `finalize`'s tie-break 2 is ASCENDING entity id, and I had made newest-first coincide with it, so any mutation that merely TIED the set still produced the asserted vector. The gate mutation proves it: reverting `needs_metadata_for_sort` to its old four-variant form yielded `left: [1, 2, 3, 4, 5]` -- literally the old fixture's expectation. No test in the suite failed if the retrieve executor stopped consulting `Sort::needs_item_metadata`, which is the exact drift this work exists to prevent. Every ordering fixture is now non-monotonic in id, so it disagrees with BOTH descending id (the old proxy) and ascending id (the tie collapse). CHANGELOG named profiles that do not exist. `recent_uploads` -- zero hits repo-wide; I invented it. `following` is Sort::New, not Hot. `brief` is Hot, not New. `related` (Hot{1.2}) and `chronological` (New) were omitted entirely. The real blast radius is SEVEN profiles, not the four-plus-three I claimed, and five of them carry a diversity block so they already loaded item metadata and change behaviour the instant this image rolls. Replaced with a verified table. Also cited the spec conformance this brings: docs/specs/09-ranking-scoring.md:1214 already specified `Sort::New` as "created_at DESC". created_at units are now trusted with a warning instead of silently. A seconds-unit value parses as u64 and was stored verbatim; `read_age_hours` divides by nanos-per-hour, so `1700000000` reads as 56 years old. `Sort::New` then scores -496731 against boost sums in single digits -- re-entering through DATA the exact "recency annihilates every boost" defect this work removed -- and Hot buries the item by a factor of 5.1e7. Pre-change both sorts ignored the value, so it was inert; this work made it live. The repo's OWN fixtures made that mistake in three places, which is the proof it is the natural one. `metadata_with_created_at` now warns when the value is too small to be nanoseconds, logging entity id, value, age_hours and age_years so an operator can act. It does NOT rewrite the value -- guessing the unit would corrupt what the `created_at` range index already reads as nanoseconds -- and does NOT reject the write, which would break an API that currently accepts it. Threshold 6e17 ns (1989): nanosecond timestamps after 1990 exceed 6.31e17 while seconds/millis/micros for any plausible date stay under 1e16, so the ranges cannot overlap. VERIFIED on a live server: seconds, millis and micros each warn with age_years=56; a real nanosecond value is silent. The nanosecond contract is now documented on the public DTO and propagates to the OpenAPI schema, where it was invisible before. Stale docs corrected: two published blog posts and two claim-verification ledgers were certifying "entity recency (higher ID = newer)"; the e2e fixture contract justified an interleaving decision with reasoning that is now false (its conclusion still holds, for a different reason); and k8s/discover/schema.yaml's "NO sort:" rationale cited behaviour this work removed, so it is now marked PENDING RE-MEASUREMENT with the three specific measurements named rather than left reading as justified. Full lib suite 2133 passed. Clippy 66 vs 66 at baseline, zero added, zero errors. Fast integration suites all green. Real-server e2e re-verified: `new` and `hot` both return newest-first, `new`'s scores now evenly spaced across evenly spaced ages.	2026-08-31 21:31:31 -06:00
jordan	6385425a92	ranking: make Hot and New age-aware; fix the same gap in three more places ... `score_hot` hardcoded `age_hours = 24.0`, so the divisor in `log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate set and `Sort::Hot` reduced EXACTLY to `log10(max(views, 1))` -- a view-count ranking wearing a recency sort's name. Four built-in profiles use it (`hot`, `for_you`, `following`, `brief`); anyone tuning `gravity` was tuning a no-op. The in-code comment justified this by saying a per-entity `created_at` lookup needs an `EntityId -> created_at_ns` reverse map that "is not built". That was stale, and it was the load-bearing claim: `created_at` has been materialized INTO item metadata on every write since `Items::metadata_with_created_at`, the executor has held an `EntityId -> metadata` map since M6p3, and the replication record carries the materialized map so replicas cannot diverge. No index, storage change, schema change or migration -- the scorer reads the map it already had, exactly the way `read_duration` does three lines away. `Sort::New` used `entity_id as f64`. Wrong twice: it assumed IDs are assigned in creation order, and it used the ID's MAGNITUDE as the base score, so on a catalog of N items the sort contributed ~N against a boost sum in single digits. Recency did not participate in the ranking, it annihilated every boost. Now negated age in hours -- same ordering, boost-comparable scale. Three more instances of the same defect class, found by auditing rather than assuming the report was complete: 1. Both age sorts were missing from `needs_metadata_for_sort`, so a profile with no session and no diversity never loaded the map the fix depends on. 2. Every metadata sort was DEAD on the SEARCH path. Its metadata pre-load was gated on `session_context.is_some()` and never consulted `profile.sort`, AND the `ProfileExecutor` it built never had `with_item_metadata` called at all -- the map it did compute went only to the keyword-hint argument, which the sort scorers do not read. `shortest`/`longest` scored NEG_INFINITY and the alphabetical sorts the missing-title sentinel, for every candidate, silently. 3. Under `ReducedCandidates` load the candidate cap kept the highest entity IDs, correct only while `Sort::New` meant "highest ID". Left alone it would discard the genuinely newest items BEFORE scoring -- wrong only when degraded, the hardest case to notice. Now keyed off the `created_at` index via the new `RangeIndex::top_n_descending`. The decision "which sorts read item metadata" now lives on `Sort` itself as an exhaustive match. It was a `matches!` in one executor while a second executor had its own different copy, which is precisely how a metadata-reading sort came to be omitted from both. MEASURED, not inferred: - Real server, 10 items, equal views, ages 2-20 days: before every score was 0.5 (all-equal set folded to the normalizer's midpoint) and the feed returned oldest-first forever; after, 1.0 -> 0.0 strictly descending, newest first. - `new` with zero signals returns the exact REVERSE of candidate-scan order. - `alphabetical_asc`, `shortest`, `longest` verified end to end with title and duration order both opposing entity id. - Metadata point-read cost at 2,000 candidates (the ceiling: `scan_candidates` caps at `max(limit*10, 200)` and `limit > 500` is rejected): 7.25ms, 3.6us per candidate. Guarded at 250ms. THE BUG REPORT'S CENTRAL PROMISE IS FALSE and the changelog says so. §7 claimed this fix lets a zero-signal corpus rank newest-first so a consumer could delete its workaround. It arithmetically cannot: the numerator `log10(max(views,1))` is exactly 0.0 for 0 OR 1 views, so the age divisor has nothing to scale and every candidate still ties -- confirmed on the live server, all ten scores 0.5. Age-awareness begins at the second view. Fixing cold-start needs recency to be ADDITIVE rather than a pure divisor, which reorders every existing Hot consumer, so it is a separate decision. `sort_hot_zero_view_corpus_still_ties_regardless_of_ age` pins the limit so it cannot be rediscovered by accident. Three existing tests asserted the old entity-ID behaviour. Inverted to assert real recency, not loosened -- and each fixture now makes id order and creation order DISAGREE, because an ordering assertion where the two candidate orderings agree is satisfied by the defect too. Three of my own new tests were vacuous for exactly that reason and were caught by mutation-testing; one was also flaky (it passed in a 12-test run and failed run alone, because retrieval order for exactly-tied vectors is not deterministic). Every new assertion is mutation-proven against the implementation it replaces. Full lib suite 2130 passed. Clippy 66 warnings vs 66 at baseline, zero added.	2026-08-31 19:58:01 -06:00