6385425a92
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
`score_hot` hardcoded `age_hours = 24.0`, so the divisor in `log10(max(views,1)) / (age_hours + 2)^gravity` was constant across the candidate set and `Sort::Hot` reduced EXACTLY to `log10(max(views, 1))` -- a view-count ranking wearing a recency sort's name. Four built-in profiles use it (`hot`, `for_you`, `following`, `brief`); anyone tuning `gravity` was tuning a no-op. The in-code comment justified this by saying a per-entity `created_at` lookup needs an `EntityId -> created_at_ns` reverse map that "is not built". That was stale, and it was the load-bearing claim: `created_at` has been materialized INTO item metadata on every write since `Items::metadata_with_created_at`, the executor has held an `EntityId -> metadata` map since M6p3, and the replication record carries the materialized map so replicas cannot diverge. No index, storage change, schema change or migration -- the scorer reads the map it already had, exactly the way `read_duration` does three lines away. `Sort::New` used `entity_id as f64`. Wrong twice: it assumed IDs are assigned in creation order, and it used the ID's MAGNITUDE as the base score, so on a catalog of N items the sort contributed ~N against a boost sum in single digits. Recency did not participate in the ranking, it annihilated every boost. Now negated age in hours -- same ordering, boost-comparable scale. Three more instances of the same defect class, found by auditing rather than assuming the report was complete: 1. Both age sorts were missing from `needs_metadata_for_sort`, so a profile with no session and no diversity never loaded the map the fix depends on. 2. Every metadata sort was DEAD on the SEARCH path. Its metadata pre-load was gated on `session_context.is_some()` and never consulted `profile.sort`, AND the `ProfileExecutor` it built never had `with_item_metadata` called at all -- the map it did compute went only to the keyword-hint argument, which the sort scorers do not read. `shortest`/`longest` scored NEG_INFINITY and the alphabetical sorts the missing-title sentinel, for every candidate, silently. 3. Under `ReducedCandidates` load the candidate cap kept the highest entity IDs, correct only while `Sort::New` meant "highest ID". Left alone it would discard the genuinely newest items BEFORE scoring -- wrong only when degraded, the hardest case to notice. Now keyed off the `created_at` index via the new `RangeIndex::top_n_descending`. The decision "which sorts read item metadata" now lives on `Sort` itself as an exhaustive match. It was a `matches!` in one executor while a second executor had its own different copy, which is precisely how a metadata-reading sort came to be omitted from both. MEASURED, not inferred: - Real server, 10 items, equal views, ages 2-20 days: before every score was 0.5 (all-equal set folded to the normalizer's midpoint) and the feed returned oldest-first forever; after, 1.0 -> 0.0 strictly descending, newest first. - `new` with zero signals returns the exact REVERSE of candidate-scan order. - `alphabetical_asc`, `shortest`, `longest` verified end to end with title and duration order both opposing entity id. - Metadata point-read cost at 2,000 candidates (the ceiling: `scan_candidates` caps at `max(limit*10, 200)` and `limit > 500` is rejected): 7.25ms, 3.6us per candidate. Guarded at 250ms. THE BUG REPORT'S CENTRAL PROMISE IS FALSE and the changelog says so. §7 claimed this fix lets a zero-signal corpus rank newest-first so a consumer could delete its workaround. It arithmetically cannot: the numerator `log10(max(views,1))` is exactly 0.0 for 0 OR 1 views, so the age divisor has nothing to scale and every candidate still ties -- confirmed on the live server, all ten scores 0.5. Age-awareness begins at the second view. Fixing cold-start needs recency to be ADDITIVE rather than a pure divisor, which reorders every existing Hot consumer, so it is a separate decision. `sort_hot_zero_view_corpus_still_ties_regardless_of_ age` pins the limit so it cannot be rediscovered by accident. Three existing tests asserted the old entity-ID behaviour. Inverted to assert real recency, not loosened -- and each fixture now makes id order and creation order DISAGREE, because an ordering assertion where the two candidate orderings agree is satisfied by the defect too. Three of my own new tests were vacuous for exactly that reason and were caught by mutation-testing; one was also flaky (it passed in a 12-test run and failed run alone, because retrieval order for exactly-tied vectors is not deterministic). Every new assertion is mutation-proven against the implementation it replaces. Full lib suite 2130 passed. Clippy 66 warnings vs 66 at baseline, zero added.
144 lines
5.5 KiB
YAML
144 lines
5.5 KiB
YAML
# tidalDB standalone server as a StatefulSet.
|
|
#
|
|
# WHY A STATEFULSET (not a Deployment): tidalDB is single-node-first and
|
|
# embeddable — the server wraps ONE engine instance whose state (WAL +
|
|
# checkpoints + indexes) lives on a durable data dir. It scales VERTICALLY
|
|
# (bigger node), not by adding active replicas. The parked source state keeps
|
|
# `replicas: 0`; `scripts/restore-fleet.sh --standalone` restores cardinality 1.
|
|
# For HA, use the DIFFERENT multi-process `cluster` deployment: one process per
|
|
# region, quorum-acked writes, and automatic failover. It ships as its own
|
|
# StatefulSet under k8s/cluster/ and does not change this manifest.
|
|
# See docs/runbooks/kubernetes.md and docs/runbooks/cluster.md.
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: tidaldb
|
|
namespace: tidaldb
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/component: server
|
|
spec:
|
|
serviceName: tidaldb # the headless Service in service.yaml — stable network id
|
|
replicas: 0 # parked; scripts/restore-fleet.sh is the only supported scale-up path
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: tidaldb
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
app.kubernetes.io/component: server
|
|
annotations:
|
|
# Plain-Prometheus scrape hints (the PodMonitor/ServiceMonitor in
|
|
# servicemonitor.yaml is the Operator-native alternative). Metrics are
|
|
# unauthenticated — keep :9091 cluster-internal (see ops/monitoring.md).
|
|
prometheus.io/scrape: "true"
|
|
prometheus.io/port: "9091"
|
|
prometheus.io/path: "/metrics"
|
|
spec:
|
|
# SIGTERM flips readiness to 503 (pod leaves Endpoints), drains in-flight
|
|
# requests, then checkpoints + fsyncs the WAL before exit. Give that room.
|
|
terminationGracePeriodSeconds: 60
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 10001 # the `tidal` user baked into docker/deploy/Dockerfile
|
|
runAsGroup: 10001
|
|
fsGroup: 10001 # makes the mounted PVC group-writable by the runtime user
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: tidaldb
|
|
# For a real cluster, replace with your registry image pinned by digest
|
|
# (e.g. registry.example.com/tidaldb@sha256:...) and set
|
|
# imagePullPolicy: IfNotPresent. `tidaldb:deploy` is the local image
|
|
# built from docker/deploy/Dockerfile and loaded via `kind load`.
|
|
image: tidaldb:deploy
|
|
imagePullPolicy: IfNotPresent
|
|
# ENTRYPOINT is the bare binary; these args override the image CMD so
|
|
# the schema comes from the mounted ConfigMap, not the baked default.
|
|
args:
|
|
- standalone
|
|
- --listen
|
|
- 0.0.0.0:9400
|
|
- --schema
|
|
- /etc/tidaldb/schema/schema.yaml
|
|
- --data-dir
|
|
- /data
|
|
- --metrics
|
|
- 0.0.0.0:9091
|
|
env:
|
|
- name: TIDAL_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: tidaldb-api-key
|
|
key: api-key
|
|
- name: TIDAL_SERVER_LOG
|
|
value: info
|
|
ports:
|
|
- name: http
|
|
containerPort: 9400
|
|
- name: metrics
|
|
containerPort: 9091
|
|
# Three distinct probes map to the three health endpoints:
|
|
# - /health/startup : always 200 once the HTTP listener is up
|
|
# - /health/live : always 200 while the process is alive
|
|
# - /health : 200 ready / 503 while draining on SIGTERM
|
|
# All are unauthenticated by design, so the probes need no token.
|
|
startupProbe:
|
|
httpGet:
|
|
path: /health/startup
|
|
port: http
|
|
periodSeconds: 5
|
|
failureThreshold: 60 # up to ~5 min for large-DB WAL replay / index load (capacity-planning.md)
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /health/live
|
|
port: http
|
|
periodSeconds: 10
|
|
timeoutSeconds: 3
|
|
failureThreshold: 3
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /health # 503 during drain -> removed from Service Endpoints
|
|
port: http
|
|
periodSeconds: 10
|
|
timeoutSeconds: 3
|
|
failureThreshold: 3
|
|
resources:
|
|
requests:
|
|
cpu: "250m"
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "2"
|
|
memory: 2Gi # size from docs/ops/capacity-planning.md for your item/embedding count
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true # the server only writes /data (PVC) and /tmp (emptyDir)
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /data
|
|
- name: schema
|
|
mountPath: /etc/tidaldb/schema
|
|
readOnly: true
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
volumes:
|
|
- name: schema
|
|
configMap:
|
|
name: tidaldb-schema
|
|
- name: tmp
|
|
emptyDir: {}
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: data
|
|
labels:
|
|
app.kubernetes.io/name: tidaldb
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
# storageClassName: "" # uncomment + set to pin a class; omitted = cluster default
|
|
resources:
|
|
requests:
|
|
storage: 10Gi
|