# Research: m1p1 Core Type System and Schema Foundation

## Question

What are the correct Rust implementation patterns for TidalDB's foundational types -- EntityId, SignalType, DecayRate, Window, Timestamp, LumenError, and the schema builder/validator -- such that they are zero-cost, serde-friendly, cache-line-aware, and forward-compatible with the atomic operations required in m1p4?

## TidalDB Context

m1p1 delivers the type system that every subsequent subsystem depends on. Schema is the root of the module dependency chain (CODING_GUIDELINES.md Section 9): storage, signals, query, and ranking all import from schema. Mistakes here propagate everywhere. The types must satisfy:

- **Hot-path performance**: EntityId, DecayRate, and Timestamp are accessed on every candidate scoring pass (~200 candidates, <5 microseconds total budget). Copy semantics, no heap allocation.
- **Atomic compatibility**: DecayRate scores stored as f64 will need atomic CAS operations in m1p4 for lock-free signal updates. The type design now must not preclude this.
- **Serde at boundaries**: API responses include signal snapshots and entity IDs. Serialization must work at API boundaries but never on the hot path.
- **Correctness under decay math**: f64 precision for exponential decay over long idle periods (days/weeks) must not produce ranking artifacts. The signal ledger research (lumens_signal_ledger.md) confirmed f64 is adequate through year 18,000 for 1-hour half-lives.

---

## 1. Newtype Pattern for EntityId

### Question
What is the best practice for a `struct EntityId(u64)` newtype that needs Display, Hash, Eq, Ord, Copy, serde support, and zero-cost conversion to/from u64?

### Approaches Surveyed

#### Approach A: Hand-implement all traits

```rust
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct EntityId(u64);

impl std::fmt::Display for EntityId {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "{}", self.0)
    }
}

impl From<u64> for EntityId {
    fn from(v: u64) -> Self { Self(v) }
}

impl From<EntityId> for u64 {
    fn from(id: EntityId) -> Self { id.0 }
}
```

**Used by:** sled's `IVec` (hand-implements Deref, PartialEq, Ord), fjall's `SeqNo` (type alias rather than newtype), DuckDB-rs bindings.

**Strengths:** Zero dependencies. Full control. No proc-macro compile time. The CODING_GUIDELINES.md explicitly warns: "Do not add dependencies for things the standard library or a 50-line util handles."

**Weaknesses:** Boilerplate for Display and From impls. For a single newtype (EntityId), this is ~25 lines. If we add UserId, CreatorId, SignalId as separate newtypes, the boilerplate multiplies.

#### Approach B: derive_more crate (v2.1.1)

```rust
use derive_more::{Display, From, Into};

#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Display, From, Into)]
pub struct EntityId(u64);
```

**Crate health:** derive_more v2.1.1 (released Feb 2025). 100M+ downloads on crates.io. Maintained by JelteF. MSRV 1.81. No unsafe code (it is a proc-macro crate generating safe Rust). Supports individual feature flags per derive, so enabling only `display`, `from`, `into` avoids pulling in the full syn `extra-traits` feature, reducing compile overhead.

**Used by:** Widely adopted across the Rust ecosystem. Not typically used by embedded database engines (sled, fjall, redb all hand-implement or use type aliases).

**Strengths:** Reduces boilerplate if TidalDB has 3+ newtype IDs. Feature-gated derives keep compile time bounded. Display, From, Into, Deref, DerefMut all available.

**Weaknesses:** Adds a proc-macro dependency. The CODING_GUIDELINES.md Section 10 says: "Do not add dependencies for things the standard library or a 50-line util handles: builder pattern macros, derive-everything crates." This is a direct citation against derive_more.

#### Approach C: nutype crate (v0.5+)

```rust
use nutype::nutype;

#[nutype(derive(Debug, Display, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, From, Into, Serialize, Deserialize))]
pub struct EntityId(u64);
```

**Crate health:** nutype v0.5.x. ~500K downloads. Actively maintained by greyblake. Supports validation constraints (min, max, finite for floats), which could be useful for DecayRate. MSRV not documented.

**Strengths:** Built-in validation for constrained newtypes. Would let DecayRate enforce `lambda > 0.0` at construction.

**Weaknesses:** Less mature than derive_more. Heavier proc-macro. Overkill for EntityId which has no constraints. The validation is useful for exactly one type (DecayRate), not enough to justify the dependency.

### Comparison

| Criterion | Hand-implement | derive_more | nutype |
|-----------|---------------|-------------|--------|
| Lines of code per newtype | ~25 | ~3 | ~3 |
| Dependencies added | 0 | 1 proc-macro | 1 proc-macro |
| Compile time impact | None | Low (feature-gated) | Moderate |
| Aligns with CODING_GUIDELINES | Yes (Section 10) | No (explicitly discouraged) | No |
| Unsafe code | None | None (proc-macro) | None (proc-macro) |
| Production database precedent | sled, fjall, redb | General Rust ecosystem | None found |

### Recommendation

**Hand-implement.** The CODING_GUIDELINES.md Section 10 explicitly discourages "derive-everything crates." TidalDB needs exactly one newtype in m1p1 (EntityId). Even if UserId and CreatorId become separate newtypes later, the total boilerplate is ~75 lines -- well under the "could we write this in 200 lines?" threshold.

The implementation for EntityId is 25 lines:

```rust
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(transparent)]
pub struct EntityId(u64);

impl EntityId {
    #[inline]
    pub const fn new(id: u64) -> Self { Self(id) }

    #[inline]
    pub const fn as_u64(self) -> u64 { self.0 }
}

impl std::fmt::Display for EntityId {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "{}", self.0)
    }
}

impl From<u64> for EntityId {
    fn from(v: u64) -> Self { Self(v) }
}

impl From<EntityId> for u64 {
    fn from(id: EntityId) -> Self { id.0 }
}
```

Note `#[repr(transparent)]` -- this guarantees the newtype has identical layout to u64, enabling zero-cost transmutation and ensuring it fits in a register. This is the pattern sled and fjall use for their semantic wrappers.

Add serde support behind a feature gate:

```rust
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
#[cfg_attr(feature = "serde", serde(transparent))]
```

The `serde(transparent)` attribute serializes EntityId as a bare u64, not as `{"0": 123}`.

---

## 2. Duration Handling for Half-Life Declarations

### Question
The schema defines half-life durations (7d, 14d, 1d, 48h). Should we use `std::time::Duration`, `chrono::Duration`, `time::Duration`, or a custom type?

### Approaches Surveyed

#### Approach A: std::time::Duration

**Representation:** u64 seconds + u32 nanoseconds. Always non-negative. Max value ~584 billion years.

**Used by:** Standard library. tokio timeouts. Most Rust crates that need duration without calendar arithmetic.

**Strengths:** Zero dependencies. Universally understood. `Duration::from_secs(7 * 24 * 3600)` for 7 days. Nanosecond precision for sub-second half-lives if ever needed. Non-negative by construction -- half-lives cannot be negative.

**Weaknesses:** No convenience constructors for days/hours in stable Rust (though `from_secs()` with multiplication is trivial). Converting to fractional seconds for the decay formula requires `duration.as_secs_f64()`, which is stable and precise.

#### Approach B: chrono::Duration (now TimeDelta)

**Representation:** i64 milliseconds internally (as of chrono 0.4.30+, this changed to their own definition superseding the old `time::Duration`-based one). Allows negative durations.

**Used by:** chrono-dependent codebases. Web frameworks (actix-web, axum with chrono feature).

**Strengths:** `TimeDelta::days(7)` convenience constructor. Calendar-aware operations. chrono is an approved dependency in CODING_GUIDELINES.md Section 10.

**Weaknesses:** Millisecond internal precision -- loses nanosecond precision. Allows negative values, which are meaningless for half-lives. Drags in the full chrono crate (~25K lines). Overkill for what is effectively a constant multiplied by ln(2).

#### Approach C: Custom HalfLife type wrapping f64 seconds

```rust
pub struct HalfLife {
    seconds: f64,
}

impl HalfLife {
    pub const fn days(d: u32) -> Self { Self { seconds: d as f64 * 86400.0 } }
    pub const fn hours(h: u32) -> Self { Self { seconds: h as f64 * 3600.0 } }
    pub fn lambda(&self) -> f64 { std::f64::consts::LN_2 / self.seconds }
}
```

**Strengths:** Domain-specific. Encodes the relationship between half-life and lambda directly. Cannot be negative (u32 input). Pre-computes lambda at construction time. No dependencies.

**Weaknesses:** Yet another custom type. Less discoverable than std::time::Duration.

### Comparison

| Criterion | std::time::Duration | chrono::TimeDelta | Custom HalfLife |
|-----------|--------------------|--------------------|-----------------|
| Dependencies | 0 | chrono (~25K LOC) | 0 |
| Precision | Nanosecond | Millisecond | f64 (~15 significant digits) |
| Negative prevention | By construction (unsigned) | Runtime check needed | By construction (u32 input) |
| lambda computation | `LN_2 / dur.as_secs_f64()` | `LN_2 / td.num_seconds() as f64` | `.lambda()` method |
| Ergonomics | `Duration::from_secs(7*86400)` | `TimeDelta::days(7)` | `HalfLife::days(7)` |

### Recommendation

**Use `std::time::Duration` for the public API, store lambda (f64) internally.** The half-life is a schema-time constant. Once declared, TidalDB only ever uses the derived lambda value (`ln(2) / half_life_seconds`). The conversion happens once at schema definition time.

```rust
pub struct DecayConfig {
    pub half_life: std::time::Duration,
}

impl DecayConfig {
    /// Pre-compute the decay constant. Called once at schema definition time.
    pub fn lambda(&self) -> f64 {
        std::f64::consts::LN_2 / self.half_life.as_secs_f64()
    }
}
```

Convenience constructors on the schema builder side can provide `days()` and `hours()`:

```rust
impl DecayConfig {
    pub const fn days(d: u64) -> Self {
        Self { half_life: std::time::Duration::from_secs(d * 86400) }
    }
    pub const fn hours(h: u64) -> Self {
        Self { half_life: std::time::Duration::from_secs(h * 3600) }
    }
}
```

This avoids adding chrono as a dependency for schema types. chrono (or the `time` crate) should enter the dependency tree only when TidalDB needs calendar-aware timestamps for API boundaries (Phase 2+), not for internal duration arithmetic.

The internally stored `DecayRate` type should hold the pre-computed lambda:

```rust
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct DecayRate {
    lambda: f64,  // ln(2) / half_life_seconds
}

impl DecayRate {
    pub fn from_half_life(half_life: std::time::Duration) -> Self {
        let lambda = std::f64::consts::LN_2 / half_life.as_secs_f64();
        debug_assert!(lambda.is_finite() && lambda > 0.0, "half_life must be positive and finite");
        Self { lambda }
    }

    #[inline]
    pub fn lambda(self) -> f64 { self.lambda }

    /// Compute decay factor for a time delta. Used on both read and write paths.
    #[inline]
    pub fn decay_factor(self, dt_seconds: f64) -> f64 {
        (-self.lambda * dt_seconds).exp()
    }
}
```

**Precision note:** `Duration::as_secs_f64()` returns an f64 with ~15 significant digits. For 7 days (604,800 seconds), the representation is exact (it fits in 20 bits of mantissa; f64 has 52). For 30 days (2,592,000 seconds), also exact. Precision is not a concern for any realistic half-life value.

---

## 3. Error Handling: LumenError

### Question
Should TidalDB use `thiserror` for the `LumenError` enum? What about `anyhow` at boundaries?

### Approaches Surveyed

#### Approach A: thiserror for derive(Error, Display)

```rust
use thiserror::Error;

#[derive(Debug, Error)]
pub enum LumenError {
    #[error("storage failure: {0}")]
    Storage(#[from] StorageError),
    #[error("entity not found: {entity}")]
    NotFound { entity: EntityId },
    #[error("schema violation: {0}")]
    Schema(#[from] SchemaError),
    #[error("durability check failed: {0}")]
    Durability(#[from] DurabilityError),
    #[error("query error: {0}")]
    Query(#[from] QueryError),
    #[error("internal error: {0}")]
    Internal(String),
}
```

**Crate health:** thiserror v2.0.18 (Jan 2026). Maintained by dtolnay (one of the most prolific and trusted Rust maintainers). 400M+ downloads. Zero unsafe code. Pure proc-macro. MSRV varies by minor version.

**Used by:** Virtually every production Rust database and library. fjall uses thiserror for its Error enum. Tantivy uses thiserror for TantivyError. DuckDB-rs uses thiserror. tikv uses thiserror. This is the de facto standard.

**Strengths:** Eliminates ~40 lines of boilerplate per error enum (Display impl + Error impl + From impls for each variant). The `#[from]` attribute auto-generates From impls, enabling the `?` operator for error propagation. The generated code is identical to what you would hand-write -- it is not a runtime abstraction, it is pure code generation.

**Weaknesses:** Proc-macro dependency. Adds ~2-3 seconds to initial compile (subsequent incremental compiles are fast). The CODING_GUIDELINES Section 10 does not list thiserror as an approved dependency, but also does not list it as prohibited.

#### Approach B: Hand-implement Error + Display + From

```rust
#[derive(Debug)]
pub enum LumenError {
    Storage(StorageError),
    NotFound { entity: EntityId },
    Schema(SchemaError),
    Durability(DurabilityError),
    Query(QueryError),
    Internal(String),
}

impl std::fmt::Display for LumenError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            Self::Storage(e) => write!(f, "storage failure: {e}"),
            Self::NotFound { entity } => write!(f, "entity not found: {entity}"),
            Self::Schema(e) => write!(f, "schema violation: {e}"),
            Self::Durability(e) => write!(f, "durability check failed: {e}"),
            Self::Query(e) => write!(f, "query error: {e}"),
            Self::Internal(msg) => write!(f, "internal error: {msg}"),
        }
    }
}

impl std::error::Error for LumenError {
    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
        match self {
            Self::Storage(e) => Some(e),
            Self::Schema(e) => Some(e),
            Self::Durability(e) => Some(e),
            Self::Query(e) => Some(e),
            _ => None,
        }
    }
}

// Plus 4 separate From impls...
```

**Used by:** Some minimalist crates. Not common in database engines.

**Strengths:** Zero dependencies. Full control over error chain.

**Weaknesses:** ~80+ lines of boilerplate for the 6-variant enum plus 4 sub-error types. Every time a variant is added or a sub-error type changes, multiple impl blocks must be updated in lockstep. This is the exact class of boilerplate thiserror was created to eliminate.

#### Approach C: snafu crate

**Crate health:** snafu v0.8.x. ~30M downloads. Maintained by shepmaster. Takes a different philosophy: error types carry context (the "situation" that caused the error), not just the underlying cause.

**Strengths:** More structured context attachment than thiserror. Encourages unique error variants per call site.

**Weaknesses:** Heavier API surface. Less ecosystem adoption than thiserror. Unfamiliar to most Rust developers. TidalDB's error model (6 categories, not per-call-site) does not benefit from snafu's context model.

#### Approach D: anyhow at boundaries

**anyhow** is for application code where errors are reported, not inspected. It provides `anyhow::Error` as an opaque wrapper. TidalDB is a library -- callers need to match on error variants to decide whether to retry (Storage, Durability), fix input (Schema, Query), handle gracefully (NotFound), or log and degrade (Internal).

**Verdict:** anyhow is inappropriate for TidalDB's public API. It may be used internally in tests or one-off scripts.

### Comparison

| Criterion | thiserror | Hand-implement | snafu | anyhow |
|-----------|-----------|----------------|-------|--------|
| Boilerplate (6 variants, 4 From) | ~15 lines | ~80+ lines | ~20 lines | ~5 lines |
| Caller can match variants | Yes | Yes | Yes | No |
| source() chain | Auto-generated | Manual | Auto-generated | Opaque |
| Ecosystem precedent (databases) | fjall, tantivy, tikv | Rare | Rare | Application-only |
| Dependencies | 1 proc-macro | 0 | 1 proc-macro | 1 crate |
| Compile time impact | ~2-3s initial | None | ~3-4s initial | ~1-2s |

### Recommendation

**Use thiserror.** The evidence is overwhelming:

1. Every comparable Rust database engine uses it: fjall, tantivy, sled (via its own Error enum pattern), tikv.
2. It generates exactly the code you would hand-write -- zero runtime cost.
3. The boilerplate savings (~65 lines for the initial enum, more as sub-errors grow) directly reduce maintenance burden.
4. dtolnay's maintenance track record is the gold standard in the Rust ecosystem.
5. The CODING_GUIDELINES.md approved dependency list includes "serde" (also a dtolnay proc-macro), setting precedent that dtolnay proc-macros are acceptable.

**Version pin:** `thiserror = "2"` (MSRV-compatible with Rust 1.85, which is the project's rust-version in Cargo.toml).

**anyhow usage:** Not in the public API. Acceptable in integration tests and benchmarks where error inspection is not needed.

Sub-error types (`StorageError`, `SchemaError`, `DurabilityError`, `QueryError`) should also use thiserror:

```rust
#[derive(Debug, thiserror::Error)]
pub enum SchemaError {
    #[error("duplicate signal definition: {name}")]
    DuplicateSignal { name: String },
    #[error("invalid half-life: must be positive, got {half_life:?}")]
    InvalidHalfLife { half_life: std::time::Duration },
    #[error("unknown entity kind: {kind}")]
    UnknownEntityKind { kind: String },
}
```

---

## 4. Schema Validation Pattern

### Question
Should the schema builder use the typestate pattern (compile-time validation) or runtime validation with Result returns?

### Approaches Surveyed

#### Approach A: Typestate builder (compile-time enforcement)

```rust
struct SignalDefBuilder<State> { ... }
struct NeedsName;
struct NeedsDecay;
struct Ready;

impl SignalDefBuilder<NeedsName> {
    fn name(self, n: &str) -> SignalDefBuilder<NeedsDecay> { ... }
}
impl SignalDefBuilder<NeedsDecay> {
    fn decay(self, d: Decay) -> SignalDefBuilder<Ready> { ... }
}
impl SignalDefBuilder<Ready> {
    fn build(self) -> SignalDef { ... }
}
```

**Used by:** hyper's `http::Request::builder()` (partially). Some Rust web frameworks. Embedded systems (state machines).

**Strengths:** Impossible to construct an invalid SignalDef at compile time. IDE autocomplete shows only valid next steps.

**Weaknesses:**
- Combinatorial explosion: if 3 fields are required and 5 are optional, you need 2^5 = 32 type states, or complex generic parameter packing.
- TidalDB's schema definitions come from user input at runtime (the `define_signal()` API in API.md accepts a `SignalDef` struct). Compile-time enforcement is irrelevant when the data arrives at runtime.
- Error messages for missing fields are cryptic ("method `build` not found for `SignalDefBuilder<NeedsDecay>`" vs "missing required field: decay").
- Tantivy, fjall, and sled all rejected this pattern in favor of runtime validation.

#### Approach B: Runtime validation with builder

```rust
pub struct SignalDefBuilder {
    name: Option<String>,
    decay: Option<Decay>,
    windows: Vec<Window>,
    velocity: bool,
}

impl SignalDefBuilder {
    pub fn new() -> Self { ... }
    pub fn name(mut self, name: impl Into<String>) -> Self { self.name = Some(name.into()); self }
    pub fn decay(mut self, decay: Decay) -> Self { self.decay = Some(decay.into()); self }
    pub fn window(mut self, w: Window) -> Self { self.windows.push(w); self }
    pub fn velocity(mut self, v: bool) -> Self { self.velocity = v; self }

    pub fn build(self) -> Result<SignalDef, SchemaError> {
        let name = self.name.ok_or(SchemaError::MissingField { field: "name" })?;
        let decay = self.decay.ok_or(SchemaError::MissingField { field: "decay" })?;
        // Additional validation...
        Ok(SignalDef { name, decay, windows: self.windows, velocity: self.velocity })
    }
}
```

**Used by:** Tantivy's SchemaBuilder (add fields, then `build()` -- panics on duplicate field names). fjall's `Database::builder(path).open()`. sled's `Config::new().path(...)`.

**Strengths:** Simple. Rust developers understand it immediately. Validation errors are human-readable strings. Works with runtime data (user-provided schema definitions). Extensible -- adding a new optional field is one method, not a new type state.

**Weaknesses:** Validation happens at runtime, not compile time. Invalid builders are caught at `build()`, not at the call site. This is acceptable because schema definitions are user-provided data, not compile-time constants.

#### Approach C: Struct with validation function

```rust
pub struct SignalDef {
    pub name: String,
    pub decay: Decay,
    pub windows: Vec<Window>,
    pub velocity: bool,
}

impl SignalDef {
    pub fn validate(&self) -> Result<(), SchemaError> {
        if self.name.is_empty() { return Err(SchemaError::EmptyName); }
        if let Decay::Exponential { half_life } = &self.decay {
            if half_life.is_zero() { return Err(SchemaError::InvalidHalfLife { ... }); }
        }
        // ...
        Ok(())
    }
}
```

**Used by:** The API.md already shows direct struct construction (`SignalDef { name: "view", ... }`). This pattern is the simplest match to the existing API design.

**Strengths:** Simplest possible implementation. User constructs the struct directly (as shown in API.md). Validation is an explicit step. No builder boilerplate.

**Weaknesses:** Nothing prevents constructing an invalid SignalDef without calling validate(). Must remember to call validate() -- but `db.define_signal()` does this internally, so the user never calls it directly.

### Comparison

| Criterion | Typestate | Runtime builder | Struct + validate |
|-----------|-----------|-----------------|-------------------|
| Compile-time safety | Full | None | None |
| Runtime data support | No | Yes | Yes |
| Implementation complexity | High | Medium | Low |
| Precedent (Rust databases) | None found | Tantivy, fjall, sled | Common in libraries |
| Error message quality | Poor (type errors) | Good (Result) | Good (Result) |
| Matches API.md | No | Partially | Yes |

### Recommendation

**Struct with validation, called internally by `db.define_signal()`.** This matches the API.md design exactly, where users construct `SignalDef` structs directly. The `define_signal()` method validates and returns `Result<(), SchemaError>`.

```rust
impl Lumen {
    pub fn define_signal(&self, def: SignalDef) -> Result<(), LumenError> {
        def.validate().map_err(LumenError::Schema)?;
        // Store the validated definition...
        Ok(())
    }
}
```

Validation rules for m1p1:
- Signal name must be non-empty and ASCII alphanumeric + underscore
- Half-life must be positive and finite (for Exponential decay)
- Windows must not contain duplicates
- At least one window is required if velocity is enabled (velocity = count / window_duration)

The Tantivy SchemaBuilder pattern (mutable builder, add fields, then `build()`) is appropriate for the EntityDef builder, where the field list is constructed incrementally. But for SignalDef, the struct-with-validation pattern is simpler and matches the API contract.

---

## 5. f64 for Decay Scores and Atomic Operations

### Question
How should f64 decay scores be typed now (m1p1) to support atomic CAS operations in m1p4?

### Background

The CODING_GUIDELINES.md Section 1 specifies:
> `AtomicF64` (via `AtomicU64` + `f64::from_bits`) with CAS loops for decay scores

The signal ledger research (lumens_signal_ledger.md) confirms f64 is the correct precision for decay scores. The hot-path update formula is:

```
S(t) = S(t_prev) * exp(-lambda * dt) + weight
```

This requires atomic read-modify-write on the decay score. The standard library does not provide `AtomicF64`.

### Approaches Surveyed

#### Approach A: Hand-roll AtomicU64 + f64::from_bits/to_bits

```rust
use std::sync::atomic::{AtomicU64, Ordering};

pub struct AtomicF64 {
    bits: AtomicU64,
}

impl AtomicF64 {
    pub fn new(val: f64) -> Self {
        Self { bits: AtomicU64::new(val.to_bits()) }
    }

    pub fn load(&self, order: Ordering) -> f64 {
        f64::from_bits(self.bits.load(order))
    }

    pub fn store(&self, val: f64, order: Ordering) {
        self.bits.store(val.to_bits(), order);
    }

    /// CAS loop for read-modify-write operations.
    pub fn fetch_update<F>(&self, set_order: Ordering, fetch_order: Ordering, mut f: F) -> Result<f64, f64>
    where F: FnMut(f64) -> Option<f64> {
        self.bits.fetch_update(set_order, fetch_order, |bits| {
            f(f64::from_bits(bits)).map(f64::to_bits)
        }).map(f64::from_bits).map_err(f64::from_bits)
    }
}
```

**Used by:** Engram (from thoughts.md: "AtomicF32 for activation levels, CAS loops"). Prometheus Rust client (internal AtomicF64 wrapper). StemeDB ("compare_and_swap_f32 for aggregate weights").

**Strengths:** Zero dependencies. ~30 lines. The pattern is well-understood and used in production by multiple systems in this codebase. `f64::from_bits` and `f64::to_bits` are const fns that compile to zero instructions (the bit pattern is the same). The `fetch_update` method on AtomicU64 handles the CAS loop correctly.

**Weaknesses:** Requires `unsafe` -- wait, no. `AtomicU64::fetch_update` is safe. `f64::from_bits` and `f64::to_bits` are safe. The entire implementation is safe Rust. The only concern is NaN bit patterns: `f64::from_bits(f64::NAN.to_bits())` is NaN, but two NaN values with different bit patterns would compare as not-equal in CAS, potentially causing infinite loops. This is a non-issue for decay scores, which are always non-negative finite values (the formula produces non-negative results from non-negative inputs, and f64 underflow to 0.0 is correct behavior).

#### Approach B: atomic_float crate (v1.1.0)

**Crate health:** atomic_float v1.1.0. ~3.5M downloads. Last updated 2024. Provides AtomicF32 and AtomicF64 with fetch_add, fetch_sub, fetch_min, fetch_max, compare_exchange. Uses `UnsafeCell<f64>` cast to `&AtomicU64` internally.

**Strengths:** Full API including fetch_add (CAS loop internally) and fetch_min/fetch_max. Well-tested.

**Weaknesses:** Contains `unsafe` (the UnsafeCell cast). TidalDB's Cargo.toml has `unsafe_code = "forbid"` at the crate level. Using this crate would not violate that lint (the unsafe is in the dependency, not in TidalDB's code), but the hand-rolled version achieves the same result without any unsafe anywhere. Moderate download count suggests it is not a widely-adopted standard.

#### Approach C: portable-atomic crate (v1.11+) with float feature

**Crate health:** portable-atomic v1.11. ~100M+ downloads. Maintained by taiki-e (extremely prolific, maintains tokio ecosystem tools). Provides AtomicF64 behind the `float` feature flag. Also provides AtomicI128, AtomicU128 for platforms that lack native support.

**Strengths:** Most widely adopted atomic extension crate. Excellent cross-platform support. Maintained by a Tier-1 Rust ecosystem contributor. `is_lock_free()` method lets you verify platform support.

**Weaknesses:** Heavier dependency than needed -- TidalDB targets x86_64 and aarch64, where AtomicU64 is natively supported and the hand-rolled approach works perfectly. The crate's value proposition is portability to exotic targets (thumbv6m, RISC-V without A-extension), which TidalDB does not need. Also contains unsafe (necessarily, for the low-level atomic operations).

#### Standard Library Status

The Rust issue #72353 (Adding AtomicF32/AtomicF64 to std) is marked "C-feature-accepted" but has no implementation timeline. It may land in 2026-2027, at which point TidalDB could migrate from the hand-rolled version with zero API changes.

### Comparison

| Criterion | Hand-roll | atomic_float | portable-atomic |
|-----------|-----------|-------------|-----------------|
| Dependencies | 0 | 1 | 1 |
| Unsafe in TidalDB | None | None (in dep) | None (in dep) |
| Unsafe in dependency | None | Yes | Yes |
| Lines of code | ~30 | 0 | 0 |
| API surface | Custom (minimal) | Full | Full |
| Cross-platform | x86_64 + aarch64 | x86_64 + aarch64 | Everything |
| Precedent in codebase | Engram, StemeDB | None | None |
| Migration to std | Trivial | Trivial | Trivial |

### Recommendation

**Hand-roll for m1p1. Define the type now; implement atomic methods in m1p4.**

In m1p1, define a non-atomic `DecayScore` as a simple f64 wrapper:

```rust
#[derive(Debug, Clone, Copy, PartialEq, PartialOrd)]
pub struct DecayScore(f64);

impl DecayScore {
    pub const ZERO: Self = Self(0.0);

    #[inline]
    pub const fn new(value: f64) -> Self { Self(value) }

    #[inline]
    pub fn value(self) -> f64 { self.0 }

    /// Apply decay over a time delta.
    #[inline]
    pub fn decayed(self, decay_rate: DecayRate, dt_seconds: f64) -> Self {
        Self(self.0 * decay_rate.decay_factor(dt_seconds))
    }

    /// Add a weighted event to the running score.
    #[inline]
    pub fn accumulate(self, weight: f64, decay_rate: DecayRate, dt_seconds: f64) -> Self {
        Self(self.0 * decay_rate.decay_factor(dt_seconds) + weight)
    }
}
```

In m1p4, introduce `AtomicDecayScore` using the hand-rolled AtomicU64 pattern:

```rust
pub struct AtomicDecayScore {
    bits: AtomicU64,
}
```

The type separation (DecayScore vs AtomicDecayScore) mirrors `u64` vs `AtomicU64` in the standard library. Non-atomic DecayScore is used in schema definitions, test fixtures, and cold-path code. AtomicDecayScore is used in the hot-path `EntitySignalState` struct.

**Why not use a crate:** The hand-rolled version is 30 lines of safe Rust, uses a pattern proven by Engram and StemeDB in this codebase, and avoids adding a dependency for something the standard library will eventually provide. The CODING_GUIDELINES.md explicitly endorses this pattern: "AtomicF64 (via AtomicU64 + f64::from_bits) with CAS loops for decay scores."

---

## 6. Timestamp Precision

### Question
Is `u64` nanoseconds since Unix epoch the correct timestamp representation? When does it overflow? What do production systems use?

### Analysis

**Overflow calculation:**
- `u64::MAX` = 18,446,744,073,709,551,615
- Nanoseconds per second = 1,000,000,000
- Seconds representable = 18,446,744,073.71 seconds
- Years representable = 18,446,744,073.71 / (365.25 * 86400) = **~584.5 years**
- Overflow date from Unix epoch (1970-01-01) = approximately **year 2554**

This is 528 years from now. Sufficient for any practical database system.

### Production System Survey

| System | Timestamp Type | Precision | Range |
|--------|---------------|-----------|-------|
| InfluxDB | i64 | Nanoseconds | 1677-2262 (signed) |
| QuestDB | i64 (microseconds by default, nanoseconds optional) | Microseconds or nanoseconds | ~292K years (microseconds) |
| TimescaleDB | PostgreSQL timestamptz | Microseconds | 4713 BC - 294276 AD |
| Tantivy | i64 (DateTime) | Microseconds (truncated from nanoseconds) | ~292K years |
| ClickHouse | UInt64 | Nanoseconds (DateTime64) | Similar to u64 |
| Sonnerie (Rust time-series DB) | u64 | Nanoseconds | 1970-2554 |
| Go time.Time | i64 + i32 | Nanoseconds (wall) | 1885-2157 (monotonic limited) |

**Key observation:** InfluxDB uses **signed** i64 nanoseconds, which halves the range to 1677-2262. This is a more constrained choice than u64. They made this decision to support pre-epoch timestamps (historical data). TidalDB does not need pre-epoch timestamps -- all signals are engagement events that happen now or in the recent past.

**ClickHouse** uses u64 nanoseconds (as DateTime64(9)), which is exactly the approach proposed for TidalDB. Sonnerie, the only Rust-native time-series database found in the survey, also uses u64 nanoseconds.

### Recommendation

**u64 nanoseconds since Unix epoch.** This is the right choice for TidalDB.

```rust
/// Nanoseconds since Unix epoch (1970-01-01T00:00:00Z).
/// Overflows in year 2554. Sufficient for any practical use.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(transparent)]
pub struct Timestamp(u64);

impl Timestamp {
    /// Create from nanoseconds since Unix epoch.
    #[inline]
    pub const fn from_nanos(nanos: u64) -> Self { Self(nanos) }

    /// Create from seconds since Unix epoch (for convenience).
    #[inline]
    pub const fn from_secs(secs: u64) -> Self { Self(secs * 1_000_000_000) }

    /// Current wall-clock time.
    pub fn now() -> Self {
        let dur = std::time::SystemTime::now()
            .duration_since(std::time::UNIX_EPOCH)
            .expect("system clock before Unix epoch");
        Self(dur.as_nanos() as u64)
    }

    /// Nanoseconds since epoch.
    #[inline]
    pub const fn as_nanos(self) -> u64 { self.0 }

    /// Seconds since epoch as f64 (for decay math: dt = (t2 - t1).as_secs_f64()).
    #[inline]
    pub fn as_secs_f64(self) -> f64 { self.0 as f64 / 1_000_000_000.0 }

    /// Time delta in seconds as f64 (for decay formula).
    #[inline]
    pub fn seconds_since(self, earlier: Timestamp) -> f64 {
        (self.0.saturating_sub(earlier.0)) as f64 / 1_000_000_000.0
    }
}
```

**Why u64, not i64:** TidalDB signals are engagement events that happen in the present. Pre-epoch timestamps (before 1970) are meaningless for "user liked item at time T." Using u64 gives 584 years of range vs i64's 292 years, and eliminates the need to handle negative timestamps.

**Why nanoseconds, not microseconds:** Nanosecond precision matches InfluxDB's native resolution and avoids precision loss when interfacing with system clocks (`SystemTime::now()` returns nanosecond precision on Linux and macOS). The storage cost is identical (both u64). For decay math, the conversion to seconds-as-f64 is a single division.

**The `as_nanos() as u64` cast in `Timestamp::now()`:** `SystemTime::duration_since()` returns a Duration whose `as_nanos()` returns u128. The cast to u64 is safe until year 2554. The `cast_possible_truncation` clippy lint is already allowed in Cargo.toml.

**Serde:** Add `#[serde(transparent)]` to serialize as a bare u64 in JSON (not a nested object). At API boundaries, consider providing ISO 8601 string formatting via a separate method, not the default serialization.

---

## 7. Window Enum

### Question
How should the Window enum be represented for efficient storage and comparison?

### Recommendation

```rust
/// Pre-defined aggregation windows.
/// Stored as the window duration in seconds for efficient comparison.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Window {
    /// Last 1 hour (3,600 seconds)
    OneHour,
    /// Last 24 hours (86,400 seconds)
    TwentyFourHours,
    /// Last 7 days (604,800 seconds)
    SevenDays,
    /// Last 30 days (2,592,000 seconds)
    ThirtyDays,
    /// All time (no window boundary)
    AllTime,
}

impl Window {
    /// Duration of this window in seconds. Returns None for AllTime.
    pub const fn duration_secs(&self) -> Option<u64> {
        match self {
            Self::OneHour => Some(3_600),
            Self::TwentyFourHours => Some(86_400),
            Self::SevenDays => Some(604_800),
            Self::ThirtyDays => Some(2_592_000),
            Self::AllTime => None,
        }
    }

    /// All windows in order from shortest to longest.
    pub const ALL: &[Window] = &[
        Self::OneHour,
        Self::TwentyFourHours,
        Self::SevenDays,
        Self::ThirtyDays,
        Self::AllTime,
    ];
}
```

**Why an enum and not a Duration:** The API.md defines exactly 5 window variants. The schema validation must reject arbitrary durations (e.g., `Window::minutes(37)` is not a valid window). An enum makes the closed set explicit. The `duration_secs()` method provides the numeric value when needed for computation.

**Why not `Window::hours(1)` as shown in API.md:** The API.md shows convenience constructors (`Window::hours(1)`, `Window::days(7)`), but these are better expressed as associated constants or enum variants. If TidalDB later needs custom windows (Phase 2+), the enum can be extended with a `Custom(u64)` variant without breaking the existing variants.

**Display:** Hand-implement to produce human-readable strings ("1h", "24h", "7d", "30d", "all_time") for schema definition output and error messages.

---

## Complete Dependency Recommendation for m1p1

| Crate | Version | Purpose | Justification |
|-------|---------|---------|---------------|
| thiserror | 2 | Error derive macros | Used by fjall, tantivy, tikv. Eliminates ~80 lines of boilerplate. dtolnay-maintained. |
| serde | 1 | Serialization (feature-gated) | Already approved in CODING_GUIDELINES. Behind `serde` feature flag. |
| serde_json | 1 | JSON serialization (dev-dependency only for m1p1) | Testing schema serialization round-trips. |

No other dependencies are needed for m1p1. All types (EntityId, Timestamp, DecayRate, DecayScore, Window, LumenError) are hand-implemented with standard derives.

---

## Open Questions

1. **EntityId uniqueness scope:** Is EntityId globally unique across all entity kinds (items, users, creators), or unique within a kind? This affects key encoding in m1p2. If globally unique, a single u64 suffices. If per-kind, the key must include `(EntityKind, EntityId)`. The API.md uses string IDs ("item_abc", "user_123") which suggests per-kind uniqueness with string keys. m1p1 should support both via `EntityId(u64)` with an `EntityKind` discriminator.

2. **Decay::Linear and Decay::Permanent:** The API.md defines three decay types (Exponential, Linear, Permanent). m1p1 should define all three in the enum but may only implement Exponential initially. Linear decay (`weight * max(0, 1 - t/lifetime)`) and Permanent (no decay, score never changes) are simpler than Exponential but should be typed now.

3. **Custom windows in the future:** If a user needs a 6-hour window for a specific signal, the current enum does not support it. Should the enum include a `Custom(std::time::Duration)` variant from day one, or is this a Phase 2 extension? Recommendation: add it now as a variant but validate that custom durations are positive, non-zero, and less than 365 days.

4. **String vs u64 entity IDs:** The API.md shows string IDs (`"item_abc"`). The type system research recommends `EntityId(u64)`. These must be reconciled. Options: (a) the public API accepts strings, internally hashes them to u64 (like DuckDB's dictionary encoding); (b) the public API accepts u64 only, the application maps strings to u64; (c) EntityId is an enum of `Numeric(u64)` and `String(Arc<str>)`. Recommendation: u64 internally, with a string-to-u64 mapping table stored in the entity metadata namespace. The mapping is a cold-path operation (entity write), not hot-path (signal write, ranking query).

5. **f64 NaN handling in DecayScore:** Should `DecayScore::new(f64::NAN)` be legal? For safety, validate at construction in debug builds (`debug_assert!(!value.is_nan())`) but skip the check in release builds for performance. NaN should never arise from the decay formula with valid inputs, but corrupted WAL replay could theoretically produce it.

6. **Benchmark the `exp()` cost assumption:** The signal ledger research claims `exp()` costs ~12ns per call. This should be benchmarked on the target hardware in m1p1 using the existing criterion setup, as it is a load-bearing assumption for the entire scoring budget.

---

## Sources

- [Effective Rust - Item 6: Embrace the newtype pattern](https://www.lurklurk.org/effective-rust/newtype.html)
- [The Ultimate Guide to Rust Newtypes](https://www.howtocodeit.com/guides/ultimate-guide-rust-newtypes)
- [derive_more documentation (v2.1.1)](https://docs.rs/derive_more)
- [derive_more GitHub releases](https://github.com/JelteF/derive_more/releases)
- [nutype: the newtype with guarantees](https://www.greyblake.com/blog/nutype-the-newtype-with-guarantees/)
- [thiserror crate (v2.0.18)](https://docs.rs/crate/thiserror/latest)
- [Error Handling in Rust - Luca Palmieri](https://lpalmieri.com/posts/error-handling-rust/)
- [Rust Error Handling: thiserror, anyhow, and When to Use Each](https://momori.dev/posts/rust-error-handling-thiserror-anyhow/)
- [Error Handling for Large Rust Projects (GreptimeDB)](https://medium.com/@greptime/error-handling-for-large-rust-projects-a-deep-dive-into-5e10ee4cbc96)
- [Typestate Builder Pattern in Rust](https://n1ghtmare.github.io/2024-05-31/typestate-builder-pattern-in-rust/)
- [Tantivy SchemaBuilder documentation](https://docs.rs/tantivy/latest/tantivy/schema/struct.SchemaBuilder.html)
- [fjall documentation](https://docs.rs/fjall/latest/fjall/)
- [Fjall 3.0 release notes](https://fjall-rs.github.io/post/fjall-3/)
- [atomic_float crate - AtomicF64](https://docs.rs/atomic_float/latest/atomic_float/struct.AtomicF64.html)
- [portable-atomic crate - AtomicF64](https://docs.rs/portable-atomic/latest/portable_atomic/struct.AtomicF64.html)
- [Rust issue #72353: Adding AtomicF32/AtomicF64 to std](https://github.com/rust-lang/rust/issues/72353)
- [std::time::Duration documentation](https://doc.rust-lang.org/std/time/struct.Duration.html)
- [Unix timestamp in nanoseconds - Rust forum](https://users.rust-lang.org/t/unix-timestamp-in-nanoseconds/73926)
- [Sonnerie: a simple timeseries database in Rust](https://github.com/njaard/sonnerie)
- [Hacker News: Timestamps are 64-bit nanoseconds overflow](https://news.ycombinator.com/item?id=14174958)
- [InfluxDB timestamp precision documentation](https://www.influxdata.com/blog/tldr-tech-tips-flux-timestamps/)
- [QuestDB timestamp functions](https://questdb.com/docs/query/functions/date-time/)
- [Forward Decay - Cormode, Shkapenyuk, Srivastava, Xu (ICDE 2009)](https://doi.org/10.1109/ICDE.2009.65)
- [Lumen Signal Ledger Research](docs/research/lumens_signal_ledger.md)
- [TidalDB CODING_GUIDELINES.md](CODING_GUIDELINES.md)
- [TidalDB API.md](API.md)
- [TidalDB thoughts.md](thoughts.md)
