tidaldb/docs/planning/milestone-3/phase-2/task-04-atomic-signal-dispatch.md

Task 04: Atomic Signal Dispatch

Context

Milestone: 3 -- Personalized Ranking Phase: m3p2 -- Feedback Loop Depends On: Task 01 (preference vector), Task 02 (interaction weight ledger), Task 03 (hard negatives), m3p1 (user entities, relationships, user state index), m1p5 (signal write API) Blocks: m3p3 (Personalized Profiles require all user state to be correctly updated), m3p4 (User State Filters require populated bitmaps) Complexity: L

Objective

Wire all user-state updates into the signal write path so that a single db.signal() call atomically updates every affected state target. This is the integration task that connects the preference vector (Task 01), interaction weight ledger (Task 02), and hard negatives (Task 03) with the existing signal ledger from m1p5.

Before this task, db.signal("view", item_id, weight, timestamp) only updates the item's signal ledger. After this task, the same call -- when user context is available -- also:

1. Marks the item as seen in UserStateIndex
2. Updates the user->creator interaction weight
3. Shifts the user preference vector toward/away from the item embedding
4. For "hide"/"block" signals, writes a permanent hard negative

The user context is provided by a new UserSignalContext parameter that the application passes alongside the signal. Without user context, the signal write behaves exactly as before (item-level only), preserving backward compatibility with M1/M2 usage.

Requirements

• UserSignalContext struct: user_id: EntityId, plus cached references to user state
• db.signal_with_user(signal_type, item_id, weight, timestamp, user_context) dispatches to all state targets
• db.signal(signal_type, item_id, weight, timestamp) still works (item-level only, no user state)
• Signal dispatch for positive signals (view, like, share, completion):
  1. Item signal ledger update (existing)
  2. UserStateIndex::mark_seen(user_id, item_id) (for view signals)
  3. InteractionWeightLedger::update_weight(user_id, creator_id, delta, ts)
  4. PreferenceVector::update_toward(item_embedding, alpha) (for like/completion/share)
  5. Persist updated preference vector to storage
• Signal dispatch for skip:
  1. Item signal ledger update
  2. InteractionWeightLedger::update_weight(user_id, creator_id, -delta, ts)
  3. PreferenceVector::update_away(item_embedding, negative_alpha)
  4. Persist updated preference vector
• Signal dispatch for hide:
  1. HardNegativeStore::hide_item(user_id, item_id, ts, ...)
  2. No item signal ledger update (hide is user-level, not item-level)
• Signal dispatch for block:
  1. HardNegativeStore::block_creator(user_id, creator_id, ts, ...)
  2. No item signal ledger update
• All state updates visible to the next query within the process
• Signal dispatch overhead < 50 microseconds beyond the base item signal write
• TidalDb holds InteractionWeightLedger and references to UserStateIndex, PreferenceConfig
• Checkpoint/restore of interaction weights and preference vectors on shutdown/startup

Technical Design

Module Structure

tidal/src/
  db/
    signal_dispatch.rs -- UserSignalContext, dispatch logic
    mod.rs             -- Extended with signal_with_user() and new fields

UserSignalContext

// === db/signal_dispatch.rs ===

use crate::schema::EntityId;

/// User context for signal dispatch.
///
/// Provides the user identity so that signal writes can update
/// user-level state (preference vector, interaction weights, seen bitmap).
///
/// Created by the application or query executor when the user is known.
/// Passed to `signal_with_user()`.
#[derive(Debug, Clone)]
pub struct UserSignalContext {
    /// The user performing the action.
    pub user_id: EntityId,
}

impl UserSignalContext {
    pub fn new(user_id: EntityId) -> Self {
        Self { user_id }
    }
}

Signal Dispatch Logic

/// Dispatch a signal with user context to all state targets.
///
/// This is the internal implementation called by `db.signal_with_user()`.
/// It coordinates updates across the item signal ledger, user state index,
/// interaction weight ledger, and preference vector.
pub(crate) fn dispatch_user_signal(
    signal_type: &str,
    item_id: EntityId,
    weight: f64,
    timestamp: Timestamp,
    user_ctx: &UserSignalContext,
    // Dependencies injected from TidalDb:
    ledger: &SignalLedger,
    user_state: &UserStateIndex,
    interaction_weights: &InteractionWeightLedger,
    pref_config: &PreferenceConfig,
    storage: &StorageBox,
    wal_rel_writer: &dyn WalRelWriter,
    read_item_embedding: &dyn Fn(EntityId) -> Option<Vec<f32>>,
    read_item_creator: &dyn Fn(EntityId) -> Option<EntityId>,
) -> crate::Result<()> {
    match signal_type {
        "hide" => {
            // Hard negative: hide item. No item signal update.
            HardNegativeStore::hide_item(
                user_ctx.user_id,
                item_id,
                timestamp,
                wal_rel_writer,
                storage.users_engine(),
                user_state,
            )?;
        }
        "block" => {
            // Hard negative: block creator. The item_id parameter
            // is reinterpreted as creator_id for block signals.
            HardNegativeStore::block_creator(
                user_ctx.user_id,
                item_id, // creator_id
                timestamp,
                wal_rel_writer,
                storage.users_engine(),
                user_state,
            )?;
        }
        _ => {
            // 1. Item signal ledger update (existing behavior).
            ledger.record_signal(signal_type, item_id, weight, timestamp)?;

            // 2. Mark as seen for "view" signals.
            if signal_type == "view" {
                user_state.mark_seen(user_ctx.user_id, item_id);
            }

            // 3. Update interaction weight (user -> creator).
            if let Some(creator_id) = read_item_creator(item_id) {
                let delta = interaction_weights.delta_for_signal(signal_type);
                if delta.abs() > f64::EPSILON {
                    interaction_weights.update_weight(
                        user_ctx.user_id,
                        creator_id,
                        delta,
                        timestamp,
                    );
                }
            }

            // 4. Update preference vector for engagement signals.
            let is_positive = matches!(signal_type, "like" | "completion" | "share");
            let is_negative = signal_type == "skip";

            if is_positive || is_negative {
                if let Some(item_embedding) = read_item_embedding(item_id) {
                    // Read current preference vector.
                    let pref_key_suffix = preference_key_suffix();
                    let pref_key = encode_key(user_ctx.user_id, Tag::Meta, &pref_key_suffix);
                    let mut pref = match storage.users_engine().get(&pref_key)? {
                        Some(bytes) => {
                            deserialize_preference(&bytes)
                                .unwrap_or_else(|| PreferenceVector::cold_start(pref_config.dimensions))
                        }
                        None => PreferenceVector::cold_start(pref_config.dimensions),
                    };

                    // Apply EMA update.
                    if is_positive {
                        pref.update_toward(&item_embedding, pref_config.alpha);
                    } else {
                        pref.update_away(&item_embedding, pref_config.negative_alpha);
                    }

                    // 5. Persist updated preference vector.
                    let value = serialize_preference(&pref);
                    storage.users_engine().put(&pref_key, &value)
                        .map_err(LumenError::from)?;
                }
            }
        }
    }

    Ok(())
}

fn preference_key_suffix() -> Vec<u8> {
    let mut suffix = vec![PREF_TAG];
    suffix.extend_from_slice(b"default");
    suffix
}

TidalDb Extensions

impl TidalDb {
    /// Record a signal event with user context.
    ///
    /// Atomically updates:
    /// - Item signal ledger (decay scores, windowed counts)
    /// - User seen bitmap (for "view" signals)
    /// - User->creator interaction weight
    /// - User preference vector (for like/skip/completion/share)
    /// - Hard negatives (for hide/block)
    ///
    /// All updates are immediately visible to the next query.
    ///
    /// # Errors
    ///
    /// - `LumenError::Internal` if no ledger is wired.
    /// - `LumenError::Schema` if signal_type is not defined.
    /// - `LumenError::Durability` if WAL write fails (hide/block only).
    pub fn signal_with_user(
        &self,
        signal_type: &str,
        item_id: EntityId,
        weight: f64,
        timestamp: Timestamp,
        user_ctx: &UserSignalContext,
    ) -> crate::Result<()> {
        dispatch_user_signal(
            signal_type,
            item_id,
            weight,
            timestamp,
            user_ctx,
            self.ledger.as_ref()
                .ok_or_else(|| LumenError::Internal("no ledger".into()))?,
            &self.user_state,
            &self.interaction_weights,
            &self.preference_config,
            self.storage.as_ref()
                .ok_or_else(|| LumenError::Internal("no storage".into()))?,
            &self.wal_rel_writer,
            &|item_id| self.read_item_embedding_internal(item_id),
            &|item_id| self.read_item_creator_id(item_id),
        )
    }

    /// Look up the creator_id for an item from its metadata.
    fn read_item_creator_id(&self, item_id: EntityId) -> Option<EntityId> {
        let storage = self.storage.as_ref()?;
        let key = encode_key(item_id, Tag::Meta, b"");
        let bytes = storage.items_engine().get(&key).ok()??;
        // Parse metadata to extract creator_id field.
        let metadata = deserialize_metadata(&bytes)?;
        metadata.get("creator_id")
            .and_then(|v| v.parse::<u64>().ok())
            .map(EntityId::new)
    }

    /// Read an item's embedding for preference vector updates.
    fn read_item_embedding_internal(&self, item_id: EntityId) -> Option<Vec<f32>> {
        // Delegate to the vector index or storage.
        // Returns None if item has no embedding.
        // Implementation reads from the embedding slot registry from m2p1.
        None // Placeholder -- real implementation reads from vector index
    }
}

TidalDb New Fields

pub struct TidalDb {
    // ... existing fields ...

    /// User state bitmaps (seen, blocked, follows).
    user_state: UserStateIndex,
    /// User->creator interaction weight ledger.
    interaction_weights: InteractionWeightLedger,
    /// Preference vector configuration.
    preference_config: PreferenceConfig,
    /// WAL writer for relationship events (hide/block).
    wal_rel_writer: Box<dyn WalRelWriter>,
}

Startup and Shutdown

impl TidalDb {
    /// Extended startup: restore user state from storage.
    ///
    /// Called after the existing m1p5 open logic.
    fn restore_user_state(&self) -> crate::Result<()> {
        if let Some(storage) = &self.storage {
            // 1. Rebuild UserStateIndex from relationship edges.
            self.user_state.rebuild_from_relationships(
                storage.users_engine(),
            )?;

            // 2. Restore interaction weights from InteractionWeight edges.
            // Scan all InteractionWeight-type relationship edges
            // and populate the interaction weight ledger.
            // Implementation uses prefix scan on Tag::Rel with type byte 0x03.
        }
        Ok(())
    }

    /// Extended shutdown: checkpoint interaction weights.
    ///
    /// Called before the existing m1p5 shutdown logic.
    fn checkpoint_user_state(&self) -> crate::Result<()> {
        if let Some(storage) = &self.storage {
            // Checkpoint interaction weights as relationship edges.
            self.interaction_weights.checkpoint(&|user_id, creator_id, weight, ts| {
                let key = encode_relationship_key(
                    user_id, RelationshipType::InteractionWeight, creator_id,
                );
                let value = encode_relationship_value(weight, ts);
                storage.users_engine().put(&key, &value)
                    .map_err(LumenError::from)
            })?;
        }
        Ok(())
    }
}

Test Strategy

Unit Tests

#[test]
fn signal_with_user_updates_item_ledger() {
    let db = open_test_db_with_schema_and_items();
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    db.signal_with_user("view", EntityId::new(42), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();

    // Item signal should be updated.
    let score = db.read_decay_score(EntityId::new(42), "view", 0).unwrap();
    assert!(score.is_some());
}

#[test]
fn signal_with_user_marks_seen() {
    let db = open_test_db_with_schema_and_items();
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    db.signal_with_user("view", EntityId::new(42), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();

    assert!(db.user_state.is_seen(EntityId::new(1), EntityId::new(42)));
}

#[test]
fn signal_like_updates_interaction_weight() {
    let db = open_test_db_with_items_and_creators();
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    // Item 42 belongs to creator 10.
    db.signal_with_user("like", EntityId::new(42), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();

    let weight = db.interaction_weights.read_weight(
        EntityId::new(1), EntityId::new(10), Timestamp::now(),
    );
    assert!(weight > 0.0, "interaction weight should increase on like");
}

#[test]
fn signal_skip_decreases_interaction_weight() {
    let db = open_test_db_with_items_and_creators();
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    // Establish a positive weight first.
    db.signal_with_user("like", EntityId::new(42), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();
    let before = db.interaction_weights.read_weight(
        EntityId::new(1), EntityId::new(10), Timestamp::now(),
    );

    db.signal_with_user("skip", EntityId::new(43), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();
    // Assuming item 43 also belongs to creator 10.
    let after = db.interaction_weights.read_weight(
        EntityId::new(1), EntityId::new(10), Timestamp::now(),
    );
    // Weight should decrease due to skip's negative delta.
    // (May not be less than before if items have different creators.)
}

#[test]
fn signal_hide_excludes_item() {
    let db = open_test_db_with_schema_and_items();
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    db.signal_with_user("hide", EntityId::new(42), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();

    assert!(db.user_state.is_hidden(EntityId::new(1), EntityId::new(42)));
    // Item should NOT appear in RETRIEVE for this user.
}

#[test]
fn signal_block_excludes_creator() {
    let db = open_test_db_with_items_and_creators();
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    // For block, item_id is reinterpreted as creator_id.
    db.signal_with_user("block", EntityId::new(77), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();

    assert!(db.user_state.is_blocked(EntityId::new(1), EntityId::new(77)));
}

#[test]
fn signal_without_user_context_is_item_only() {
    let db = open_test_db_with_schema_and_items();

    // No user context: use the original signal() method.
    db.signal("view", EntityId::new(42), 1.0, Timestamp::now()).unwrap();

    // Item signal updated.
    let score = db.read_decay_score(EntityId::new(42), "view", 0).unwrap();
    assert!(score.is_some());

    // No user state affected.
    // (UserStateIndex is empty because no user context was provided.)
}

#[test]
fn signal_like_updates_preference_vector() {
    let db = open_test_db_with_items_and_embeddings();
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    // User starts with no preference (cold start).
    let pref_before = db.read_user_preference(EntityId::new(1)).unwrap();
    assert!(pref_before.is_none() || pref_before.as_ref().unwrap().is_cold_start());

    // Like an item that has an embedding.
    db.signal_with_user("like", EntityId::new(42), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();

    // Preference vector should now be initialized from item 42's embedding.
    let pref_after = db.read_user_preference(EntityId::new(1)).unwrap();
    assert!(pref_after.is_some());
    assert!(!pref_after.unwrap().is_cold_start());
}

#[test]
fn shutdown_and_restart_preserves_interaction_weights() {
    let dir = tempdir().unwrap();
    let db = open_persistent_db_with_schema(&dir);
    let user_ctx = UserSignalContext::new(EntityId::new(1));

    db.signal_with_user("like", EntityId::new(42), 1.0, Timestamp::now(), &user_ctx)
        .unwrap();

    let weight_before = db.interaction_weights.read_weight(
        EntityId::new(1), EntityId::new(10), Timestamp::now(),
    );
    db.close().unwrap();

    let db2 = open_persistent_db_with_schema(&dir);
    let weight_after = db2.interaction_weights.read_weight(
        EntityId::new(1), EntityId::new(10), Timestamp::now(),
    );

    assert!((weight_before - weight_after).abs() < 0.1,
        "interaction weight should survive restart");
}

Property Tests

use proptest::prelude::*;

/// The critical invariant: hidden items and blocked creators NEVER appear
/// in query results.
proptest! {
    #[test]
    fn hidden_items_never_in_results(
        hide_ids in proptest::collection::vec(1u64..100, 1..20),
        all_ids in proptest::collection::vec(1u64..200, 50..100),
    ) {
        let db = open_ephemeral_test_db();
        let user_ctx = UserSignalContext::new(EntityId::new(1));

        // Write all items.
        for &id in &all_ids {
            let mut meta = HashMap::new();
            meta.insert("creator_id".into(), "1".into());
            db.write_item(EntityId::new(id), &meta).unwrap();
        }

        // Hide some items.
        for &id in &hide_ids {
            if all_ids.contains(&id) {
                let _ = db.signal_with_user(
                    "hide", EntityId::new(id), 1.0, Timestamp::now(), &user_ctx,
                );
            }
        }

        // Query: unseen predicate allows all, but unblocked should exclude hidden.
        let unblocked = db.user_state.unblocked_predicate(EntityId::new(1));
        for &id in &all_ids {
            let hidden = hide_ids.contains(&id);
            let passes = unblocked(id, Some(1)); // creator_id doesn't matter for hide check
            if hidden {
                prop_assert!(!passes,
                    "hidden item {} should NOT pass unblocked filter", id);
            }
        }
    }

    #[test]
    fn blocked_creators_items_never_in_results(
        block_ids in proptest::collection::vec(1u64..20, 1..5),
        item_ids in proptest::collection::vec(1u64..200, 20..50),
    ) {
        let db = open_ephemeral_test_db();
        let user_ctx = UserSignalContext::new(EntityId::new(1));

        // Write items with creator assignments.
        for (i, &id) in item_ids.iter().enumerate() {
            let creator = (i % 20) as u64 + 1;
            let mut meta = HashMap::new();
            meta.insert("creator_id".into(), creator.to_string());
            db.write_item(EntityId::new(id), &meta).unwrap();
        }

        // Block some creators.
        for &cid in &block_ids {
            let _ = db.signal_with_user(
                "block", EntityId::new(cid), 1.0, Timestamp::now(), &user_ctx,
            );
        }

        // Verify: items from blocked creators are excluded.
        let unblocked = db.user_state.unblocked_predicate(EntityId::new(1));
        for (i, &id) in item_ids.iter().enumerate() {
            let creator = (i % 20) as u64 + 1;
            let passes = unblocked(id, Some(creator));
            if block_ids.contains(&creator) {
                prop_assert!(!passes,
                    "item {} from blocked creator {} should NOT pass", id, creator);
            }
        }
    }
}

Acceptance Criteria

• UserSignalContext struct with user_id field
• db.signal_with_user() dispatches to all state targets based on signal type
• db.signal() (without user context) still works as before (backward compatible)
• "view" signal marks item as seen in UserStateIndex
• "like"/"share"/"completion" signals update preference vector via EMA
• "skip" signal updates preference vector via negative EMA
• Positive engagement signals increment user->creator interaction weight
• "skip" signal decrements user->creator interaction weight
• "hide" signal writes permanent hard negative (WAL + storage + bitmap)
• "block" signal writes permanent block (WAL + storage + bitmap)
• Cold-start user gets preference vector initialized from first liked item
• All updates visible to next query (in-process consistency)
• Interaction weights survive shutdown and restart (checkpoint/restore)
• TidalDb struct extended with user_state, interaction_weights, preference_config, wal_rel_writer fields
• Startup rebuilds user state from storage (relationships + interaction weights)
• Property test: hidden items NEVER pass unblocked filter
• Property test: blocked creators' items NEVER pass unblocked filter
• Signal dispatch overhead < 50 microseconds beyond base signal write
• cargo clippy -- -D warnings passes
• All tests pass

Research References

• VISION.md -- "One write, multiple state updates, no application logic"
• SEQUENCE.md -- Core Feedback Loop sequence diagram
• thoughts.md -- Part V.16 (user preference vector), Part V.5 (WAL-first)

Implementation Notes

• The dispatch_user_signal function takes dependency references rather than holding them internally. This avoids lifetime complexity and makes testing easier -- each dependency can be mocked independently.
• The signal_with_user method on TidalDb constructs the dependency references from its fields and calls dispatch_user_signal. This is the only place where the full dependency graph is assembled.
• For "block" signals, the item_id parameter is reinterpreted as creator_id. This is a deliberate API design choice: the application calls db.signal_with_user("block", creator_id, ...) where the second parameter is the creator being blocked. An alternative would be a separate db.block_creator() method, but keeping it within the signal system is more consistent.
• The read_item_creator_id helper extracts the creator_id from item metadata. This requires that items are written with a "creator_id" key in their metadata map. If the key is missing, interaction weight updates are skipped (graceful degradation).
• The read_item_embedding_internal method is a placeholder in this task. It must be wired to the EmbeddingSlotRegistry from m2p1 or read embeddings from storage. If the item has no embedding, preference vector updates are skipped (graceful degradation).
• The TidalDb::from_parts constructor must be extended to accept the new fields. The open_with_schema method must be extended to create UserStateIndex, InteractionWeightLedger, and PreferenceConfig instances.
• For ephemeral mode, wal_rel_writer uses NoopWalRelWriter. For persistent mode, it wraps the WAL handle.
• Preference vector persistence: the vector is written to storage on every positive/negative signal. This is acceptable because preference updates are infrequent (one per engagement event). If write amplification becomes a concern at scale, buffering can be added in M7.
• Integration tests for this task should set up a full TidalDb with items, creators, embeddings, and users, then execute signal sequences and verify all state targets are correctly updated. The test helper open_test_db_with_items_and_creators() must be created as part of this task.