# Task 03: Observability + Export UAT Tests ## Delivers Three integration tests in `tidal/tests/m7_uat.rs` proving operational visibility features work end-to-end: 1. **`uat_query_stats_populated`** -- Execute RETRIEVE and SEARCH; verify the `stats` field on both results contains non-zero `candidates_considered`, `scoring_time_us`, `total_time_us`, and a valid `profile_name`. 2. **`uat_metrics_content`** -- Verify Prometheus text output contains expected metric names (`tidaldb_signal_hot_entries`, `tidaldb_wal_lag_bytes`, etc.) and is well-formed. 3. **`uat_export_and_session_aggregation`** -- Write session signals, close the session, call `export_signals()` to verify events are present, call `user_session_summary()` to verify aggregated counts. ## Complexity: M ## Dependencies - m7p4 complete (`QueryStats` struct, `Results.stats`, `SearchResults.stats`, Prometheus metrics export, `db.export_signals()`, `db.user_session_summary()`) - m7p2 complete (`DegradationLevel` used in `QueryStats`) - m7p1 complete (WAL segments readable for `export_signals`) ## Technical Design ### Test 7: QueryStats populated on RETRIEVE and SEARCH ```rust #[test] fn uat_query_stats_populated() { let db = TidalDb::builder() .ephemeral() .with_schema(m7_uat_schema()) .open() .unwrap(); let now = Timestamp::now(); // Write 100 items with metadata + signals so both RETRIEVE and SEARCH // have non-trivial work to do. for id in 1..=100u64 { let mut meta = HashMap::new(); meta.insert("title".to_string(), format!("Stats Item {id}")); meta.insert("description".to_string(), format!("Description for item {id}")); meta.insert("category".to_string(), "test".to_string()); db.write_item_with_metadata(EntityId::new(id), &meta).unwrap(); } for id in 1..=50u64 { for _ in 0..5 { db.signal("view", EntityId::new(id), 1.0, now).unwrap(); } } // Flush text index so SEARCH has results. db.flush_text_index().unwrap(); // -- RETRIEVE stats -- let retrieve_results = db .retrieve( &Retrieve::builder() .profile("trending") .limit(20) .build() .unwrap(), ) .unwrap(); let stats = &retrieve_results.stats; assert!( stats.candidates_considered > 0, "RETRIEVE stats.candidates_considered should be > 0, got {}", stats.candidates_considered ); assert!( stats.total_time_us > 0, "RETRIEVE stats.total_time_us should be > 0, got {}", stats.total_time_us ); assert!( stats.scoring_time_us > 0, "RETRIEVE stats.scoring_time_us should be > 0, got {}", stats.scoring_time_us ); assert!( !stats.profile_name.is_empty(), "RETRIEVE stats.profile_name should not be empty" ); assert_eq!( stats.profile_name, "trending", "RETRIEVE stats.profile_name should be 'trending'" ); // -- SEARCH stats -- let search_results = db .search( &Search::builder() .query("Stats Item") .limit(20) .build() .unwrap(), ) .unwrap(); let search_stats = &search_results.stats; assert!( search_stats.candidates_considered > 0, "SEARCH stats.candidates_considered should be > 0, got {}", search_stats.candidates_considered ); assert!( search_stats.total_time_us > 0, "SEARCH stats.total_time_us should be > 0, got {}", search_stats.total_time_us ); } ``` ### Test 8: Prometheus metrics content ```rust #[test] fn uat_metrics_content() { let db = TidalDb::builder() .ephemeral() .with_schema(m7_uat_schema()) .open() .unwrap(); let now = Timestamp::now(); // Write some data so metrics have non-zero values. for id in 1..=50u64 { let mut meta = HashMap::new(); meta.insert("title".to_string(), format!("Metrics Item {id}")); db.write_item_with_metadata(EntityId::new(id), &meta).unwrap(); } for id in 1..=20u64 { db.signal("view", EntityId::new(id), 1.0, now).unwrap(); } // Render Prometheus text metrics. let metrics_text = db.render_metrics(); // Verify the output is non-empty. assert!( !metrics_text.is_empty(), "metrics output should not be empty" ); // Verify expected metric names are present. let expected_metrics = [ "tidaldb_signal_hot_entries", "tidaldb_signal_writes_total", ]; for metric_name in &expected_metrics { assert!( metrics_text.contains(metric_name), "metrics output should contain '{metric_name}', got:\n{metrics_text}" ); } // Verify Prometheus text format: lines starting with # are comments/HELP/TYPE, // data lines are "metric_name{labels} value" or "metric_name value". for line in metrics_text.lines() { if line.is_empty() || line.starts_with('#') { continue; } // Data line: should have at least a metric name and a numeric value. let parts: Vec<&str> = line.split_whitespace().collect(); assert!( parts.len() >= 2, "malformed Prometheus line (expected 'name value'): {line}" ); // The last token should be parseable as a number. let value_str = parts.last().unwrap(); assert!( value_str.parse::().is_ok(), "Prometheus value should be numeric, got '{value_str}' in line: {line}" ); } } ``` ### Test 9: RLHF export + cross-session aggregation ```rust #[test] fn uat_export_and_session_aggregation() { let mut builder = SchemaBuilder::new(); for &(name, half_life_days) in &[("view", 7), ("like", 14), ("skip", 1)] { let _ = builder .signal( name, EntityKind::Item, DecaySpec::Exponential { half_life: Duration::from_secs(half_life_days * 24 * 3600), }, ) .windows(&[Window::AllTime]) .velocity(false) .add(); } builder.session_policy( "default", tidaldb::schema::AgentPolicy::builder() .max_session_duration(Duration::from_secs(300)) .build(), ); let schema = builder.build().unwrap(); let db = TidalDb::builder() .ephemeral() .with_schema(schema) .open() .unwrap(); let now = Timestamp::now(); // Write items. for id in 1..=10u64 { let mut meta = HashMap::new(); meta.insert("title".to_string(), format!("Export Item {id}")); db.write_item_with_metadata(EntityId::new(id), &meta).unwrap(); } let user_id = 42u64; // -- Session 1: 5 views + 2 likes -- let session_1 = db .start_session(user_id, "agent_export", "default", HashMap::new()) .unwrap(); for id in 1..=5u64 { db.session_signal(&session_1, "view", EntityId::new(id), 1.0, now, None) .unwrap(); } db.session_signal(&session_1, "like", EntityId::new(1), 1.0, now, Some("loved it".into())) .unwrap(); db.session_signal(&session_1, "like", EntityId::new(2), 1.0, now, None) .unwrap(); let summary_1 = db.close_session(session_1).unwrap(); assert_eq!(summary_1.signals_written, 7, "session 1 should have 7 signals"); // -- Session 2: 3 views + 1 skip -- let session_2 = db .start_session(user_id, "agent_export", "default", HashMap::new()) .unwrap(); for id in 6..=8u64 { db.session_signal(&session_2, "view", EntityId::new(id), 1.0, now, None) .unwrap(); } db.session_signal(&session_2, "skip", EntityId::new(9), 1.0, now, None) .unwrap(); let summary_2 = db.close_session(session_2).unwrap(); assert_eq!(summary_2.signals_written, 4, "session 2 should have 4 signals"); // -- Export signals -- let exported = db .export_signals(tidaldb::ExportRequest { user_id: Some(user_id), signal_types: vec!["view".into(), "like".into(), "skip".into()], since: Timestamp::from_nanos(0), until: Timestamp::from_nanos(u64::MAX), format: tidaldb::ExportFormat::JsonLines, }) .unwrap(); // Should have at least 11 exported events (7 from session 1 + 4 from session 2). assert!( exported.len() >= 11, "export should return at least 11 signal events, got {}", exported.len() ); // Verify the annotation survived export. let annotated = exported .iter() .filter(|e| e.annotation.is_some()) .count(); assert!( annotated >= 1, "at least one exported event should have an annotation" ); // Verify signal types are correct. let view_count = exported.iter().filter(|e| e.signal_type == "view").count(); let like_count = exported.iter().filter(|e| e.signal_type == "like").count(); let skip_count = exported.iter().filter(|e| e.signal_type == "skip").count(); assert!(view_count >= 8, "should have at least 8 view events, got {view_count}"); assert!(like_count >= 2, "should have at least 2 like events, got {like_count}"); assert!(skip_count >= 1, "should have at least 1 skip event, got {skip_count}"); // -- Cross-session aggregation -- let summary = db .user_session_summary(user_id, Timestamp::from_nanos(0)) .unwrap(); assert_eq!( summary.sessions_count, 2, "user should have 2 closed sessions" ); assert_eq!( summary.total_signals, 11, "total signals across sessions should be 11" ); assert_eq!( summary.total_rejections, 0, "no policy rejections should have occurred" ); // top_signal_types should include "view" as the most frequent. assert!( !summary.top_signal_types.is_empty(), "top_signal_types should not be empty" ); let top_type = &summary.top_signal_types[0]; assert_eq!( top_type.0, "view", "most frequent signal type should be 'view', got '{}'", top_type.0 ); assert!( top_type.1 >= 8, "view count should be at least 8, got {}", top_type.1 ); } ``` ### Imports needed (additions to the shared header) ```rust use tidaldb::{ExportRequest, ExportFormat}; // From m7p4 ``` ### Note on `render_metrics` The test calls `db.render_metrics()` which is the m7p4 API for producing Prometheus text. If the actual API name differs (e.g., `db.metrics_text()` or requires the `metrics` feature), the test should be gated accordingly: ```rust #[test] #[cfg(feature = "metrics")] fn uat_metrics_content() { ... } ``` The task document uses `render_metrics()` as the expected name per the m7p4 spec. The implementer should match whatever m7p4 actually delivers. ## Acceptance Criteria - [ ] `uat_query_stats_populated` passes: both RETRIEVE and SEARCH return `stats` with non-zero `candidates_considered`, `total_time_us`, `scoring_time_us`; RETRIEVE `profile_name == "trending"` - [ ] `uat_metrics_content` passes: Prometheus text output contains `tidaldb_signal_hot_entries` and `tidaldb_signal_writes_total`; all data lines have numeric values; output is non-empty - [ ] `uat_export_and_session_aggregation` passes: export returns >= 11 events with correct signal type distribution; annotation preserved; `user_session_summary` returns `sessions_count == 2`, `total_signals == 11`, `total_rejections == 0`; `top_signal_types[0]` is `("view", >= 8)` - [ ] Each test completes in under 10 seconds (no wall-clock waits) - [ ] `cargo clippy --manifest-path tidal/Cargo.toml -- -D warnings` passes ## Test Strategy **QueryStats test** validates the instrumentation wiring, not the exact values. The assertions check `> 0` rather than specific numbers because execution time varies by machine. The `profile_name` assertion verifies the stats carry semantic context. **Metrics test** validates Prometheus text format compliance at a basic level: lines are either comments (`#`) or data lines with `name value` structure where value is numeric. The test does not exhaustively check every metric defined in m7p4 -- it verifies the two most important signal system metrics to confirm the pipeline is wired correctly. Additional metrics can be asserted as the m7p4 implementation stabilizes. **Export + aggregation test** writes a deliberate signal pattern across two sessions with known types and counts, then verifies the export and aggregation APIs return matching numbers. The annotation on one `like` signal verifies that annotations flow through the WAL and into the export path. The `user_session_summary` is tested with `since: 0` to capture all historical sessions.