jx12n 4d9a26498e hush: one-time secret links the server cannot read

Paste a secret, get a link, send it. The first person to open it and press
Reveal sees the secret; the link dies at that moment. The recipient needs a
browser and nothing else — no account, no client, no installed tooling.

The server cannot read what it stores. AES-256-GCM happens in the browser and
the key lives in the URL fragment, which browsers never transmit, so hushd
holds ciphertext and no key material. That is a property of where the key sits
rather than a promise about our conduct, which is why there is deliberately no
endpoint accepting a plaintext secret and no server-side-encryption fallback:
two guarantees behind one URL would be worse than one honest guarantee.

Three decisions carry the design:

  * GET /s/{id} touches NO storage, not even to check existence. Slack, Teams,
    WhatsApp, iMessage and Outlook Safe Links all fetch a URL before a human
    sees it, so destroying on GET would destroy most secrets in transit and the
    recipient's "already used" would be indistinguishable from interception.
    Only POST /reveal consumes. Bot user-agent detection is an arms race;
    removing the side effect from GET is not. Pinned by
    TestGettingTheRevealPageNeverConsumesTheSecret.
  * Destruction is one Redis GETDEL, which is atomic. GET-then-DEL has a window
    where two simultaneous readers both win, and for a one-time secret that
    window is the product. The store contract demands atomicity and the same
    concurrency test runs against both implementations.
  * Missing, already-revealed, expired and evicted are ONE indistinguishable
    410. Separating them would confirm to a prober that a given link was real.

The secret id IS the capability, so secret.ID is a struct whose every
accidental path — %v, %s, String(), slog, json.Marshal — emits a redacted
handle or refuses, and the raw value needs an explicit Value(). The first
version tried to prevent leaks by implementing no String() at all; its own test
caught that Go's fmt prints unexported fields anyway, so forbidding the method
had removed the control rather than the leak.

Operationally: structured JSON on stdout in the fleet's wire format, which
Vector already collects with no annotation; six hush_* metrics on the chassis
registry with no id, IP or path in any label; five alert rules wired into
vmalert. The public Ingress enumerates /, /s/ and /api/ so /metrics, /healthz
and /readyz share the port but are unreachable from the internet — no
basic-auth middleware to maintain and get wrong.

Dependencies are vendored because go-chassis is private: the Woodpecker test
step and the in-cluster Kaniko build both run -mod=vendor with GOPROXY=off and
hold no git credential.

cmd/hush-mcp is a stdio MCP server doing the same client-side crypto locally,
so using hush from an agent preserves the same guarantee as using it from a
browser.

2026-09-03 00:08:38 -06:00

9.2 KiB

Raw Blame History

Maintenance Notifications - FEATURES

Overview

The Maintenance Notifications feature enables seamless Redis connection handoffs during cluster maintenance operations without dropping active connections. This feature leverages Redis RESP3 push notifications to provide zero-downtime maintenance for Redis Enterprise and compatible Redis deployments.

Important

Using Maintenance Notifications may affect the read and write timeouts by relaxing them during maintenance operations. This is necessary to prevent false failures due to increased latency during handoffs. The relaxed timeouts are automatically applied and removed as needed.

Key Features

Seamless Connection Handoffs

Zero-Downtime Maintenance: Automatically handles connection transitions during cluster operations
Active Operation Preservation: Transfers in-flight operations to new connections without interruption
Graceful Degradation: Falls back to standard reconnection if handoff fails

Push Notification Support

Supports all Redis Enterprise maintenance notification types:

MOVING - Slot moving to a new node
MIGRATING - Slot in migration state
MIGRATED - Migration completed
FAILING_OVER - Node failing over
FAILED_OVER - Failover completed

Circuit Breaker Pattern

Endpoint-Specific Failure Tracking: Prevents repeated connection attempts to failing endpoints
Automatic Recovery Testing: Half-open state allows gradual recovery validation
Configurable Thresholds: Customize failure thresholds and reset timeouts

Flexible Configuration

Auto-Detection Mode: Automatically detects server support for maintenance notifications
Multiple Endpoint Types: Support for internal/external IP/FQDN endpoint resolution
Auto-Scaling Workers: Automatically sizes worker pool based on connection pool size
Timeout Management: Separate timeouts for relaxed (during maintenance) and normal operations

Extensible Hook System

Pre/Post Processing Hooks: Monitor and customize notification handling
Built-in Hooks: Logging and metrics collection hooks included
Custom Hook Support: Implement custom business logic around maintenance events

Comprehensive Monitoring

Metrics Collection: Track notification counts, processing times, and error rates
Circuit Breaker Stats: Monitor endpoint health and circuit breaker states
Operation Tracking: Track active handoff operations and their lifecycle

Architecture Highlights

Event-Driven Handoff System

Asynchronous Processing: Non-blocking handoff operations using worker pool pattern
Queue-Based Architecture: Configurable queue size with auto-scaling support
Retry Mechanism: Configurable retry attempts with exponential backoff

Connection Pool Integration

Pool Hook Interface: Seamless integration with go-redis connection pool
Connection State Management: Atomic flags for connection usability tracking
Graceful Shutdown: Ensures all in-flight handoffs complete before shutdown

Thread-Safe Design

Lock-Free Operations: Atomic operations for high-performance state tracking
Concurrent-Safe Maps: sync.Map for tracking active operations
Minimal Lock Contention: Read-write locks only where necessary

Configuration Options

Operation Modes

ModeDisabled: Maintenance notifications completely disabled
ModeEnabled: Forcefully enabled (fails if server doesn't support)
ModeAuto: Auto-detect server support (recommended default)

Endpoint Types

EndpointTypeAuto: Auto-detect based on current connection
EndpointTypeInternalIP: Use internal IP addresses
EndpointTypeInternalFQDN: Use internal fully qualified domain names
EndpointTypeExternalIP: Use external IP addresses
EndpointTypeExternalFQDN: Use external fully qualified domain names
EndpointTypeNone: No endpoint (reconnect with current configuration)

Timeout Configuration

RelaxedTimeout: Extended timeout during maintenance operations (default: 10s)
HandoffTimeout: Maximum time for handoff completion (default: 15s)
PostHandoffRelaxedDuration: Relaxed period after handoff (default: 2×RelaxedTimeout)

Worker Pool Configuration

MaxWorkers: Maximum concurrent handoff workers (auto-calculated if 0)
HandoffQueueSize: Handoff queue capacity (auto-calculated if 0)
MaxHandoffRetries: Maximum retry attempts for failed handoffs (default: 3)

Circuit Breaker Configuration

CircuitBreakerFailureThreshold: Failures before opening circuit (default: 5)
CircuitBreakerResetTimeout: Time before testing recovery (default: 60s)
CircuitBreakerMaxRequests: Max requests in half-open state (default: 3)

Auto-Scaling Formulas

Worker Pool Sizing

When MaxWorkers = 0 (auto-calculate):

MaxWorkers = min(PoolSize/2, max(10, PoolSize/3))

Queue Sizing

When HandoffQueueSize = 0 (auto-calculate):

QueueSize = max(20 × MaxWorkers, PoolSize)
Capped by: min(MaxActiveConns + 1, 5 × PoolSize)

Examples

Pool Size 100: 33 workers, 660 queue (capped at 500)
Pool Size 100 + MaxActiveConns 150: 33 workers, 151 queue
Pool Size 50: 16 workers, 320 queue (capped at 250)

Performance Characteristics

Throughput

Non-Blocking Handoffs: Client operations continue during handoffs
Concurrent Processing: Multiple handoffs processed in parallel
Minimal Overhead: Lock-free atomic operations for state tracking

Latency

Relaxed Timeouts: Extended timeouts during maintenance prevent false failures
Fast Path: Connections not undergoing handoff have zero overhead
Graceful Degradation: Failed handoffs fall back to standard reconnection

Resource Usage

Memory Efficient: Bounded queue sizes prevent memory exhaustion
Worker Pool: Fixed worker count prevents goroutine explosion
Connection Reuse: Handoff reuses existing connection objects

Testing

Unit Tests

Comprehensive unit test coverage for all components
Mock-based testing for isolation
Concurrent operation testing

Integration Tests

Pool integration tests with real connection handoffs
Circuit breaker behavior validation
Hook system integration testing

E2E Tests

Real Redis Enterprise cluster testing
Multiple scenario coverage (timeouts, endpoint types, stress tests)
Fault injection testing
TLS configuration testing

Compatibility

Requirements

Redis Protocol: RESP3 required for push notifications
Redis Version: Redis Enterprise or compatible Redis with maintenance notifications
Go Version: Go 1.18+ (uses generics and atomic types)

Client Support

Currently Supported

Standalone Client (redis.NewClient) - Full support for MOVING, MIGRATING, MIGRATED, FAILING_OVER, FAILED_OVER notifications
Cluster Client (redis.NewClusterClient) - Support for SMIGRATING and SMIGRATED notifications for hitless slot migrations

Will Not Support

Failover Client (no planned support)
Ring Client (no planned support)

Migration Guide

Enabling Maintenance Notifications (Standalone Client)

Before:

client := redis.NewClient(&redis.Options{
    Addr:     "localhost:6379",
    Protocol: 2, // RESP2
})

After:

client := redis.NewClient(&redis.Options{
    Addr:     "localhost:6379",
    Protocol: 3, // RESP3 required
    MaintNotificationsConfig: &maintnotifications.Config{
        Mode: maintnotifications.ModeAuto,
    },
})

Enabling Hitless Upgrades (Cluster Client)

For Redis Cluster with hitless slot migration support:

client := redis.NewClusterClient(&redis.ClusterOptions{
    Addrs:    []string{"localhost:7000", "localhost:7001", "localhost:7002"},
    Protocol: 3, // RESP3 required for push notifications
    MaintNotificationsConfig: &maintnotifications.Config{
        Mode:           maintnotifications.ModeAuto,
        RelaxedTimeout: 10 * time.Second, // Extended timeout during slot migrations
    },
})

The cluster client automatically handles:

SMIGRATING: Relaxes timeouts when slots are being migrated
SMIGRATED: Triggers lazy cluster state reload when migration completes
SeqID Deduplication: Same notification from multiple nodes triggers only one reload

Adding Monitoring

// Get the manager from the client
manager := client.GetMaintNotificationsManager()
if manager != nil {
    // Add logging hook
    loggingHook := maintnotifications.NewLoggingHook(2) // Info level
    manager.AddNotificationHook(loggingHook)
    
    // Add metrics hook
    metricsHook := maintnotifications.NewMetricsHook()
    manager.AddNotificationHook(metricsHook)
}

Known Limitations

RESP3 Required: Push notifications require RESP3 protocol
Server Support: Requires Redis Enterprise or compatible Redis with maintenance notifications
Single Connection Commands: Some commands (MULTI/EXEC, WATCH) may need special handling
No Failover/Ring Client Support: Failover and Ring clients are not supported and there are no plans to add support

Future Enhancements

Enhanced metrics and observability
TTL-based cleanup for SeqID deduplication map

9.2 KiB Raw Blame History Unescape Escape