stemedb/docs/operations/deployment/tls-setup.md
jml 3e7eddc074 feat: add enterprise production readiness infrastructure
This commit implements comprehensive production hardening across multiple
layers to prepare StemeDB for enterprise pilot deployments:

## API Layer
- Add rate limiting middleware with configurable limits per endpoint
- Enhance error handling with detailed context and proper HTTP status codes
- Add security hardening tests for input validation and boundary conditions
- Create store_helpers module for defensive storage access patterns

## Storage & WAL
- Optimize group commit batching for higher throughput
- Add defensive error handling in hybrid backend with proper fallbacks
- Enhance WAL journal durability guarantees with fsync validation
- Improve index store query performance with better caching

## Operations & Deployment
- Add comprehensive operations documentation (deployment, monitoring, DR)
- Create systemd units for backup, WAL archival, and verification
- Add monitoring configs (Prometheus alerts, metrics exporters)
- Implement backup/restore scripts with verification and S3 archival
- Add DR drill automation and runbook procedures
- Create load balancer configs (nginx, envoy) with health checks

## Documentation
- Update CLAUDE.md with operations and troubleshooting guides
- Expand roadmap with production readiness milestones
- Add pilot success criteria and deployment reference architecture
- Document TLS setup, monitoring integration, and incident response

## Configuration
- Add .env.example with all required environment variables
- Document resource sizing for different deployment scales
- Add configuration examples for various deployment topologies

This positions StemeDB for successful enterprise pilots with proper
operational discipline, monitoring, backup/DR, and security hardening.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-12 06:08:15 +00:00

8.0 KiB

TLS/HTTPS Setup Guide

This guide covers setting up TLS/HTTPS for StemeDB API server in production.

Overview

StemeDB supports TLS 1.3 for encrypted communication. When TLS is enabled:

  • All traffic is encrypted using TLS 1.3 (TLS 1.2 and below are disabled)
  • Server listens on HTTPS instead of HTTP
  • Self-signed certificates work for development
  • Let's Encrypt certificates are recommended for production

Prerequisites

  • A domain name pointing to your server (for Let's Encrypt)
  • Root or sudo access to install certbot
  • Ports 80 and 443 accessible from the internet

Quick Start (Let's Encrypt)

1. Install Certbot

Ubuntu/Debian:

sudo apt update
sudo apt install certbot

RHEL/CentOS:

sudo yum install certbot

macOS:

brew install certbot

2. Obtain Certificate

Standalone mode (stops existing web servers):

sudo certbot certonly --standalone -d stemedb.example.com

Webroot mode (if you have a web server running):

sudo certbot certonly --webroot -w /var/www/html -d stemedb.example.com

Certificates will be stored at:

  • Certificate: /etc/letsencrypt/live/stemedb.example.com/fullchain.pem
  • Private Key: /etc/letsencrypt/live/stemedb.example.com/privkey.pem

3. Configure StemeDB

Set environment variables:

export STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
export STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
export STEMEDB_BIND_ADDR=0.0.0.0:443

Or add to .env file:

STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
STEMEDB_BIND_ADDR=0.0.0.0:443

4. Start Server

# If running as systemd service:
sudo systemctl start stemedb-api

# Or run directly:
sudo ./target/release/stemedb-api

Note: Port 443 requires root/sudo privileges. Use sudo or configure the binary with setcap:

sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/stemedb-api

5. Verify HTTPS

curl https://stemedb.example.com/v1/health

Expected response:

{
  "status": "healthy",
  "version": "0.1.0"
}

Self-Signed Certificates (Development)

For local development or testing without a domain name:

1. Generate Self-Signed Certificate

openssl req -x509 -newkey rsa:4096 \
  -keyout key.pem -out cert.pem \
  -days 365 -nodes \
  -subj "/CN=localhost"

This creates:

  • cert.pem - Self-signed certificate
  • key.pem - Private key

2. Configure StemeDB

export STEMEDB_TLS_CERT_PATH=./cert.pem
export STEMEDB_TLS_KEY_PATH=./key.pem
export STEMEDB_BIND_ADDR=127.0.0.1:443

3. Test with Curl

# Accept self-signed cert with -k flag:
curl -k https://localhost:443/v1/health

4. Import Certificate (Optional)

To avoid -k flag, import the certificate:

macOS:

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain cert.pem

Linux:

sudo cp cert.pem /usr/local/share/ca-certificates/stemedb.crt
sudo update-ca-certificates

Certificate Renewal (Let's Encrypt)

Let's Encrypt certificates expire after 90 days. Certbot can auto-renew them.

Setup Auto-Renewal

Test renewal:

sudo certbot renew --dry-run

Add cron job (runs twice daily):

sudo crontab -e

Add line:

0 0,12 * * * certbot renew --quiet --deploy-hook "systemctl reload stemedb-api"

Manual Renewal

sudo certbot renew
sudo systemctl reload stemedb-api

Important: StemeDB needs to be reloaded/restarted after certificate renewal to pick up the new certificate.

Systemd Service Integration

Create Service File

/etc/systemd/system/stemedb-api.service:

[Unit]
Description=StemeDB API Server
After=network.target

[Service]
Type=simple
User=stemedb
Group=stemedb
WorkingDirectory=/opt/stemedb
EnvironmentFile=/opt/stemedb/.env
ExecStart=/opt/stemedb/stemedb-api
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/stemedb/data

[Install]
WantedBy=multi-user.target

Configure Permissions

Let's Encrypt certificates are owned by root. Grant read access to stemedb user:

# Create stemedb user
sudo useradd -r -s /bin/false stemedb

# Grant read access to certificates
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/live
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/archive

Enable and Start

sudo systemctl daemon-reload
sudo systemctl enable stemedb-api
sudo systemctl start stemedb-api
sudo systemctl status stemedb-api

Reverse Proxy with Nginx (Alternative)

Instead of running StemeDB with TLS directly, you can use Nginx as a TLS termination proxy.

Nginx Configuration

/etc/nginx/sites-available/stemedb:

server {
    listen 443 ssl http2;
    server_name stemedb.example.com;

    # TLS Configuration
    ssl_certificate /etc/letsencrypt/live/stemedb.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/stemedb.example.com/privkey.pem;
    ssl_protocols TLSv1.3;
    ssl_prefer_server_ciphers off;

    # Proxy to StemeDB (running on localhost:18180 without TLS)
    location / {
        proxy_pass http://127.0.0.1:18180;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeouts
        proxy_connect_timeout 30s;
        proxy_send_timeout 30s;
        proxy_read_timeout 30s;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name stemedb.example.com;
    return 301 https://$server_name$request_uri;
}

Enable and reload:

sudo ln -s /etc/nginx/sites-available/stemedb /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Troubleshooting

Server Won't Start

Check certificate paths:

ls -la $STEMEDB_TLS_CERT_PATH
ls -la $STEMEDB_TLS_KEY_PATH

Verify permissions:

sudo -u stemedb cat $STEMEDB_TLS_CERT_PATH > /dev/null

If permission denied, grant access:

sudo setfacl -m u:stemedb:r $STEMEDB_TLS_CERT_PATH
sudo setfacl -m u:stemedb:r $STEMEDB_TLS_KEY_PATH

Check logs:

sudo journalctl -u stemedb-api -f

Certificate Expired

sudo certbot renew --force-renewal
sudo systemctl reload stemedb-api

Clients Can't Connect

Check firewall:

sudo ufw status
sudo ufw allow 443/tcp

Verify DNS:

dig stemedb.example.com

Test from external host:

curl -v https://stemedb.example.com/v1/health

TLS Handshake Failures

Check TLS version:

openssl s_client -connect stemedb.example.com:443 -tls1_3

If connection fails, client may not support TLS 1.3. Verify client TLS support:

curl --tlsv1.3 https://stemedb.example.com/v1/health

Security Best Practices

  1. Use Strong Certificates

    • Let's Encrypt certificates are free and automatically renew
    • Minimum 2048-bit RSA keys (4096-bit recommended)
  2. Keep Certificates Updated

    • Set up auto-renewal
    • Monitor expiration dates
    • Test renewal process regularly
  3. Restrict Private Key Access

    • Private key should be readable only by stemedb user and root
    • Never commit private keys to version control
  4. Use HTTPS Everywhere

    • Redirect all HTTP traffic to HTTPS
    • Use HSTS headers to force HTTPS
  5. Monitor Certificate Expiration

    • Set up alerts for certificate expiration (30 days before)
    • Test renewal process monthly
  6. Audit TLS Configuration

    • Use SSL Labs to test configuration
    • Aim for A+ rating

See Also