This commit implements comprehensive production hardening across multiple layers to prepare StemeDB for enterprise pilot deployments: ## API Layer - Add rate limiting middleware with configurable limits per endpoint - Enhance error handling with detailed context and proper HTTP status codes - Add security hardening tests for input validation and boundary conditions - Create store_helpers module for defensive storage access patterns ## Storage & WAL - Optimize group commit batching for higher throughput - Add defensive error handling in hybrid backend with proper fallbacks - Enhance WAL journal durability guarantees with fsync validation - Improve index store query performance with better caching ## Operations & Deployment - Add comprehensive operations documentation (deployment, monitoring, DR) - Create systemd units for backup, WAL archival, and verification - Add monitoring configs (Prometheus alerts, metrics exporters) - Implement backup/restore scripts with verification and S3 archival - Add DR drill automation and runbook procedures - Create load balancer configs (nginx, envoy) with health checks ## Documentation - Update CLAUDE.md with operations and troubleshooting guides - Expand roadmap with production readiness milestones - Add pilot success criteria and deployment reference architecture - Document TLS setup, monitoring integration, and incident response ## Configuration - Add .env.example with all required environment variables - Document resource sizing for different deployment scales - Add configuration examples for various deployment topologies This positions StemeDB for successful enterprise pilots with proper operational discipline, monitoring, backup/DR, and security hardening. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
8.0 KiB
TLS/HTTPS Setup Guide
This guide covers setting up TLS/HTTPS for StemeDB API server in production.
Overview
StemeDB supports TLS 1.3 for encrypted communication. When TLS is enabled:
- All traffic is encrypted using TLS 1.3 (TLS 1.2 and below are disabled)
- Server listens on HTTPS instead of HTTP
- Self-signed certificates work for development
- Let's Encrypt certificates are recommended for production
Prerequisites
- A domain name pointing to your server (for Let's Encrypt)
- Root or sudo access to install certbot
- Ports 80 and 443 accessible from the internet
Quick Start (Let's Encrypt)
1. Install Certbot
Ubuntu/Debian:
sudo apt update
sudo apt install certbot
RHEL/CentOS:
sudo yum install certbot
macOS:
brew install certbot
2. Obtain Certificate
Standalone mode (stops existing web servers):
sudo certbot certonly --standalone -d stemedb.example.com
Webroot mode (if you have a web server running):
sudo certbot certonly --webroot -w /var/www/html -d stemedb.example.com
Certificates will be stored at:
- Certificate:
/etc/letsencrypt/live/stemedb.example.com/fullchain.pem - Private Key:
/etc/letsencrypt/live/stemedb.example.com/privkey.pem
3. Configure StemeDB
Set environment variables:
export STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
export STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
export STEMEDB_BIND_ADDR=0.0.0.0:443
Or add to .env file:
STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
STEMEDB_BIND_ADDR=0.0.0.0:443
4. Start Server
# If running as systemd service:
sudo systemctl start stemedb-api
# Or run directly:
sudo ./target/release/stemedb-api
Note: Port 443 requires root/sudo privileges. Use sudo or configure the binary with setcap:
sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/stemedb-api
5. Verify HTTPS
curl https://stemedb.example.com/v1/health
Expected response:
{
"status": "healthy",
"version": "0.1.0"
}
Self-Signed Certificates (Development)
For local development or testing without a domain name:
1. Generate Self-Signed Certificate
openssl req -x509 -newkey rsa:4096 \
-keyout key.pem -out cert.pem \
-days 365 -nodes \
-subj "/CN=localhost"
This creates:
cert.pem- Self-signed certificatekey.pem- Private key
2. Configure StemeDB
export STEMEDB_TLS_CERT_PATH=./cert.pem
export STEMEDB_TLS_KEY_PATH=./key.pem
export STEMEDB_BIND_ADDR=127.0.0.1:443
3. Test with Curl
# Accept self-signed cert with -k flag:
curl -k https://localhost:443/v1/health
4. Import Certificate (Optional)
To avoid -k flag, import the certificate:
macOS:
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain cert.pem
Linux:
sudo cp cert.pem /usr/local/share/ca-certificates/stemedb.crt
sudo update-ca-certificates
Certificate Renewal (Let's Encrypt)
Let's Encrypt certificates expire after 90 days. Certbot can auto-renew them.
Setup Auto-Renewal
Test renewal:
sudo certbot renew --dry-run
Add cron job (runs twice daily):
sudo crontab -e
Add line:
0 0,12 * * * certbot renew --quiet --deploy-hook "systemctl reload stemedb-api"
Manual Renewal
sudo certbot renew
sudo systemctl reload stemedb-api
Important: StemeDB needs to be reloaded/restarted after certificate renewal to pick up the new certificate.
Systemd Service Integration
Create Service File
/etc/systemd/system/stemedb-api.service:
[Unit]
Description=StemeDB API Server
After=network.target
[Service]
Type=simple
User=stemedb
Group=stemedb
WorkingDirectory=/opt/stemedb
EnvironmentFile=/opt/stemedb/.env
ExecStart=/opt/stemedb/stemedb-api
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/stemedb/data
[Install]
WantedBy=multi-user.target
Configure Permissions
Let's Encrypt certificates are owned by root. Grant read access to stemedb user:
# Create stemedb user
sudo useradd -r -s /bin/false stemedb
# Grant read access to certificates
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/live
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/archive
Enable and Start
sudo systemctl daemon-reload
sudo systemctl enable stemedb-api
sudo systemctl start stemedb-api
sudo systemctl status stemedb-api
Reverse Proxy with Nginx (Alternative)
Instead of running StemeDB with TLS directly, you can use Nginx as a TLS termination proxy.
Nginx Configuration
/etc/nginx/sites-available/stemedb:
server {
listen 443 ssl http2;
server_name stemedb.example.com;
# TLS Configuration
ssl_certificate /etc/letsencrypt/live/stemedb.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/stemedb.example.com/privkey.pem;
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers off;
# Proxy to StemeDB (running on localhost:18180 without TLS)
location / {
proxy_pass http://127.0.0.1:18180;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
}
}
# Redirect HTTP to HTTPS
server {
listen 80;
server_name stemedb.example.com;
return 301 https://$server_name$request_uri;
}
Enable and reload:
sudo ln -s /etc/nginx/sites-available/stemedb /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
Troubleshooting
Server Won't Start
Check certificate paths:
ls -la $STEMEDB_TLS_CERT_PATH
ls -la $STEMEDB_TLS_KEY_PATH
Verify permissions:
sudo -u stemedb cat $STEMEDB_TLS_CERT_PATH > /dev/null
If permission denied, grant access:
sudo setfacl -m u:stemedb:r $STEMEDB_TLS_CERT_PATH
sudo setfacl -m u:stemedb:r $STEMEDB_TLS_KEY_PATH
Check logs:
sudo journalctl -u stemedb-api -f
Certificate Expired
sudo certbot renew --force-renewal
sudo systemctl reload stemedb-api
Clients Can't Connect
Check firewall:
sudo ufw status
sudo ufw allow 443/tcp
Verify DNS:
dig stemedb.example.com
Test from external host:
curl -v https://stemedb.example.com/v1/health
TLS Handshake Failures
Check TLS version:
openssl s_client -connect stemedb.example.com:443 -tls1_3
If connection fails, client may not support TLS 1.3. Verify client TLS support:
curl --tlsv1.3 https://stemedb.example.com/v1/health
Security Best Practices
-
Use Strong Certificates
- Let's Encrypt certificates are free and automatically renew
- Minimum 2048-bit RSA keys (4096-bit recommended)
-
Keep Certificates Updated
- Set up auto-renewal
- Monitor expiration dates
- Test renewal process regularly
-
Restrict Private Key Access
- Private key should be readable only by stemedb user and root
- Never commit private keys to version control
-
Use HTTPS Everywhere
- Redirect all HTTP traffic to HTTPS
- Use HSTS headers to force HTTPS
-
Monitor Certificate Expiration
- Set up alerts for certificate expiration (30 days before)
- Test renewal process monthly
-
Audit TLS Configuration
- Use SSL Labs to test configuration
- Aim for A+ rating