stemedb/docs/operations/deployment/tls-setup.md
jml 3e7eddc074 feat: add enterprise production readiness infrastructure
This commit implements comprehensive production hardening across multiple
layers to prepare StemeDB for enterprise pilot deployments:

## API Layer
- Add rate limiting middleware with configurable limits per endpoint
- Enhance error handling with detailed context and proper HTTP status codes
- Add security hardening tests for input validation and boundary conditions
- Create store_helpers module for defensive storage access patterns

## Storage & WAL
- Optimize group commit batching for higher throughput
- Add defensive error handling in hybrid backend with proper fallbacks
- Enhance WAL journal durability guarantees with fsync validation
- Improve index store query performance with better caching

## Operations & Deployment
- Add comprehensive operations documentation (deployment, monitoring, DR)
- Create systemd units for backup, WAL archival, and verification
- Add monitoring configs (Prometheus alerts, metrics exporters)
- Implement backup/restore scripts with verification and S3 archival
- Add DR drill automation and runbook procedures
- Create load balancer configs (nginx, envoy) with health checks

## Documentation
- Update CLAUDE.md with operations and troubleshooting guides
- Expand roadmap with production readiness milestones
- Add pilot success criteria and deployment reference architecture
- Document TLS setup, monitoring integration, and incident response

## Configuration
- Add .env.example with all required environment variables
- Document resource sizing for different deployment scales
- Add configuration examples for various deployment topologies

This positions StemeDB for successful enterprise pilots with proper
operational discipline, monitoring, backup/DR, and security hardening.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-12 06:08:15 +00:00

381 lines
8.0 KiB
Markdown

# TLS/HTTPS Setup Guide
This guide covers setting up TLS/HTTPS for StemeDB API server in production.
## Overview
StemeDB supports TLS 1.3 for encrypted communication. When TLS is enabled:
- All traffic is encrypted using TLS 1.3 (TLS 1.2 and below are disabled)
- Server listens on HTTPS instead of HTTP
- Self-signed certificates work for development
- Let's Encrypt certificates are recommended for production
## Prerequisites
- A domain name pointing to your server (for Let's Encrypt)
- Root or sudo access to install certbot
- Ports 80 and 443 accessible from the internet
## Quick Start (Let's Encrypt)
### 1. Install Certbot
**Ubuntu/Debian:**
```bash
sudo apt update
sudo apt install certbot
```
**RHEL/CentOS:**
```bash
sudo yum install certbot
```
**macOS:**
```bash
brew install certbot
```
### 2. Obtain Certificate
**Standalone mode** (stops existing web servers):
```bash
sudo certbot certonly --standalone -d stemedb.example.com
```
**Webroot mode** (if you have a web server running):
```bash
sudo certbot certonly --webroot -w /var/www/html -d stemedb.example.com
```
Certificates will be stored at:
- **Certificate:** `/etc/letsencrypt/live/stemedb.example.com/fullchain.pem`
- **Private Key:** `/etc/letsencrypt/live/stemedb.example.com/privkey.pem`
### 3. Configure StemeDB
Set environment variables:
```bash
export STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
export STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
export STEMEDB_BIND_ADDR=0.0.0.0:443
```
Or add to `.env` file:
```bash
STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
STEMEDB_BIND_ADDR=0.0.0.0:443
```
### 4. Start Server
```bash
# If running as systemd service:
sudo systemctl start stemedb-api
# Or run directly:
sudo ./target/release/stemedb-api
```
**Note:** Port 443 requires root/sudo privileges. Use `sudo` or configure the binary with `setcap`:
```bash
sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/stemedb-api
```
### 5. Verify HTTPS
```bash
curl https://stemedb.example.com/v1/health
```
Expected response:
```json
{
"status": "healthy",
"version": "0.1.0"
}
```
## Self-Signed Certificates (Development)
For local development or testing without a domain name:
### 1. Generate Self-Signed Certificate
```bash
openssl req -x509 -newkey rsa:4096 \
-keyout key.pem -out cert.pem \
-days 365 -nodes \
-subj "/CN=localhost"
```
This creates:
- `cert.pem` - Self-signed certificate
- `key.pem` - Private key
### 2. Configure StemeDB
```bash
export STEMEDB_TLS_CERT_PATH=./cert.pem
export STEMEDB_TLS_KEY_PATH=./key.pem
export STEMEDB_BIND_ADDR=127.0.0.1:443
```
### 3. Test with Curl
```bash
# Accept self-signed cert with -k flag:
curl -k https://localhost:443/v1/health
```
### 4. Import Certificate (Optional)
To avoid `-k` flag, import the certificate:
**macOS:**
```bash
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain cert.pem
```
**Linux:**
```bash
sudo cp cert.pem /usr/local/share/ca-certificates/stemedb.crt
sudo update-ca-certificates
```
## Certificate Renewal (Let's Encrypt)
Let's Encrypt certificates expire after 90 days. Certbot can auto-renew them.
### Setup Auto-Renewal
**Test renewal:**
```bash
sudo certbot renew --dry-run
```
**Add cron job** (runs twice daily):
```bash
sudo crontab -e
```
Add line:
```
0 0,12 * * * certbot renew --quiet --deploy-hook "systemctl reload stemedb-api"
```
### Manual Renewal
```bash
sudo certbot renew
sudo systemctl reload stemedb-api
```
**Important:** StemeDB needs to be reloaded/restarted after certificate renewal to pick up the new certificate.
## Systemd Service Integration
### Create Service File
`/etc/systemd/system/stemedb-api.service`:
```ini
[Unit]
Description=StemeDB API Server
After=network.target
[Service]
Type=simple
User=stemedb
Group=stemedb
WorkingDirectory=/opt/stemedb
EnvironmentFile=/opt/stemedb/.env
ExecStart=/opt/stemedb/stemedb-api
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/stemedb/data
[Install]
WantedBy=multi-user.target
```
### Configure Permissions
Let's Encrypt certificates are owned by root. Grant read access to stemedb user:
```bash
# Create stemedb user
sudo useradd -r -s /bin/false stemedb
# Grant read access to certificates
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/live
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/archive
```
### Enable and Start
```bash
sudo systemctl daemon-reload
sudo systemctl enable stemedb-api
sudo systemctl start stemedb-api
sudo systemctl status stemedb-api
```
## Reverse Proxy with Nginx (Alternative)
Instead of running StemeDB with TLS directly, you can use Nginx as a TLS termination proxy.
### Nginx Configuration
`/etc/nginx/sites-available/stemedb`:
```nginx
server {
listen 443 ssl http2;
server_name stemedb.example.com;
# TLS Configuration
ssl_certificate /etc/letsencrypt/live/stemedb.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/stemedb.example.com/privkey.pem;
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers off;
# Proxy to StemeDB (running on localhost:18180 without TLS)
location / {
proxy_pass http://127.0.0.1:18180;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
}
}
# Redirect HTTP to HTTPS
server {
listen 80;
server_name stemedb.example.com;
return 301 https://$server_name$request_uri;
}
```
Enable and reload:
```bash
sudo ln -s /etc/nginx/sites-available/stemedb /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```
## Troubleshooting
### Server Won't Start
**Check certificate paths:**
```bash
ls -la $STEMEDB_TLS_CERT_PATH
ls -la $STEMEDB_TLS_KEY_PATH
```
**Verify permissions:**
```bash
sudo -u stemedb cat $STEMEDB_TLS_CERT_PATH > /dev/null
```
If permission denied, grant access:
```bash
sudo setfacl -m u:stemedb:r $STEMEDB_TLS_CERT_PATH
sudo setfacl -m u:stemedb:r $STEMEDB_TLS_KEY_PATH
```
**Check logs:**
```bash
sudo journalctl -u stemedb-api -f
```
### Certificate Expired
```bash
sudo certbot renew --force-renewal
sudo systemctl reload stemedb-api
```
### Clients Can't Connect
**Check firewall:**
```bash
sudo ufw status
sudo ufw allow 443/tcp
```
**Verify DNS:**
```bash
dig stemedb.example.com
```
**Test from external host:**
```bash
curl -v https://stemedb.example.com/v1/health
```
### TLS Handshake Failures
**Check TLS version:**
```bash
openssl s_client -connect stemedb.example.com:443 -tls1_3
```
If connection fails, client may not support TLS 1.3. Verify client TLS support:
```bash
curl --tlsv1.3 https://stemedb.example.com/v1/health
```
## Security Best Practices
1. **Use Strong Certificates**
- Let's Encrypt certificates are free and automatically renew
- Minimum 2048-bit RSA keys (4096-bit recommended)
2. **Keep Certificates Updated**
- Set up auto-renewal
- Monitor expiration dates
- Test renewal process regularly
3. **Restrict Private Key Access**
- Private key should be readable only by stemedb user and root
- Never commit private keys to version control
4. **Use HTTPS Everywhere**
- Redirect all HTTP traffic to HTTPS
- Use HSTS headers to force HTTPS
5. **Monitor Certificate Expiration**
- Set up alerts for certificate expiration (30 days before)
- Test renewal process monthly
6. **Audit TLS Configuration**
- Use [SSL Labs](https://www.ssllabs.com/ssltest/) to test configuration
- Aim for A+ rating
## See Also
- [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
- [Certbot User Guide](https://eff-certbot.readthedocs.io/)
- [Mozilla SSL Configuration Generator](https://ssl-config.mozilla.org/)
- [StemeDB Operations Guide](../README.md)