This commit implements comprehensive production hardening across multiple layers to prepare StemeDB for enterprise pilot deployments: ## API Layer - Add rate limiting middleware with configurable limits per endpoint - Enhance error handling with detailed context and proper HTTP status codes - Add security hardening tests for input validation and boundary conditions - Create store_helpers module for defensive storage access patterns ## Storage & WAL - Optimize group commit batching for higher throughput - Add defensive error handling in hybrid backend with proper fallbacks - Enhance WAL journal durability guarantees with fsync validation - Improve index store query performance with better caching ## Operations & Deployment - Add comprehensive operations documentation (deployment, monitoring, DR) - Create systemd units for backup, WAL archival, and verification - Add monitoring configs (Prometheus alerts, metrics exporters) - Implement backup/restore scripts with verification and S3 archival - Add DR drill automation and runbook procedures - Create load balancer configs (nginx, envoy) with health checks ## Documentation - Update CLAUDE.md with operations and troubleshooting guides - Expand roadmap with production readiness milestones - Add pilot success criteria and deployment reference architecture - Document TLS setup, monitoring integration, and incident response ## Configuration - Add .env.example with all required environment variables - Document resource sizing for different deployment scales - Add configuration examples for various deployment topologies This positions StemeDB for successful enterprise pilots with proper operational discipline, monitoring, backup/DR, and security hardening. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
381 lines
8.0 KiB
Markdown
381 lines
8.0 KiB
Markdown
# TLS/HTTPS Setup Guide
|
|
|
|
This guide covers setting up TLS/HTTPS for StemeDB API server in production.
|
|
|
|
## Overview
|
|
|
|
StemeDB supports TLS 1.3 for encrypted communication. When TLS is enabled:
|
|
- All traffic is encrypted using TLS 1.3 (TLS 1.2 and below are disabled)
|
|
- Server listens on HTTPS instead of HTTP
|
|
- Self-signed certificates work for development
|
|
- Let's Encrypt certificates are recommended for production
|
|
|
|
## Prerequisites
|
|
|
|
- A domain name pointing to your server (for Let's Encrypt)
|
|
- Root or sudo access to install certbot
|
|
- Ports 80 and 443 accessible from the internet
|
|
|
|
## Quick Start (Let's Encrypt)
|
|
|
|
### 1. Install Certbot
|
|
|
|
**Ubuntu/Debian:**
|
|
```bash
|
|
sudo apt update
|
|
sudo apt install certbot
|
|
```
|
|
|
|
**RHEL/CentOS:**
|
|
```bash
|
|
sudo yum install certbot
|
|
```
|
|
|
|
**macOS:**
|
|
```bash
|
|
brew install certbot
|
|
```
|
|
|
|
### 2. Obtain Certificate
|
|
|
|
**Standalone mode** (stops existing web servers):
|
|
```bash
|
|
sudo certbot certonly --standalone -d stemedb.example.com
|
|
```
|
|
|
|
**Webroot mode** (if you have a web server running):
|
|
```bash
|
|
sudo certbot certonly --webroot -w /var/www/html -d stemedb.example.com
|
|
```
|
|
|
|
Certificates will be stored at:
|
|
- **Certificate:** `/etc/letsencrypt/live/stemedb.example.com/fullchain.pem`
|
|
- **Private Key:** `/etc/letsencrypt/live/stemedb.example.com/privkey.pem`
|
|
|
|
### 3. Configure StemeDB
|
|
|
|
Set environment variables:
|
|
|
|
```bash
|
|
export STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
|
|
export STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
|
|
export STEMEDB_BIND_ADDR=0.0.0.0:443
|
|
```
|
|
|
|
Or add to `.env` file:
|
|
|
|
```bash
|
|
STEMEDB_TLS_CERT_PATH=/etc/letsencrypt/live/stemedb.example.com/fullchain.pem
|
|
STEMEDB_TLS_KEY_PATH=/etc/letsencrypt/live/stemedb.example.com/privkey.pem
|
|
STEMEDB_BIND_ADDR=0.0.0.0:443
|
|
```
|
|
|
|
### 4. Start Server
|
|
|
|
```bash
|
|
# If running as systemd service:
|
|
sudo systemctl start stemedb-api
|
|
|
|
# Or run directly:
|
|
sudo ./target/release/stemedb-api
|
|
```
|
|
|
|
**Note:** Port 443 requires root/sudo privileges. Use `sudo` or configure the binary with `setcap`:
|
|
|
|
```bash
|
|
sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/stemedb-api
|
|
```
|
|
|
|
### 5. Verify HTTPS
|
|
|
|
```bash
|
|
curl https://stemedb.example.com/v1/health
|
|
```
|
|
|
|
Expected response:
|
|
```json
|
|
{
|
|
"status": "healthy",
|
|
"version": "0.1.0"
|
|
}
|
|
```
|
|
|
|
## Self-Signed Certificates (Development)
|
|
|
|
For local development or testing without a domain name:
|
|
|
|
### 1. Generate Self-Signed Certificate
|
|
|
|
```bash
|
|
openssl req -x509 -newkey rsa:4096 \
|
|
-keyout key.pem -out cert.pem \
|
|
-days 365 -nodes \
|
|
-subj "/CN=localhost"
|
|
```
|
|
|
|
This creates:
|
|
- `cert.pem` - Self-signed certificate
|
|
- `key.pem` - Private key
|
|
|
|
### 2. Configure StemeDB
|
|
|
|
```bash
|
|
export STEMEDB_TLS_CERT_PATH=./cert.pem
|
|
export STEMEDB_TLS_KEY_PATH=./key.pem
|
|
export STEMEDB_BIND_ADDR=127.0.0.1:443
|
|
```
|
|
|
|
### 3. Test with Curl
|
|
|
|
```bash
|
|
# Accept self-signed cert with -k flag:
|
|
curl -k https://localhost:443/v1/health
|
|
```
|
|
|
|
### 4. Import Certificate (Optional)
|
|
|
|
To avoid `-k` flag, import the certificate:
|
|
|
|
**macOS:**
|
|
```bash
|
|
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain cert.pem
|
|
```
|
|
|
|
**Linux:**
|
|
```bash
|
|
sudo cp cert.pem /usr/local/share/ca-certificates/stemedb.crt
|
|
sudo update-ca-certificates
|
|
```
|
|
|
|
## Certificate Renewal (Let's Encrypt)
|
|
|
|
Let's Encrypt certificates expire after 90 days. Certbot can auto-renew them.
|
|
|
|
### Setup Auto-Renewal
|
|
|
|
**Test renewal:**
|
|
```bash
|
|
sudo certbot renew --dry-run
|
|
```
|
|
|
|
**Add cron job** (runs twice daily):
|
|
```bash
|
|
sudo crontab -e
|
|
```
|
|
|
|
Add line:
|
|
```
|
|
0 0,12 * * * certbot renew --quiet --deploy-hook "systemctl reload stemedb-api"
|
|
```
|
|
|
|
### Manual Renewal
|
|
|
|
```bash
|
|
sudo certbot renew
|
|
sudo systemctl reload stemedb-api
|
|
```
|
|
|
|
**Important:** StemeDB needs to be reloaded/restarted after certificate renewal to pick up the new certificate.
|
|
|
|
## Systemd Service Integration
|
|
|
|
### Create Service File
|
|
|
|
`/etc/systemd/system/stemedb-api.service`:
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=StemeDB API Server
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=stemedb
|
|
Group=stemedb
|
|
WorkingDirectory=/opt/stemedb
|
|
EnvironmentFile=/opt/stemedb/.env
|
|
ExecStart=/opt/stemedb/stemedb-api
|
|
ExecReload=/bin/kill -HUP $MAINPID
|
|
Restart=on-failure
|
|
RestartSec=5s
|
|
|
|
# Security hardening
|
|
NoNewPrivileges=true
|
|
PrivateTmp=true
|
|
ProtectSystem=strict
|
|
ProtectHome=true
|
|
ReadWritePaths=/opt/stemedb/data
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
### Configure Permissions
|
|
|
|
Let's Encrypt certificates are owned by root. Grant read access to stemedb user:
|
|
|
|
```bash
|
|
# Create stemedb user
|
|
sudo useradd -r -s /bin/false stemedb
|
|
|
|
# Grant read access to certificates
|
|
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/live
|
|
sudo setfacl -R -m u:stemedb:rX /etc/letsencrypt/archive
|
|
```
|
|
|
|
### Enable and Start
|
|
|
|
```bash
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable stemedb-api
|
|
sudo systemctl start stemedb-api
|
|
sudo systemctl status stemedb-api
|
|
```
|
|
|
|
## Reverse Proxy with Nginx (Alternative)
|
|
|
|
Instead of running StemeDB with TLS directly, you can use Nginx as a TLS termination proxy.
|
|
|
|
### Nginx Configuration
|
|
|
|
`/etc/nginx/sites-available/stemedb`:
|
|
|
|
```nginx
|
|
server {
|
|
listen 443 ssl http2;
|
|
server_name stemedb.example.com;
|
|
|
|
# TLS Configuration
|
|
ssl_certificate /etc/letsencrypt/live/stemedb.example.com/fullchain.pem;
|
|
ssl_certificate_key /etc/letsencrypt/live/stemedb.example.com/privkey.pem;
|
|
ssl_protocols TLSv1.3;
|
|
ssl_prefer_server_ciphers off;
|
|
|
|
# Proxy to StemeDB (running on localhost:18180 without TLS)
|
|
location / {
|
|
proxy_pass http://127.0.0.1:18180;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
|
|
# Timeouts
|
|
proxy_connect_timeout 30s;
|
|
proxy_send_timeout 30s;
|
|
proxy_read_timeout 30s;
|
|
}
|
|
}
|
|
|
|
# Redirect HTTP to HTTPS
|
|
server {
|
|
listen 80;
|
|
server_name stemedb.example.com;
|
|
return 301 https://$server_name$request_uri;
|
|
}
|
|
```
|
|
|
|
Enable and reload:
|
|
|
|
```bash
|
|
sudo ln -s /etc/nginx/sites-available/stemedb /etc/nginx/sites-enabled/
|
|
sudo nginx -t
|
|
sudo systemctl reload nginx
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Server Won't Start
|
|
|
|
**Check certificate paths:**
|
|
```bash
|
|
ls -la $STEMEDB_TLS_CERT_PATH
|
|
ls -la $STEMEDB_TLS_KEY_PATH
|
|
```
|
|
|
|
**Verify permissions:**
|
|
```bash
|
|
sudo -u stemedb cat $STEMEDB_TLS_CERT_PATH > /dev/null
|
|
```
|
|
|
|
If permission denied, grant access:
|
|
```bash
|
|
sudo setfacl -m u:stemedb:r $STEMEDB_TLS_CERT_PATH
|
|
sudo setfacl -m u:stemedb:r $STEMEDB_TLS_KEY_PATH
|
|
```
|
|
|
|
**Check logs:**
|
|
```bash
|
|
sudo journalctl -u stemedb-api -f
|
|
```
|
|
|
|
### Certificate Expired
|
|
|
|
```bash
|
|
sudo certbot renew --force-renewal
|
|
sudo systemctl reload stemedb-api
|
|
```
|
|
|
|
### Clients Can't Connect
|
|
|
|
**Check firewall:**
|
|
```bash
|
|
sudo ufw status
|
|
sudo ufw allow 443/tcp
|
|
```
|
|
|
|
**Verify DNS:**
|
|
```bash
|
|
dig stemedb.example.com
|
|
```
|
|
|
|
**Test from external host:**
|
|
```bash
|
|
curl -v https://stemedb.example.com/v1/health
|
|
```
|
|
|
|
### TLS Handshake Failures
|
|
|
|
**Check TLS version:**
|
|
```bash
|
|
openssl s_client -connect stemedb.example.com:443 -tls1_3
|
|
```
|
|
|
|
If connection fails, client may not support TLS 1.3. Verify client TLS support:
|
|
```bash
|
|
curl --tlsv1.3 https://stemedb.example.com/v1/health
|
|
```
|
|
|
|
## Security Best Practices
|
|
|
|
1. **Use Strong Certificates**
|
|
- Let's Encrypt certificates are free and automatically renew
|
|
- Minimum 2048-bit RSA keys (4096-bit recommended)
|
|
|
|
2. **Keep Certificates Updated**
|
|
- Set up auto-renewal
|
|
- Monitor expiration dates
|
|
- Test renewal process regularly
|
|
|
|
3. **Restrict Private Key Access**
|
|
- Private key should be readable only by stemedb user and root
|
|
- Never commit private keys to version control
|
|
|
|
4. **Use HTTPS Everywhere**
|
|
- Redirect all HTTP traffic to HTTPS
|
|
- Use HSTS headers to force HTTPS
|
|
|
|
5. **Monitor Certificate Expiration**
|
|
- Set up alerts for certificate expiration (30 days before)
|
|
- Test renewal process monthly
|
|
|
|
6. **Audit TLS Configuration**
|
|
- Use [SSL Labs](https://www.ssllabs.com/ssltest/) to test configuration
|
|
- Aim for A+ rating
|
|
|
|
## See Also
|
|
|
|
- [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
|
|
- [Certbot User Guide](https://eff-certbot.readthedocs.io/)
|
|
- [Mozilla SSL Configuration Generator](https://ssl-config.mozilla.org/)
|
|
- [StemeDB Operations Guide](../README.md)
|