diff --git a/.claude/guides/ops/deploying.md b/.claude/guides/ops/deploying.md new file mode 100644 index 0000000..cc46b3d --- /dev/null +++ b/.claude/guides/ops/deploying.md @@ -0,0 +1,215 @@ +# Deploying to k3s + +**When to use:** Deploying rdev-api or claudebox pods to the k3s cluster. + +## Prerequisites + +- Access to orchard9 k3s cluster +- kubectl installed +- kubeconfig file at `~/.kube/orchard9-k3sf.yaml` + +## Quick Deploy + +```bash +# CRITICAL: Always set kubeconfig first +export KUBECONFIG=~/.kube/orchard9-k3sf.yaml + +# Deploy all resources +kubectl apply -k deployments/k8s/base + +# Verify deployment +kubectl get pods -n rdev +kubectl get svc -n rdev +``` + +## What Gets Deployed + +The Kustomize base deploys: + +| Resource | Name | Purpose | +|----------|------|---------| +| Namespace | `rdev` | Isolation | +| Deployment | `rdev-api` | API server | +| StatefulSet | `claudebox` | Claude Code pods | +| Service | `rdev-api` | Internal service | +| ServiceAccount | `rdev-api` | RBAC identity | +| Role/RoleBinding | `rdev-api` | Namespace permissions | +| ClusterRole/Binding | `rdev-api` | Cluster-wide permissions | +| PVC | `workspace-pvc` | Shared workspace volume | +| Secret | `claude-credentials` | Claude auth (manual) | + +## Deployment Steps + +### 1. Set Kubeconfig + +```bash +export KUBECONFIG=~/.kube/orchard9-k3sf.yaml + +# Verify connection +kubectl cluster-info +``` + +### 2. Create Secrets (First Time Only) + +```bash +# Claude credentials secret +kubectl create secret generic claude-credentials \ + -n rdev \ + --from-file=credentials.json=/path/to/credentials.json + +# Database credentials (if using external postgres) +kubectl create secret generic rdev-db \ + -n rdev \ + --from-literal=host=postgres.example.com \ + --from-literal=password=secret +``` + +### 3. Apply Manifests + +```bash +# Full deployment +kubectl apply -k deployments/k8s/base + +# Or apply individually +kubectl apply -f deployments/k8s/base/namespace.yaml +kubectl apply -f deployments/k8s/base/rdev-api.yaml +kubectl apply -f deployments/k8s/base/claudebox.yaml +``` + +### 4. Verify + +```bash +# Check pods +kubectl get pods -n rdev +# Expected: rdev-api-xxx Running, claudebox-0 Running + +# Check logs +kubectl logs -n rdev deployment/rdev-api + +# Test API +kubectl port-forward -n rdev svc/rdev-api 8080:8080 +curl http://localhost:8080/health +``` + +## Updating Deployments + +### Rolling Update + +```bash +# Update image tag in manifests, then: +kubectl apply -k deployments/k8s/base + +# Or force rollout +kubectl rollout restart deployment/rdev-api -n rdev +``` + +### Watch Rollout + +```bash +kubectl rollout status deployment/rdev-api -n rdev +``` + +## Manifest Structure + +``` +deployments/k8s/ +└── base/ + ├── kustomization.yaml # Kustomize config + ├── namespace.yaml # rdev namespace + ├── rdev-api.yaml # API deployment + RBAC + ├── claudebox.yaml # StatefulSet for Claude pods + └── pvc-workspace.yaml # Shared workspace PVC +``` + +## RBAC Permissions + +rdev-api requires: +- **Namespace-scoped:** Pod exec, ConfigMap read +- **Cluster-scoped:** Deployments, Services, Ingresses (for project deployment feature) + +### Woodpecker CI Deployer RBAC + +The `woodpecker-deployer-rbac.yaml` manifest grants Woodpecker CI permission to deploy projects. Without this, deploy steps fail with permission errors. + +**Why it's needed:** Woodpecker pipeline steps run as the `default` ServiceAccount in the `threesix` namespace, but need to `kubectl set image` on deployments in the `projects` namespace. + +**Permissions granted:** +- `get`, `list`, `patch` on `deployments` (apps API group) + +This follows least-privilege principles - only the minimum permissions needed for `kubectl set image` to work. + +## Troubleshooting + +### Pod not starting + +```bash +kubectl describe pod -n rdev +kubectl logs -n rdev --previous +``` + +### Permission denied + +Check RBAC: +```bash +kubectl auth can-i exec pods -n rdev --as=system:serviceaccount:rdev:rdev-api +``` + +### Image pull error + +Verify registry access: +```bash +kubectl get events -n rdev --sort-by='.lastTimestamp' +``` + +### Database connection failed + +Check secret and network policy: +```bash +kubectl get secret rdev-db -n rdev -o yaml +kubectl exec -n rdev deployment/rdev-api -- env | grep DB_ +``` + +## Automated CI/CD via Woodpecker + +rdev now uses Woodpecker CI for automated builds and deploys: + +``` +git push → Gitea → Woodpecker → kaniko → registry.threesix.ai → kubectl deploy +``` + +### How It Works + +1. Push to `git.threesix.ai/jordan/rdev` triggers Woodpecker +2. Woodpecker runs `.woodpecker.yml`: + - Tests with `go test ./...` + - Builds images via `woodpeckerci/plugin-kaniko` + - Pushes to `registry.threesix.ai/rdev/{api,worker,claudebox}` + - Deploys via `kubectl set image` + +### Manual Deploy (if needed) + +```bash +export KUBECONFIG=~/.kube/orchard9-k3sf.yaml +kubectl apply -f deployments/k8s/base/rdev-api.yaml +kubectl rollout restart -n rdev deployment/rdev-api +``` + +### Image Registry + +Images are stored in Zot at `registry.threesix.ai/rdev/`: +- `rdev/api:latest` - API server +- `rdev/worker:latest` - Worker pool +- `rdev/claudebox:latest` - Claude Code container + +## Constraints + +- **ON-PREM k3s** - Not GKE/EKS, always use local kubeconfig +- **Kustomize only** - No ArgoCD or Helm +- **Woodpecker CI** - Automated builds on push to main + +## Related + +- [Releasing](./releasing.md) - Build and publish new versions +- [Database Guide](./database.md) +- [Kubernetes Adapter](../services/kubernetes.md) +- [External Health Diagnostics](./external-health-diagnostics.md) - Debug registry/CI/git issues diff --git a/.claude/guides/ops/releasing.md b/.claude/guides/ops/releasing.md new file mode 100644 index 0000000..dc68c72 --- /dev/null +++ b/.claude/guides/ops/releasing.md @@ -0,0 +1,135 @@ +# Releasing rdev-api + +**When to use:** Creating a new versioned release with changelog, container image, and git tag. + +## Automated Releases (Recommended) + +Push to main branch triggers Woodpecker CI to build and deploy automatically: + +```bash +# Just push - Woodpecker handles the rest +git push origin main + +# Or push to both remotes +GITEA_TOKEN=$(kubectl get secret rdev-credentials -n rdev -o jsonpath='{.data.GITEA_TOKEN}' | base64 -d) +git push https://jordan:${GITEA_TOKEN}@git.threesix.ai/jordan/rdev.git main +git push origin main +``` + +Images are built via kaniko and pushed to `registry.threesix.ai/rdev/*`. + +## Prerequisites (Manual Releases) + +- Go installed (for local binary builds if needed) +- KUBECONFIG set: `export KUBECONFIG=~/.kube/orchard9-k3sf.yaml` +- Access to Gitea (`git.threesix.ai/jordan/rdev`) + +## Quick Start + +```bash +# Release and deploy in one command +./scripts/release.sh v0.8.1 "Fix worker ID config bug" --deploy + +# Release only (no deploy) +./scripts/release.sh v0.8.1 "Fix worker ID config bug" +``` + +## What the Release Script Does + +The script (`scripts/release.sh`) performs these steps in order: + +1. **Creates changelog** - Writes `changelog/.md` with date and message +2. **Updates deployment** - Patches `deployments/k8s/base/rdev-api.yaml` with new image tag +3. **Commits and pushes** - Commits changelog and deployment changes to main +4. **Builds binary** - Cross-compiles for linux/amd64 with version embedded +5. **Builds container** - Creates `ghcr.io/orchard9/rdev-api:` image +6. **Pushes image** - Uploads to GitHub Container Registry +7. **Tags release** - Creates annotated git tag and pushes it + +With `--deploy` flag, it also: +8. **Runs migrations** - Executes all SQL migrations as `rdev` superuser +9. **Applies manifest** - `kubectl apply` the deployment YAML +10. **Restarts deployment** - Triggers rollout with new image + +## Usage + +```bash +./scripts/release.sh "" [--deploy] +``` + +| Argument | Description | Example | +|----------|-------------|---------| +| `version` | Semver tag (with or without `v` prefix) | `v0.8.1` or `0.8.1` | +| `message` | Changelog entry describing the release | `"Add worker pool support"` | +| `--deploy` | Also run migrations and deploy to k3s | (optional flag) | + +## Examples + +```bash +# Full release + deploy (recommended) +./scripts/release.sh v0.9.0 "Add project templates" --deploy + +# Bug fix release + deploy +./scripts/release.sh v0.8.2 "Fix nil pointer in command handler" --deploy + +# Release only (deploy later manually) +./scripts/release.sh v0.8.3 "Update dependency versions" +``` + +## Manual Deploy (if not using --deploy) + +```bash +export KUBECONFIG=~/.kube/orchard9-k3sf.yaml +kubectl apply -f deployments/k8s/base/rdev-api.yaml +kubectl rollout restart -n rdev deployment/rdev-api +``` + +## Artifacts Created + +| Artifact | Location | +|----------|----------| +| Changelog | `changelog/.md` | +| Container images | `registry.threesix.ai/rdev/{api,worker,claudebox}:` | +| Git tag | `` (annotated) | +| Updated deployment | `deployments/k8s/base/rdev-api.yaml` | + +## Image Registry + +Images are now stored in Zot (`registry.threesix.ai`) instead of ghcr.io: + +| Image | Path | +|-------|------| +| API server | `registry.threesix.ai/rdev/api:latest` | +| Worker | `registry.threesix.ai/rdev/worker:latest` | +| Claudebox | `registry.threesix.ai/rdev/claudebox:latest` | + +Tags: `latest` and `${CI_COMMIT_SHA:0:8}` (8-char commit hash) + +## Troubleshooting + +### Docker push fails + +Ensure you're authenticated to ghcr.io: +```bash +echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin +``` + +### Build fails + +Check Go environment: +```bash +go version +go env GOOS GOARCH +``` + +### Git push rejected + +Ensure you have push access and main is up to date: +```bash +git pull origin main +``` + +## Related + +- [Deploying to k3s](./deploying.md) +- [Credentials Management](./credentials.md) diff --git a/CLAUDE.md b/CLAUDE.md index 3bf1ed4..653f0ad 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -93,18 +93,14 @@ go run ./cmd/rdev-api # Run tests go test ./... -# Release + deploy (one command) -./scripts/release.sh v0.10.1 "Description of changes" --deploy +# Automated deploy (push triggers Woodpecker CI) +git push origin main # Builds and deploys automatically via Woodpecker -# Release only (no deploy) -./scripts/release.sh v0.10.1 "Description of changes" - -# Manual deploy (if needed) +# Manual deploy (if Woodpecker unavailable) kubectl apply -f deployments/k8s/base/rdev-api.yaml kubectl rollout restart -n rdev deployment/rdev-api -# Deploy claudebox worker (when Dockerfile changes) -./scripts/build-push.sh v0.4.0 claudebox && kubectl apply -f deployments/k8s/base/claudebox.yaml && kubectl rollout restart -n rdev statefulset/claudebox +# Images are at registry.threesix.ai/rdev/{api,worker,claudebox} # Verify pods kubectl get pods -n rdev diff --git a/docs/RELEASE_CHECKLIST.md b/docs/RELEASE_CHECKLIST.md index 8e9a6b0..389d6bd 100644 --- a/docs/RELEASE_CHECKLIST.md +++ b/docs/RELEASE_CHECKLIST.md @@ -48,29 +48,30 @@ Note: Some coverage targets not met, but core functionality is well-tested. ## Release -### Build +### Automated (Recommended) + +Push to main triggers Woodpecker CI to build and deploy: + ```bash -# Build binary -GOOS=linux GOARCH=amd64 go build -o rdev-api ./cmd/rdev-api - -# Build Docker image -docker build -t ghcr.io/orchard9/rdev-api:1.0.0 . - -# Push image -docker push ghcr.io/orchard9/rdev-api:1.0.0 +# Push to both remotes - Woodpecker builds and deploys automatically +git push origin main +GITEA_TOKEN=$(kubectl get secret rdev-credentials -n rdev -o jsonpath='{.data.GITEA_TOKEN}' | base64 -d) +git push https://jordan:${GITEA_TOKEN}@git.threesix.ai/jordan/rdev.git main ``` +Images are built via kaniko and pushed to `registry.threesix.ai/rdev/*`. + ### Tag ```bash git tag -a v1.0.0 -m "Release v1.0.0" git push origin v1.0.0 ``` -### Deploy +### Manual Deploy (if needed) ```bash -# Update image tag in kustomization -# Apply to cluster -kubectl apply -k deployments/k8s/overlays/prod +export KUBECONFIG=~/.kube/orchard9-k3sf.yaml +kubectl apply -f deployments/k8s/base/rdev-api.yaml +kubectl rollout restart -n rdev deployment/rdev-api # Verify deployment kubectl -n rdev rollout status deployment/rdev-api