rdev/.claude/guides/ops/deploying.md

# Deploying to k3s

**When to use:** Deploying rdev-api or claudebox pods to the k3s cluster.

## Prerequisites

- Access to orchard9 k3s cluster
- kubectl installed
- kubeconfig file at `~/.kube/orchard9-k3sf.yaml`

## Quick Deploy

```bash
# CRITICAL: Always set kubeconfig first
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml

# Deploy all resources
kubectl apply -k deployments/k8s/base

# Verify deployment
kubectl get pods -n rdev
kubectl get svc -n rdev
```

## What Gets Deployed

The Kustomize base deploys:

| Resource | Name | Purpose |
|----------|------|---------|
| Namespace | `rdev` | Isolation |
| Deployment | `rdev-api` | API server |
| StatefulSet | `claudebox` | Claude Code pods |
| Service | `rdev-api` | Internal service |
| ServiceAccount | `rdev-api` | RBAC identity |
| Role/RoleBinding | `rdev-api` | Namespace permissions |
| ClusterRole/Binding | `rdev-api` | Cluster-wide permissions |
| PVC | `workspace-pvc` | Shared workspace volume |
| Secret | `claude-credentials` | Claude auth (manual) |

## Deployment Steps

### 1. Set Kubeconfig

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml

# Verify connection
kubectl cluster-info
```

### 2. Create Secrets (First Time Only)

```bash
# Claude credentials secret
kubectl create secret generic claude-credentials \
  -n rdev \
  --from-file=credentials.json=/path/to/credentials.json

# Database credentials (if using external postgres)
kubectl create secret generic rdev-db \
  -n rdev \
  --from-literal=host=postgres.example.com \
  --from-literal=password=secret
```

### 3. Apply Manifests

```bash
# Full deployment
kubectl apply -k deployments/k8s/base

# Or apply individually
kubectl apply -f deployments/k8s/base/namespace.yaml
kubectl apply -f deployments/k8s/base/rdev-api.yaml
kubectl apply -f deployments/k8s/base/claudebox.yaml
```

### 4. Verify

```bash
# Check pods
kubectl get pods -n rdev
# Expected: rdev-api-xxx Running, claudebox-0 Running

# Check logs
kubectl logs -n rdev deployment/rdev-api

# Test API
kubectl port-forward -n rdev svc/rdev-api 8080:8080
curl http://localhost:8080/health
```

## Updating Deployments

### Rolling Update

```bash
# Update image tag in manifests, then:
kubectl apply -k deployments/k8s/base

# Or force rollout
kubectl rollout restart deployment/rdev-api -n rdev
```

### Watch Rollout

```bash
kubectl rollout status deployment/rdev-api -n rdev
```

## Manifest Structure

```
deployments/k8s/
└── base/
    ├── kustomization.yaml    # Kustomize config
    ├── namespace.yaml        # rdev namespace
    ├── rdev-api.yaml         # API deployment + RBAC
    ├── claudebox.yaml        # StatefulSet for Claude pods
    └── pvc-workspace.yaml    # Shared workspace PVC
```

## RBAC Permissions

rdev-api requires:
- **Namespace-scoped:** Pod exec, ConfigMap read
- **Cluster-scoped:** Deployments, Services, Ingresses (for project deployment feature)

### Woodpecker CI Deployer RBAC

The `woodpecker-deployer-rbac.yaml` manifest grants Woodpecker CI permission to deploy projects. Without this, deploy steps fail with permission errors.

**Why it's needed:** Woodpecker pipeline steps run as the `default` ServiceAccount in the `threesix` namespace, but need to `kubectl set image` on deployments in the `projects` namespace.

**Permissions granted:**
- `get`, `list`, `patch` on `deployments` (apps API group)

This follows least-privilege principles - only the minimum permissions needed for `kubectl set image` to work.

## Troubleshooting

### Pod not starting

```bash
kubectl describe pod -n rdev <pod-name>
kubectl logs -n rdev <pod-name> --previous
```

### Permission denied

Check RBAC:
```bash
kubectl auth can-i exec pods -n rdev --as=system:serviceaccount:rdev:rdev-api
```

### Image pull error

Verify registry access:
```bash
kubectl get events -n rdev --sort-by='.lastTimestamp'
```

### Database connection failed

Check secret and network policy:
```bash
kubectl get secret rdev-db -n rdev -o yaml
kubectl exec -n rdev deployment/rdev-api -- env | grep DB_
```

## Automated CI/CD via Woodpecker

rdev now uses Woodpecker CI for automated builds and deploys:

```
git push → Gitea → Woodpecker → kaniko → registry.threesix.ai → kubectl deploy
```

### How It Works

1. Push to `git.threesix.ai/jordan/rdev` triggers Woodpecker
2. Woodpecker runs `.woodpecker.yml`:
   - Tests with `go test ./...`
   - Builds images via `woodpeckerci/plugin-kaniko`
   - Pushes to `registry.threesix.ai/rdev/{api,worker,claudebox}`
   - Deploys via `kubectl set image`

### Manual Deploy (if needed)

```bash
export KUBECONFIG=~/.kube/orchard9-k3sf.yaml
kubectl apply -f deployments/k8s/base/rdev-api.yaml
kubectl rollout restart -n rdev deployment/rdev-api
```

### Image Registry

Images are stored in Zot at `registry.threesix.ai/rdev/`:
- `rdev/api:latest` - API server
- `rdev/worker:latest` - Worker pool
- `rdev/claudebox:latest` - Claude Code container

## Constraints

- **ON-PREM k3s** - Not GKE/EKS, always use local kubeconfig
- **Kustomize only** - No ArgoCD or Helm
- **Woodpecker CI** - Automated builds on push to main

## Related

- [Releasing](./releasing.md) - Build and publish new versions
- [Database Guide](./database.md)
- [Kubernetes Adapter](../services/kubernetes.md)
- [External Health Diagnostics](./external-health-diagnostics.md) - Debug registry/CI/git issues