rdev/docs/citadel-integration.md

# Citadel Integration Plan

rdev integrates with a **partner-hosted Citadel** instance at `citadel-staging.orchard9.ai` for centralized log aggregation, querying, and alerting across both the platform and generated projects.

## Environment Architecture

```
Organization: orchard9 (on citadel-staging.orchard9.ai)
│
├── Environment: "rdev-platform"
│   ├── rdev-api logs (stdout via agent)
│   ├── rdev-worker logs (stdout via agent)
│   ├── claudebox logs (stdout via agent)
│   └── audit events (shipped from AuditLogger)
│
├── Environment: "<project-slug>"       (auto-created per project)
│   └── Project pod logs (stdout via agent, routed by k8s labels)
│
├── Tenant Group: "platform"            → rdev-platform
└── Tenant Group: "projects"            → all project-* environments
```

Each project gets its own Citadel environment, matching rdev's isolation model (each project gets its own DB, Redis, DNS, registry namespace).

## Integration Points

### 1. Agent DaemonSet (Log Collection)

**What:** A `citadel-agent` DaemonSet on every k3s node collects container stdout/stderr and ships to Citadel via HTTPS.

**Why:** rdev already outputs structured JSON slog to stdout. Zero code changes needed.

**Routing:** Agent reads k8s labels to route logs to the correct Citadel environment:
- `citadel.io/environment` label → determines target environment
- `citadel.io/service` label → tags the service name in Citadel

**Manifests:** `deployments/k8s/base/citadel-agent/`

### 2. Citadel Client Adapter (API Integration)

**What:** Go HTTP client for Citadel's API, following rdev's hexagonal architecture (port interface + adapter).

**Why:** Needed for auto-provisioning environments and shipping audit logs.

**Files:**
- `internal/port/citadel.go` — Port interface
- `internal/adapter/citadel/client.go` — HTTP client implementation
- `internal/adapter/citadel/audit_shipper.go` — Audit log shipping

### 3. Project Provisioning Step

**What:** When `ProjectInfraService.CreateProject()` runs, a new step creates a Citadel environment for the project.

**Why:** Each project needs its own isolated log environment. Auto-provisioning eliminates manual setup.

**Where in the flow:**
```
CreateProject()
  1. Generate slug
  2. Create project in DB
  3. Create git repo
  4. Create DNS
  5. Activate CI
  6. Seed template
  7. Provision DB + cache
  8. *** Create Citadel environment ***  ← NEW
  9. Create initial deployment
  10. Trigger CI build
```

**Rollback:** On project deletion, the Citadel environment is deleted too.

### 4. Skeleton Template Labels

**What:** Add `citadel.io/*` labels to the skeleton's k8s deployment templates.

**Why:** The agent uses these labels to route project logs to the correct Citadel environment.

**Template vars:** `{{CITADEL_TENANT_ID}}` injected during provisioning.

### 5. Audit Log Shipping

**What:** Hook into rdev's existing `AuditLogger` to also ship audit events to Citadel.

**Why:** Unified search across application logs and security events.

**Pattern:** Wrap the existing PostgreSQL `AuditLogger` with a multi-writer that sends to both PostgreSQL and Citadel.

## Configuration

### Environment Variables (rdev-api)

| Variable | Description | Example |
|----------|-------------|---------|
| `CITADEL_URL` | Partner Citadel instance URL | `https://citadel-staging.orchard9.ai` |
| `CITADEL_API_KEY` | API key for environment management | `ck_live_...` |
| `CITADEL_TENANT_ID` | Platform environment tenant ID | `uuid` |
| `CITADEL_ENABLED` | Enable/disable Citadel integration | `true` |

### Secrets (k8s)

```yaml
# deployments/k8s/base/citadel-agent/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: citadel-agent
  namespace: observability
type: Opaque
stringData:
  api-key: "ck_live_..."
  tenant-id: "..."
```

## Queries (What You Get)

### Platform Operations

```bash
# All errors across platform
citadel query "level:error" --tenant rdev-platform --last 1h

# Track a project lifecycle
citadel query "project_id:my-project" --tenant rdev-platform --last 24h

# Build failures
citadel query "component:build level:error" --tenant rdev-platform

# Saga failures
citadel query "component:saga result:failure" --tenant rdev-platform

# Slow API requests
citadel query "duration_ms:>5000 component:http" --tenant rdev-platform

# Audit trail
citadel query "action:project.create" --tenant rdev-platform --last 7d
```

### Generated Projects

```bash
# Query a specific project's logs
citadel query "level:error" --tenant <project-slug> --last 1h

# All project errors at once
citadel query "level:error" --group projects --last 1h

# Correlate platform + project
citadel query "project_id:my-project" --last 24h  # org-wide
```

## Implementation Order

| Phase | Task | Effort |
|-------|------|--------|
| 1 | Write Citadel client adapter (`internal/adapter/citadel/`) | 2h |
| 1 | Write port interface (`internal/port/citadel.go`) | 30m |
| 2 | Deploy agent DaemonSet (`deployments/k8s/base/citadel-agent/`) | 1h |
| 2 | Add citadel labels to existing rdev-api/worker manifests | 15m |
| 3 | Add Citadel env creation to `ProjectInfraService.CreateProject()` | 2h |
| 3 | Add Citadel env deletion to `ProjectInfraService.DeleteProject()` | 30m |
| 3 | Add migration for `citadel_tenant_id` column on projects table | 15m |
| 4 | Add `citadel.io/*` labels to skeleton k8s templates | 30m |
| 4 | Add `CITADEL_TENANT_ID` to template variables | 15m |
| 5 | Implement audit log shipper | 2h |
| 5 | Wire into main.go startup | 30m |

## Files Created/Modified

### New Files
- `internal/port/citadel.go`
- `internal/adapter/citadel/client.go`
- `internal/adapter/citadel/audit_shipper.go`
- `deployments/k8s/base/citadel-agent/kustomization.yaml`
- `deployments/k8s/base/citadel-agent/daemonset.yaml`
- `deployments/k8s/base/citadel-agent/serviceaccount.yaml`
- `deployments/k8s/base/citadel-agent/configmap.yaml`
- `internal/db/migrations/024_citadel_tenant_id.sql`

### Modified Files
- `deployments/k8s/base/kustomization.yaml` — include citadel-agent
- `deployments/k8s/base/rdev-api.yaml` — add citadel labels + env vars
- `internal/service/project_infra.go` — add CitadelClient field
- `internal/service/project_infra_crud.go` — add provisioning + cleanup steps
- `internal/adapter/templates/templates/skeleton/` — k8s template labels
- `cmd/rdev-api/main.go` — wire Citadel adapter