rdev/docs/citadel-integration.md

Citadel Integration Plan

rdev integrates with a partner-hosted Citadel instance at citadel-staging.orchard9.ai for centralized log aggregation, querying, and alerting across both the platform and generated projects.

Environment Architecture

Organization: orchard9 (on citadel-staging.orchard9.ai)
│
├── Environment: "rdev-platform"
│   ├── rdev-api logs (stdout via agent)
│   ├── rdev-worker logs (stdout via agent)
│   ├── claudebox logs (stdout via agent)
│   └── audit events (shipped from AuditLogger)
│
├── Environment: "<project-slug>"       (auto-created per project)
│   └── Project pod logs (stdout via agent, routed by k8s labels)
│
├── Tenant Group: "platform"            → rdev-platform
└── Tenant Group: "projects"            → all project-* environments

Each project gets its own Citadel environment, matching rdev's isolation model (each project gets its own DB, Redis, DNS, registry namespace).

Integration Points

1. Agent DaemonSet (Log Collection)

What: A citadel-agent DaemonSet on every k3s node collects container stdout/stderr and ships to Citadel via HTTPS.

Why: rdev already outputs structured JSON slog to stdout. Zero code changes needed.

Routing: Agent reads k8s labels to route logs to the correct Citadel environment:

• citadel.io/environment label → determines target environment
• citadel.io/service label → tags the service name in Citadel

Manifests: deployments/k8s/base/citadel-agent/

2. Citadel Client Adapter (API Integration)

What: Go HTTP client for Citadel's API, following rdev's hexagonal architecture (port interface + adapter).

Why: Needed for auto-provisioning environments and shipping audit logs.

Files:

• internal/port/citadel.go — Port interface
• internal/adapter/citadel/client.go — HTTP client implementation
• internal/adapter/citadel/audit_shipper.go — Audit log shipping

3. Project Provisioning Step

What: When ProjectInfraService.CreateProject() runs, a new step creates a Citadel environment for the project.

Why: Each project needs its own isolated log environment. Auto-provisioning eliminates manual setup.

Where in the flow:

CreateProject()
  1. Generate slug
  2. Create project in DB
  3. Create git repo
  4. Create DNS
  5. Activate CI
  6. Seed template
  7. Provision DB + cache
  8. *** Create Citadel environment ***  ← NEW
  9. Create initial deployment
  10. Trigger CI build

Rollback: On project deletion, the Citadel environment is deleted too.

4. Skeleton Template Labels

What: Add citadel.io/* labels to the skeleton's k8s deployment templates.

Why: The agent uses these labels to route project logs to the correct Citadel environment.

Template vars: {{CITADEL_TENANT_ID}} injected during provisioning.

5. Audit Log Shipping

What: Hook into rdev's existing AuditLogger to also ship audit events to Citadel.

Why: Unified search across application logs and security events.

Pattern: Wrap the existing PostgreSQL AuditLogger with a multi-writer that sends to both PostgreSQL and Citadel.

Configuration

Environment Variables (rdev-api)

Variable	Description	Example
CITADEL_URL	Partner Citadel instance URL	https://citadel-staging.orchard9.ai
CITADEL_API_KEY	API key for environment management	ck_live_...
CITADEL_TENANT_ID	Platform environment tenant ID	uuid
CITADEL_ENABLED	Enable/disable Citadel integration	true

Secrets (k8s)

# deployments/k8s/base/citadel-agent/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: citadel-agent
  namespace: observability
type: Opaque
stringData:
  api-key: "ck_live_..."
  tenant-id: "..."

Queries (What You Get)

Platform Operations

# All errors across platform
citadel query "level:error" --tenant rdev-platform --last 1h

# Track a project lifecycle
citadel query "project_id:my-project" --tenant rdev-platform --last 24h

# Build failures
citadel query "component:build level:error" --tenant rdev-platform

# Saga failures
citadel query "component:saga result:failure" --tenant rdev-platform

# Slow API requests
citadel query "duration_ms:>5000 component:http" --tenant rdev-platform

# Audit trail
citadel query "action:project.create" --tenant rdev-platform --last 7d

Generated Projects

# Query a specific project's logs
citadel query "level:error" --tenant <project-slug> --last 1h

# All project errors at once
citadel query "level:error" --group projects --last 1h

# Correlate platform + project
citadel query "project_id:my-project" --last 24h  # org-wide

Implementation Order

Phase	Task	Effort
1	Write Citadel client adapter (internal/adapter/citadel/)	2h
1	Write port interface (internal/port/citadel.go)	30m
2	Deploy agent DaemonSet (deployments/k8s/base/citadel-agent/)	1h
2	Add citadel labels to existing rdev-api/worker manifests	15m
3	Add Citadel env creation to ProjectInfraService.CreateProject()	2h
3	Add Citadel env deletion to ProjectInfraService.DeleteProject()	30m
3	Add migration for citadel_tenant_id column on projects table	15m
4	Add citadel.io/* labels to skeleton k8s templates	30m
4	Add CITADEL_TENANT_ID to template variables	15m
5	Implement audit log shipper	2h
5	Wire into main.go startup	30m

Files Created/Modified

New Files

• internal/port/citadel.go
• internal/adapter/citadel/client.go
• internal/adapter/citadel/audit_shipper.go
• deployments/k8s/base/citadel-agent/kustomization.yaml
• deployments/k8s/base/citadel-agent/daemonset.yaml
• deployments/k8s/base/citadel-agent/serviceaccount.yaml
• deployments/k8s/base/citadel-agent/configmap.yaml
• internal/db/migrations/024_citadel_tenant_id.sql

Modified Files

• deployments/k8s/base/kustomization.yaml — include citadel-agent
• deployments/k8s/base/rdev-api.yaml — add citadel labels + env vars
• internal/service/project_infra.go — add CitadelClient field
• internal/service/project_infra_crud.go — add provisioning + cleanup steps
• internal/adapter/templates/templates/skeleton/ — k8s template labels
• cmd/rdev-api/main.go — wire Citadel adapter