rdev/docs/plans/THREESIX_INFRASTRUCTURE.md

threesix.ai Infrastructure Implementation Plan

Self-hosted git, CI/CD, and deployment infrastructure for agent-driven development.

Overview

Replace GitHub dependency with self-hosted infrastructure on k3s:

• soft-serve - Git server (SSH-based, minimal)
• Zot - Container registry (OCI-native)
• Woodpecker - CI/CD pipelines
• rdev-api - Orchestration layer with DNS management

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                              threesix.ai                                     │
│                                                                              │
│  git.threesix.ai ──────▶ soft-serve (SSH :22)                               │
│  registry.threesix.ai ─▶ zot (internal only, HTTPS for UI)                  │
│  ci.threesix.ai ───────▶ woodpecker (web UI)                                │
│  *.threesix.ai ────────▶ project deployments                                │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│                           k3s cluster                                        │
│                                                                              │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐                   │
│  │  soft-serve  │───▶│  woodpecker  │───▶│     zot      │                   │
│  │  (git repos) │    │  (CI/CD)     │    │  (registry)  │                   │
│  └──────────────┘    └──────────────┘    └──────────────┘                   │
│         │                   │                   │                            │
│         │                   ▼                   │                            │
│         │            ┌──────────────┐           │                            │
│         └───────────▶│   rdev-api   │◀──────────┘                           │
│                      │              │                                        │
│                      │ - Create repos                                        │
│                      │ - Deploy apps                                         │
│                      │ - Manage DNS                                          │
│                      └──────────────┘                                        │
│                             │                                                │
│                             ▼                                                │
│                      ┌──────────────┐                                        │
│                      │  Cloudflare  │                                        │
│                      │  DNS API     │                                        │
│                      └──────────────┘                                        │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Configuration

Credentials (from .secrets)

Key	Value	Purpose
CLOUDFLARE_API_TOKEN	nGoDhG6Za...	DNS management
CLOUDFLARE_ZONE_ID	e0bc8d51...	threesix.ai zone

Network

Resource	Value
External IP	208.122.204.172
Let's Encrypt Email	jordan@threesix.ai
Domain	threesix.ai

Admin Access

SSH Public Key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDZwQF0Ro0E0foFo0oro/NrfUb5abEec/A0OP2qO8dVn jordanwashburn@jordanmacstudio.lan

---

Phase 1: Foundation (K8s Infrastructure)

1.1 Create Namespace and Secrets

# deployments/k8s/base/threesix/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: threesix
---
# Cloudflare API secret for cert-manager and rdev-api
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api
  namespace: threesix
type: Opaque
stringData:
  api-token: "${CLOUDFLARE_API_TOKEN}"
  zone-id: "${CLOUDFLARE_ZONE_ID}"

1.2 Configure cert-manager for Wildcard Certs

# deployments/k8s/base/threesix/cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-threesix
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: jordan@threesix.ai
    privateKeySecretRef:
      name: letsencrypt-threesix-account
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api
            key: api-token
      selector:
        dnsZones:
        - "threesix.ai"
---
# Wildcard certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: threesix-wildcard
  namespace: threesix
spec:
  secretName: threesix-wildcard-tls
  issuerRef:
    name: letsencrypt-threesix
    kind: ClusterIssuer
  dnsNames:
  - "threesix.ai"
  - "*.threesix.ai"

1.3 Deploy soft-serve

# deployments/k8s/base/threesix/soft-serve.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: soft-serve-config
  namespace: threesix
data:
  config.yaml: |
    name: threesix
    log_format: text
    ssh:
      listen_addr: :22
      public_url: ssh://git.threesix.ai
      max_timeout: 30
      idle_timeout: 120
    http:
      listen_addr: :23231
      public_url: https://git.threesix.ai
    stats:
      listen_addr: :23233
    initial_admin_keys:
      - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDZwQF0Ro0E0foFo0oro/NrfUb5abEec/A0OP2qO8dVn jordanwashburn"
    # Allow anyone to read public repos, admins can create
    anon_access: read-only    
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: soft-serve
  namespace: threesix
spec:
  serviceName: soft-serve
  replicas: 1
  selector:
    matchLabels:
      app: soft-serve
  template:
    metadata:
      labels:
        app: soft-serve
    spec:
      containers:
      - name: soft-serve
        image: charmcli/soft-serve:latest
        ports:
        - containerPort: 22
          name: ssh
        - containerPort: 23231
          name: http
        - containerPort: 23233
          name: stats
        volumeMounts:
        - name: data
          mountPath: /soft-serve
        - name: config
          mountPath: /soft-serve/config.yaml
          subPath: config.yaml
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "256Mi"
            cpu: "500m"
      volumes:
      - name: config
        configMap:
          name: soft-serve-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: longhorn
      resources:
        requests:
          storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: soft-serve
  namespace: threesix
spec:
  selector:
    app: soft-serve
  ports:
  - name: ssh
    port: 22
    targetPort: 22
  - name: http
    port: 80
    targetPort: 23231
  - name: stats
    port: 23233
    targetPort: 23233
---
# External SSH access via LoadBalancer
apiVersion: v1
kind: Service
metadata:
  name: soft-serve-ssh
  namespace: threesix
spec:
  type: LoadBalancer
  selector:
    app: soft-serve
  ports:
  - name: ssh
    port: 22
    targetPort: 22
---
# HTTP access via Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: soft-serve
  namespace: threesix
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-threesix
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - git.threesix.ai
    secretName: git-threesix-tls
  rules:
  - host: git.threesix.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: soft-serve
            port:
              number: 80

1.4 Deploy Zot Registry

# deployments/k8s/base/threesix/zot.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: zot-config
  namespace: threesix
data:
  config.json: |
    {
      "distSpecVersion": "1.1.0",
      "storage": {
        "rootDirectory": "/var/lib/zot",
        "gc": true,
        "gcDelay": "1h"
      },
      "http": {
        "address": "0.0.0.0",
        "port": "5000"
      },
      "log": {
        "level": "info"
      },
      "extensions": {
        "search": {
          "enable": true
        },
        "ui": {
          "enable": true
        }
      }
    }    
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zot
  namespace: threesix
spec:
  serviceName: zot
  replicas: 1
  selector:
    matchLabels:
      app: zot
  template:
    metadata:
      labels:
        app: zot
    spec:
      containers:
      - name: zot
        image: ghcr.io/project-zot/zot-linux-amd64:latest
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: data
          mountPath: /var/lib/zot
        - name: config
          mountPath: /etc/zot/config.json
          subPath: config.json
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "1000m"
      volumes:
      - name: config
        configMap:
          name: zot-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: longhorn
      resources:
        requests:
          storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
  name: zot
  namespace: threesix
spec:
  selector:
    app: zot
  ports:
  - port: 5000
    targetPort: 5000
---
# Internal DNS name for cluster access
# Pods can pull from: zot.threesix.svc.cluster.local:5000/image:tag
---
# Optional: External UI access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: zot
  namespace: threesix
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-threesix
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - registry.threesix.ai
    secretName: registry-threesix-tls
  rules:
  - host: registry.threesix.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: zot
            port:
              number: 5000

1.5 Initial DNS Records

Create via Cloudflare API or dashboard:

Type	Name	Value	Proxy
A	git	208.122.204.172	No (SSH needs direct)
A	registry	208.122.204.172	No
A	ci	208.122.204.172	Yes (optional)
A	*	208.122.204.172	Yes (optional)

---

Phase 2: CI/CD (Woodpecker)

2.1 Deploy Woodpecker Server

# deployments/k8s/base/threesix/woodpecker-server.yaml
apiVersion: v1
kind: Secret
metadata:
  name: woodpecker-secrets
  namespace: threesix
type: Opaque
stringData:
  # Generate with: openssl rand -hex 32
  WOODPECKER_AGENT_SECRET: "${WOODPECKER_AGENT_SECRET}"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: woodpecker-server
  namespace: threesix
spec:
  replicas: 1
  selector:
    matchLabels:
      app: woodpecker-server
  template:
    metadata:
      labels:
        app: woodpecker-server
    spec:
      containers:
      - name: woodpecker
        image: woodpeckerci/woodpecker-server:latest
        ports:
        - containerPort: 8000
        env:
        - name: WOODPECKER_HOST
          value: "https://ci.threesix.ai"
        - name: WOODPECKER_OPEN
          value: "false"
        - name: WOODPECKER_ADMIN
          value: "jordan"
        # Soft-serve / generic git integration
        - name: WOODPECKER_GITEA
          value: "false"
        - name: WOODPECKER_WEBHOOK_HOST
          value: "http://woodpecker-server.threesix.svc:8000"
        envFrom:
        - secretRef:
            name: woodpecker-secrets
        volumeMounts:
        - name: data
          mountPath: /var/lib/woodpecker
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: woodpecker-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: woodpecker-data
  namespace: threesix
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
  name: woodpecker-server
  namespace: threesix
spec:
  selector:
    app: woodpecker-server
  ports:
  - name: http
    port: 8000
    targetPort: 8000
  - name: grpc
    port: 9000
    targetPort: 9000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: woodpecker
  namespace: threesix
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-threesix
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - ci.threesix.ai
    secretName: ci-threesix-tls
  rules:
  - host: ci.threesix.ai
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: woodpecker-server
            port:
              number: 8000

2.2 Deploy Woodpecker Agent (with Kaniko)

# deployments/k8s/base/threesix/woodpecker-agent.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: woodpecker-agent
  namespace: threesix
spec:
  replicas: 2
  selector:
    matchLabels:
      app: woodpecker-agent
  template:
    metadata:
      labels:
        app: woodpecker-agent
    spec:
      containers:
      - name: agent
        image: woodpeckerci/woodpecker-agent:latest
        env:
        - name: WOODPECKER_SERVER
          value: "woodpecker-server.threesix.svc:9000"
        - name: WOODPECKER_BACKEND
          value: "kubernetes"
        - name: WOODPECKER_BACKEND_K8S_NAMESPACE
          value: "threesix"
        - name: WOODPECKER_BACKEND_K8S_STORAGE_CLASS
          value: "longhorn"
        - name: WOODPECKER_BACKEND_K8S_VOLUME_SIZE
          value: "10Gi"
        envFrom:
        - secretRef:
            name: woodpecker-secrets
      serviceAccountName: woodpecker-agent
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: woodpecker-agent
  namespace: threesix
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: woodpecker-agent
  namespace: threesix
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "secrets", "configmaps", "persistentvolumeclaims"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: woodpecker-agent
  namespace: threesix
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: woodpecker-agent
subjects:
- kind: ServiceAccount
  name: woodpecker-agent
  namespace: threesix

---

Phase 3: rdev-api Extensions

3.1 New Port Interfaces

// internal/port/git.go
package port

import "context"

// GitRepository manages git repositories.
type GitRepository interface {
    // CreateRepo creates a new git repository.
    CreateRepo(ctx context.Context, name, description string) (*Repo, error)

    // DeleteRepo deletes a repository.
    DeleteRepo(ctx context.Context, name string) error

    // ListRepos returns all repositories.
    ListRepos(ctx context.Context) ([]*Repo, error)

    // GetRepo returns a single repository.
    GetRepo(ctx context.Context, name string) (*Repo, error)

    // AddCollaborator adds a user's SSH key to a repo.
    AddCollaborator(ctx context.Context, repo, keyName, publicKey string) error

    // AddWebhook adds a webhook to trigger on push.
    AddWebhook(ctx context.Context, repo, url, secret string) error
}

// Repo represents a git repository.
type Repo struct {
    Name        string
    Description string
    CloneSSH    string // ssh://git@git.threesix.ai/name.git
    CloneHTTP   string // https://git.threesix.ai/name.git
    CreatedAt   time.Time
}

// internal/port/dns.go
package port

import "context"

// DNSProvider manages DNS records.
type DNSProvider interface {
    // CreateRecord creates a DNS record.
    CreateRecord(ctx context.Context, record DNSRecord) error

    // DeleteRecord removes a DNS record.
    DeleteRecord(ctx context.Context, recordType, name string) error

    // ListRecords returns all records for the zone.
    ListRecords(ctx context.Context) ([]*DNSRecord, error)
}

// DNSRecord represents a DNS record.
type DNSRecord struct {
    Type    string // A, CNAME, TXT
    Name    string // subdomain or @ for root
    Content string // IP or target
    TTL     int    // seconds, 1 = auto
    Proxied bool   // Cloudflare proxy
}

// internal/port/deployer.go
package port

import "context"

// Deployer manages application deployments.
type Deployer interface {
    // Deploy creates or updates a deployment.
    Deploy(ctx context.Context, spec DeploySpec) error

    // Undeploy removes a deployment.
    Undeploy(ctx context.Context, projectName string) error

    // GetStatus returns deployment status.
    GetStatus(ctx context.Context, projectName string) (*DeployStatus, error)
}

// DeploySpec defines a deployment.
type DeploySpec struct {
    ProjectName string
    Image       string
    Domain      string   // e.g., "myapp.threesix.ai"
    Port        int      // container port
    Replicas    int
    EnvVars     map[string]string
    Secrets     map[string]string
}

// DeployStatus represents current deployment state.
type DeployStatus struct {
    ProjectName    string
    Image          string
    Replicas       int
    ReadyReplicas  int
    URL            string
    Status         string // "running", "pending", "failed"
}

3.2 New Adapters

internal/adapter/
├── softserve/       # soft-serve SSH/API client
│   └── client.go
├── cloudflare/      # Cloudflare DNS API client
│   └── client.go
├── deployer/        # K8s deployment manager
│   └── deployer.go
└── registry/        # Zot registry client (optional)
    └── client.go

3.3 New Handlers

// internal/handlers/projects_git.go

// POST /projects/{id}/repo - Create git repo for project
// DELETE /projects/{id}/repo - Delete git repo
// GET /projects/{id}/repo - Get repo info

// POST /projects/{id}/deploy - Deploy project
// DELETE /projects/{id}/deploy - Undeploy project
// GET /projects/{id}/deploy/status - Get deployment status

// POST /projects/{id}/domain - Set custom domain
// DELETE /projects/{id}/domain - Remove custom domain

3.4 New API Endpoints

Method	Path	Description
POST	/projects/{id}/repo	Create git repo
DELETE	/projects/{id}/repo	Delete git repo
GET	/projects/{id}/repo	Get repo info (clone URLs)
POST	/projects/{id}/deploy	Deploy from image
DELETE	/projects/{id}/deploy	Remove deployment
GET	/projects/{id}/deploy/status	Deployment status
POST	/projects/{id}/domain	Add custom domain
DELETE	/projects/{id}/domain	Remove custom domain

---

Phase 4: Database Schema

4.1 Migration: Add Git and Deployment Fields

-- migrations/010_project_infrastructure.up.sql

-- Add infrastructure fields to projects
ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    git_repo_name VARCHAR(255);

ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    git_clone_ssh VARCHAR(512);

ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    git_clone_http VARCHAR(512);

ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    domain VARCHAR(255);

ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    custom_domain VARCHAR(255);

ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    deployment_image VARCHAR(512);

ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    deployment_status VARCHAR(50) DEFAULT 'none';

ALTER TABLE projects ADD COLUMN IF NOT EXISTS
    deployment_replicas INTEGER DEFAULT 1;

-- Index for domain lookups
CREATE INDEX IF NOT EXISTS idx_projects_domain ON projects(domain);
CREATE INDEX IF NOT EXISTS idx_projects_custom_domain ON projects(custom_domain);

---

Phase 5: Pantheon Integration

5.1 New Commands for Agents

/project create <name>
  → Creates project in DB
  → Creates git repo in soft-serve
  → Creates DNS record (<name>.threesix.ai)
  → Returns clone URL

/project deploy <name>
  → Triggers build from latest commit
  → Deploys to k8s
  → Returns live URL

/project status <name>
  → Shows git repo, deployment status, URLs

/project domain <name> <custom-domain>
  → Adds custom domain to project
  → Instructions for DNS pointing

5.2 Webhook Flow

Agent pushes code
        │
        ▼
soft-serve receives push
        │
        ▼
Webhook fires to Woodpecker
        │
        ▼
Woodpecker reads .woodpecker.yml
        │
        ▼
Kaniko builds image, pushes to zot
        │
        ▼
Woodpecker calls rdev-api: POST /projects/{id}/deploy
        │
        ▼
rdev-api creates/updates K8s resources
        │
        ▼
Project live at https://{name}.threesix.ai

---

Implementation Checklist

Phase 1: Foundation

• Create threesix namespace
• Create Cloudflare API secret
• Configure ClusterIssuer for DNS-01 challenge
• Request wildcard certificate
• Deploy soft-serve StatefulSet
• Configure soft-serve LoadBalancer for SSH
• Deploy Zot registry
• Create initial DNS records (git, registry, ci, wildcard)
• Test: ssh git@git.threesix.ai works
• Test: https://registry.threesix.ai shows Zot UI

Phase 2: CI/CD

• Generate Woodpecker agent secret
• Deploy Woodpecker server
• Deploy Woodpecker agents
• Configure soft-serve webhook to Woodpecker
• Test: push triggers build
• Test: Kaniko builds and pushes to Zot

Phase 3: rdev-api

• Add GitRepository port interface
• Add DNSProvider port interface
• Add Deployer port interface
• Implement soft-serve adapter
• Implement Cloudflare adapter
• Implement K8s deployer adapter
• Add database migration
• Add new handlers
• Test: API can create repos
• Test: API can manage DNS
• Test: API can deploy apps

Phase 4: Integration

• Wire up webhook: build → deploy
• Add project commands to Pantheon
• Test: end-to-end "create project" → "push code" → "live site"

Phase 5: Polish

• Custom domain support
• Build notifications to Pantheon
• Deployment logs streaming
• Resource limits per project
• Usage metrics

---

Resource Estimates

Component	CPU Request	Memory Request	Storage
soft-serve	50m	64Mi	10Gi
Zot	100m	128Mi	50Gi
Woodpecker Server	100m	128Mi	5Gi
Woodpecker Agent (x2)	200m each	256Mi each	-
Total	~650m	~832Mi	65Gi

---

Security Considerations

1. soft-serve admin key - Only jordan's key is admin initially
2. Registry access - Internal only, no auth needed (ClusterIP)
3. Woodpecker - Closed registration, admin-only access
4. Cloudflare token - Scoped to DNS edit only
5. Deploy permissions - rdev-api ServiceAccount limited to threesix and projects namespaces

---

Next Steps

1. Review this plan
2. I deploy Phase 1 infrastructure
3. Test git and registry
4. Deploy Phase 2 CI/CD
5. Implement Phase 3 rdev-api changes
6. Integration testing
7. Pantheon integration