rdev/.claude/guides/services/composable-monorepo.md

Composable Monorepo Templates

When to use: Implementing the monorepo skeleton, component templates, or the component addition API.

Overview

Composable Monorepo Templates evolve rdev from single-template projects to full monorepo scaffolding:

1. Skeleton: Every POST /projects creates the monorepo base (shared pkg/, scripts, CI)
2. Component Templates: POST /projects/{id}/components adds services/workers/apps/cli
3. Deployment: Target whole monorepo or individual components

Terminology: "skeleton" = generated project code; "platform" = rdev itself. See ai-lookup/terminology.md.

Plan Document: tmp/template-monorepo-plan.md

Implementation Phases

Phase 1: Skeleton Template

Create templates/skeleton/ with base monorepo structure.

Files to create:

internal/adapter/templates/templates/skeleton/
├── CLAUDE.md.tmpl
├── README.md.tmpl
├── Procfile.tmpl
├── docker-compose.yml.tmpl
├── go.work.tmpl
├── .golangci.yml
├── .gitignore
├── .woodpecker.yml.tmpl        # ⚠️ Template-provided CI (never AI-generated)
├── .claude/
│   ├── settings.local.json
│   ├── guides/
│   └── skills/
├── scripts/
│   ├── install.sh
│   ├── quality.sh
│   ├── dev.sh
│   └── discover.sh
└── pkg/
    ├── go.mod.tmpl
    ├── app/
    ├── middleware/
    ├── httpcontext/
    ├── httpresponse/
    ├── httpvalidation/
    ├── logging/
    ├── config/
    └── httpclient/

Critical: CI Must Be Templated

AI-generated .woodpecker.yml produces invalid YAML (broken anchor syntax). All CI comes from templates:

• Skeleton provides base .woodpecker.yml.tmpl
• Components provide .woodpecker.step.yml.tmpl
• AddComponent service inserts component steps into main pipeline

Template Variables:

{{.ProjectName}}     // "acme"
{{.GoModule}}        // "github.com/orchard9/acme"
{{.Description}}     // "Project description"

Provider Changes:

// internal/port/template_provider.go
type TemplateProvider interface {
    // Existing
    GetTemplate(name string) (*Template, error)
    ListTemplates() []Template

    // New
    GetSkeleton() (*Template, error)
    GetComponentTemplate(componentType, templateName string) (*Template, error)
    ListComponentTemplates(componentType string) []Template
}

Phase 1.5: Shared Packages (pkg/)

Extract and combine packages from Aeries + Colix:

Package	Source	Key Files
app/	Aeries pkg/chassis/	app.go
middleware/	Colix pkg/middleware/	cors.go, recovery.go, request_id.go, logger.go
httpcontext/	Colix pkg/httpcontext/	keys.go
httpresponse/	Both	response.go, envelope.go
httpvalidation/	Colix pkg/httpvalidation/	validator.go, errors.go
logging/	Both	logger.go
config/	Aeries pkg/config/	config.go
httpclient/	Both	client.go

Phase 2: Component Templates

Migrate existing templates to component format:

internal/adapter/templates/templates/components/
├── service/           # Renamed from go-api
│   ├── cmd/server/main.go.tmpl
│   ├── internal/
│   ├── Makefile.tmpl
│   ├── Dockerfile.tmpl
│   ├── component.yaml.tmpl
│   └── .woodpecker.step.yml.tmpl  # CI step for this component type
├── worker/            # NEW
├── app-astro/         # Renamed from astro-landing
├── app-react/         # NEW
└── cli/               # NEW

Each component includes .woodpecker.step.yml.tmpl that defines its Kaniko build step. When components are added, the service renders this template and inserts it into the main pipeline.

Component CI Step Rules:

• ALL component build steps MUST use depends_on: [preflight] (not [deps]) — this ensures registry health is verified before any image build
• Components with Docker images (service, app-*, worker) MUST have a 3-step chain: build → verify (registry check) → deploy with explicit depends_on between each
• Build steps MUST NOT use failure: ignore — build failures should be visible, not swallowed
• The verify step uses failure: ignore so deploy is skipped (not failed) when the image doesn't exist
• updateWoodpeckerYml automatically wires each new deploy-{name} step into build-complete's depends_on via the # BUILD_COMPLETE_DEPS marker

Component Variables:

{{.ProjectName}}        // "acme"
{{.ComponentName}}      // "auth-api"
{{.ComponentNameCamel}} // "AuthApi"
{{.GoModule}}           // "github.com/orchard9/acme"
{{.Port}}               // 8080

Phase 3: Add Component API

Create endpoint for adding components to existing projects.

Handler:

// internal/handlers/components.go
func (h *Handler) AddComponent(w http.ResponseWriter, r *http.Request) {
    projectID := chi.URLParam(r, "projectID")

    var req AddComponentRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        api.Error(w, http.StatusBadRequest, "invalid request")
        return
    }

    component, err := h.svc.AddComponent(r.Context(), projectID, req.Type, req.Name, req.Template)
    if err != nil {
        api.Error(w, http.StatusInternalServerError, err.Error())
        return
    }

    api.OK(w, component)
}

Request/Response:

type AddComponentRequest struct {
    Type     string `json:"type"`     // service, worker, app, cli
    Name     string `json:"name"`     // auth-api, dashboard, etc.
    Template string `json:"template"` // optional: specific variant
}

Service Logic:

func (s *Service) AddComponent(ctx context.Context, projectID, compType, name, template string) (*domain.Component, error) {
    // 1. Render component template
    // 2. Commit files to project repo
    // 3. Update Procfile with new entry
    // 4. Update go.work if Go component
    // 5. Update CLAUDE.md routing
    // 6. Update .woodpecker.yml with component's build step
    // 7. Return component metadata
}

Phase 4: Component-Aware Deployment

Extend deploy to support component targets:

type DeployRequest struct {
    Component string `json:"component,omitempty"` // e.g., "services/auth-api"
}

// POST /projects/{id}/deploy                    → full monorepo
// POST /projects/{id}/deploy/services/auth-api  → single component

Phase 5: Discovery Scripts

Scripts that walk the monorepo and operate on all components:

# scripts/discover.sh
#!/bin/bash
for type in services workers apps cli; do
  for dir in "$type"/*/; do
    [ -d "$dir" ] && echo "$type/$(basename $dir)"
  done
done

# scripts/install.sh
#!/bin/bash
for service in services/*/; do
  [ -f "$service/go.mod" ] && (cd "$service" && go mod download)
done
for app in apps/*/; do
  [ -f "$app/package.json" ] && (cd "$app" && npm install)
done

# scripts/quality.sh
#!/bin/bash
golangci-lint run ./services/... ./workers/... ./cli/...
for app in apps/*/; do
  [ -f "$app/package.json" ] && (cd "$app" && npm run lint)
done

# scripts/dev.sh
#!/bin/bash
docker-compose up -d
overmind start

Phase 6: Quality Hooks

Pre-commit hooks for monorepo quality:

# .githooks/pre-commit
#!/bin/bash
# File length check (500 lines)
for file in $(git diff --cached --name-only --diff-filter=ACM | grep -E '\.(go|ts|tsx)$'); do
  lines=$(wc -l < "$file")
  if [ "$lines" -gt 500 ]; then
    echo "Error: $file exceeds 500 lines ($lines)"
    exit 1
  fi
done

# Go formatting
gofmt -l -w $(git diff --cached --name-only --diff-filter=ACM | grep '\.go$')

# Linting
golangci-lint run ./services/... ./workers/... ./cli/...

Domain Model

// internal/domain/component.go
type ComponentType string

const (
    // Code components (scaffold template files)
    ComponentTypeService  ComponentType = "service"
    ComponentTypeWorker   ComponentType = "worker"
    ComponentTypeAppAstro ComponentType = "app-astro"
    ComponentTypeAppReact ComponentType = "app-react"
    ComponentTypeCLI      ComponentType = "cli"

    // Infrastructure components (provision resources)
    ComponentTypePostgres ComponentType = "postgres"
    ComponentTypeRedis    ComponentType = "redis"
)

type Component struct {
    Type         ComponentType
    Name         string
    Template     string
    Path         string            // "services/auth-api" or "infra/postgres"
    Port         int               // from component.yaml or auto-assigned
    Dependencies []string          // postgres, redis, etc.
    BuildOrder   int
}

Infrastructure Components

Infrastructure components (postgres, redis) don't scaffold files — they provision actual resources:

Adding Database (CockroachDB)

POST /projects/acme/components
{
  "type": "postgres",
  "name": "main-db"
}

This:

1. Creates a CockroachDB database: project_acme
2. Creates a database user with full permissions
3. Stores DATABASE_URL and DATABASE_URL_STAGING in credential store

Adding Cache (Redis)

POST /projects/acme/components
{
  "type": "redis",
  "name": "job-queue"
}

This:

1. Creates a Redis ACL user: proj-acme
2. Scopes access to keys matching project:acme:*
3. Stores REDIS_URL, REDIS_URL_STAGING, and REDIS_PREFIX in credential store

Credential Injection

Critical: When code components are deployed, credentials are automatically injected.

The createInitialComponentDeployment() function:

1. Calls fetchProjectCredentials(projectID) to retrieve stored credentials
2. Populates DeploySpec.Secrets with DATABASE_URL, REDIS_URL, etc.
3. Deployer creates K8s Secret and mounts it as environment variables

Available credentials (if provisioned):

• DATABASE_URL - CockroachDB connection string
• DATABASE_URL_STAGING - Staging database (same as prod currently)
• REDIS_URL - Redis connection string with auth
• REDIS_URL_STAGING - Staging Redis (same as prod currently)
• REDIS_PREFIX - Key prefix for isolation (e.g., project:acme:)

Service Discovery: Sibling services are also injected as env vars:

• AUTH_SVC_URL=http://acme-auth-svc:8001
• CHAT_SVC_URL=http://acme-chat-svc:8002

File Pointers:

• Credential injection: internal/service/component_deploy.go:35-48, 160-212
• Infrastructure provisioning: internal/service/component_infra.go
• Deployer secret creation: internal/adapter/deployer/resources.go:81-135

## API Reference

**Important Route Note:** Project CRUD uses `/project` (singular), but component/build/pipeline operations use `/projects/{id}/...` (plural).

### Create Project (Skeleton)

```bash
POST /project
{
  "name": "acme",
  "description": "Acme Corp platform"
}

# Response: Creates monorepo skeleton in repo

Add Component

POST /projects/acme/components
{
  "type": "service",
  "name": "auth-api",
  "template": "service"  # optional
}

# Response:
{
  "type": "service",
  "name": "auth-api",
  "path": "services/auth-api",
  "port": 8001
}

Deploy

# Component deploy (single component at a time)
POST /projects/acme/deploy
{
  "component": "services/auth-api"
}

# Note: Full monorepo deploy is handled by CI pipeline, not a single API call

List Components

GET /projects/acme/components

# Response:
{
  "components": [
    {"type": "service", "name": "auth-api", "path": "services/auth-api", "port": 8001},
    {"type": "app-react", "name": "dashboard", "path": "apps/dashboard", "port": 3001}
  ]
}

Delete Project

DELETE /project/acme

List Component Templates

GET /templates/components

# Response shows available component types: service, worker, app-astro, app-react, cli

Routing Patterns

Service URL Convention

Services are accessed via a consistent URL pattern:

https://{project-domain}/api/{service-name}/...

Examples:

• https://acme.threesix.ai/api/auth/login → auth-api service
• https://acme.threesix.ai/api/chat/messages → chat-api service
• https://acme.threesix.ai/ → frontend app (no /api/ prefix)

How Routing Works

[Client] → [Cloudflare DNS] → [Ingress Controller] → [K8s Service] → [Pod]

1. DNS: *.threesix.ai → cluster IP
2. Ingress: Routes by host + path prefix
3. Service: Load balances to pods
4. Pod: Runs your container on assigned port

Configuring Routes in Trees

When adding services via cookbook trees, routes are configured automatically based on component type and name:

# Adding a service creates these routes:
add-auth-service:
  action: api
  method: POST
  endpoint: "/projects/{{ .outputs.create-project.project_id }}/components"
  body:
    type: service
    name: auth  # Routes to /api/auth/*

Port assignment: Services get auto-assigned ports (8001, 8002, ...). The ingress handles external-to-internal port mapping.

Common Routing Mistakes

Mistake	Symptom	Fix
Missing /api/ prefix in client	404 on service calls	Use /api/{service}/...
Hardcoded localhost:8001	Works locally, fails in K8s	Use relative paths or env vars
Wrong service name in path	404 or wrong service	Match name from component add
CORS errors	Browser blocks requests	Ensure middleware/cors.go is configured
Trailing slash mismatch	301 redirect loops	Be consistent: /api/auth not /api/auth/

Multi-Service Routing

When multiple services exist, they share the domain but have isolated path prefixes:

# Project with 3 services
components:
  - type: service, name: auth    # /api/auth/*
  - type: service, name: chat    # /api/chat/*
  - type: service, name: billing # /api/billing/*
  - type: app-react, name: web   # /* (catch-all for frontend)

Frontend API calls:

// In React app - use relative paths
fetch('/api/auth/login', { method: 'POST', body: ... })
fetch('/api/chat/messages')
fetch('/api/billing/invoices')

Internal Service Communication

Services communicate internally via K8s service names, not external URLs:

# Service discovery environment variables (auto-injected)
AUTH_SVC_URL=http://acme-auth-svc:8001
CHAT_SVC_URL=http://acme-chat-svc:8002

# In code - use env vars
authURL := os.Getenv("AUTH_SVC_URL")
resp, err := http.Get(authURL + "/internal/validate-token")

Internal vs External:

• External (from browser): https://acme.threesix.ai/api/auth/...
• Internal (service-to-service): http://acme-auth-svc:8001/...

Internal routes can have endpoints not exposed externally (e.g., /internal/*).

Testing

func TestAddComponent(t *testing.T) {
    // Setup project first
    project := createTestProject(t, "test-monorepo")

    // Add service component
    req := AddComponentRequest{
        Type:     "service",
        Name:     "auth-api",
        Template: "go-api",
    }

    component, err := svc.AddComponent(ctx, project.ID, req.Type, req.Name, req.Template)
    require.NoError(t, err)

    assert.Equal(t, "service", string(component.Type))
    assert.Equal(t, "auth-api", component.Name)
    assert.Equal(t, "services/auth-api", component.Path)

    // Verify files created in repo
    files, err := gitea.ListFiles(project.Owner, project.Name, "services/auth-api")
    require.NoError(t, err)
    assert.Contains(t, files, "cmd/server/main.go")
    assert.Contains(t, files, "Makefile")
}

Troubleshooting

Component endpoint returns 404

The component service is only initialized when GITEA_URL, GITEA_TOKEN, and template provider are all configured. Check startup logs for "component service initialized":

kubectl logs -n rdev deployment/rdev-api | grep "component service"

If missing, verify the rdev-api deployment has the required env vars.

Component addition fails with "directory exists"

The component path already exists. Either delete it or choose a different name.

Component addition fails with "connection refused"

Hairpin NAT issue. rdev-api is trying to reach Gitea via external URL which doesn't work from within the cluster. Use internal service hostnames:

# In deployments/k8s/base/rdev-api.yaml
- name: GITEA_URL
  value: "http://gitea.threesix.svc.cluster.local"

See ops/networking.md for details.

CI pipeline linter errors

Woodpecker may show linter warnings about oneOf/anyOf validation. These are usually benign. Check for actual schema violations like invalid when.event values.

Critical: The .woodpecker.yml marker must be exactly # COMPONENT_STEPS_BELOW on its own line. If the marker has trailing text (like - Do not remove), it will be appended to the last line of inserted component steps, corrupting the YAML.

File Pointer: internal/adapter/templates/templates/skeleton/.woodpecker.yml.tmpl:11

Procfile not updating

Check that the template includes the Procfile.tmpl and the service has write access to the repo.

go.work not syncing

Run ./scripts/discover.sh to verify component detection, then manually run go work sync.

Related

• Project Templates - Current single-template system
• Build Orchestration - Component builds
• ai-lookup: Composable Monorepo - Quick facts