rdev/.claude/guides/services/composable-monorepo.md

# Composable Monorepo Templates

**When to use:** Implementing the monorepo skeleton, component templates, or the component addition API.

## Overview

Composable Monorepo Templates evolve rdev from single-template projects to full monorepo scaffolding:

1. **Skeleton**: Every `POST /projects` creates the monorepo base (shared `pkg/`, scripts, CI)
2. **Component Templates**: `POST /projects/{id}/components` adds services/workers/apps/cli
3. **Deployment**: Target whole monorepo or individual components

**Terminology:** "skeleton" = generated project code; "platform" = rdev itself. See [ai-lookup/terminology.md](../../../ai-lookup/terminology.md).

**Plan Document:** `tmp/template-monorepo-plan.md`

## Implementation Phases

### Phase 1: Skeleton Template

Create `templates/skeleton/` with base monorepo structure.

**Files to create:**
```
internal/adapter/templates/templates/skeleton/
├── CLAUDE.md.tmpl
├── README.md.tmpl
├── Procfile.tmpl
├── docker-compose.yml.tmpl
├── go.work.tmpl
├── .golangci.yml
├── .gitignore
├── .woodpecker.yml.tmpl        # ⚠️ Template-provided CI (never AI-generated)
├── .claude/
│   ├── settings.local.json
│   ├── guides/
│   └── skills/
├── scripts/
│   ├── install.sh
│   ├── quality.sh
│   ├── dev.sh
│   └── discover.sh
└── pkg/
    ├── go.mod.tmpl
    ├── app/
    ├── middleware/
    ├── httpcontext/
    ├── httpresponse/
    ├── httpvalidation/
    ├── logging/
    ├── config/
    └── httpclient/
```

**Critical: CI Must Be Templated**

AI-generated `.woodpecker.yml` produces invalid YAML (broken anchor syntax). All CI comes from templates:
- Skeleton provides base `.woodpecker.yml.tmpl`
- Components provide `.woodpecker.step.yml.tmpl`
- AddComponent service inserts component steps into main pipeline

**Template Variables:**
```go
{{.ProjectName}}     // "acme"
{{.GoModule}}        // "github.com/orchard9/acme"
{{.Description}}     // "Project description"
```

**Provider Changes:**
```go
// internal/port/template_provider.go
type TemplateProvider interface {
    // Existing
    GetTemplate(name string) (*Template, error)
    ListTemplates() []Template

    // New
    GetSkeleton() (*Template, error)
    GetComponentTemplate(componentType, templateName string) (*Template, error)
    ListComponentTemplates(componentType string) []Template
}
```

### Phase 1.5: Shared Packages (pkg/)

Extract and combine packages from Aeries + Colix:

| Package | Source | Key Files |
|---------|--------|-----------|
| `app/` | Aeries `pkg/chassis/` | `app.go` |
| `middleware/` | Colix `pkg/middleware/` | `cors.go`, `recovery.go`, `request_id.go`, `logger.go` |
| `httpcontext/` | Colix `pkg/httpcontext/` | `keys.go` |
| `httpresponse/` | Both | `response.go`, `envelope.go` |
| `httpvalidation/` | Colix `pkg/httpvalidation/` | `validator.go`, `errors.go` |
| `logging/` | Both | `logger.go` |
| `config/` | Aeries `pkg/config/` | `config.go` |
| `httpclient/` | Both | `client.go` |

### Phase 2: Component Templates

Migrate existing templates to component format:

```
internal/adapter/templates/templates/components/
├── service/           # Renamed from go-api
│   ├── cmd/server/main.go.tmpl
│   ├── internal/
│   ├── Makefile.tmpl
│   ├── Dockerfile.tmpl
│   ├── component.yaml.tmpl
│   └── .woodpecker.step.yml.tmpl  # CI step for this component type
├── worker/            # NEW
├── app-astro/         # Renamed from astro-landing
├── app-react/         # NEW
└── cli/               # NEW
```

Each component includes `.woodpecker.step.yml.tmpl` that defines its Kaniko build step.
When components are added, the service renders this template and inserts it into the main pipeline.

**Component Variables:**
```go
{{.ProjectName}}        // "acme"
{{.ComponentName}}      // "auth-api"
{{.ComponentNameCamel}} // "AuthApi"
{{.GoModule}}           // "github.com/orchard9/acme"
{{.Port}}               // 8080
```

### Phase 3: Add Component API

Create endpoint for adding components to existing projects.

**Handler:**
```go
// internal/handlers/components.go
func (h *Handler) AddComponent(w http.ResponseWriter, r *http.Request) {
    projectID := chi.URLParam(r, "projectID")

    var req AddComponentRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        api.Error(w, http.StatusBadRequest, "invalid request")
        return
    }

    component, err := h.svc.AddComponent(r.Context(), projectID, req.Type, req.Name, req.Template)
    if err != nil {
        api.Error(w, http.StatusInternalServerError, err.Error())
        return
    }

    api.OK(w, component)
}
```

**Request/Response:**
```go
type AddComponentRequest struct {
    Type     string `json:"type"`     // service, worker, app, cli
    Name     string `json:"name"`     // auth-api, dashboard, etc.
    Template string `json:"template"` // optional: specific variant
}
```

**Service Logic:**
```go
func (s *Service) AddComponent(ctx context.Context, projectID, compType, name, template string) (*domain.Component, error) {
    // 1. Render component template
    // 2. Commit files to project repo
    // 3. Update Procfile with new entry
    // 4. Update go.work if Go component
    // 5. Update CLAUDE.md routing
    // 6. Update .woodpecker.yml with component's build step
    // 7. Return component metadata
}
```

### Phase 4: Component-Aware Deployment

Extend deploy to support component targets:

```go
type DeployRequest struct {
    Component string `json:"component,omitempty"` // e.g., "services/auth-api"
}

// POST /projects/{id}/deploy                    → full monorepo
// POST /projects/{id}/deploy/services/auth-api  → single component
```

### Phase 5: Discovery Scripts

Scripts that walk the monorepo and operate on all components:

```bash
# scripts/discover.sh
#!/bin/bash
for type in services workers apps cli; do
  for dir in "$type"/*/; do
    [ -d "$dir" ] && echo "$type/$(basename $dir)"
  done
done

# scripts/install.sh
#!/bin/bash
for service in services/*/; do
  [ -f "$service/go.mod" ] && (cd "$service" && go mod download)
done
for app in apps/*/; do
  [ -f "$app/package.json" ] && (cd "$app" && npm install)
done

# scripts/quality.sh
#!/bin/bash
golangci-lint run ./services/... ./workers/... ./cli/...
for app in apps/*/; do
  [ -f "$app/package.json" ] && (cd "$app" && npm run lint)
done

# scripts/dev.sh
#!/bin/bash
docker-compose up -d
overmind start
```

### Phase 6: Quality Hooks

Pre-commit hooks for monorepo quality:

```bash
# .githooks/pre-commit
#!/bin/bash
# File length check (500 lines)
for file in $(git diff --cached --name-only --diff-filter=ACM | grep -E '\.(go|ts|tsx)$'); do
  lines=$(wc -l < "$file")
  if [ "$lines" -gt 500 ]; then
    echo "Error: $file exceeds 500 lines ($lines)"
    exit 1
  fi
done

# Go formatting
gofmt -l -w $(git diff --cached --name-only --diff-filter=ACM | grep '\.go$')

# Linting
golangci-lint run ./services/... ./workers/... ./cli/...
```

## Domain Model

```go
// internal/domain/component.go
type ComponentType string

const (
    // Code components (scaffold template files)
    ComponentTypeService  ComponentType = "service"
    ComponentTypeWorker   ComponentType = "worker"
    ComponentTypeAppAstro ComponentType = "app-astro"
    ComponentTypeAppReact ComponentType = "app-react"
    ComponentTypeCLI      ComponentType = "cli"

    // Infrastructure components (provision resources)
    ComponentTypePostgres ComponentType = "postgres"
    ComponentTypeRedis    ComponentType = "redis"
)

type Component struct {
    Type         ComponentType
    Name         string
    Template     string
    Path         string            // "services/auth-api" or "infra/postgres"
    Port         int               // from component.yaml or auto-assigned
    Dependencies []string          // postgres, redis, etc.
    BuildOrder   int
}
```

## Infrastructure Components

Infrastructure components (`postgres`, `redis`) don't scaffold files — they provision actual resources:

### Adding Database (CockroachDB)

```bash
POST /projects/acme/components
{
  "type": "postgres",
  "name": "main-db"
}
```

This:
1. Creates a CockroachDB database: `project_acme`
2. Creates a database user with full permissions
3. Stores `DATABASE_URL` and `DATABASE_URL_STAGING` in credential store

### Adding Cache (Redis)

```bash
POST /projects/acme/components
{
  "type": "redis",
  "name": "job-queue"
}
```

This:
1. Creates a Redis ACL user: `proj-acme`
2. Scopes access to keys matching `project:acme:*`
3. Stores `REDIS_URL`, `REDIS_URL_STAGING`, and `REDIS_PREFIX` in credential store

## Credential Injection

**Critical:** When code components are deployed, credentials are automatically injected.

The `createInitialComponentDeployment()` function:
1. Calls `fetchProjectCredentials(projectID)` to retrieve stored credentials
2. Populates `DeploySpec.Secrets` with `DATABASE_URL`, `REDIS_URL`, etc.
3. Deployer creates K8s Secret and mounts it as environment variables

**Available credentials (if provisioned):**
- `DATABASE_URL` - CockroachDB connection string
- `DATABASE_URL_STAGING` - Staging database (same as prod currently)
- `REDIS_URL` - Redis connection string with auth
- `REDIS_URL_STAGING` - Staging Redis (same as prod currently)
- `REDIS_PREFIX` - Key prefix for isolation (e.g., `project:acme:`)

**Service Discovery:** Sibling services are also injected as env vars:
- `AUTH_SVC_URL=http://acme-auth-svc:8001`
- `CHAT_SVC_URL=http://acme-chat-svc:8002`

**File Pointers:**
- Credential injection: `internal/service/component_deploy.go:35-48, 160-212`
- Infrastructure provisioning: `internal/service/component_infra.go`
- Deployer secret creation: `internal/adapter/deployer/resources.go:81-135`
```

## API Reference

**Important Route Note:** Project CRUD uses `/project` (singular), but component/build/pipeline operations use `/projects/{id}/...` (plural).

### Create Project (Skeleton)

```bash
POST /project
{
  "name": "acme",
  "description": "Acme Corp platform"
}

# Response: Creates monorepo skeleton in repo
```

### Add Component

```bash
POST /projects/acme/components
{
  "type": "service",
  "name": "auth-api",
  "template": "service"  # optional
}

# Response:
{
  "type": "service",
  "name": "auth-api",
  "path": "services/auth-api",
  "port": 8001
}
```

### Deploy

```bash
# Component deploy (single component at a time)
POST /projects/acme/deploy
{
  "component": "services/auth-api"
}

# Note: Full monorepo deploy is handled by CI pipeline, not a single API call
```

### List Components

```bash
GET /projects/acme/components

# Response:
{
  "components": [
    {"type": "service", "name": "auth-api", "path": "services/auth-api", "port": 8001},
    {"type": "app-react", "name": "dashboard", "path": "apps/dashboard", "port": 3001}
  ]
}
```

### Delete Project

```bash
DELETE /project/acme
```

### List Component Templates

```bash
GET /templates/components

# Response shows available component types: service, worker, app-astro, app-react, cli
```

## Routing Patterns

### Service URL Convention

Services are accessed via a consistent URL pattern:

```
https://{project-domain}/api/{service-name}/...
```

**Examples:**
- `https://acme.threesix.ai/api/auth/login` → auth-api service
- `https://acme.threesix.ai/api/chat/messages` → chat-api service
- `https://acme.threesix.ai/` → frontend app (no `/api/` prefix)

### How Routing Works

```
[Client] → [Cloudflare DNS] → [Ingress Controller] → [K8s Service] → [Pod]

1. DNS: *.threesix.ai → cluster IP
2. Ingress: Routes by host + path prefix
3. Service: Load balances to pods
4. Pod: Runs your container on assigned port
```

### Configuring Routes in Trees

When adding services via cookbook trees, routes are configured automatically based on component type and name:

```yaml
# Adding a service creates these routes:
add-auth-service:
  action: api
  method: POST
  endpoint: "/projects/{{ .outputs.create-project.project_id }}/components"
  body:
    type: service
    name: auth  # Routes to /api/auth/*
```

**Port assignment:** Services get auto-assigned ports (8001, 8002, ...). The ingress handles external-to-internal port mapping.

### Common Routing Mistakes

| Mistake | Symptom | Fix |
|---------|---------|-----|
| Missing `/api/` prefix in client | 404 on service calls | Use `/api/{service}/...` |
| Hardcoded localhost:8001 | Works locally, fails in K8s | Use relative paths or env vars |
| Wrong service name in path | 404 or wrong service | Match name from component add |
| CORS errors | Browser blocks requests | Ensure middleware/cors.go is configured |
| Trailing slash mismatch | 301 redirect loops | Be consistent: `/api/auth` not `/api/auth/` |

### Multi-Service Routing

When multiple services exist, they share the domain but have isolated path prefixes:

```yaml
# Project with 3 services
components:
  - type: service, name: auth    # /api/auth/*
  - type: service, name: chat    # /api/chat/*
  - type: service, name: billing # /api/billing/*
  - type: app-react, name: web   # /* (catch-all for frontend)
```

**Frontend API calls:**
```typescript
// In React app - use relative paths
fetch('/api/auth/login', { method: 'POST', body: ... })
fetch('/api/chat/messages')
fetch('/api/billing/invoices')
```

### Internal Service Communication

Services communicate internally via K8s service names, not external URLs:

```yaml
# Service discovery environment variables (auto-injected)
AUTH_SVC_URL=http://acme-auth-svc:8001
CHAT_SVC_URL=http://acme-chat-svc:8002

# In code - use env vars
authURL := os.Getenv("AUTH_SVC_URL")
resp, err := http.Get(authURL + "/internal/validate-token")
```

**Internal vs External:**
- **External** (from browser): `https://acme.threesix.ai/api/auth/...`
- **Internal** (service-to-service): `http://acme-auth-svc:8001/...`

Internal routes can have endpoints not exposed externally (e.g., `/internal/*`).

## Testing

```go
func TestAddComponent(t *testing.T) {
    // Setup project first
    project := createTestProject(t, "test-monorepo")

    // Add service component
    req := AddComponentRequest{
        Type:     "service",
        Name:     "auth-api",
        Template: "go-api",
    }

    component, err := svc.AddComponent(ctx, project.ID, req.Type, req.Name, req.Template)
    require.NoError(t, err)

    assert.Equal(t, "service", string(component.Type))
    assert.Equal(t, "auth-api", component.Name)
    assert.Equal(t, "services/auth-api", component.Path)

    // Verify files created in repo
    files, err := gitea.ListFiles(project.Owner, project.Name, "services/auth-api")
    require.NoError(t, err)
    assert.Contains(t, files, "cmd/server/main.go")
    assert.Contains(t, files, "Makefile")
}
```

## Troubleshooting

### Component endpoint returns 404

The component service is only initialized when `GITEA_URL`, `GITEA_TOKEN`, and template provider are all configured. Check startup logs for "component service initialized":

```bash
kubectl logs -n rdev deployment/rdev-api | grep "component service"
```

If missing, verify the rdev-api deployment has the required env vars.

### Component addition fails with "directory exists"

The component path already exists. Either delete it or choose a different name.

### Component addition fails with "connection refused"

**Hairpin NAT issue.** rdev-api is trying to reach Gitea via external URL which doesn't work from within the cluster. Use internal service hostnames:

```yaml
# In deployments/k8s/base/rdev-api.yaml
- name: GITEA_URL
  value: "http://gitea.threesix.svc.cluster.local"
```

See [ops/networking.md](../ops/networking.md) for details.

### CI pipeline linter errors

Woodpecker may show linter warnings about oneOf/anyOf validation. These are usually benign. Check for actual schema violations like invalid `when.event` values.

**Critical:** The `.woodpecker.yml` marker must be exactly `# COMPONENT_STEPS_BELOW` on its own line. If the marker has trailing text (like `- Do not remove`), it will be appended to the last line of inserted component steps, corrupting the YAML.

**File Pointer:** `internal/adapter/templates/templates/skeleton/.woodpecker.yml.tmpl:11`

### Procfile not updating

Check that the template includes the Procfile.tmpl and the service has write access to the repo.

### go.work not syncing

Run `./scripts/discover.sh` to verify component detection, then manually run `go work sync`.

## Related

- [Project Templates](./templates.md) - Current single-template system
- [Build Orchestration](./build-orchestration.md) - Component builds
- [ai-lookup: Composable Monorepo](../../ai-lookup/features/composable-monorepo.md) - Quick facts