sp2-verify-1770324794/.sdlc/features/async-jobs/design.md

Technical Design: Async Jobs

Feature: async-jobs Status: approved Author: Claude Created: 2026-02-05

Architecture Overview

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   API Service   │────▶│      Redis      │◀────│ Background      │
│  (services/api) │     │                 │     │ Worker          │
└─────────────────┘     └─────────────────┘     └─────────────────┘
       │                        │                      │
       │ POST /jobs             │ jobs:queue           │ BLPOP
       │ GET /jobs/{id}         │ jobs:data:{id}       │ Update status
       └────────────────────────┴──────────────────────┘

Component Design

1. Redis Job Queue Package (pkg/redisqueue)

A new shared package providing Redis-based job queue operations:

// pkg/redisqueue/queue.go
type RedisQueue struct {
    client *redis.Client
    logger *logging.Logger
}

func NewRedisQueue(client *redis.Client, logger *logging.Logger) *RedisQueue

// Producer operations (for API)
func (q *RedisQueue) Enqueue(ctx context.Context, job *Job) error
func (q *RedisQueue) GetJob(ctx context.Context, jobID string) (*Job, error)

// Consumer operations (for Worker)
func (q *RedisQueue) Dequeue(ctx context.Context, timeout time.Duration) (*Job, error)
func (q *RedisQueue) UpdateStatus(ctx context.Context, jobID string, status JobStatus, err string) error

2. API Service Modifications

New Files:

• services/api/internal/api/handlers/job.go - Job HTTP handlers
• services/api/internal/port/job.go - JobQueue port interface
• services/api/internal/service/job.go - Job business logic

Modified Files:

• services/api/cmd/server/main.go - Add Redis client, job service initialization
• services/api/internal/api/routes.go - Register job routes
• services/api/internal/config/config.go - Add Redis URL config

3. Worker Modifications

Modified Files:

• workers/background-processor/cmd/worker/main.go - Add Redis client, job handler
• workers/background-processor/internal/config/config.go - Add Redis URL, work simulation config
• workers/background-processor/internal/handlers/jobs.go - Async job handler

Redis Data Structure

Queue List: jobs:queue

• Type: List
• Operations: RPUSH (enqueue), BLPOP (dequeue)
• Contains: Job IDs only (lightweight)

Job Data: jobs:data:{id}

• Type: Hash (stored as JSON string for simplicity)
• Fields: id, type, payload, status, created_at, started_at, completed_at, error
• TTL: 24 hours after completion (configurable)

Sequence Diagrams

Create Job Flow

Client → API → JobService.Create()
                    │
                    ├── Generate UUID
                    ├── Create Job struct
                    ├── SET jobs:data:{id} (JSON)
                    ├── RPUSH jobs:queue (id only)
                    └── Return job with pending status

Get Job Flow

Client → API → JobService.Get(id)
                    │
                    ├── GET jobs:data:{id}
                    └── Return job or 404

Worker Processing Flow

Worker → RedisQueue.Dequeue()
              │
              ├── BLPOP jobs:queue
              ├── GET jobs:data:{id}
              ├── Update status to "running"
              ├── Simulate work (sleep)
              └── Update status to "completed"

Configuration

API Service

REDIS_URL=redis://localhost:6379

Worker

REDIS_URL=redis://localhost:6379
JOB_SIMULATION_DURATION=2s  # Duration to simulate work

Error Handling

Scenario	Behavior
Redis connection failure	Return 503 Service Unavailable
Job not found	Return 404 Not Found
Invalid job payload	Return 400 Bad Request
Worker crash during processing	Job remains in "running" (future: add timeout/recovery)

Testing Strategy

1. Unit Tests: Mock Redis client, test service logic
2. Integration Tests: Real Redis (via testcontainers or local), test full flow

Files to Create/Modify

New Files

1. pkg/redisqueue/queue.go - Redis queue implementation
2. pkg/redisqueue/job.go - Job struct and status constants
3. services/api/internal/api/handlers/job.go - Job handlers
4. services/api/internal/api/handlers/job_test.go - Handler tests
5. services/api/internal/port/job.go - JobQueue interface
6. services/api/internal/service/job.go - Job service
7. workers/background-processor/internal/handlers/jobs.go - Job processor

Modified Files

1. services/api/cmd/server/main.go - Add Redis setup
2. services/api/internal/api/routes.go - Add job routes
3. services/api/internal/config/config.go - Add Redis config
4. workers/background-processor/cmd/worker/main.go - Add Redis queue processing
5. workers/background-processor/internal/config/config.go - Add Redis config