152 lines
5.2 KiB
Markdown
152 lines
5.2 KiB
Markdown
# Technical Design: Async Jobs
|
|
|
|
**Feature:** async-jobs
|
|
**Status:** approved
|
|
**Author:** Claude
|
|
**Created:** 2026-02-05
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ API Service │────▶│ Redis │◀────│ Background │
|
|
│ (services/api) │ │ │ │ Worker │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
│ │ │
|
|
│ POST /jobs │ jobs:queue │ BLPOP
|
|
│ GET /jobs/{id} │ jobs:data:{id} │ Update status
|
|
└────────────────────────┴──────────────────────┘
|
|
```
|
|
|
|
## Component Design
|
|
|
|
### 1. Redis Job Queue Package (`pkg/redisqueue`)
|
|
|
|
A new shared package providing Redis-based job queue operations:
|
|
|
|
```go
|
|
// pkg/redisqueue/queue.go
|
|
type RedisQueue struct {
|
|
client *redis.Client
|
|
logger *logging.Logger
|
|
}
|
|
|
|
func NewRedisQueue(client *redis.Client, logger *logging.Logger) *RedisQueue
|
|
|
|
// Producer operations (for API)
|
|
func (q *RedisQueue) Enqueue(ctx context.Context, job *Job) error
|
|
func (q *RedisQueue) GetJob(ctx context.Context, jobID string) (*Job, error)
|
|
|
|
// Consumer operations (for Worker)
|
|
func (q *RedisQueue) Dequeue(ctx context.Context, timeout time.Duration) (*Job, error)
|
|
func (q *RedisQueue) UpdateStatus(ctx context.Context, jobID string, status JobStatus, err string) error
|
|
```
|
|
|
|
### 2. API Service Modifications
|
|
|
|
**New Files:**
|
|
- `services/api/internal/api/handlers/job.go` - Job HTTP handlers
|
|
- `services/api/internal/port/job.go` - JobQueue port interface
|
|
- `services/api/internal/service/job.go` - Job business logic
|
|
|
|
**Modified Files:**
|
|
- `services/api/cmd/server/main.go` - Add Redis client, job service initialization
|
|
- `services/api/internal/api/routes.go` - Register job routes
|
|
- `services/api/internal/config/config.go` - Add Redis URL config
|
|
|
|
### 3. Worker Modifications
|
|
|
|
**Modified Files:**
|
|
- `workers/background-processor/cmd/worker/main.go` - Add Redis client, job handler
|
|
- `workers/background-processor/internal/config/config.go` - Add Redis URL, work simulation config
|
|
- `workers/background-processor/internal/handlers/jobs.go` - Async job handler
|
|
|
|
## Redis Data Structure
|
|
|
|
### Queue List: `jobs:queue`
|
|
- Type: List
|
|
- Operations: RPUSH (enqueue), BLPOP (dequeue)
|
|
- Contains: Job IDs only (lightweight)
|
|
|
|
### Job Data: `jobs:data:{id}`
|
|
- Type: Hash (stored as JSON string for simplicity)
|
|
- Fields: id, type, payload, status, created_at, started_at, completed_at, error
|
|
- TTL: 24 hours after completion (configurable)
|
|
|
|
## Sequence Diagrams
|
|
|
|
### Create Job Flow
|
|
```
|
|
Client → API → JobService.Create()
|
|
│
|
|
├── Generate UUID
|
|
├── Create Job struct
|
|
├── SET jobs:data:{id} (JSON)
|
|
├── RPUSH jobs:queue (id only)
|
|
└── Return job with pending status
|
|
```
|
|
|
|
### Get Job Flow
|
|
```
|
|
Client → API → JobService.Get(id)
|
|
│
|
|
├── GET jobs:data:{id}
|
|
└── Return job or 404
|
|
```
|
|
|
|
### Worker Processing Flow
|
|
```
|
|
Worker → RedisQueue.Dequeue()
|
|
│
|
|
├── BLPOP jobs:queue
|
|
├── GET jobs:data:{id}
|
|
├── Update status to "running"
|
|
├── Simulate work (sleep)
|
|
└── Update status to "completed"
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### API Service
|
|
```bash
|
|
REDIS_URL=redis://localhost:6379
|
|
```
|
|
|
|
### Worker
|
|
```bash
|
|
REDIS_URL=redis://localhost:6379
|
|
JOB_SIMULATION_DURATION=2s # Duration to simulate work
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
| Scenario | Behavior |
|
|
|----------|----------|
|
|
| Redis connection failure | Return 503 Service Unavailable |
|
|
| Job not found | Return 404 Not Found |
|
|
| Invalid job payload | Return 400 Bad Request |
|
|
| Worker crash during processing | Job remains in "running" (future: add timeout/recovery) |
|
|
|
|
## Testing Strategy
|
|
|
|
1. **Unit Tests**: Mock Redis client, test service logic
|
|
2. **Integration Tests**: Real Redis (via testcontainers or local), test full flow
|
|
|
|
## Files to Create/Modify
|
|
|
|
### New Files
|
|
1. `pkg/redisqueue/queue.go` - Redis queue implementation
|
|
2. `pkg/redisqueue/job.go` - Job struct and status constants
|
|
3. `services/api/internal/api/handlers/job.go` - Job handlers
|
|
4. `services/api/internal/api/handlers/job_test.go` - Handler tests
|
|
5. `services/api/internal/port/job.go` - JobQueue interface
|
|
6. `services/api/internal/service/job.go` - Job service
|
|
7. `workers/background-processor/internal/handlers/jobs.go` - Job processor
|
|
|
|
### Modified Files
|
|
1. `services/api/cmd/server/main.go` - Add Redis setup
|
|
2. `services/api/internal/api/routes.go` - Add job routes
|
|
3. `services/api/internal/config/config.go` - Add Redis config
|
|
4. `workers/background-processor/cmd/worker/main.go` - Add Redis queue processing
|
|
5. `workers/background-processor/internal/config/config.go` - Add Redis config
|