sp4-fresh/.claude/skills/microservices/SKILL.md

name	description
microservices	Inter-service communication patterns using pkg/svc for service discovery and circuit breaker protection. Use when implementing service-to-service calls.

Microservices Communication

Identity

You are a distributed systems engineer who understands the pitfalls of microservice communication. You prioritize resilience, observability, and graceful degradation over feature velocity.

Service Discovery

Services discover siblings via environment variables injected automatically by the platform.

How It Works

When a component is deployed, it receives env vars for all sibling services:

• AUTH_SVC_URL=http://myproject-auth-svc:8001
• CHAT_SVC_URL=http://myproject-chat-svc:8002

The naming convention: {COMPONENT_NAME}_URL where COMPONENT_NAME is UPPER_SNAKE_CASE.

Using pkg/svc

import "git.threesix.ai/jordan/sp4-fresh/pkg/svc"

// Simple lookup
url := svc.ServiceURL("auth-svc")
if url == "" {
    // Service not configured
}

// Check availability
if svc.ServiceConfigured("auth-svc") {
    // Safe to call
}

// For required dependencies (panics if missing)
url := svc.MustServiceURL("auth-svc")

Service Client

Use svc.NewClient() for a pre-configured HTTP client with circuit breaker protection.

Basic Usage

import "git.threesix.ai/jordan/sp4-fresh/pkg/svc"

// Create client (returns error if service not configured)
authClient, err := svc.NewClient("auth-svc")
if err != nil {
    return fmt.Errorf("auth service unavailable: %w", err)
}

// Make requests
resp, err := authClient.Get(ctx, "/users/123")
if err != nil {
    if errors.Is(err, httpclient.ErrCircuitOpen) {
        // Circuit breaker is open - service is unhealthy
        return ErrAuthServiceDown
    }
    return fmt.Errorf("auth request failed: %w", err)
}
defer resp.Body.Close()

// JSON POST
resp, err := authClient.Post(ctx, "/validate", ValidateRequest{Token: token})

Custom Configuration

client, err := svc.NewClientWithConfig("auth-svc", svc.ClientConfig{
    Timeout:    5 * time.Second,  // Shorter timeout for fast-fail
    MaxRetries: 2,                // Fewer retries
    CircuitBreaker: &httpclient.CircuitBreakerConfig{
        FailureThreshold: 3,      // Open after 3 failures
        ResetTimeout:     15 * time.Second,
    },
})

Circuit Breaker

The circuit breaker prevents cascading failures by failing fast when a service is unhealthy.

States

State	Behavior
Closed	Normal operation, requests pass through
Open	Blocks all requests, returns ErrCircuitOpen immediately
Half-Open	Allows one test request to check if service recovered

Default Thresholds

• Opens after 5 consecutive failures
• Waits 30s before attempting recovery (half-open)
• Closes after one successful request in half-open state

What Affects Circuit State

The circuit breaker tracks transient failures only:

Response	Affects Circuit?	Reason
HTTP 2xx/3xx	✅ RecordSuccess	Service is healthy
HTTP 5xx	✅ RecordFailure	Server error - transient
HTTP 429	✅ RecordFailure	Rate limited - transient
HTTP 4xx (except 429)	❌ No effect	Client error - not service's fault
Network error	✅ RecordFailure	Connection failed
Context cancelled	❌ No effect	User/caller initiated
Timeout	✅ RecordFailure	Service too slow

Key insight: 4xx responses (bad requests, not found, unauthorized) don't trip the circuit because they indicate a problem with the request, not the service. A service returning 400s is still "healthy" from a circuit breaker perspective.

Handling Circuit Open

resp, err := authClient.Get(ctx, "/users/123")
if errors.Is(err, httpclient.ErrCircuitOpen) {
    // Option 1: Return degraded response
    return CachedUserData(userID)

    // Option 2: Propagate as service unavailable
    return nil, ErrServiceTemporarilyUnavailable

    // Option 3: Use fallback service
    return fallbackClient.Get(ctx, "/users/123")
}

Patterns

Initialization Pattern

Initialize service clients at startup, not on-demand:

type Server struct {
    authClient *svc.Client
    chatClient *svc.Client
}

func NewServer() (*Server, error) {
    authClient, err := svc.NewClient("auth-svc")
    if err != nil {
        return nil, fmt.Errorf("auth service required: %w", err)
    }

    // Optional dependency - check but don't fail
    var chatClient *svc.Client
    if svc.ServiceConfigured("chat-svc") {
        chatClient, _ = svc.NewClient("chat-svc")
    }

    return &Server{
        authClient: authClient,
        chatClient: chatClient,
    }, nil
}

Response Decoding

type User struct {
    ID   string `json:"id"`
    Name string `json:"name"`
}

resp, err := authClient.Get(ctx, "/users/123")
if err != nil {
    return nil, err
}

user, err := svc.DecodeResponse[User](resp)
if err != nil {
    return nil, fmt.Errorf("decode user: %w", err)
}

Graceful Degradation

func (s *Server) GetUserProfile(ctx context.Context, userID string) (*Profile, error) {
    // Required call
    user, err := s.fetchUser(ctx, userID)
    if err != nil {
        return nil, err
    }

    profile := &Profile{User: user}

    // Optional enrichment - don't fail if chat service is down
    if s.chatClient != nil {
        messages, err := s.fetchRecentMessages(ctx, userID)
        if err != nil {
            s.logger.Warn("failed to fetch messages", "error", err)
            // Continue without messages
        } else {
            profile.RecentMessages = messages
        }
    }

    return profile, nil
}

Anti-Patterns

Hardcoded URLs

// BAD: Hardcoded URLs break when services move
client := httpclient.New(httpclient.Config{})
resp, err := client.Get(ctx, "http://auth-svc:8001/users")

// GOOD: Use service discovery
authClient, _ := svc.NewClient("auth-svc")
resp, err := authClient.Get(ctx, "/users")

Ignoring Circuit Breaker Errors

// BAD: Retrying forever when circuit is open
for {
    resp, err := authClient.Get(ctx, "/users")
    if err != nil {
        time.Sleep(time.Second)
        continue
    }
}

// GOOD: Detect circuit open and handle gracefully
resp, err := authClient.Get(ctx, "/users")
if errors.Is(err, httpclient.ErrCircuitOpen) {
    return nil, ErrServiceUnavailable
}

On-Demand Client Creation

// BAD: Creating client on every request
func (h *Handler) GetUser(w http.ResponseWriter, r *http.Request) {
    client, _ := svc.NewClient("auth-svc")  // Wastes resources
    // ...
}

// GOOD: Reuse client instance
type Handler struct {
    authClient *svc.Client
}

func (h *Handler) GetUser(w http.ResponseWriter, r *http.Request) {
    resp, _ := h.authClient.Get(r.Context(), "/users")
    // ...
}

Silent Failures

// BAD: Swallowing errors
resp, _ := authClient.Get(ctx, "/validate")
if resp != nil && resp.StatusCode == 200 {
    // Assume success
}

// GOOD: Explicit error handling
resp, err := authClient.Get(ctx, "/validate")
if err != nil {
    return fmt.Errorf("auth validation: %w", err)
}
if resp.StatusCode != http.StatusOK {
    return fmt.Errorf("auth validation failed: %d", resp.StatusCode)
}

Checklist

When implementing inter-service calls:

• Use svc.NewClient() instead of raw HTTP clients
• Handle ErrCircuitOpen explicitly
• Initialize clients at startup, not on-demand
• Log service call failures with context
• Consider graceful degradation for optional dependencies
• Set appropriate timeouts (shorter than HTTP handler timeout)
• Propagate trace IDs for distributed tracing

Files

File	Purpose
pkg/svc/discovery.go	Service URL lookup from env vars
pkg/svc/client.go	Pre-configured service client
pkg/httpclient/circuit.go	Circuit breaker implementation
pkg/httpclient/client.go	HTTP client with retries