Scaling Event-Driven Background Workers with Go and Redis
In my experience building distributed web applications, one architectural truth always emerges: synchronous HTTP lifecycles do not scale. Early in our journey, we made the mistake of processing expensive tasks—such as generating detailed PDF invoices, hitting third-party payment gateways, and compiling massive analytics reports—directly inside the request-response thread. When traffic spiked, our application servers routinely choked, database connections saturated, and response latencies climbed into the tens of seconds.
To resolve this, we moved to an asynchronous processing model using an event-driven task queue. We needed a system that was fast, lightweight, and capable of handling thousands of concurrent jobs per second without burning a hole in our infrastructure budget. After evaluating several options, we chose to combine the raw, concurrent processing power of Go (Golang) with the blazing-fast in-memory storage of Redis. This is the step-by-step blueprint of how we built, optimized, and scaled that pipeline to process millions of critical background jobs every day.
Step 1: Designing the Core Architecture
When implementing a task queue, the core objective is to decouple the producer (the web server taking the user request) from the consumer (the worker executing the heavy task). To keep things clean, we avoided bulky message brokers like RabbitMQ or Kafka in favor of Redis. Redis is uniquely suited for this due to its native list and sorted set structures, which let us implement FIFO (First-In-First-Out) queues, delayed jobs, and retries with minimal overhead.
For our Go consumer service, we integrated Asynq, a Go library built on top of Redis that handles task scheduling, retries, and concurrency out of the box. Asynq uses Redis sorted sets to manage scheduled tasks and a list structure for immediate processing queues, meaning our infrastructure footprint stayed incredibly small.
Below is a high-level conceptual overview of our event-driven flow:
The Producer: A Go API endpoint receives a request, generates an event payload, and writes it to Redis.
The Broker: Redis stores the tasks in a queued state, maintaining priority levels.
The Consumer: A pool of Go workers continuously polls Redis, safely claiming jobs and executing them within isolated goroutines.
Step 2: Tuning Redis for Durability and Performance
A default Redis installation is optimized for speed, not necessarily data durability. If a server suddenly power-cycles under default settings, you risk losing the background jobs currently sitting in memory. In our production testing, we had to carefully tune our Redis configuration to prevent job loss.
To balance lightning-fast throughput with robust resilience, we enabled both AOF (Append Only File) and RDB (Redis Database) snapshots. Here are the core parameters we added to our production redis.conf:
Append Only File configuration for durability
appendonly yes appendfsync everysecSnapshotting rules as a safety net
save 900 1 save 300 10Keep memory usage under control
maxmemory 4gb maxmemory-policy noevictionSetting the maxmemory-policy to noeviction is critical. If Redis runs out of memory, we want it to return an error to our API producers rather than silently deleting scheduled background jobs to make room. This forces our application layers to backpressure gracefully rather than dropping jobs silently.
Step 3: Implementing the Go Task Producer
Next, let's write the code to enqueue tasks. In our Go web server, we initialize an Asynq Client. Each task must have an explicit string identifier (e.g., "email:welcome") and a binary payload. We serialize our data payloads using JSON for ease of debugging.
Here is how we set up the task package and enqueued a sample welcome email job:
package tasks
import (
"encoding/json"
"fmt"
"github.com/hibiken/asynq"
)
const TypeEmailWelcome = "email:welcome"
type EmailWelcomePayload struct {
UserID int
Email string
FullName string
}
// NewEmailWelcomeTask packages our data into an Asynq Task instance
func NewEmailWelcomeTask(userID int, email, name string) (*asynq.Task, error) {
payload := EmailWelcomePayload{UserID: userID, Email: email, FullName: name}
bytes, err := json.Marshal(payload)
if err != nil {
return nil, err
}
return asynq.NewTask(TypeEmailWelcome, bytes), nil
}
On our API controller side, triggering this background task is highly efficient and non-blocking:
package main
import (
"log"
"github.com/hibiken/asynq"
"yourproject/tasks"
)
func main() {
// Connect to our configured Redis instance
client := asynq.NewClient(asynq.RedisClientOpt{Addr: "127.0.0.1:6379"})
defer client.Close()
task, err := tasks.NewEmailWelcomeTask(42, "user@example.com", "John Doe")
if err != nil {
log.Fatalf("Failed to create task: %v", err)
}
// Enqueue the task to be processed immediately by workers
info, err := client.Enqueue(task)
if err != nil {
log.Fatalf("Failed to enqueue task: %v", err)
}
log.Printf("Successfully enqueued task with ID: %s to queue: %s", info.ID, info.Queue)
}
Step 4: Writing the Go Worker Pool
With our API successfully pushing jobs to Redis, we need a dedicated worker process to ingest and run them. Go shine here because we can run thousands of concurrent goroutines under a controlled pool without hitting system memory exhaustion limits.
Here is our robust background worker implementation that registers task handlers and executes them safe from sudden runtime panics:
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"github.com/hibiken/asynq"
"yourproject/tasks"
)
func handleWelcomeEmailTask(ctx context.Context, t *asynq.Task) error {
var p tasks.EmailWelcomePayload
if err := json.Unmarshal(t.Payload(), &p); err != nil {
return fmt.Errorf("json.Unmarshal failed: %v: %w", err, asynq.SkipRetry)
}
log.Printf("Processing welcome email for User #%d: Send to %s", p.UserID, p.Email)
// Simulate actual work (e.g., SMTP call, API invocation)
return nil
}
func main() {
srv := asynq.NewServer(
asynq.RedisClientOpt{Addr: "127.0.0.1:6379"},
asynq.Config{
Concurrency: 10, // Max concurrent workers
Queues: map[string]int{
"critical": 6, // High priority
"default": 3,
"low": 1,
},
},
)
mux := asynq.NewServeMux()
mux.HandleFunc(tasks.TypeEmailWelcome, handleWelcomeEmailTask)
log.Println("Worker process running, waiting for background tasks...")
if err := srv.Run(mux); err != nil {
log.Fatalf("Could not run worker server: %v", err)
}
}
Step 5: Resiliency Patterns for Real-World Failures
In a real-world production environment, jobs will fail. Network interfaces drop, external APIs go down, and database connections time out. Designing with resiliency in mind is what separates fragile architectures from production-grade engineering.
- Implement Idempotency Keys
- Since task queue mechanisms sometimes operate on an "at-least-once" delivery guarantee, a transient error might cause a job to run twice. To avoid charging a customer twice or sending duplicate emails, we implement idempotency keys. Before executing a job, workers write a unique transaction identifier to Redis with an expiration limit (e.g., 24 hours). If the key already exists, the worker skips the task entirely.
- Fine-Tuned Backoffs
- By default, retrying failed tasks immediately can overwhelm downstream services (often called a "thundering herd" problem). We configure exponential backoff delays with random jitter so that retry spikes do not result in a self-inflicted denial of service.
- Dead Letter Queues (DLQ)
- If a task fails 10 consecutive times, we stop retrying and move it automatically to a Dead Letter Queue (DLQ). This prevents broken payloads from permanently clogging up active workers. We configure alerts so engineers can inspect these failed payloads manually, fix the root database or code bugs, and safely re-queue the tasks.
Step 6: Monitoring and Production Scaling
You cannot scale what you do not measure. In production, we keep close tabs on queue lengths, latency, and system resource limits.
For observability, we export metrics from Redis and Asynq using the Prometheus exporter. We track three primary indicators:
Queue Latency: How long does a task sit in Redis before a Go worker picks it up? If latency rises above 5 seconds, it means our worker pool is starved for resources and we must scale out additional pods.
Error Rate: A sharp spike in background errors usually indicates an expired third-party API token or a faulty deployment.
Redis Memory Fragmentation: High fragmentation suggests that Redis is aggressively allocating and deallocating memory, which might require tuning your memory allocator settings.
Because Go compiles down to a single, statically-linked binary, containerization is incredibly simple. We deploy our worker code inside lightweight, multi-stage Docker scratch containers. We use Kubernetes Horizontal Pod Autoscaling (HPA) to monitor queue sizes via custom Prometheus metrics, spinning up extra worker pods dynamically when workloads spike.
Frequently Asked Questions (FAQs)
Why choose Redis over RabbitMQ or Kafka for a task queue?
For small to medium workloads (up to tens of millions of jobs daily), Redis offers dramatically lower operational overhead, simpler setup, and lower latency. While Kafka is exceptional for persistent data logging and massive streaming architectures, Redis is faster, easier to maintain, and requires far less server resource allocation.
How do you handle tasks that hang or run indefinitely?
You should always establish strict processing deadlines. When enqueuing a task in Go, you can set a maximum runtime context limit. If the task exceeds this timeout limit, the parent context is cancelled, halting execution immediately and preventing deadlocked workers from exhausting your resource pools.
Can I scale Go workers independently from the web API?
Yes, absolutely. This is one of the major benefits of this pattern. Our web APIs run inside highly optimized, memory-efficient pods focused solely on handling web traffic. The workers run in an isolated deployment group, enabling us to scale up worker capacity independently when batch jobs are running without affecting the frontend user experience.
