Today's Agenda

• Part 1: Prep Work Recap - Why Background Processing?
• Part 2: Task Queue Architecture
• Part 3: Redis & rq Deep Dive
• Part 4: Worker Processes
• Part 5: Production Patterns
• Demo: Building a Task Queue with rq
• In-Class Exploration: Notification Service

The Problem: Slow Operations

• Some operations take seconds or minutes
• Sending emails via external API (500ms-3s)
• Processing uploaded images (2-10s)
• Generating PDF reports (5-30s)
• Video transcoding (minutes to hours)
• If Flask waits, the user waits
• If Flask waits, other users can't connect
• This doesn't scale

The Solution: Process Later

What is a Queue?

• A data structure for ordered items
• FIFO: First In, First Out
• Add items to the back (enqueue)
• Remove items from the front (dequeue)
• Like a line at a coffee shop
• The first person in line gets served first

Key Vocabulary

From the Prep Videos

• We use rq (Redis Queue)
• Simple Python library for task queues
• Uses Redis as the broker
• Alternative to Celery (more complex)
• Tasks go in a separate tasks.py file
• Worker runs as a separate process
• Uses the decorator pattern - just like Flask!

The Decorator Pattern: A Familiar Friend

# Flask: Execute NOW when HTTP request arrives
@app.route('/notifications')
def handle_notification():
    return {"status": "ok"}

# rq: Execute LATER when worker picks it up
@job('notifications', connection=redis)
def send_notification(email, message):
    return {"status": "sent"}

• Same pattern, different timing
• Flask decorator → function becomes HTTP endpoint
• rq decorator → function becomes background task
• The function is the building block

Building Blocks: Now vs Later

Three Components

3. Consumer (Worker)

• Polls Redis for new jobs
• Executes task functions
• Stores results back in Redis

HTTP 202 Accepted

• HTTP has a status code for this pattern!
• 202 Accepted means:
• "I received your request"
• "Processing has been started"
• "But it's not complete yet"
• Different from 200 OK (completed now)
• Different from 201 Created (resource exists now)

@app.post('/videos/<int:id>/process')
def process_video(id):
    job = transcode_video.delay(id)  # Queue it, don't wait!
    return {"job_id": job.id}, 202   # Accepted, not done!

The Job ID Pattern

• When you enqueue a task, you get a job ID
• Return this ID to the client
• Client can check status: GET /jobs/{id}
• This is called polling

// POST /videos/42/process
{
    "message": "Processing started",
    "job_id": "a1b2c3d4-e5f6-7890",
    "status_url": "/jobs/a1b2c3d4-e5f6-7890"
}

// GET /jobs/a1b2c3d4-e5f6-7890
{
    "job_id": "a1b2c3d4-e5f6-7890",
    "status": "finished",
    "result": {"formats": ["720p", "1080p"]}
}

Job Lifecycle

Scaling with Workers

• One worker = one job at a time
• Need more throughput? Add more workers!
• Workers can run on different machines
• Redis handles coordination automatically

What is Redis?

• In-memory key-value data store
• Extremely fast (all data in RAM)
• Used for: caching, sessions, queues, pub/sub
• We use it as our message broker

# Start Redis with Docker
docker run -d -p 6379:6379 --name redis redis:alpine

# Test connection
docker exec -it redis redis-cli ping
# PONG

Redis in Docker Compose

# docker-compose.yml
version: '3.8'

services:
  redis:
    image: redis:alpine
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 3

• redis:alpine - Small, fast Redis image
• Healthcheck ensures Redis is ready before other services start
• Other services connect via redis://redis:6379

Installing rq

# requirements.txt
flask==3.0.0
redis==5.0.1
rq==1.15.1

• redis - Python Redis client
• rq - Redis Queue library
• rq depends on redis, but be explicit

Connecting to Redis

# app.py
from redis import Redis
from rq import Queue
import os

# Connect to Redis
redis_url = os.getenv('REDIS_URL', 'redis://localhost:6379/0')
redis_conn = Redis.from_url(redis_url)

# Create a queue
queue = Queue('notifications', connection=redis_conn)

• Redis.from_url() parses the connection string
• Queue name ('notifications') groups related tasks
• You can have multiple queues for different priorities

Defining Tasks with @job

# tasks.py
from redis import Redis
from rq.decorators import job
import time
import os

redis_conn = Redis.from_url(os.getenv('REDIS_URL', 'redis://localhost:6379/0'))

@job('notifications', connection=redis_conn)
def send_notification(user_email, message):
    """
    Send a notification email.
    This runs in the WORKER, not in Flask!
    """
    print(f"Sending to {user_email}...")
    time.sleep(3)  # Simulate slow email API
    print(f"Sent: {message}")
    return {"email": user_email, "status": "sent"}

• @job('queue_name') - just like @app.route('/path')
• Decorator specifies which queue this task belongs to
• Function can now be called with .delay()

Enqueueing Tasks with .delay()

# app.py
from tasks import send_notification

@app.post('/notifications')
def create_notification():
    data = request.get_json()

    # .delay() queues the task - returns IMMEDIATELY
    job = send_notification.delay(
        data['email'],          # First argument
        data['message']         # Second argument
    )

    return {
        "job_id": job.id,
        "status": "queued"
    }, 202

• .delay(*args) - queues the task, returns Job object
• Just like calling the function, but it runs later
• Returns immediately - does NOT wait for completion

Checking Job Status

# app.py
from rq.job import Job

@app.get('/jobs/<job_id>')
def get_job_status(job_id):
    try:
        job = Job.fetch(job_id, connection=redis_conn)
    except Exception:
        return {"error": "Job not found"}, 404

    response = {
        "job_id": job_id,
        "status": job.get_status()
    }

    if job.is_finished:
        response["result"] = job.result
    elif job.is_failed:
        response["error"] = str(job.exc_info)

    return response

Why a Separate Process?

• Flask handles HTTP requests
• Worker handles background tasks
• They run independently
• Benefits:
• Flask stays responsive (never blocked)
• Workers can run on different machines
• Scale workers independently of Flask
• Crash isolation (worker crash doesn't kill API)

Running the rq Worker

# Basic command
rq worker --url redis://localhost:6379 queue_name

# With Docker Compose
services:
  worker:
    build: .
    environment:
      - REDIS_URL=redis://redis:6379/0
    command: rq worker --url redis://redis:6379/0 notifications
    depends_on:
      - redis

• --url specifies the Redis connection
• Last argument is the queue name to watch
• Worker polls Redis continuously for new jobs

Worker in Docker Compose

# docker-compose.yml
services:
  flask:
    build: .
    ports:
      - "5000:5000"
    environment:
      - REDIS_URL=redis://redis:6379/0
    command: python app.py

  worker:
    build: .
    environment:
      - REDIS_URL=redis://redis:6379/0
    command: rq worker --url redis://redis:6379/0 notifications
    depends_on:
      redis:
        condition: service_healthy

• Same Docker image as Flask, different command
• No ports needed - worker doesn't serve HTTP
• depends_on ensures Redis starts first

Critical: tasks.py Isolation

# tasks.py - runs in WORKER, not Flask!

import os
from dotenv import load_dotenv
from redis import Redis
from rq.decorators import job

# Worker needs its own environment loading
load_dotenv()

redis_conn = Redis.from_url(os.getenv('REDIS_URL'))

@job('emails', connection=redis_conn)
def send_email(to, subject, body):
    api_key = os.getenv('MAILGUN_API_KEY')
    # ... send email using api_key

• Worker is a separate Python process
• It imports tasks.py, NOT app.py
• Must load environment variables separately
• Cannot access Flask's app context

What Can Tasks Access?

Passing Arguments

# Good - pass simple values
job = send_email.delay(
    user.email,      # string
    user.id,         # integer
    message_text     # string
)

# Bad - pass complex objects
job = send_email.delay(
    user,           # SQLAlchemy object - problematic!
    request         # Flask request - won't work!
)

• Pass strings, numbers, lists, dicts
• Avoid passing ORM objects or Flask objects
• Arguments get serialized (pickled) into Redis

Common Use Cases

• Email/SMS notifications - external API latency
• Image/video processing - CPU intensive
• PDF generation - memory intensive
• Data imports - CSV/Excel processing
• Report generation - complex queries
• Webhooks - calling external services
• Cache warming - pre-computing expensive data

Error Handling

# tasks.py
def send_notification(email, message):
    try:
        # Attempt to send
        response = mailgun.send(email, message)
        return {"status": "sent", "id": response.id}
    except Exception as e:
        # Log the error
        print(f"Failed to send to {email}: {e}")
        # Re-raise so rq marks job as failed
        raise

• Exceptions in tasks mark the job as failed
• Failed jobs can be retried or inspected
• rq stores the exception info in Redis

Retries

# Configure retries in the decorator
from rq.decorators import job
from rq import Retry

@job('notifications', connection=redis_conn,
     retry=Retry(max=3, interval=60))  # 3 retries, 60s apart
def send_notification(email, message):
    # If this fails, rq retries up to 3 times
    response = mailgun.send(email, message)
    return {"status": "sent"}

• Network errors and timeouts are common
• Automatic retries handle transient failures
• interval adds delay between retries
• After max retries, job stays in failed queue

Multiple Queues

# tasks.py - queue name in decorator
@job('urgent', connection=redis_conn)
def send_password_reset(email):
    # Password resets go to "urgent" queue
    ...

@job('low', connection=redis_conn)
def send_newsletter(subscriber_ids):
    # Newsletters go to "low" queue
    ...

# In app.py - just call .delay()
send_password_reset.delay(user.email)  # → urgent queue
send_newsletter.delay(subscriber_ids)   # → low queue

# Worker processes queues in order listed
rq worker urgent default low

Scaling Workers

# docker-compose.yml
services:
  worker:
    build: .
    command: rq worker --url redis://redis:6379/0 tasks
    deploy:
      replicas: 4  # Run 4 worker containers!

• More workers = more parallel processing
• Each worker handles one job at a time
• Redis coordinates - no duplicate processing
• Scale based on queue depth and latency requirements

Industry Alternative: Celery

• More popular in large enterprises
• More features: scheduling, workflows, monitoring
• More complex configuration
• Same core concepts as rq!

What We Learned

• Task queues separate fast responses from slow processing
• Producer → Broker → Consumer architecture
• Redis as the message broker
• rq for simple Python task queues
• Workers run as separate processes
• HTTP 202 for accepted-but-not-complete
• Job IDs let clients check status

Assignment 2 Preview

• Build a production Task Manager API
• Uses everything we've learned:
• Flask REST API (Week 3)
• Docker containers (Week 5)
• Data validation (Week 6)
• SQLAlchemy database (Week 7)
• Background tasks (Today!)
• Released today, due Week 11

Next Week

• Week 9: Spring Break - No Class
• Week 10: System Design Theory
• Scalability fundamentals
• Reliability and fault tolerance
• CAP theorem
• Work on Assignment 2!

What We'll Build

• Flask API that queues video processing jobs
• Redis as the message broker
• rq worker processing jobs in background
• Status endpoint to check job progress

Project Overview

• Scenario: Convert a slow notification API to async
• Starter: Working API that blocks for 3 seconds per notification
• Goal: Use rq so POST returns instantly
• Success: Multiple notifications queue up and process in background

Getting Started

1. Clone/download the starter code
2. Start Redis: docker-compose up -d redis
3. Run the starter API: python app.py
4. Test it - notice the 3-second delay!
5. Now add rq to make it async...

# Test the slow version
time curl -X POST http://localhost:5000/notifications \
  -H "Content-Type: application/json" \
  -d '{"email": "test@example.com", "message": "Hello!"}'
# Takes 3+ seconds...