Active Job Patterns

Active Job Patterns: Continuations vs. Checkpointing

This document explains two distinct patterns for managing complex background jobs in Rails.

Pattern 1: Job Chaining (Continuations)

Purpose: Schedule follow-up jobs after the current job completes successfully.

Use Case: Sequential workflows where Job A must complete before Job B starts.

Implementation: Continuable concern (app/jobs/concerns/continuable.rb)

Example:

class SendWelcomeEmailJob < ApplicationJob
  include Continuable

  # These jobs run AFTER welcome email is sent successfully
  continuations do |user_id|
    SendOnboardingTipsJob.set(wait: 5.minutes).perform_later(user_id)
    NotifyAdminJob.perform_later(user_id)
  end

  def perform(user_id)
    UserMailer.welcome_email(user_id).deliver_now
  end
end

Flow:

SendWelcomeEmailJob completes
    ↓
Continuations triggered
    ↓
├── SendOnboardingTipsJob scheduled (5 min delay)
└── NotifyAdminJob scheduled (immediate)

Key Features:

• ✅ Success-only execution - Continuations only run if parent succeeds
• ✅ Job chaining - Build multi-step workflows
• ✅ Error isolation - Continuation failures don’t affect parent
• ✅ Declarative - Clear job dependencies at class level

Pattern 2: Job Checkpointing (Steps)

Purpose: Break long-running jobs into resumable steps with progress tracking.

Use Case: Processing large datasets, imports, bulk operations that can be interrupted.

Implementation: Steppable concern (app/jobs/concerns/steppable.rb)

Example:

class ProcessImportJob < ApplicationJob
  include Steppable

  def perform(import_id)
    @import = Import.find(import_id)

    # Step 1: Initialize
    step :initialize do
      @import.update!(status: 'processing')
    end

    # Step 2: Process with checkpointing
    step :process do |step|
      @import.records.find_each(start: step.cursor) do |record|
        process_record(record)
        step.advance! from: record.id  # Save progress
      end
    end

    # Step 3: Finalize
    step :finalize
  end

  private

  def finalize
    @import.update!(status: 'completed')
  end
end

Flow:

Step 1: Initialize ✓
Step 2: Process
    ├── Record 1 ✓ (checkpoint saved)
    ├── Record 2 ✓ (checkpoint saved)
    ├── Record 3 ❌ (job crashes/interrupted)
    └── [Job restarts from Record 3]
Step 3: Finalize ✓

Key Features:

• ✅ Resumable - Jobs can be interrupted and resumed from last checkpoint
• ✅ Fault-tolerant - Partial progress is never lost
• ✅ Progress tracking - Know exactly where you are in a long process
• ✅ Idempotent - Safe to re-run from any checkpoint

When to Use Each Pattern

Use Continuations When:

• ✓ You need to chain multiple jobs sequentially
• ✓ Each job is independent and relatively quick
• ✓ You want clean separation between jobs
• ✓ You need conditional job chains

Examples:

• Email workflows (welcome → onboarding → metrics)
• Post-registration tasks
• Multi-stage notifications
• Async API calls with callbacks

Use Checkpointing When:

• ✓ Processing large datasets (thousands+ records)
• ✓ Jobs that take minutes/hours to complete
• ✓ Risk of interruption (server restarts, timeouts)
• ✓ Need to show progress to users

Examples:

• Bulk email sending
• Data imports/exports
• Report generation
• Database migrations
• Image/video processing pipelines

Combining Both Patterns

You can use both patterns together for complex workflows:

class BulkWelcomeEmailJob < ApplicationJob
  include Steppable    # For resumable processing
  include Continuable  # For follow-up jobs

  # Continuation: runs AFTER all steps complete
  continuations do |options|
    TrackBulkEmailMetricsJob.perform_later(options)
  end

  def perform(options)
    # Step 1: Initialize
    step :initialize_scope do
      @users = User.where(...)
    end

    # Step 2: Process with checkpointing
    step :send_emails do |step|
      @users.find_each(start: step.cursor) do |user|
        send_email(user)
        step.advance! from: user.id
      end
    end

    # Step 3: Finalize
    step :finalize
  end
end

Flow:

Step 1: Initialize ✓
Step 2: Send emails (resumable)
    ├── User 1 ✓
    ├── User 2 ✓
    └── ... (thousands more)
Step 3: Finalize ✓
    ↓
Continuation triggered
    ↓
TrackBulkEmailMetricsJob scheduled

Implementation Details

Continuable Concern

File: app/jobs/concerns/continuable.rb

module Continuable
  extend ActiveSupport::Concern

  class_methods do
    def continuations(&block)
      @continuation_block = block
      after_perform :execute_continuations
    end
  end

  private

  def execute_continuations
    instance_exec(*arguments, &self.class.continuation_block)
  end
end

How it works:

1. continuations block is defined at class level
2. after_perform callback is registered
3. When job completes successfully, callback executes the block
4. Block schedules follow-up jobs

Steppable Concern

File: app/jobs/concerns/steppable.rb

module Steppable
  class Step
    def advance!(from:)
      @cursor = from
      Rails.cache.write(cursor_key, from, expires_in: 24.hours)
    end

    def cursor
      Rails.cache.read(cursor_key)
    end
  end

  def step(name, &block)
    step_obj = Step.new(name, self)
    block.call(step_obj)
    step_obj.complete!
  end
end

How it works:

1. step method creates a Step object
2. Step object tracks cursor in Rails.cache
3. step.advance! saves progress
4. On restart, step.cursor loads last position
5. find_each(start: step.cursor) resumes from checkpoint

Testing

Testing Continuations:

RSpec.describe SendWelcomeEmailJob do
  it "schedules continuation jobs after success" do
    expect(SendOnboardingTipsJob).to receive(:set)
      .with(wait: 5.minutes)
      .and_return(double(perform_later: true))

    SendWelcomeEmailJob.perform_now(user.id)
  end

  it "does not schedule continuations if job fails" do
    allow(UserMailer).to receive(:welcome_email).and_raise(StandardError)

    expect(SendOnboardingTipsJob).not_to receive(:set)

    expect { SendWelcomeEmailJob.perform_now(user.id) }.to raise_error
  end
end

Testing Checkpointing:

RSpec.describe ProcessImportJob do
  it "can resume from checkpoint after failure" do
    # First attempt: fails midway
    expect { ProcessImportJob.perform_now(import.id) }.to raise_error

    # Verify checkpoint was saved
    step = Steppable::Step.new(:process, job)
    expect(step.cursor).to eq(last_processed_id)

    # Second attempt: resumes from checkpoint
    ProcessImportJob.perform_now(import.id)
    expect(import.reload.status).to eq('completed')
  end
end

Real-World Examples in Our App

Continuations (Already Implemented):

• ✅ SendWelcomeEmailJob → SendOnboardingTipsJob → TrackSignupMetricsJob
• ✅ SendWelcomeEmailJob → NotifyAdminJob (parallel)

Checkpointing (Example Implementations):

• 📧 BulkWelcomeEmailJob - Send welcome emails to thousands of users
• 📦 ProcessImportJob - Process large data imports
• 📊 GenerateReportJob - Generate reports from large datasets

Performance Considerations

Continuations:

• Overhead: Minimal (just scheduling next jobs)
• Scalability: Excellent (each job is independent)
• Resource usage: Low (jobs are small and quick)

Checkpointing:

• Overhead: Small (cache writes for checkpoints)
• Scalability: Good (single job processes many records)
• Resource usage: Moderate (long-running, but resumable)

Summary

Bottom line:

• Use Continuations for job workflows
• Use Checkpointing for resumable processing
• Use Both for complex, resumable workflows

See also:

• docs/ACTIVE_JOBS_EMAIL_SETUP.md - Email signup implementation guide
• app/jobs/concerns/continuable.rb - Continuations implementation
• app/jobs/concerns/steppable.rb - Checkpointing implementation