I assume you've been coding in Python for a while, and by now, you feel pretty confident in your stack. You know how to structure your backend, you know how to build worker pools, and you've probably spent your fair share of late nights debugging distributed locks on Redis.
You think your architecture is rock solid.
Until one day, you look at your codebase and realize that half of the engineering hours you spent over the last six months weren't spent on your actual product. They were spent building an intricate, fragile house of cards just to handle background jobs, task scheduling, retries, and failure states.
That was me.
We were building an internal data processing engine that fetched thousands of customer records, transformed them, ran validation checks, called third-party enrichment APIs, and updated our analytical warehouse.
On paper, the flow was straightforward. In reality, it was an absolute nightmare of edge cases:
- What happens if the third-party API rate-limits us midway through step 4?
- What happens if our server restarts while a 20-minute batch sync is halfway done?
- How do we ensure idempotent retries without reprocessing duplicate records?
- Why do we have 1,200 lines of custom Redis code just to keep track of job state transitions?
We built custom polling loops. We built database tables just to store intermediate step states. We wired up Celery, Redis, and a custom state machine.
It worked, but it was bloated, brittle, and terrifying to touch.
Then, I stumbled across Temporal (and its native Python SDK).
One library. One paradigm shift.
Within a week, our entire custom-built task orchestrator was gone. We deleted hundreds of lines of boilerplate infrastructure code and replaced it with clean, deterministic Python functions.
The Mistake We All Make: Reinventing the State Machine
Every developer eventually hits the point where a simple function call isn't enough. You have tasks that take minutes, hours, or even days to complete.
Normally, the Python ecosystem teaches us to reach for the classic recipe: Celery + Redis/RabbitMQ + PostgreSQL.
Here is what that usually looks like in production code:
# The classic, painful way we all build distributed background tasks
import time
import requests
from celery import Celery
app = Celery('tasks', broker='redis://localhost:6379/0')
@app.task(bind=True, max_retries=3, default_retry_delay=60)
def process_customer_pipeline(self, customer_id):
try:
# Step 1: Fetch raw data
raw_data = fetch_customer_data(customer_id)
save_step_state(customer_id, step=1, status="FETCHED")
# Step 2: Third-party API call (fragile)
enriched_data = call_third_party_api(raw_data)
save_step_state(customer_id, step=2, status="ENRICHED")
# Step 3: Heavy analytical computation
results = compute_analytics(enriched_data)
save_step_state(customer_id, step=3, status="COMPLETED")
return results
except requests.exceptions.RequestException as exc:
# What happens if the worker dies right here?
# You have to manually inspect DB state to resume from step 2!
raise self.retry(exc=exc)Look closely at what is happening here:
- State tracking is your problem: You have to write boilerplate database calls (
save_step_state) to know where a job failed. - Resuming is nearly impossible: If your worker crashes during step 3, you either re-run step 1 and step 2 (wasting API credits and compute), or you write an enormous conditional router to inspect the database and skip completed steps.
- Distributed locking is painful: If two workers pick up the same job, you have to write custom Redis lock mechanisms with TTLs to prevent race conditions.
You didn't just build a feature. You accidentally built an incomplete, buggy workflow engine.
The Paradigm Shift: Durable Execution in Python
Temporal flips the entire model on its head using a concept called Durable Execution.
Instead of manually managing state, retries, timeouts, and queues, you write standard, linear Python code known as a Workflow.
If the server crashes, if the power goes out, or if an API goes down for three days, the Temporal runtime captures the exact state of your code down to local variables and pauses it. When the system comes back up, your function resumes from the exact line where it stopped — without re-running previous steps.
Let's look at how the exact same pipeline looks using Temporal's Python SDK:
from datetime import timedelta
from temporalio import activity, workflow
# 1. Activities represent the actual work (can fail, retry, or touch external APIs)
@activity.defn
async def fetch_customer_data(customer_id: str) -> dict:
# Standard I/O call
return {"id": customer_id, "tier": "enterprise"}
@activity.defn
async def call_third_party_api(data: dict) -> dict:
# If this fails with a network error, Temporal automatically retries
# based on the policy defined in the workflow
return {**data, "score": 98.5}
@activity.defn
async def save_to_warehouse(data: dict) -> None:
print(f"Data safely written to database: {data}")
# 2. Workflows represent the orchestration logic (pure, deterministic Python)
@workflow.defn
class CustomerDataWorkflow:
@workflow.run
async def run(self, customer_id: str) -> dict:
# Step 1: Run activity with explicit timeouts & automatic exponential backoff
raw = await workflow.execute_activity(
fetch_customer_data,
customer_id,
start_to_close_timeout=timedelta(seconds=30),
)
# Step 2: Run external call with robust retry policy
enriched = await workflow.execute_activity(
call_third_party_api,
raw,
start_to_close_timeout=timedelta(minutes=2),
)
# Step 3: Write final result
await workflow.execute_activity(
save_to_warehouse,
enriched,
start_to_close_timeout=timedelta(seconds=10),
)
return enrichedNotice what is missing from this code:
- Zero database writes for state management.
- Zero custom retry handling loops.
- Zero distributed lock handling.
If call_third_party_api fails because the service is undergoing scheduled maintenance for two hours, the activity simply pauses, backs off exponentially, and retries until it succeeds.
The fetch_customer_data step is never re-executed. Its result was already durably saved.
Companies like Netflix, Stripe, and Datadog run massive portions of their mission-critical infrastructure on this exact durable execution pattern.
What Happened When We Deleted Our Old System
When we refactored our pipeline to use this library:
- We deleted over 800 lines of custom polling and database lock logic.
- Bug tickets dropped to almost zero. Almost every previous production bug was related to race conditions or unhandled intermediate states during worker restarts.
- Human-in-the-loop workflows became trivial. We had one requirement where high-tier enterprise accounts needed manual admin approval before syncing. In the old system, that required a separate database schema, an email notifier, and a cron scanner. With Temporal, we literally wrote
await workflow.wait_condition(lambda: self.is_approved)directly inside the function.
Stop Writing Infrastructure You Don't Need
As Python developers, our first instinct when faced with complex multi-step pipelines is to start designing database schemas for job states, setting up Redis queues, and writing retry wrappers.
We think that writing complex infrastructure makes us senior developers.
In reality, the most senior thing you can do is recognize when an entire category of problems has already been solved by modern tooling.
The next time you find yourself writing a custom background worker loop or wrestling with distributed state, stop building a bespoke orchestration engine. Grab a durable execution SDK, write clean Python functions, and get back to building the features that actually matter.
Enjoyed this one? Show some love with 50 claps 👏 and hit Follow to stay tuned for upcoming posts packed with fresh perspectives. Appreciate your time — see you in the next article! 🌟 Thanks a lot for reading! 🙌