Distributed speech-to-image pipeline

The problem

My role · Designed and built it: the architecture, the migration from a NestJS/BullMQ stack to Python FastAPI + Celery, the queue router and workers, the live story pipeline, and the AWS deployment and operations scripts.

How it works

1. 1

   Stream the narration

   The browser captures microphone audio with an AudioWorklet (16 kHz PCM) and streams it over a WebSocket to a FastAPI session; Sarvam's streaming API turns the speech, including Hindi, into English text.

2. 2

   An LLM directs each scene

   An utterance buffer flushes complete thoughts to a scene director (OpenAI gpt-4o-mini or a self-hosted Qwen3 on vLLM) that registers new characters and decides when a scene is ready to draw.

3. 3

   The producer fans out

   FastAPI writes job rows to PostgreSQL and dispatches Celery tasks: character portraits go to a single text-to-image queue, and each scene panel fans out as one candidate per image-to-image pod, each with its own seed and the characters' reference images.

4. 4

   An in-flight-aware router

   A Redis-backed router counts in-flight jobs per pod and orders pods least-busy first, rotating ties, so back-to-back requests land on different GPUs and adding a pod adds a candidate automatically.

5. 5

   One job per GPU, never lost

   Each Celery worker drives exactly one ComfyUI pod with concurrency 1, prefetch 1 and acks_late, so a crashed worker's job is redelivered rather than dropped. Reference images are deduplicated per pod by content hash, and results land in MinIO.

6. 6

   Progress you can watch

   Workers publish progress on Redis pub/sub; the API streams it to the browser over SSE and WebSocket events, multiplexing every candidate in a batch into one stream.

7. 7

   A GPU fleet on AWS

   A central EC2 instance runs the API and workers behind an ALB; four GPU instances on private subnets run only ComfyUI from a pinned AMI. Scripts start the fleet and warm every model, and on shutdown drain Celery before stopping the GPUs.

Key decisions

• One queue per pod, behind an in-flight-aware router

  instead of: One shared queue every worker pulls from

  A scene's candidates must land on different GPUs to render in parallel; per-pod queues make placement explicit, and the router keeps load even.

• Concurrency 1, prefetch 1, acks_late

  instead of: Several workflows per GPU, acknowledged on receipt

  Heavy workflows compete for GPU memory, so one at a time keeps each job predictable; acknowledging late means a crash redelivers the job instead of losing it.

• Fan candidates out in parallel

  instead of: Rendering one image at a time

  Three idle pods produce three options in about one job's wall-clock time, at three times the GPU cost, and the best one is kept.

• Push progress over Redis pub/sub and SSE

  instead of: Clients polling job status

  People watch panels fill in live, and one batch stream carries every candidate.

• Rewrite in FastAPI + Celery, keeping the schema

  instead of: Extending the NestJS/BullMQ stack

  The existing PostgreSQL schema and storage layout were preserved, and the new stack used its own Redis database, so the cutover could run alongside the old one.

• GPU hosts run only ComfyUI

  instead of: Celery workers on every GPU machine (the earlier topology)

  GPU machines stay stateless, so the whole fleet can be started, warmed, drained and stopped by script to control cost.

Outcomes

• ✓A live narration-to-illustration pipeline: speech in, character portraits and scene panels out.
• ✓Scene candidates rendered in parallel across GPU pods in about one job's time.
• ✓Production on AWS: four 96 GB RTX PRO 6000 GPU instances behind a central API and worker host, started, warmed and drained by script.
• ✓115 tests across the producer, the workers and the image-generation package.