Deploying Kokoro TTS to RunPod Serverless

We wanted a fast, free, self-hosted TTS tier for OpenAudio Suite, ForeA’s self-hosted voice/TTS stack — something that didn’t depend on a cloud API key and could run at near-zero marginal cost. Kokoro fit the brief: 82M parameters, Apache 2.0, CPU-friendly, and it beats much larger models in blind listening tests. Here’s what actually happened getting it from “runs on my laptop” to “runs on a RunPod GPU endpoint that a real user hit through the browser.”

The easy part came first

Kokoro is CPU-friendly enough to run locally with no GPU at all — pip install kokoro, brew install espeak-ng for phonemization, and you have real 24kHz speech in a few seconds. We built that first: a tts_local.py wrapper with a pipeline-caching singleton, wired into a new /api/tts/generate-local endpoint, tested end-to-end with real audio.

That part was easy. The interesting part was moving it off a laptop.

Docker isn’t always the path of least resistance

The original plan was a Dockerfile-based RunPod Serverless worker — the traditional way. Except docker info on the dev machine returned a client that was installed but a daemon that wasn’t running. So the Dockerfile approach was a dead end before it began.

The fix was RunPod’s newer Flash CLI — a code-first serverless framework. You write a plain Python function, decorate it with @Endpoint(...), and flash dev/flash deploy build and run everything on RunPod’s own infrastructure. No local Docker daemon required, no image to push. This turned out to matter more than expected: it’s the difference between blocked and shipped on a machine where Docker Desktop simply isn’t part of the workflow.

flash dev only ships the function body

The first deploy attempt threw a NameError on a module-level constant:

# defined at module scope — looks fine, isn't
VALID_VOICES = {"af_heart", "af_bella", ...}

@Endpoint(...)
async def synthesize(text: str, voice: str = "af_heart", speed: float = 1.0) -> dict:
    if voice not in VALID_VOICES:  # NameError, under flash dev only
        ...

The fix is mechanical once you know the rule: everything the function needs — imports, constants, helper dicts — has to live inside the function body.

@Endpoint(...)
async def synthesize(text: str, voice: str = "af_heart", speed: float = 1.0) -> dict:
    VALID_VOICES = {"af_heart", "af_bella", ...}  # now survives flash dev
    ...

CPU-friendly doesn’t mean deploy-to-CPU

Kokoro runs fine on CPU locally, so the obvious move was a cheap cpu= Flash endpoint instead of paying for a GPU nobody strictly needs. That produced a much less obvious error:

…even with torch explicitly listed in dependencies=[]. Worker logs (via RunPod’s stream-worker-logs) confirmed it: Flash’s CPU base image doesn’t ship torch pre-baked, and the build step excludes torch from the packaged artifact regardless of whether you asked for it. Every torch example in RunPod’s own Flash docs uses gpu=, never cpu= — GPU base images carry a matching torch build; the CPU one doesn’t.

The fix was deploying on the cheapest available GPU tier (GpuGroup.AMPERE_16 — RTX A4000/A4500-class, 16GB VRAM) even though an 82M-param model needs nowhere near that much headroom. Massive overkill for the model, but it sidesteps the packaging gap entirely, and it’s still cheap at that tier.

A working endpoint isn’t a shipped feature

Once the endpoint worked — verified with a direct curl producing a valid RIFF/WAVE file — it felt done. It wasn’t. The actual user tried the feature in the browser, clicked “Generate Speech,” and heard nothing, because the frontend’s engine toggle only ever called the cloud path. The backend endpoint existed and worked perfectly; there was simply no UI route to it.

A backend endpoint that only a curl command can reach isn’t a feature yet.

The fix was adding a Cloud/Kokoro toggle to the Studio page, wiring the endpoint through the frontend’s API client, and — this part matters — actually clicking through it in a browser via Playwright before calling it done. Zero console errors, real audio played back. Then it was done.

flash dev provisions a real endpoint — and keeps billing

The most expensive lesson. Checking in on cost later, we found two live endpoints instead of the one we expected:

• The deployed kokoro-tts endpoint — 5 workers running, despite workers.max=3 in config.
• A second, entirely separate endpoint named live-kokoro-tts, created the instant flash dev had first started, running 5 CPU workers for about 34 minutes, with nobody watching it.

Where it landed

The final path was verified through three independent layers before we deliberately tore it down again (to stop paying for a GPU we weren’t actively using): a raw curl against the RunPod endpoint, the actual backend’s /api/tts/generate-local route under the full pytest suite, and a real browser click through the Studio UI via Playwright — audio generated, played back, zero console errors. Warm-worker latency landed around 2–3 seconds; a cold start (model download + CUDA init) was closer to 15–25 seconds.

tts_local.py now auto-detects a RUNPOD_ENDPOINT_ID + RUNPOD_API_KEY pair in the environment and calls the remote endpoint; unset, it falls back to local CPU inference with the exact same code path. The whole thing — including the RunPod fallback — is one contained, additive change.