Every model in this stack so far earned its place by closing a gap — a language nobody could hear, a clone nobody could make. StyleTTS2 was different. Kokoro and Chatterbox already sit near the top of the realistic open-model quality range; StyleTTS2 was added purely as a fourth narration option, chosen for one thing: style-diffusion prosody, a genuinely different way of sounding natural. No new language, no new capability. Just a better ear for delivery, if it worked.
It became the hardest deployment this project has done — eight distinct, real bugs, roughly twenty full rebuild cycles, spanning a packaging bug in a third-party PyPI wrapper, three separate rounds of NumPy fighting itself, a security default that shipped in a PyTorch minor version, and four packages that simply don’t publish a wheel anywhere. None of it was in the paper. All of it was findable only by actually trying to run the thing.
The bug that wasn’t the bug it looked like
First crash, first flash dev call:
The textbook read: NumPy 2.x installed, something compiled against 1.x, pin an old NumPy and move on. Pinning numpy==1.26.4 didn’t fix it. Neither did --force-reinstall. Neither, eventually, did installing a clean NumPy into its own directory and forcing it to the front of sys.path — a trick that had actually worked earlier this same session for a different model, and this time produced a new, more interesting error: TypeError: expected np.ndarray (got numpy.ndarray). Two classes, same name, refusing to recognize each other.
That error is the tell. RunPod’s own worker harness runs a GPU health-check — visible in every log as GPU compute benchmark passed — and it runs before the deployed function ever executes. That benchmark imports torch. torch imports NumPy. By the time this worker’s own code got a chance to swap NumPy out from under itself, torch had already bound its C extensions to whatever NumPy the base image shipped.
Not downgrading NumPy at all — pinning numba (the actual package with the 1.x-compiled extension, pulled in transitively through librosa) to a release built for NumPy 2.x, and leaving NumPy exactly where torch already expected it. Which surfaced a second, unrelated fact about the base image: its pre-installed scipy expected a private NumPy 1.x attribute 2.x had removed. Pinning scipy>=1.13 resolved the whole ABI maze — three packages, two of them innocent bystanders — at once.
A progress bar that took the whole page down with it
Past NumPy, the next crash came from checking a download’s progress:
rich — the terminal-formatting library, several dependency-hops removed from anything StyleTTS2 actually cares about — was missing part of its own bundled data in whatever release resolved. Pinning a known-good version didn’t help; a from-scratch reinstall into its own directory came back missing the same files, which pointed at something wrong with the release itself, not the version choice.
Skipped the library entirely — one line, Console.print = lambda self, *a, **k: None, silencing the exact call the crash traced through. Cosmetic output disabled, not functionality. The checkpoint still downloaded; nobody was going to see the progress bar on a serverless worker anyway.
The default that changed under a pinned model
Model loaded, weights loading, new error:
PyTorch 2.6 flipped torch.load()’s weights_only argument from False to True — a real security improvement, and a breaking one for any checkpoint pickled before the flag existed, which describes every pretrained research checkpoint from before 2024. StyleTTS2’s own loading code never sets the argument, so it silently inherits whichever default the installed PyTorch carries.
A one-line monkeypatch defaulting it back to False — deliberately scoped to checkpoints this project already downloads directly and had already verified the license terms of, not arbitrary input.
The deploy that worked locally and failed for real
flash dev had, by this point, produced a genuine success: real audio, a valid WAV header, three and a half seconds of narration. flash deploy — the step that turns a dev session into a persistent, billable endpoint — failed immediately, on a dependency it had never complained about:
PyPI plainly has gruut. A plain pip install on the same machine proved it. The difference: flash deploy resolves dependencies for the target platform — Linux, from a Mac — using a wheels-only cross-compilation mode that can’t build a package from source for a machine it isn’t running on. gruut has never published a wheel, sdist only.
Reproducing that exact resolution locally — pip install --dry-run --platform manylinux2014_x86_64 --only-binary=:all: — surfaced the real dependency graph pip’s own error message never showed: gruut needed gruut-ipa and gruut_lang_en, neither of which has a wheel either, and one level further down, docopt — a dependency of a dependency of a dependency — didn’t have one either.
All four are pure Python. All four got built into wheels by hand and hosted on a new public repo, referenced by direct URL ahead of everything else in the dependency list.
The job that looked stuck and wasn’t
Deploy succeeded. First real job against the new endpoint sat IN_QUEUE for over a minute with the endpoint’s own health check reporting plenty of idle, healthy workers — which looked exactly like a capacity problem and wasn’t one. The actual worker logs told a different story:
A race: a worker had scaled up and tried to unpack the deploy’s build artifact before it had finished propagating to the shared storage volume. Cancel the stuck job, resubmit a few seconds later, and the second attempt found the artifact exactly where it should have been. Not a bug in the code — a timing window in the platform doing the deploying.
What eight bugs bought
Every fix here came from an actual traceback, checked against the real infrastructure, never guessed at from a hunch. Two separate points along the way produced a false “it’s working” — a healthy-looking log tail that turned out to precede a crash a few lines later — and both got caught before being written down as done, because the discipline this project runs on is checking the response body, not the vibe of the log.
What came out the other side: a raw call to the deployed endpoint returning real, decodable audio; the full backend dispatch path returning the right model name under a real login session; a Playwright pass through the actual Studio page, StyleTTS2 selected, narration generated and played back, zero console errors.
Then, per the standing rule for every model in this stack, the endpoint came back down — verified once, not run forever on a GPU bill nobody’s watching. The fourth engine exists, it works, and the next person who hits a NumPy ABI error chasing a completely different model now has eight fewer ways to lose an afternoon.