StoryGraph scales self-hosted LLM and ML models to 300 million monthly requests

Overview

Book recommendation platform StoryGraph, run by a two-person team, scaled AI/ML infrastructure to handle 300 million monthly requests by self-hosting large language models rather than using external APIs like ChatGPT. Self-hosting was driven by data privacy and cost: using GPT-4 at their scale was calculated to cost approximately $770,000 per month, versus a fraction of that self-hosted. The company processes roughly 1 million LLM requests per day using Redis-based queuing across GPU servers and migrated its database to a 60-server YugabyteDB cluster.

The challenge

StoryGraph needed to deliver AI-powered features, including book mood classification, recommendation engines, personalized previews, and content moderation, to millions of users while maintaining profitability as a freemium app where only a small fraction of users pay. Using GPT-4 for their LLM use cases at their scale was calculated to cost approximately $770,000 per month, which would make the business model unviable; the company's community also values data privacy, which self-hosting guarantees since user data never leaves their infrastructure.

The solution

StoryGraph, run by a two-person team, self-hosts its large language models and roughly a dozen specialized ML models rather than using external APIs like ChatGPT. The company processes approximately 1 million LLM requests per day using a Redis-based queuing system (leveraging Redis's BRPOP command) to distribute ML inference jobs across GPU servers, allowing any machine with a GPU to join the cluster. StoryGraph also migrated its database from PostgreSQL to a distributed YugabyteDB cluster to handle scale, using CTE rewriting, additional indexes, and denormalized tables to optimize query performance.

Reported business value

By self-hosting, StoryGraph achieves the functionality of large-scale LLM inference at a fraction of the approximately $770,000 per month that using GPT-4 at their scale would cost. The company scaled from about 1,000 users in June 2020 to over 6 million monthly active users by January 2024, handling 300 million monthly requests. The YugabyteDB cluster scaled to 60 servers with 960 cores, 2.5 TB RAM, and 40 TB storage for approximately $2,000 per month.