Large language models (LLMs) like OpenAI’s GPT-4 and Anthropic’s Claude 2 have captured the public’s imagination with their ability to generate human-like text. Enterprises are just as enthusiastic, with many exploring how to leverage LLMs to improve products and services. However, a major bottleneck is severely constraining the adoption of the most advanced LLMs in production environments: rate limits. There are ways to get past these rate limit toll booths, but real progress may not come without improvements in compute resources.
Public LLM APIs that give access to models from companies like OpenAI and Anthropic impose strict limits on the number of tokens (units of text) that can be processed per minute, the number of requests per minute, and the number of requests per day. This sentence, for example, would consume nine tokens.
