Batch inference

1 · What it is

Normally an app sends one request to an AI model and waits for the reply. That makes sense for a chat. It makes less sense when you have ten thousand product descriptions to write and nobody is watching the screen.

Batch inference is the other way to ask. You gather your requests into a single job. The provider works through it in the background. Later you download a file of answers. Because you are not waiting, OpenAI, Anthropic and Google charge half the usual price for this kind of work.

The catch is time. OpenAI says a batch is done within 24 hours, and Anthropic says most of its batches finish in under an hour. At Anthropic, a batch that has not finished after 24 hours expires. So batch inference fits jobs where nobody is waiting: testing a model on many questions, sorting large piles of posts, or embedding a library of documents. It is not for a live conversation.

For engineers: at OpenAI, requests go in as a JSON Lines file, one request per line, each tagged with a unique custom_id. The output file may come back in a different order, so you match answers to requests by that id, not by position. An Anthropic batch can hold up to 100,000 requests or 256 MB. Its results can be downloaded for 29 days. On Amazon Bedrock you collect the output files from S3 storage. On Google Cloud, cache discounts and the batch discount do not stack.