Large language model operations

1 · What it is

LLMOps is short for “large language model operations”. LLMOps is the set of habits and tools for running an app built on one once real people depend on it. It covers the everyday work around the model: keeping prompts under control, testing answers, and watching speed and cost. Microsoft calls the same work generative AI operations, or GenAIOps, and treats it as an extension of MLOps.

Google Cloud describes LLMOps as a specialized subset of MLOps, short for machine learning operations. Much stays the same, but the focus moves. Many language models start as a general foundation model, a big model trained for many jobs. It is then fine-tuned, meaning trained a little more for one job, instead of being built from zero. Often, an app team never trains anything. They work on the pipeline around the model. That means the steps that prepare a question, send it and handle the answer.

That shifts what needs managing. In classic MLOps, the app that calls the model is often left outside the process. With a language model app, you manage that part too, because it writes the prompt, the instructions sent to the model. Prompt templates matter for accurate, reliable answers.

Picture a school that runs a homework-help chatbot. A teacher asks for friendlier replies, and a developer rewrites one sentence of the prompt. With LLMOps, the new wording is saved as version 3 in a prompt registry, a store that lets teams version, track and reuse prompts. Versions cannot be changed after saving, so version 2 always means the same text.

Next comes evaluation. Evals, short for evaluations, test the model’s answers against rules the team chooses. OpenAI calls them essential, especially when you upgrade or try a new model. The team runs version 2 and version 3 on the same set of real homework questions and compares the scores. Only if version 3 holds up does the production alias, a movable label saying which version is live, point to it. If something goes wrong later, moving the label back is a rollback.

Measuring quality also works differently. Classic models have clear scores, such as accuracy (how often they are right). A chatbot that answers from school notes needs measures like groundedness, meaning whether the answer sticks to its source, and relevancy. These tasks have no single right answer, so feedback from real users is often critical. Feeding it back into the pipeline helps both testing and later fine-tuning.

Once live, the app needs watching. Microsoft suggests watching response time, how many tokens (small chunks of text) are used, and 429 (too many requests) errors. Tracing tools such as MLflow record the time and tokens each step used.

Cost is part of the job as well. If you use a model as a hosted service, someone else runs the servers. Instead you care about how much work the service handles, your quota (your usage allowance), and being slowed down when you hit limits. Azure OpenAI bills by tokens, so watching quota usage is how a team keeps spending in check.

How much of this a project needs depends on the project.

	MLOps	LLMOps
Typical quality measures	Accuracy, F1 score	Groundedness, relevancy, human feedback