Semantic caching saves a model's past answers and reuses one when a new prompt means the same thing, even if it is worded differently.
In plain words
Calling a language model costs money and time. A normal cache only helps when a request is exactly the same as before. A semantic cache uses an embedding model to compare meanings, and looks for a stored prompt that is close enough to the new one. If it finds one, it returns the saved answer without calling the model. How close counts as a match is a threshold you have to tune for your app.
A quick example
Imagine a help bot that has already answered "How do I reset my password?". Later someone asks "I forgot my password, how can I change it?". The cache sees the two questions are very similar in meaning and returns the stored answer instantly, with no model call.
Where you'll meet it
- Customer support bots with many repeated questions.
- Cutting cost and latency in busy LLM apps.
Related
Sources
- DocsCache LLM responses (SemanticCache), Redis documentation · read 29 Sept 2026
- Docszilliztech/GPTCache, Zilliz (GitHub) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.