The Cost of Repetition
Our customer support chatbot was answering the exact same questions hundreds of times a day ("How do I reset my password?", "Where is my refund?"). Every single query was being sent to a frontier LLM API, racking up massive token generation bills.
Semantic Caching
We implemented a Semantic Cache using Redis and a fast, local embedding model (all-MiniLM-L6-v2).
When a user asks a question, we instantly embed it locally. We then do a vector search in Redis against the embeddings of previously answered questions.
from sentence_transformers import SentenceTransformer
import redis
embedder = SentenceTransformer('all-MiniLM-L6-v2')
redis_client = redis.Redis(host='localhost', port=6379, db=0)
user_query = "How can I reset my forgotten password?"
query_vector = embedder.encode(user_query).tobytes()
# Search Redis for semantically similar past queries
results = redis_client.ft('idx:cache').search(...)
if results.docs and results.docs[0].score > 0.95:
# Cache Hit! Return the saved LLM response instantly.
return results.docs[0].cached_response
else:
# Cache Miss. Call expensive LLM API, then save to cache.
response = call_llm(user_query)
save_to_cache(user_query, response)If the cosine similarity is above a strict threshold (0.95), we return the cached LLM response instantly. This architectural change cut our API costs by 40% and reduced latency for common questions to under 50ms.