Stanford CS224V · Conversational Virtual Assistants with Deep Learning · Fall 2023
Cache for Cash
Department of Computer Science, Stanford University
Abstract
We evaluate the cost-saving potential of cached response recycling and revision to answer incoming user questions for LLM chatbot use cases. We implement a transformer-based vector retrieval system for mapping user queries to semantically similar queries in our cache. We then employ a novel, logit-based technique inspired by speculative decoding to evaluate response appropriateness without additional prompt overhead. Alongside high potential for cost savings, we discover an intriguing approach to the Quora problem of semantic equivalence.