FinCacheServe: Dependency-Consistent Answer Reuse for RAG on Mutable Documents
FinCacheServe is a service that enhances the handling of changing enterprise documents by using retrieval-augmented generation (RAG). It smartly reuses answers for similar questions, which cuts down on the heavy GPU tasks needed for generating new responses. Each answer is carefully indexed to reflect the company's goals and comes with safeguards like document versions and fingerprints to ensure reliability. A trial involving financial documents from the SEC utilized Qwen2.5 models in a hosted 7B trace, where it managed to avoid 53.27% of LLM calls without any outdated results. In another test with 32B operators, it skipped 53.31% of requests, outperforming previous caching methods significantly.
Key facts
- FinCacheServe treats each generated answer as a serving object indexed by enterprise intent.
- Guards include document versions, evidence fingerprints, tool fingerprints, model identity, and decoding configuration.
- Evaluated on SEC-derived financial-document workloads with Qwen2.5 models.
- On a 2,230-request hosted 7B trace, 53.27% of LLM calls skipped with zero stale outputs.
- Across three hosted 32B operator-suite seeds, 53.31% of 544 requests skipped.
- Versioned semantic caching achieved 38.97% skip rate; grounded-style reuse achieved 22.43%.
- Capacity, backend, and SLO replays show oracle-bounded cache performance.
- System designed for mutable enterprise documents where filings, evidence chunks, and tool outputs change.
Entities
Institutions
- arXiv
- SEC