在商用 CPU 硬體上實現低延遲 LLM 網頁搜尋之三層快取架構

摘要

諸如 ChatGPT 搜尋、Google 的 AI Overviews 與 Perplexity 等 AI 驅動搜尋產品，會提供以即時網路結果為依據、由 LLM 合成的答案。我們開發了 OreoLook（前身為 lixSearch），這是一個開源答案引擎，使用自動化瀏覽器代理程式與由供應商路由的 LLM 推論。其本機搜尋、快取、工作階段管理與嵌入堆疊可在商用 CPU 硬體上執行；答案合成則由遠端推論供應商執行。隨著使用量成長，工作階段會遺失上下文，等效查詢會觸發重複工作，且 URL 會在不同工作階段間被重複嵌入。我們提出一個三層快取架構：(1) 工作階段上下文視窗，在 Redis 中維護近期訊息的滾動視窗，並在溢位時自動轉存至經 Huffman 壓縮的磁碟封存檔；(2) 語意查詢快取，透過嵌入向量上的餘弦相似度捕捉改寫後的查詢，消除重複的 LLM 呼叫；以及 (3) URL 嵌入快取，可跨工作階段對嵌入計算進行去重。受評估系統部署於單一 8-vCPU Intel Cascade Lake 伺服器（2 GHz、32 GB RAM），該伺服器在三個容器化副本間執行 30 個 Hypercorn 工作程序；其報告顯示 89.3% 的 Redis 鍵空間整體命中率、0.1 毫秒讀取延遲，以及僅 1.38 MB 的記憶體額外負擔。背景 LRU 逐出守護程序會將閒置工作階段從 Redis 遷移至磁碟，並隨需重新還原，使對話能在設定的保留政策下於數小時或數天後恢復。

English

AI-powered search products such as ChatGPT search, Google's AI Overviews, and Perplexity provide LLM-synthesized answers grounded in live web results. We developed OreoLook (formerly lixSearch), an open-source answer engine using automated browser agents and provider-routed LLM inference. Its local search, caching, session-management, and embedding stack runs on commodity CPU hardware; answer synthesis is performed by a remote inference provider. As usage grew, sessions lost context, equivalent queries triggered redundant work, and URLs were repeatedly embedded across sessions. We present a three-layer caching architecture: (1) a Session Context Window maintaining a rolling window of recent messages in Redis with automatic overflow to Huffman-compressed disk archives; (2) a Semantic Query Cache catches rephrasings via cosine similarity on embedding vectors, eliminating redundant LLM invocations; and (3) a URL Embedding Cache that deduplicates embedding computations across sessions. Deployed on a single 8-vCPU Intel Cascade Lake server (2 GHz, 32 GB RAM) running 30 Hypercorn worker processes across three containerized replicas, the evaluated system reported an 89.3% aggregate Redis keyspace hit rate with 0.1 ms read latency and just 1.38 MB of memory overhead. A background LRU eviction daemon migrates idle sessions from Redis to disk and re-hydrates them on demand, enabling conversations that can be resumed hours or days later under the configured retention policy.