How to tune a demo cache

The author built chat.betterdb.com as a retrieval-based chat over Valkey, Redis and Dragonfly documentation to publicly exercise caching libraries. Its tool-result cache normalizes calls before execution. A separate semantic cache embeds prompts and searches with KNN through valkey-search, allowing paraphrased questions to reuse responses when sufficiently similar.

The libraries were extended to store metadata in Valkey or Redis. Monitoring analyzed that information and suggested improvements, and an MCP server exposed it to the chat’s agent. In this demo, the agent could both create and approve its own suggestions. Configuration was read from the database, so changes required no restart.

The author scheduled the process with cron inside Vercel and left it running overnight and into the next day. The first run applied several TTL changes. Similar suggestions followed because TTL was the wrong control for natural-language tools whose inputs differed in wording. The author identified a routing change into semantic caching as the actual fix, requiring code changes beyond the system’s capabilities.

Across three runs, tool calls fell from fifteen to thirteen to eight; the third run proposed nothing new. Who it's for: developers experimenting with limited configuration autonomy in a demo and investigating what an agent cannot repair.

Prompt
Suggested prompt (written by Agents In Use, not the author's)

Source: Hacker News · kaliades