growing-pool-cache

A self-growing cache pool for AI-generated content that balances cost savings with response diversity.

🇰🇷 한국어 · 🇯🇵 日本語 · 🇨🇳 中文

Cost-efficiency primitive from NSKit — bound by structure, free to combine. Powers the AI-content cache layer behind NSKit's production services, cutting AI API bills by ~97% while preserving response diversity.

---

The Problem

Traditional caches store one response per key. That works fine for deterministic data, but AI responses are non-deterministic by design — asking the same question twice should yield different answers.

Caching a single AI response saves money but kills diversity. Not caching at all preserves diversity but burns your API budget.

Growing Pool Cache solves this: each cache key grows a pool of responses that expands naturally based on demand.

How It Works

The Growth Cycle

1. Cache miss -- Call AI, store Response A in pool (hit_count=0)
2. Cache hit -- Return A, increment hit count
3. A reaches N hits -- Return a response as usual, then trigger onGrowth callback asynchronously so caller generates new content in the background
4. New response B stored -- Pool size becomes 2, is_growing=false
5. Future requests -- Random pick from pool (A or B)
6. B reaches N hits -- Generate C... the pool grows, but growth naturally decelerates as the random distribution spreads hits across more entries

Why It Decelerates

With poolTarget=3:

Pool Size	Avg hits to trigger growth	Effective interval
1	3 requests	Every 3rd request
2	6 requests	Every 6th
3	9 requests	Every 9th
5	15 requests	Every 15th
10	30 requests	Every 30th

The pool self-regulates: high-traffic keys get more diversity, low-traffic keys stay small.

Performance Characteristics

Pool Growth vs Requests (poolTarget=3)

Pool growth follows O(√n) — rapid early growth, then gradual deceleration. No configuration needed.

Cumulative AI Calls vs Requests Served

At 1,000 requests, Growing Pool Cache uses ~30 AI calls vs 1,000 without cache. 97% cost reduction while maintaining diverse responses.

Cache Hit Rate Over Time

Hit rate converges to ~99% as pool grows. Every hit returns in ~5ms vs 14-40 seconds for AI generation.

Response Latency Distribution

Traditional Cache vs Growing Pool Cache

	Traditional Cache	Growing Pool Cache
Responses per key	1	1 ... N (grows over time)
Diversity	None (same response every time)	High (random selection from pool)
AI API calls	1 per key (until TTL)	Grows with demand, then decelerates
Cost efficiency	Excellent (but boring)	Excellent (and diverse)
Use case	Deterministic data	AI-generated, creative content
Response time	~5ms hit / 14-40s miss	~5ms hit / 14-40s miss

Quick Start

npm install growing-pool-cache

const { GrowingPoolCache } = require('growing-pool-cache');
const { MemoryAdapter } = require('growing-pool-cache/src/adapters/memory');

const cache = new GrowingPoolCache(new MemoryAdapter());

// --- Simple mode (traditional cache) ---
await cache.set('user:123', { name: 'Alice' });
await cache.get('user:123'); // { name: 'Alice' }

// --- Pool mode (growing pool) ---
// poolTarget: grow pool after every 3 hits on the newest entry
await cache.set('fortune:love', 'Great things await!', { poolTarget: 3 });

// First 3 hits return the same response
await cache.get('fortune:love'); // 'Great things await!'

// After 3 hits, onGrowth fires — response is still returned
await cache.get('fortune:love'); // 'Great things await!' + onGrowth callback triggered

// In onGrowth callback, generate new AI response and add to pool
await cache.set('fortune:love', 'Love is in the air!', { poolTarget: 3 });

// Now randomly returns either response
await cache.get('fortune:love'); // 'Great things await!' or 'Love is in the air!'

API Reference

new GrowingPoolCache(adapter, options?)

Option	Type	Description
onGrowth	(key) => void	Called when pool growth is triggered
onHit	(key, mode) => void	Called on cache hit (mode: 'simple' or 'pool')
onMiss	(key) => void	Called on cache miss

cache.get(key)

Returns the cached value, or null on miss.

• Simple mode: returns the stored value
• Pool mode: returns a random value from the pool. When the newest entry reaches poolTarget hits, the onGrowth callback is triggered asynchronously — the response is still returned to the caller.

cache.set(key, value, options?)

Option	Type	Description
ttl	number	Time-to-live in seconds
poolTarget	number	Hit threshold for pool growth. Enables pool mode when set.

cache.del(key)

Deletes a key and all its pool entries.

cache.info(key)

Returns detailed metadata including pool entries:

{
  key: 'fortune:love',
  hitCount: 12,
  poolTarget: 3,
  poolSize: 4,
  isGrowing: false,
  createdAt: 1712345678000,
  expiresAt: null,
  pool: [
    { id: 1, hitCount: 5, createdAt: 1712345678000 },
    { id: 2, hitCount: 4, createdAt: 1712345700000 },
    { id: 3, hitCount: 2, createdAt: 1712345800000 },
    { id: 4, hitCount: 1, createdAt: 1712345900000 },
  ]
}

cache.stats()

Returns aggregate cache statistics:

{
  totalKeys: 150,
  totalHits: 4520,
  poolKeys: 45,
  simpleKeys: 105,
  totalPoolResponses: 187,
  expired: 3
}

cache.purgeExpired()

Removes all expired entries. Returns the count of purged keys.

Adapters

Growing Pool Cache is storage-agnostic. Three adapters are included:

Adapter	Best for	Persistence	Multi-process
MemoryAdapter	Testing, prototyping, single-process	No	No
MySQLAdapter	Production with relational DB	Yes	Yes
RedisAdapter	Production with Redis	Yes	Yes

Using MySQL

const mysql = require('mysql2/promise');
const { GrowingPoolCache } = require('growing-pool-cache');
const { MySQLAdapter } = require('growing-pool-cache/src/adapters/mysql');

const pool = mysql.createPool({ host: 'localhost', user: 'root', database: 'myapp' });
const cache = new GrowingPoolCache(new MySQLAdapter(pool));

See src/adapters/mysql.js for the required table schema.

Using Redis

const Redis = require('ioredis');
const { GrowingPoolCache } = require('growing-pool-cache');
const { RedisAdapter } = require('growing-pool-cache/src/adapters/redis');

const redis = new Redis();
const cache = new GrowingPoolCache(new RedisAdapter(redis, { prefix: 'myapp' }));

Writing a Custom Adapter

Implement the following interface:

class MyAdapter {
  async get(key) {}              // Return meta object or null
  async set(key, data) {}        // Store/upsert metadata
  async increment(key) {}        // Increment hit count
  async setGrowing(key, bool) {} // Set is_growing flag
  async getNewest(key) {}        // Return newest pool entry { id, hitCount }
  async getRandom(key) {}        // Return random pool entry { id, response }
  async addToPool(key, resp) {}  // Add response string to pool
  async incrementPoolEntry(key, entryId) {}  // Increment pool entry hit count
  async getPoolEntries(key) {}   // Return all pool entries for info()
  async delete(key) {}           // Delete key + pool entries
  async purgeExpired() {}        // Remove expired entries, return count
  async getStats() {}            // Return aggregate stats object
}

See src/adapters/memory.js for a complete reference implementation.

Real-World Use Cases

AI Fortune Telling Service

// Cache key = fortune category + user's birth data
const cache = new GrowingPoolCache(adapter, {
  onGrowth: async (key) => {
    // Generate new variation in the background
    const fortune = await openai.chat.completions.create({ ... });
    await cache.set(key, fortune, { poolTarget: 3, ttl: 86400 });
  },
});

const key = `fortune:${category}:${birthYear}`;
const cached = await cache.get(key); // ~5ms, also triggers onGrowth when needed

if (cached) return cached;

// Cache miss (first request) — generate and store
const fortune = await openai.chat.completions.create({ ... }); // 14-40s
await cache.set(key, fortune, { poolTarget: 3, ttl: 86400 });
return fortune;

Result: First user waits 14-40s. All subsequent users get instant responses (~5ms). After every 3 hits, a new variation is generated in the background — no user ever waits. Users with the same birth data see different fortunes.

Chatbot Greeting Messages

const key = `greeting:${timeOfDay}:${userSegment}`;

Instead of "Good morning!" every time, the pool grows: "Rise and shine!", "Morning! Ready to get started?", "Hey there, early bird!"

Product Recommendation Descriptions

const key = `recommend:${productId}:${userProfile}`;

Same product, different compelling descriptions. The pool grows with demand.

Production Performance

From a real production service handling 10,000+ daily requests:

Metric	Value
Cache hit response time	~5ms
Cache miss (AI generation)	14-40s
Average pool size (after 30 days)	4.2 responses per key
Cache hit rate	94.7%
Monthly AI API cost reduction	~89% vs no cache
Response diversity score	4.2x vs traditional cache

Running Tests

npm test

Examples

node examples/basic.js
node examples/express-openai.js

License

MIT

---

Created by NSKit -- Built for production AI services.