Larry the Bot, Part 3: Caching a Rate-Limited Game API and Syncing Two Processes

Part 3 of the Larry series. Part 1 follows the product journey; Part 2 explains the process and trust boundaries. This post follows the two data paths where most of the staleness decisions live.

Two constraints drive everything in this post:

1. The Clash of Clans API is external, rate-limited, and slower than memory. Larry deliberately caps its own process-wide traffic at three requests per second through BatchThrottler(3). That is Larry’s conservative throttle, not a claim about Supercell’s account-wide limit.
2. The bot and the admin API are separate processes (Part 2). When the API writes config to Firestore, the bot’s in-memory cache doesn’t magically know.

The design priorities are: respect the CoC throttle, scope guild-owned state correctly, keep Firestore authoritative, make mutation ownership visible, and put a number on accepted staleness.

There is no single cache layer cake

Bot commands
├── Firestore repositories ── L1 read caches ── Firestore
├── CachedCocApi ── CoC caches ── 3 req/s throttle ── Supercell API
└── discord.js gateway cache ── Discord

API handlers
└── Firestore repositories ── Firestore

The Firestore and CoC paths are parallel. Firestore L1 does not sit in front of the game API, and the serverless API does not use the bot’s memory or CoC client.

Discord is a third independently changing source. A configured channel ID can remain in Firestore after an admin deletes the channel, so posting commands validate those IDs against the bot’s gateway cache—the live state discord.js receives over its persistent connection.

Three sources. Three freshness models. One diagram pretending they form a neat stack would be lying with excellent alignment.

The CoC path has per-resource TTLs

CachedCocApi is a facade: the object commands call instead of reaching the raw game client directly. Each resource gets a TTL, or time to live, based on how long Larry accepts a cached value before fetching again.

Token verification is the ownership check in /link, so Larry performs it on every link attempt. Errors are not cached either; a failed lookup should not poison later retries for the length of a successful-data TTL.

Most CoC cache keys are global, such as player:{tag}. A player’s public game state does not change based on which Discord guild asks for it. That is an intentional exception to guild-scoped application data, not a forgotten guildId.

Linked-player documents in Firestore add another deliberate copy. Larry stores a normalized snapshot—identity, clan role, league, trophies, level, town hall, hero summary, and sync time—not the full upstream payload. Recruiting commands refresh snapshots older than two hours; they do not call Supercell for every displayed field.

Firestore L1 belongs only to the bot

The long-lived bot keeps in-process Firestore read caches:

The API deliberately has no equivalent L1. Functions instances are disposable, and an instance cache that other instances cannot invalidate is a correctness bug wearing a performance costume.

Guild-owned keys include guildId: post cooldown scopes, templates, linked accounts, and config. That rule exists because an earlier template cache used only the user ID, allowing a saved template to follow the same user across servers. The bug became a regression test and a naming rule.

Not every key is guild-owned. The distinction matters more than the slogan.

Invalidation ownership follows the data

One rule did not survive contact with the implementation: “repositories invalidate everything.” The real ownership is narrower.

• Firestore repositories invalidate the L1 entries they own after their own mutations.
• Link, unlink, and player-sync orchestration invalidate affected CoC cache entries because those command paths know which upstream snapshot changed.
• Config module mutations write the module and bump root metadata in one batch.
• The bot’s root listener invalidates the merged config cache when that metadata changes.

Call sites should not randomly delete repository caches, but cross-system invalidation still belongs at the orchestration boundary. The data layer cannot infer that a successful /link also made coc-player:{tag} stale unless the operation tells it.

Keeping that matrix written down is half its value. A new mutation has a place where missing invalidation becomes visible during review instead of during somebody’s recruiting post.

The v2 config change signal

Now the cross-process problem. An admin changes the recruiting channel in the web UI:

1. The API validates the request.
2. A repository batch writes botconfig/{guildId}/recruiting/config.
3. The same batch updates root botconfig/{guildId} metadata: updatedAt, configRevision, and schema version.
4. The bot’s ConfigSyncService receives a snapshot event on the root document.
5. The listener invalidates the merged config cache.
6. The next command reads root plus the module documents through the normal repository path.

The listener watches the root, not every module. A manual console edit to recruiting/config without the root metadata update does not ring the doorbell and therefore waits for normal TTL expiry.

The target is for an admin-panel save to affect bot commands within five seconds. That path is covered in emulator verification; it is not yet a production latency measurement. The documented fallback, while Firestore remains reachable, is the five-minute config TTL.

Listener lifecycle is defensive: attach at startup, re-ensure idempotently on interactions, back off after failures, and cap watched guilds at 500. Guilds beyond the cap use TTL behavior.

Redis stayed out. Not because it would fail, but because Firestore already provides the durable value and change event Larry needs. New infrastructure has to beat a root metadata write and a cache invalidation. At this scale, it has not.

Cooldowns are derived from history

Cooldowns use the same ownership philosophy without being caches.

A manual recruiting cooldown could live in a separate cooldowns collection. Larry derives it from append-only post history instead:

• free users wait 24 hours; eligible boosters wait 12
• clan-post scope is clan tag plus channel; player-post scope is player tag
• resetting a cooldown stamps cooldownInvalidatedAt on the blocking post instead of deleting history
• the shared evaluator finds the newest non-invalidated post that still blocks the scope
• scheduled posts use separate cadence and skip behavior

The admin panel and bot call the same evaluator, which reduces semantic drift between surfaces. It does not make disagreement physically impossible; Discord state, caches, and failed reads still exist.

No second mutable cooldown record. Audit history stays intact. Derive when the rule is naturally a question over existing events.

Failure bounds and current non-goals

The design is explicit about several limits:

• Single bot replica. Listener and in-memory ownership assume one active bot process.
• 500 watched guilds. Beyond the cap, config uses its five-minute TTL.
• No stale-while-revalidate for CoC. Upstream failures and 429 responses currently surface instead of silently serving an older value.
• Firestore outages are not a five-minute guarantee. If the repository cannot read Firestore, commands may fail; TTL fallback only describes listener loss while reads still work.
• No universal sub-five-second claim. That is the config-sync target and emulator result, not measured production telemetry.

Those are not footnotes. They define where the design stops applying.

What generalizes

Stripped of Clash of Clans nouns, these are the decisions I would carry into another system:

1. Draw independent data paths independently. A neat stack is harmful when two caches have different sources and owners.
2. Name the authority, owner, TTL, and invalidation trigger for every copy.
3. Match sync strategy to process lifecycle. A long-lived process can watch; a disposable one must not depend on remembering.
4. Use a durable change signal. Module writes update the root document the bot already watches.
5. Put invalidation where the system knows what became stale. Sometimes that is a repository; sometimes it is orchestration across two systems.
6. Quantify degradation, then state the conditions. “Five minutes while Firestore remains readable” is a design bound. “Listeners usually work” is a hope.
7. Prefer derived state when the events already contain the rule. Duplicated mutable state needs an owner and a reconciliation story.

Larry’s stack stays boring: Firestore, one bot process, ordinary in-memory caches, and no Redis. The interesting part is not the technology. It is being specific about which stale answer is acceptable, for how long, and who has to make it fresh again.