Bring your own key: adding an LLM feature without betting the product on one model

If you are new to building with AI, the most common feature request you will ever get -- from a client, a teammate, or yourself at 11pm -- is some version of "can we add an LLM to this?" Summarize the game. Explain the move. Coach the player. Narrate the recap. The model is good at all of it, the SDK call is four lines, and the demo lands. So you wire it in, ship it, and discover the two problems nobody warned you about: every call now costs you money, and your whole product is quietly married to one vendor's one model.

This post is about how this project added an LLM feature without either of those problems. It is grounded entirely in code that has already shipped -- the AI advisor framework in the advisor framework, marked COMPLETE as Phase 4 in the overview docs. The lesson is portable. You can apply the same three moves to whatever you are building.

What Phase 4 actually is

The platform is a LAN board-game system: a TV is the shared table, phones are private hands, and a server holds the authoritative game state. Most of the catalog works with no LLM at all. The AI opponents are deterministic search -- Monte Carlo Tree Search and its hidden-information cousin -- and they play just fine offline. (For why those opponents have personalities instead of just playing optimally, see AI opponents that play like people, not optimizers.)

Phase 4 is a separate, horizontal layer that sits on top of every game: the LLM advisor. It is the part that turns numbers into sentences. There are four advisor surfaces, defined as a small enum in the advisor framework:

• Recap -- a narrative summary after the game ends.
• Live commentary -- occasional in-game color, drama-tiered so it stays quiet most of the time.
• Coaching -- a private, plain-English "here is what that move was really worth" whisper to one player's phone.
• Rules Q&A -- answering "wait, can I do that?" from a dense rulebook.

The reason it is built as one horizontal layer instead of being re-implemented per game is the same reason any shared infrastructure exists: every later game in the catalog wants to land with these surfaces wired in, not bolt them on afterward. Werewolf reuses it for the moderator's narrative voice and its post-game lie detector (that story is told in Werewolf without a moderator); Hold'em reuses it for the strategic advisor; Bluff reuses it for coaching. Build the layer once, correctly, and every game inherits it.

Three design decisions make that layer worth copying. Here they are, in order of how much grief they will save you.

Move 1: put the model behind a seam

The single most important line in the whole subsystem is the smallest. In the advisor framework, there is an abstract base class with exactly one method:

class LLMProvider(ABC): @abstractmethod async def complete(self, *, system: str, user: str, max_output_tokens: int) -> LLMResponse: ...

That is the seam. Every piece of game code that wants prose from an LLM talks to this interface -- complete(system, user, max_output_tokens) in, text out -- and never to a vendor SDK directly. Behind the interface live the concrete implementations in the advisor framework:

• claude.py -- an adapter over Anthropic's SDK.
• openai.py -- an adapter over OpenAI's SDK.
• ollama.py -- an adapter over a local model served by Ollama on the same machine, no hosted API at all.
• fake.py -- a canned-response provider so the entire test suite can run with zero network calls and zero spend.

The payoff is stated as a hard invariant in the ROADMAP: "No single backend is load-bearing." Because every caller depends on the interface and not on Anthropic-or-bust, swapping providers is a configuration choice, not a rewrite. A tiny factory (in the advisor framework) maps a (provider, model) pair to the right adapter, and the game code upstream never notices which one it got.

If you are new to this, internalize the move before anything else: the first thing you do when adding an LLM feature is hide the LLM. Not because vendors are bad, but because the moment your business logic imports a specific SDK and a specific model name, you have made a bet you cannot easily unwind -- on pricing, on availability, on that model still existing in a year. The Fake provider is the tell that the seam is real: if you can run your whole product against a fake LLM in tests, you have genuinely decoupled "what the feature does" from "which model does it."

Move 2: bring your own key (and why that is the v1 business model, not a cop-out)

Now the money problem. Every complete() call costs something. Who pays?

The answer this project ships is BYOK -- bring your own key. The user supplies their own API key; the platform never bills anyone for tokens. The key lives in memory for the active room only (see the KeyStore in the advisor framework), resolved one of a few ways: a host can provide one key for the whole table, or an individual player can plug in their own key for private coaching that the rest of the table never touches.

It is tempting to read BYOK as "we were too lazy to build billing." It is the opposite -- it is a deliberate sequencing decision, and the overview docs lays out the ladder it sits on:

• Free tier -- deterministic AI only (the MCTS opponents), template commentary, fully offline. No LLM, no spend.
• Platform tier (future) -- the platform holds a key and bills a subscription for it.
• Premium tier -- the user brings their own key and pays their own LLM costs directly.

The non-obvious insight is that the premium tier is the one where platform cost goes down. A user on their own key is a user the platform spends nothing on. So v1 ships only BYOK: every user is effectively premium, the platform defers building any subscription or metering infrastructure until it is actually worth it, and a hosted platform tier can be added later to ride alongside BYOK whenever the economics justify it.

And notice what makes this possible: it is the seam from Move 1. The provider abstraction is exactly what lets the project defer billing. Because no single backend is load-bearing, a user can point the feature at a hosted Claude or OpenAI key or at a free local Ollama model, and the host can choose a table-wide key while one privacy-minded player overrides it with their own. None of that flexibility requires the platform to operate a payment system. BYOK aligns with the rest of the product's posture, too -- this is a platform that deliberately holds as little of yours as possible, the same instinct behind device profiles instead of user accounts. Your key is yours; it is held only as long as the room is live.

Move 3: let deterministic code own anything that must be correct

Here is the move that separates a toy LLM feature from a trustworthy one, and it is the one beginners skip.

An LLM is a language model. It is superb at language and unreliable at arithmetic. So the rule this project enforces is a clean division of labor: deterministic Python owns every number that has to be right; the LLM only renders prose around numbers it is handed.

Bluff is the cleanest illustration. Bluff is a game of lies -- you claim a rank, you may be lying, and the skill is deciding whether to call. A strong coach can compute, against tempo, the Bayesian probability that the current claim is true, given the cards in your own hand and the public history of claims. Almost no human can do that fast enough at a real table. (The full design of why Bluff has a coaching whisper but deliberately no chat panel is in A game built on lying, with no chat panel.)

So who computes the probability? Not the LLM. Look at the advisor framework: a function called _decision_facts does the actual reasoning in plain Python. It counts how many copies of the claimed rank the player already holds, reads the public claim history, checks whether the claim is arithmetically impossible given those known cards, and decides whether a call was a good call or a thin one. Those are facts, computed deterministically, the same way every time. Only then is the LLM invoked -- and its instructions (in the sibling prompts.py) are narrow: write one second-person paragraph explaining the reasoning in natural language. The model never decides the probability. It dresses a number it was given.

This boundary shows up across the whole catalog. Coaching prose is layered on top of a deterministic MoveAssessment (in the relevant AI module) that already graded the move as a blunder or a brilliancy; the LLM only explains the grade it was handed. Hold'em's equity is computed, then narrated. The discipline is always the same: if it has to be correct, a deterministic function computes it and a test pins it; the LLM is the last mile that turns a correct fact into a sentence a human enjoys reading. (The broader version of this argument -- which failures the deterministic layer must absorb so the LLM never becomes load-bearing -- is in When a stage fails: what is mechanical and what is the LLM?.)

There is a quiet bonus here. Because the math is deterministic and the LLM is the optional garnish, a session with no key at all does not crash -- it simply degrades to the deterministic output, with a clear status indicator saying the LLM features are off. The numbers still work. You just do not get the prose.

On demand, not nagging

One more thing that is easy to get wrong: when the model speaks. The lazy default is to narrate everything -- a comment on every move, a tip after every turn -- and it is exhausting within five minutes. This project treats restraint as a feature. Live commentary is drama-tiered, so most moves get silence and only the genuinely dramatic ones get color. Coaching is a whisper to the one phone that asked for it, never a broadcast. The advisor never plays for you and never interrupts to remind you it exists. It answers when asked and otherwise stays quiet.

For a beginner the heuristic is simple: an LLM feature that talks constantly reads as a gimmick; one that talks exactly when it has something worth saying reads as a tool. Default to quiet.

The honest edge

BYOK is not free magic, and selling it as free magic would be dishonest. The trade-off is real and worth naming plainly: BYOK pushes both cost and quality variance onto the user.

Cost, because the user supplies and pays for the key. The platform's bill goes to zero precisely because the user's does not. For someone running a free local Ollama model the marginal cost is just their own electricity; for someone on a hosted Claude or OpenAI key, every recap and every coaching tip is a charge on their account, not yours.

Quality, because output tracks whatever provider and model the user chose. A small local model and a frontier hosted model are both valid behind the same seam -- that is the whole point of the abstraction -- but they do not write equally well. The coaching paragraph a user reads is only as good as the model behind their key. The seam guarantees the feature works across providers; it cannot guarantee the prose is uniformly excellent, because that is the user's choice to make and the user's cost to bear.

That is the honest shape of the deal: the user trades "the platform handles it and charges me a subscription" for "I bring my own key, I pay my own way, and I pick my own quality-cost point." For a v1 that wants to ship LLM features without building a billing system, it is a genuinely good trade -- as long as you say so out loud.

The one lesson to take with you

If you remember nothing else from this post, remember the three moves in order, because they compose:

1. Put the model behind an interface. Depend on a complete() seam, never on a vendor SDK directly. A Fake provider in your tests proves the seam is real.
2. Let deterministic code own anything that must be correct. Compute the numbers in code you can test; hand them to the LLM only to turn into sentences.
3. Invoke the LLM only where natural-language judgment genuinely helps -- on demand, not constantly.

The seam is what lets you defer the business decision (who pays) without rewriting code. The deterministic boundary is what lets you trust the feature. The restraint is what makes it pleasant. Adding an LLM is the easy part -- four lines and a demo. Adding one responsibly, so that it does not bankrupt you, lie to your users, or chain your product to one model, is the part worth learning. This is the most transferable thing in the whole repository, and it is why Phase 4 was built the way it was.

---