In our last post, How LLMs Work, we went over the basic building blocks of LLMs. We started by going over how the next word is generated, and then how basic chats are handled. But Franz does much more than just talk back and forth with you. How does he handle remembering things over time? How does he create new items? How does he move characters from point A to point B? In this post, we’ll dive into how Franz works specifically.
In other AI chat applications, building context is quite simple. It is usually just your chat history + basic details and instructions. But Friends & Fables is much more than a chat app, we have an entire game engine to run. Your stats need to be tracked. Your inventory, map position, and quest progress all need to be updated! On top of that, worlds in Friends & Fables have an immense amount of data. A single world can have thousands of characters, items, monsters, and locations. The world builder might have written hundreds of detailed lore entires. On top of that, some of our players play months long campaigns that have hundreds of thousands of messages.
To make all this possible, we can’t rely on a single LLM to handle all of this information. When you send a message to Franz, we don’t just make one request to an LLM, but instead we might make a dozen different requests to various LLMs purpose built for certain tasks. Each request has its own context derived from your game state, all orchestrated with the purpose of generating the best narration possible, as fast as possible.
Your campaign is composed of a bunch of different context such as map information, items, characters, lore entries, and more. We could just send all of this data to the LLM every time we generate Franz’s narration, but that wouldn’t be a good idea for a few reasons:
The key to success here is to make sure the narration step has the right context at the right time and nothing more. Here’s why it is so important: Imagine you are an LLM trying to predict the next sequence of words in a tavern scene, but in your context you have a bunch of irrelevant information about various monsters, spells, and combat instructions. That extra information is just irrelevant noise that makes it harder to focus on what’s important, and decreases the probability that the “right” tokens will show up given the current moment.
In order to make sure Franz has the most relevant context, we’ve created a feature called Working Context. Working Context is a shared workspace between you and Franz. It is the context that both of you can see and edit. Before Franz writes his narration, he conducts research where he looks at your message and then searches across his long term memories, lore, and entities to pull in the most relevant information he can find into working context. You can think of all of your campaign data as the documents and books a regular GM might have, with the working context being the specific notes from those documents that are relevant for the current scene.
Franz tries his best during the research step, but he can mistakes or miss information in his searches. He also can’t always know upfront what players want out of their campaigns. That’s why we’ve designed working context to be transparent and editable, so if you want you can have full control and steer Franz into the right directions with the right context at the right time.
Narration is just one piece of the puzzle for an AI role-playing game. Franz also needs to consider and keep track of things like your stats, current location, items, available spells, etc. To make these things possible, Franz offloads it to other LLMs for post processing which take a look at the current state of your game, your most recent message and Franz’s response, and then decides to take appropriate actions like moving you to a different location or adding items to your inventory.
Unlike traditional games where the mechanics and outputs of actions are fixed, in Friends & Fables these are probabilistic as the decisions need to be made by the AI on the fly depending on the current state of your campaign. Each campaign is a completely unique playthrough depending on the actions of the player and has its own unique combination of game state and decisions to be made. This means the AI can make mistakes given the complexity of the task and will likely never be 100% perfect, but our goal is to gradually increase the quality floor over time so that more often than not, it gets it right.
Deciding to build an actual AI game instead of a chat app means that certain trade offs have to be made. For instance, since we’re loading much more context about the world and not just a chat history, and since that context needs to be shared across dozens of requests for a single response, we can't use the largest frontier language models. Large frontier models are too slow and costly to operate. If we were using the largest models, then you could be waiting minutes for a response and the subscription plans would be way too high for the average consumer.
Instead, most of the system is composed of medium size models which run extremely fast and are more economical than the largest frontier models. We think this trade off is worth it since it allows us to provide an experience grounded in an entire world you can build and explore with your friends or AI companions, rather than a one dimensional character chat.
Creating the framework of a game and a world for the AI to use also means that we’ll be able to introduce game mechanics and experiences that are simply impossible within a single model’s context window. We also believe that over time, the intelligence of models will continue to get smarter, cheaper, and faster, which will naturally fill in the gaps and improve the experience as those capabilities become available.
That being said, not everyone is looking to play an AI game. If you want to dive deep purely into a pure text roleplay, then you may be better off with another service that uses less context overall but in a singular larger model with more control over the text.
Thank you for reading! If you have any more questions about how Franz works, feel free to hop into our Discord or Subreddit and ask questions.
