5 min read 2 wks ago

Tokens, Context and Messages

Let's tackle words and concepts you might not know about

Tokens 

Think of tokens as the currency AI models use to digest and generate text. Each piece of text is broken down into these units called tokens. The number of tokens can vary based on how common a sequence of characters is and how the AI model has been trained to understand it. It’s like the AI’s vocabulary!

To get a better idea, why not give it a go yourself? Head over to the OpenAI Tokenizer and plug in some text to see how it’s tokenized. It’s pretty cool to see how the AI breaks down your words!

Context 

Imagine having a conversation with someone who forgets everything you said a minute ago. Frustrating, right? That’s where context comes into play. The AI keeps track of what has been said previously, storing it in its memory bank to maintain a coherent conversation. This history, or “Context”, isn’t just limited to what you say, though. It also includes instructions, known as system messages, which can be set up when you’re configuring your chatbot or when you’re pulling in dynamic content like a web search. This way, the AI isn’t flying blind: it’s got the backstory!

Messages 

Each entry in the context is a message. Think of messages as pieces of a puzzle that the AI puts together to see the full picture. Each message takes up a certain number of tokens, so you’ll want to balance the length and number of messages to avoid overloading the AI, especially since there’s a maximum context size it can handle.

Understanding this in AI Engine 

Let’s break down some of the technical jargon and make it easier to understand:

  • Max Tokens: This is the maximum length of the AI’s reply, in tokens. Different models have different limits. Most users just keep a generous value and don’t need to worry about this setting too much. You will find it in the AI Model section of your chatbot. If a reply reaches this limit, it ends with a short notice instead of stopping in the middle of a sentence, so you know it’s time to raise it.
  • Max Messages: This is all about the conversation history. Think of it like the AI’s short-term memory of what you’ve been discussing. Remember the puzzle pieces we talked about ? This setting limits how many of them are sent to the AI with each new message. It’s in the Thresholds section of your chatbot, like the two settings below.
  • Input Max Length: This one refers to the length of the new message you’re sending to the AI. Keep it concise to stay within limits! This is the number of characters allowed in the input box of your chatbot.
  • Context Max Length: This isn’t about tokens anymore; it’s the maximum size, in characters, of the extra context added to the conversation: the page content with Content-Aware, the embeddings found in your Knowledge, or other dynamic content like web search results. If there is more, it’s cut to fit. The conversation history is controlled by Max Messages, not by this setting.

And a little heads up: it’s now called Context Max Length, not Context Max Tokens, for a bunch of reasons. It’s simpler and faster this way. Plus, calculating tokens on the fly can be tricky and might vary between models and systems, so let’s stick to what’s stable and clear: text length.

Thresholds section of the chatbot editor with Input Max Length, Max Messages and Context Max Length

Modern AI models have very large context windows, so the default values in AI Engine work well for most use cases. As a rule of thumb, 4 characters is roughly equal to 1 token in English. The values below are a good starting point (as of late 2026), but always check the latest model specs from each provider. When you pick a model in the AI Model section, AI Engine also shows its limits under Max Tokens (Contextual and Completion).

Here are some recommended starting values depending on the model you’re using:

OpenAI Models

GPT-6 (Astra, Sol, Luna) / GPT-5.6 / GPT-5.5 / GPT-5.4: these models support around 1,000,000 tokens of context and up to 128,000 tokens of output. Smaller models like GPT-5.4 Mini have a smaller window (400,000 tokens), which is still a lot. Safe starting values:

Max Tokens: 4096 tokens, Max Messages: 15 to 32 messages, Context Max Length: 16384 characters (~4096 tokens)

With that much context, you have a lot of room. Feel free to increase Context Max Length or Max Messages based on your needs.

Reasoning models: most current GPT models think before they answer, and that thinking also counts toward Max Tokens. If you pick a high Reasoning level, give them a higher Max Tokens, or the whole budget can be spent on thinking before the reply is written.

Anthropic Models

Claude (Opus, Sonnet, Haiku, Fable): the current Claude models support 1,000,000 tokens of context and up to 128,000 tokens of output. Older ones, like Claude Haiku 4.5, have 200,000 tokens of context. The recommended values are the same as above.

Google Models

Gemini (Pro, Flash): Gemini models support very large context windows (up to 1 million tokens for most of them). AI Engine reads their limits directly from Google. The same recommended values apply, and you can comfortably increase them.

General Tips

For Max Tokens, which is only used for the reply, a generous value means the AI can use as many tokens as it needs to build its answer. If you want shorter answers, lower it; if you need detailed responses, increase it. If you see the notice that the reply was cut, raise it.

For Max Messages, this controls how much conversation history is kept. More messages means the AI remembers more of the conversation, but also uses more tokens (and costs more). New chatbots start at 15, and 15 to 32 messages is a good balance for most chatbot use cases.

For Context Max Length, you can increase this significantly with modern models. If you’re using embeddings or long pages with Content-Aware, consider raising it to 32,000 characters or more.