Context Window Trimming

The problem, from first principles

A conversation grows with every turn, but the model's context window is finite and every token costs money and latency. The simplest form of agent memory is a sliding window: keep the most recent messages that fit in the budget and drop the oldest. Two rules keep it from breaking things. The system message holds the agent's instructions and must never be dropped. And the kept messages must be a contiguous suffix of the history: skipping a short recent message to keep a longer older one would leave the model with a conversation that has holes in it.

From theory to code

Implement trim_messages.

Constraints

• messages is a list of dicts with role and content. A message's cost is len(content.split()) tokens.
• All messages with role 'system' are always kept. The remaining messages are considered from newest to oldest and added while the running total stays within max_tokens (system messages count toward the total); stop at the first message that does not fit, so the kept non-system messages form a contiguous suffix.
• Return the kept messages in their original order. If the system messages alone exceed max_tokens, raise ValueError.
• Do not modify the input.

Hints