How Models Read the World: Tokens, Context Windows, and Embeddings

Part two of my write-up of the AI foundations path. Part one landed on a definition: software that uses statistical patterns learned from data. This post is about the plumbing, how information actually gets into a model, and why so much of your development time goes somewhere you didn’t expect.

Tokens are the unit, not words

When you use a language model, it isn’t reading your text the way you do. Think of predictive text on your phone: you type “How are” and it suggests “you.” Same fundamental mechanism, scaled up enormously.

The model operates on tokens. A token is roughly four characters, sometimes a whole word, sometimes a fragment. “Understanding” might be two tokens: under + standing. The model doesn’t see words or sentences. It sees tokens.

This matters for two practical reasons. First, you’re billed per token, so token count is your cost model. Second, it explains a whole family of failures. Asking a model to count the letter “r” in “strawberry” is asking it to do character-level work on something it perceives as a couple of tokens. It’s not a reasoning failure so much as a representation mismatch.

Context windows are hard walls

Models can only process a limited number of tokens at once. The ceiling has climbed steadily, a bit over 4,000, then 8,000, then 100,000, then 200,000, and it keeps going. One of the courses put the jump starkly: around 10,000 tokens of practical context in 2024, up to a million in 2025. A hundredfold in a year.

That sounds like plenty until you try to process a 50-page document or hold a long conversation. The model literally cannot see past its context window. When a chat model “forgets” what you said earlier, that’s not forgetfulness. It’s a structural limit.

The dangerous part is the failure mode. If you overload the window, information gets dropped, silently. You don’t get an error. You get a confident answer based on partial input. This is one of the clearest examples of the pattern from part one: AI doesn’t fail loudly, it fails plausibly.

Treat context as a budget

The most useful reframing I picked up: every token you send costs money and consumes the model’s limited attention. So spend it on the task, not on boilerplate.

• Keep the system prompt short.
• Remove duplicates and compress long quotes.
• Keep retrieved chunks small, with slight overlap so sentences aren’t sliced in half.
• Order matters. Put the task first, then recent conversation, then the most relevant facts, then everything else. Models attend more to what’s at the beginning and end, and less to the middle.
• Consider summarizing the conversation behind the scenes once it grows, to keep the window focused.
• Cite sources, so a human can verify what the model actually used.

If you’re dropping inputs to make things fit, you’re buying confidently wrong answers. Fix retrieval to be more selective instead.

Embeddings: why “car” finds “automobile”

Everything, text, images, audio, has to become numbers before a model can work with it. The word “king” becomes a list of numbers. “Queen” becomes a different list. These numerical representations are embeddings.

The interesting part is that the numbers capture relationships. The classic demonstration:

KING - MAN + WOMAN ≈ QUEEN

That actually works, because the vectors encode semantic relationships rather than just identity. It’s why search over embeddings finds related concepts instead of exact keyword matches. Search for “car” and it surfaces “automobile” and “vehicle”, not because the system understands transportation, but because those vectors sit close together.

This is the machinery underneath semantic search and RAG. It’s genuinely powerful, and it’s still pattern matching.

Preprocessing is where your time actually goes

Here’s the part nobody warns you about. Models are extraordinarily picky about input format, and raw data almost never works.

A vision model trained on 224×224 pixel images wants exactly 224×224. Not 225. Not 223. Color channels must match, RGB versus grayscale isn’t a preference, it’s a requirement. Pixel values need normalizing to whatever range the model expects, maybe 0 to 1, maybe -1 to +1. One channel wrong, one normalization mismatch, and it fails.

Expect to spend roughly as much time formatting data for the model as integrating the model itself. I’m flagging it now so it isn’t a surprise when half the sprint disappears into it.

Vision has its own set of frustrations

You might assume vision is simpler, show it pictures, done. It’s differently hard.

To us, a cat is a cat whether it’s sleeping, jumping, or half-hidden behind furniture. To a model, those are different pattern sets. Rotate a cat 45 degrees and it may as well be a different animal, because the pixel patterns changed.

The failures are predictable once you know this: rotate an image and accuracy drops; partially occlude an object and the model gets confused.

It also opens an attack surface. You can alter a handful of pixels, invisible to a human, and make a model read a stop sign as a speed limit sign. The model isn’t seeing objects, it’s seeing numbers, and numbers can be nudged deliberately.

The tradeoff you can’t engineer away

Across every modality, the same triangle shows up: capability, speed, and cost. Bigger models are more capable, slower, and more expensive. Small models are fast and cheap and miss nuance.

This isn’t a limitation waiting to be solved next year. Better accuracy means more computation, which means more latency and more cost. You will constantly balance good enough against fast enough against cheap enough, and doing it deliberately is the job.

A concrete way to think about it, from the courses: in e-commerce, product search should be fast even if some matches are slightly off, users are browsing. The actual purchase flow should use the accurate model, because precision matters when money moves. Same product, different tradeoff per surface.

A rough sketch of where things land:

Surface	Priority	Reasonable approach
Interactive UI (search, autocomplete)	Speed	Small model, minimal context, aggressive caching
Recommendations, summaries, support	Balance	Precompute embeddings, cache by meaning, stream output
Financial or medical analysis	Quality	Large model, longer latency is acceptable and even expected

That last row has a subtlety I liked: for high-stakes analysis, users expect thoughtfulness. An instant answer can actually feel less trustworthy than one that took ten seconds. Sometimes the vibe is part of the design.

Multimodal collapses the boundaries

Formats that used to need separate models, text, images, audio, code, increasingly run through one. Modern multimodal models read PDFs, interpret charts, generate tables from images and images from tables. One model, multiple input and output types.

The mechanism hasn’t changed. It’s all still being converted to numbers and matched against learned patterns.

The one idea to keep

Models process information by converting it to numerical patterns and finding statistical relationships. Text, images, audio, it all becomes numbers.

That single fact explains both sides of the behavior. A model generates fluent text because it learned those patterns. It fails at counting characters because counting is computation, not pattern matching.

Though note the caveat, because it’s a real one: if the model can call a tool that runs code, it can compute. Many of today’s “it can’t do X” limitations are really “the model alone can’t do X.” The capability boundary moves when you give it tools, which is exactly why understanding where the boundary sits matters more than memorizing a list of things models can’t do.

Next: how a model gets from these patterns to an actual output, temperature, thresholds, confidence scores, and the compounding error problem that quietly wrecks multi-step pipelines.