Token
The processing unit of an LLM — approximately 0.75 words in English. LLM API pricing is calculated in input and output tokens.
Definition
A token is the atomic processing unit of an LLM. Models don't read words or characters — they read tokens, which are character sequences defined by a tokenizer specific to each model family. As a rule of thumb: 1 token ≈ 0.75 words in English, and source code is generally less token-dense than natural text (spaces, braces, and keywords all count). All LLM API pricing is expressed in tokens: X dollars per million input tokens (what you send), Y dollars per million output tokens (what the model generates).
postcursors perspective
The token is the cost variable to master in an agentic workflow. A session where you load 10 PHP files of 300 lines consumes ~30k tokens in input alone before the agent responds to anything. Multiplied by Sonnet 4.5's price ($3/M input), that's ~$0.09 per session. With 50 sessions per day across a team of 5 devs, you're looking at ~$225/month just in input. Understanding token mechanics helps you make model and workflow choices that have a real impact on costs.
In practice
Useful tool: OpenAI's tokenizer is open-source and visualizes how text gets split (tiktoken). To estimate session cost: paste the contents of files to be loaded into the tokenizer, multiply by the chosen model's input price. The agent's response will typically be 2-5x shorter than the context, so output costs less than input.
Common misconceptions
- ✗ Confusing tokens with words — the conversion varies by language and content
- ✗ Ignoring input tokens in cost estimates — they often represent 80-90% of a coding session's cost