Pelajaran 2 dari 49 menit
Language models and tokens
Models read and write tokens, not words. Learn what tokens are, how next-token prediction and temperature shape answers, and what a context window limits.
Pelajaran ini hanya tersedia dalam bahasa Inggris. Antarmuka halaman tetap dalam bahasa Anda.
Dalam pelajaran ini
- Explain tokens and why they matter for cost and limits
- Describe next-token prediction and sampling
- Explain what a context window is
Tokens: the units a model sees#
Before a model reads text, a tokenizer splits it into tokens: whole words, pieces of words, punctuation or spaces. Common words are often one token; rarer words are split into several. Each token maps to a number, and the model works only with those numbers.
| Text | A plausible split |
|---|---|
coffee | coffee |
unbelievable | un believ able |
Selamat pagi! | Sel amat ·pagi ! (the dot marks a leading space) |
Exact splits depend on the model's tokenizer. A useful rule of thumb for English is roughly four characters per token. Languages that were less common in the training data, Indonesian included, often need more tokens for the same meaning.
Why you care
Providers price usage and set limits in tokens, both for your input and for the model's output. Longer prompts and longer answers cost more and take longer.
function roughTokenEstimate(text: string): number { return Math.ceil(text.length / 4);1}console.log(roughTokenEstimate("Generative models predict the next token.")); // 11Only an estimate for English text. For exact counts, use the tokenizer or token-counting endpoint your provider offers.
Next-token prediction#
Given the tokens so far, the model computes a score for every token in its vocabulary. Those scores become probabilities, one token is chosen, appended, and the process repeats until the answer is complete. A long answer is thousands of these small steps.
Temperature: how adventurous the choice is#
Picking the single most likely token every time gives predictable but sometimes repetitive text. Instead, models usually sample: they pick randomly according to the probabilities. Temperature reshapes those probabilities before sampling.
function softmax(scores: number[], temperature: number): number[] { const scaled = scores.map((score) => score / temperature);1 const max = Math.max(...scaled);2 const exps = scaled.map((value) => Math.exp(value - max)); const total = exps.reduce((sum, value) => sum + value, 0); return exps.map((value) => Number((value / total).toFixed(2)));}const candidates = ["coffee", "tea", "juice"];const scores = [2.0, 1.0, 0.1];console.log(candidates, softmax(scores, 0.5)); // [0.86, 0.12, 0.02]console.log(candidates, softmax(scores, 1.5)); // [0.56, 0.29, 0.16]Dividing by a low temperature stretches the gaps between scores, so the favourite dominates. A high temperature flattens them.
Subtracting the maximum before Math.exp avoids overflow with large scores without changing the result.
Low or high temperature?
Low temperature suits extraction, classification and code, where you want consistent answers. Higher temperature suits brainstorming and creative writing. Even at low settings, outputs can vary between runs, so do not build features that assume identical answers.
The context window#
A model can only consider a limited number of tokens at once: its context window. Everything it uses to answer has to fit: instructions, the conversation so far, documents you include, and the answer it is writing. When a conversation outgrows the window, apps drop or summarise older parts, which is why long chats can "forget" early details.
Peringatan
A bigger window is not a reason to paste everything in. Irrelevant text costs tokens and can distract the model from what matters. Include what the task needs.
LatihanBudget a prompt
You have a document of about 24,000 English characters and want a 300-token summary. Roughly how many tokens will the request and response use together?
Tampilkan solusi
About 24,000 ÷ 4 = 6,000 tokens for the document, plus a few dozen for your instructions, plus about 300 for the answer: roughly 6,400 tokens. Check the real count with your provider's tokenizer before relying on it.
Recap#
- Models read and write tokens; costs and limits are counted in tokens.
- Answers are built one token at a time from probabilities; temperature controls how varied the choices are.
- The context window caps everything the model can consider at once.