How does ChatGPT actually work?

It is a machine that guesses the next word.

This is the main goal of AI. The essays, code, and debates all come back to one question: given everything so far, what comes next?

Watch the sentence on the left. It is trying to finish it, and it has opinions.

It learned by being corrected, trillions of times.

Give the model a huge amount of text. Hide the next word and ask it to guess. If it is wrong, adjust it slightly.

Repeat this again and again across billions of examples.

It does not store facts like a database. It learns patterns that help it predict what comes next, which is why it can sound right even when it is wrong.

Your sentence is split into tokens.

Before the model can read text, it turns it into numbers. The text is split into small chunks called tokens.

Common words may stay whole. Unusual words are often split into smaller pieces.

The model works with tokens, not letters. That is why simple tasks like counting letters in a word can sometimes confuse it.

Each token becomes a position in space.

Each token is turned into a list of numbers that represents its meaning.

Tokens with similar meanings often end up close to each other.

That is why a search for “time off” can also find something about “annual leave”. They are close in this number space.

But “bank” can mean different things.

The meaning of a word depends on the words around it. “Bank” could mean money or the side of a river.

So the model updates a token’s meaning based on its context every time.

The next step is what helps the model understand that context.

Attention decides what matters.

The model looks at the earlier words and decides which ones matter most for what comes next.

This helps “bank” mean money in one sentence and a river bank in another. It also helps the model understand what words like “it” refer to.

It learned these connections from patterns in the data, not from grammar rules written by people.

Different attention heads watch for different things.

Attention runs many times at once. Each part can learn to focus on something different, like nearby words, subjects, verbs or pronouns.

Some attention heads are much more useful than others.

Nobody assigns these jobs. The model learns them during training, and researchers study them afterwards.

These layers are stacked on top of each other.

The output of one layer becomes the input to the next. Each layer looks at the information again and improves its understanding.

A large model can repeat this dozens of times before producing one word.

Earlier layers often learn simpler patterns. Deeper layers can capture more complex meaning.

It gives every possible token a probability.

The model does not produce an answer immediately. It gives every possible next token a score.

Most have almost no chance. A few are much more likely.

The model is predicting what is likely to come next, not deciding what is true or false.

Temperature controls how adventurous it is.

Low temperature makes the model choose safer and more predictable words.

Higher temperature gives less likely words a better chance, creating more variety but also more mistakes.

This is one reason the same question can produce different answers.

It does the whole thing again.

The chosen token is added to the text. Then the model predicts the next one.

It repeats this process one token at a time until the answer is finished.

The response is built step by step, with each new token based on everything that came before it.

Everything must fit inside the context window.

Your prompt, previous messages and any documents all have to fit inside the model’s context window.

Anything outside that window is no longer available to the model while generating the answer.

More context also means more information for the model to process, which can increase cost and computation.

Then people rated its answers.

Predicting the next word alone does not make a useful assistant. So people rated different answers and chose which ones were better.

The model was then trained to produce answers people found more helpful and appropriate.

This helps shape things like helpfulness, tone and refusals. It can also make models more likely to agree with users than they should.

It cannot count letters

The letters were discarded at step one. It is recalling what it has read about spelling, not looking at the word.

It invents citations

Step four always returns something. With nothing well-grounded available, the most plausible-looking option wins anyway.

It contradicts itself

Nothing was planned at word one. Each word is chosen given the last, so a long answer can drift somewhere its opening never intended.

It forgets the beginning

Only what fits in the context window exists. Past that edge there is nothing to attend to.

It agrees with you too readily

Raters preferred agreeable answers, so tuning pushed it that way. Confidence is a style it learned, not a signal of accuracy.

It is bad at arithmetic

Numbers are split into token chunks like everything else. It is pattern-matching the shape of sums, not calculating.