Trick question — it has none. An AI has no neurons, no cells, no spark. It has numbers arranged in grids, multiplied together billions of times. This guide takes you from a single pretend "neuron" all the way to how a language model writes a sentence — in plain English, with pieces you can poke.
It starts with a "neuron" that isn't one
Your brain runs on roughly 86 billion real neurons — living cells that fire electrochemical pulses at each other. An artificial neuron borrows the word and almost nothing else. It is a tiny piece of arithmetic: take some numbers coming in, multiply each by a weight that says how much it matters, add them all up, and pass the total through one simple curve that "squashes" it. That's the whole cell. Drive one yourself:
That is the entire neuron: multiply each input by a weight, add them up (plus a bias), then squash the result through one simple curve. No biology, no spark — three lines of arithmetic.
The weights are the only thing that makes one neuron different from another. Slide them and you change what the neuron "cares about." Everything a neural network knows lives in numbers exactly like those — and "training" is just the slow process of nudging billions of them until the answers come out right.
Advertisement
AdSense Slot: content_top
Stack them in layers — and it becomes multiplication
One neuron is nearly useless. The power comes from wiring thousands of them into layers, where every neuron in one layer connects to every neuron in the next layer — and only the next. That single rule is the reason people say neural networks are "just matrix multiplication": the full set of connections between two layers is exactly a grid of weights, and pushing a signal through it is one matrix multiply. Fire the signal, then hit "see the matrix":
Every node connects to every node in the next layer only — never sideways, never skipping ahead. Those connections between two layers are just a grid of numbers:
So a network is layers of neurons, and each gap between layers is a matrix. Run the input through matrix after matrix — with a little squash in between so it can learn curves and not just straight lines — and you have the entire engine. Depth is just how many of these matrices you stack; width is how big each one is.
So how big do the real ones get?
For a large language model, "width" is the size of the vector flowing through the stack, and "depth" is the number of layers. The honest currency isn't neurons but parameters — the individual weights — because a layer of 1,000 units feeding 1,000 units is already a million of them. Here are the public ones:
A few thousand to ~16,000 wide, and tens to about 130 layers deep — not absurdly deep. The frontier models (GPT-4-class, Claude, Gemini) keep their exact shapes secret, and many now use mixture-of-experts, where each layer holds many feed-forward blocks but only a couple fire per word — which is why a single "neuron count" stops meaning anything.
What actually goes in: tokens, not letters
A language model doesn't read letters or whole words — it reads tokens, sub-word chunks from a fixed vocabulary of roughly 50,000 to 128,000 pieces. Each token is just an ID number. To turn that ID into something the matrices can chew on, the model looks it up in a giant table and pulls out a row of numbers — the token's embedding. In GPT-3 that row is 12,288 numbers long.
This is the part people most often get backwards: those 12,288 numbers are not 12,288 words. They are 12,288 learned coordinates describing one token — a fingerprint of its meaning, placing it in a vast "meaning space" where similar words sit near each other. Feed in a ten-token sentence and you get a grid: ten of these fingerprints, stacked, all flowing up through the layers together.
The word is split into 3 tokens, each just an ID number. Every token becomes one row of numbers — its embedding. We show 10 cells; a real model's row is 12,288 long. The numbers are not words; they are coordinates that place the token in a vast "meaning space" (illustrative values).
Advertisement
AdSense Slot: content_mid
Attention: how the words talk to each other
Here's a puzzle. If each token just flows up its own stack of matrices, the words never meet — "bank" by the river and "bank" with your money would be processed identically. The fix, and the single idea that made modern AI work, is attention. Each transformer layer has two steps: a feed-forward step where every token is processed in its own lane (the matrix multiplies above), and an attention step where every token gets to look at the other tokens and pull in what's relevant. Mix, process, mix, process — dozens of times up the stack.
In the models that write text, attention is deliberately one-way: each token can look backward at earlier tokens but never forward at words that haven't been written yet. That restriction is exactly what lets the model generate left-to-right, one word at a time.
Try "it" — it leans hardest on "cat", which is how the model knows what "it" means. Every word builds its meaning from the words behind it, and never from the ones ahead — that one-way rule is what lets the model write left-to-right.
How it writes a sentence — and talks to itself
To pick the next word, the model takes the final token's row of numbers — which by now has absorbed the whole sentence through attention — and multiplies it by one last matrix that scores every word in the vocabulary at once. The highest-scoring word (or a weighted random pick from the top ones) becomes the next word. Then the trick: that new word is appended to the input and fed back in, and the whole process repeats for the word after it.
This feedback loop is why "thinking out loud" works. When a model reasons step by step, or drafts an answer and then critiques its own draft, it is literally reading its own earlier output as new input — its words and yours merge into one stream. The honest limit: the part doing the checking is the same predictor that wrote the text, so self-correction helps but never guarantees — there's no separate truth-checking organ inside.
The randomness dial: temperature
Remember the model produces a probability for every possible next word. A setting called temperature decides how boldly it picks. At temperature 0, it always takes the most likely word — completely deterministic, but often boring. Turn the temperature up, and it starts taking risks on lower-probability words.
At 0 it always says "blue" — deterministic. Crank it up and "purple" or "crying" start winning: more surprising, more creative, and eventually nonsense.
So, how many neurons?
Zero. But it has billions of parameters, arranged in matrices, multiplying inputs and passing them through simple curves, layer after layer. It turns words into numbers, lets them look at each other through attention, and predicts the next number. It is just math — but stack enough of it, and the math learns to speak.