There are plenty of things to say about Large Language Models and the discourse around AI. I may not be able to resist contributing to that discourse. But this post (and probably some others to follow it up) is an attempt to step away from the Discourse, and just try to better understand this technology.

That choice itself can be reasonably critiqued. Can we afford to look away from the ways that these technologies exacerbate climate collapse and prop up hypercapitalist totalitarianism? Can one use a technology that is largely controlled by people who seem somewhere between comfortable and downright eager to placate the current US presidential administration? Besides, how interesting can a tool be that is undermining education and facilitating illiteracy and cheating? What could conceivably be learned from a technology that is little more than an act of mass intellectual property theft? A technology that is an insult to the very idea of creativity? To the very notion of humanity? Can it possibly be worth scrutinizing the details of the valve timings on an engine that is shredding the already-frayed fabric of collective life?I don’t think all these critiques are equally reasonable though.

My modest, tentative answer is: yes, it is worth it to better understand these things. In part this is because I don’t fully agree with all the critiques (perhaps unfairly) summarized above. One can be curious without being uncritical. And most fundamentally, what I think I don’t share with AI skeptics, is utter shock that these machines work as well as they do. It is very hard for me not to remain surprised that a next-token predictor can do the things that it does. If you had told me ten years ago that a technology built by digesting every book and reddit post would be able to pass the Turing test, I would have been deeply skeptical—and it is my sense that most other people would have been too. It was simply not obvious that this technology could produce these results. The shape of the political moment and the massive investment in these technologies (and those are not uncorrelated forces) have produced strongly polarized responses, with accordingly stark utopian/dystopian visions. But I remain surprised, and curious, how software, trained this way, produces these outputs.

Below, I want to talk about a more specific instance of that surprise concerning how LLMs represent individual tokens. That realization can be summarized: the “meaning” of each token as represented in an LLM is unique to its context. The phrases “I saw a bear in the woods” and “a bear was running through the woods,” will use the same token (“bear”) with the same token ID. They may be describing the same particular bear. But the context reshapes the representation of that word so that the token “bear” will have a unique value, or meaning, in these two cases. Equating the value of some vector—some list of numbers—with the “meaning” of the term is tricky. Another way to say this: meanings, in an LLM, are not points, they’re regions. In one way, this is just repeating the premise of contextual word vectors. But it still struck me as odd. So I wanted to try to walk through it carefully. This post is my attempt to do so.

Another Attempt to Explain how LLMs Work

Understanding my point requires trying to understand how LLMs work. I’ve found different explanations of LLMs stress different elements of the machinery. This one will try to focus on how vector representations of tokens are reshaped. Recall, that the first step when an LLM processes text (I’ll talk about inference, and ignore training here) is to split the input into terms from the model’s vocabulary—to tokenize the input. After the text is tokenized, each token is replaced with an initial vector of some high dimensionality. For gpt2-small (the smallest version of the gpt2 model, and the one I’ll be using as an example), those vectors have 768 numeric values. That means each token can now be imagined as a point in 768-dimensional space. For larger open models, that dimensionality can be larger. The embedding vector dimensionality for the most current models from Google, OpenAI, Anthropic, and others is not publicly disclosed. Whatever its size, this initial “static” embedding is basically the same as the word embeddings of word2vec or GloVe. LLMs learn this embedding during model training. I think this initial embedding value can be loosely described as the average meaning of a token across the training data. After this vector look up, all tokens of the same type (i.e. having the same ID), will start with the same value. However, even at this first stage, positional information is added to the vector. This “positional information” is not just an index of the position itself, but another vector, of the same dimensionality as the embedding, which is, like the initial embedding, learned during training.

Consider the text input: “dog dog.” That looks like two instances of the token (dog).It could of course be the token “dog” and the token ” dog,” but we’ll ignore that. Even in this first stage all tokens of the same type (ID) will probably not have the same vector value, because to a simple token value is added “positional information.” This though is the trivial way that token values are essentially unique.

The work of the LLM is to then reshape this vector based on context, so that the “best” possible next token can be predicted. To do so, all those vectors values (that have been looked up and altered by position vectors) are passed through a series of “layers.” A layer is simply a set of mathematical functions to reshape the vector values. gpt-2 has 12 layers. (Again, details like “number of layers” are not typically disclosed for the chatbots most folks think of as LLMs: ChatGPT, Claude, etc.) The neural network alters each vector based on all the preceding tokens in the context window. Knowing exactly which of those preceding tokens should influence any particular token, and how (how many, and which of those 768 numbers to change; and how to change them—increase or decrease), is of course non-trivial. It’s the whole (decidedly black-box-y) mechanism. This is the “attention” mechanism at work: a system of keys, values, and queries that is the backbone of transformer-based LLMs. The most relevant terms in the context push or pull that initial, layer-0 vector value in different directions. And then you do it again… and again, for each layer in the model (along with some simpler “feed forward” layers in between). I struggle to have a meaningful, intuitive understanding of this process; and I suspect that most other people do too.(I, like literally millions of others, have found 3Blue1Brown’s videos illuminating—on LLMs and much else besides.) But ultimately, however baroquely complex, this is a matter of lots of cross products passing through some non-linear functions, with the effect of reshaping each vector according to values learned from the training data. That training process (frustratingly called “pretraining”) has “learned” the particular values that will reshape the input vectors in ways that are meaningful. Or, perhaps more accurately, values that will reshape the input vectors to be most consistent with the training data. (Keep in mind: the task during training is accurately predicting the next token given the input.)This has led to the description of LLMs as “autocomplete on steroids,” a description that helpfully gives one an intuition for the main goal of LLM training, but may underplay the particular means of how they pursue this task.

So, text becomes tokens; tokens becomes vectors; and the vector values change based on the other vectors in the context window (or, more precisely, the preceding vectors). And all these values, in all these vectors, are changing in parallel. The “product” or “goal” of this process, in typical use, is to predict the next token after the last token in the context window. Doing so involves reshaping all the vectors, multiple times. The autoregressive character of LLMs means that all vectors representing the context window reshape themselves in light of all the preceding vectors. That this process could write even a bad poem or produce an inaccurate summary of the Krebs cycle still strikes me as deeply weird.

After all the tokens have been reshaped across all the layers, but before the next token is generated, we have a reshaped set of vectors that all been changed based on the values/weights/parameters learned during training. How do we get from these values to the next token or word? We take our reshaped final vector and turn it back into a token by “unembedding” it. As the name suggests, this is essentially the reverse of the embedding stage. In gpt2, it is done by comparing our final 768-dimension vector to the initial vectors representing each token in the vocabulary. (Whether the “frontier” models use the same matrix for embedding and unembedding is another undisclosed detail.) This comparison is done with that staple of linear algebra, the dot product. One dot product is calculated between the final vector and a vector representing each term in the vocabulary. (That is a lot of dot products—tens of thousands at least.) Dot products with higher numeric values mean greater similarity. (These numbers, to the frustration of neophytes and nonexperts, are called logits—despite having nothing to do with logistic regression or with the log-odds.) Using softmax, this list of numbers can easily be turned into a probability distribution across the whole vocabulary of the model.Or, you could use a setting like “temperature” to reshape this distribution, and so likelihood of tokens being sampled (about which, more below). In typical inference (when you’re using a chatbot like Claude or ChatGPT), there is complicated (and stochastic) logic to select a token from this distribution. In practice, always choosing the top “most likely” next token produces bad outputs. And so, there is a deliberate injection of randomness to what is otherwise a deterministic processes built on a probabilistic representation of the training data.Technically, this determinism can be lost just through matrix multiplication because of the way that floating point rounding works in digital computers. If LLMs are stochastic parrots, they are stochastic in two ways: their training attempts to model the conditional probabilities of a distribution that is only very loosely sampled by even their enormous training data; but also, the text they generate from these distributions involves deliberately injecting randomness because, empirically, it makes the output “better.”¹

However it is selected, the new token is added onto the input. If this selected token isn’t some special “end of response” token, the whole thing starts over, now with that one added token that will, in turn, reshapes the values that come after it in the window. And you do the whole thing again.

You might note that only the last token of the last layer is unembedded in order to predict the next token. So why did we do all those other calculations—for each token at each layer? This is the autogressive character of the LLM: in order to get that last token’s vector representation, we need all the others, and each of those need all the layers, all reshaped as if they were trying to predict the next token. But each token in the context window is being reshaped.² So that is my own, overlong, explanation of LLMs.

Each Meaning is Utterly Unique

After struggling to understand this process, I was surprised to notice that the value of any contextualized vector is almost certainly unique. I don’t know that this is counterintuitive exactly, because I have no intuitions about how meanings represented as points in high-dimensional space. The naive sense that the word “jello” is the same word in every sentence is belied by a system which rearranges its representation of “jello” in every appearance. Context reshapes meaning, so that the representation of any token within that message is unique to the message.

Should that be surprising? After all, every unique sentence is describing, or doing, something unique. Doesn’t my point reduce to the rather obvious observation that unique sentences are unique? Maybe! I think my surprise stems from a naive sense of the discreteness of words. There is a tension between the abstraction that language requires by carving the world up into manipulable meaning-units like words, and any specific use of it. I am ready to concede that “unique sentences are unique.” I was surprised to realize that in an LLM every unique sentence (or unique context) is made up of unique vectors.

An LLM enacts this tension between the abstract and the particular, between the token-type and the contextualized, vectorized token value, in a particular way. After the initial embeddings, the LLM simply is a series of vectors in a continuous high-dimensional space such that each token-representation is unique. To find two vectors that are identical in any position (other than the first!) seems impossibly unlikely. Because computers represent things in digital bits, the space is not truly continuous. But consider: each token position in gpt2 (a “small” “large language model”) is represented by 768 32-bit precision floating point numbers. That means each of those 768 numbers can take on one of 2³² values. 768 numbers, each capable of 2³² unique values, means 2^(32*768), or 2^24,576 unique combinations. That is so many possible values. So many, that if you, try to print the number, Python returns a ValueError.³

My point is simply that the in an LLM is no longer what I naively assume “a word” is. To say, as LLM explainers often do, that in an LLM each word is replaced with a set of numbers is not untrue. But the description is too coarse to be useful. More useful is thinking about how words are first mapped in, and then back out, of the model’s high-dimensional space. In this model words and meanings are less discrete points than neighborhoods in the high-dimensional space that is constructed by the LLM and its training. Even though all the tokens will have the same token ID, and will begin with the same initial embedding (even though modified by their position in the context window)—everything else will be different. Those vectors will be moved around, so when the is output, it is because the predicted vector has ended up in some region or neighborhood of high-dimensional space, near to other thes, such that it unembeds as the. A vector becomes a word in an LLM not because of some essential quality, but simply because that token-representation is the closest one based on a dot product. This idea, that meaning is ultimately simply a matter of proximity (of near and far) is just very strange.

Consider this very basic illustration. I fed each of the following sentences into gpt2 independently (i.e. the model only ever saw one line from this passage at a time).

I went to the bank.
Joe went to the bank.
I went to the store.
Joe went to the store.
I am going to the store.
I will eat the elephant.
Green ideas sleep furiously in the dull mortgage.

Each of these sentences was independently tokenized, replaced with a set of 768-dimension vectors (one per token), and then processed, layer by layer. At each layer, the weights that the model learned during training dictate how each of these vectors modify each of the other in each layer. Now imagine stopping the machine, opening it up mid-processing, and looking at those lists of numbers. This sort of inspection has been done in a more rigorous way by serious researchers. But I grabbed a layer in the middle (layer 6, of this 12 layer network). From this layer I get one vector per token for each sentence. I then used Principal Component Analysis (which I think of as the most boring, straightforward method of dimensionality reduction) to reduce the 768-dimensional vectors for each term to 2 dimensions and created the plot below.There was an additional element of normalization because otherwise GPT2’s layers behave oddly. I only learned that in the course of trying to produce this plot. While the typical use of gpt2 would be to predict the next token, I haven’t retrieved that. For this illustration, it doesn’t matter.

Each point in the plot above is represents a token from one of the above listed sentences (hover over to see the sentence, or sentences, from which it was drawn). I’ve color coded the most frequent words here, to show how in the LLM even the “same” token is mapped to different places within the model’s space. Consider the word/token the above, from the sentences “I went to the store” and “Joe went to the store.” These two sentences feature the same token, in the same position, with a meaning that is as similar as I could imagine. But the vectors representing the in these two sentences are still separated a little. On this 2-dimensional reduction, they’re close. Most instances of the are in the lower right of this plot, representing sentences like “I am going to the store” and “Joe went to the bank.” But some end up in the top left. The the of “I will eat the elephant” is closer to ideas and sleep than the the of “I went to the bank.” I often see plots like these stress the proximity—the nearness of similar terms. We normally plot a bunch of terms and note how semantically similar terms appear near each other, but far from dissimilar terms. Just as important as the nearness of similar terms is the distance a model creates between similar, and even “identical,” tokens. Note, though, that the values for the in “I went to the store” and “I went to the bank” will be identical. That’s because all the tokens and positions to the left are identical, the “context” is identical. (The plot represents identical vector representations with a larger point.) This again is because of “predict the next token” task. When you’re predicting the next token, then “I went to the bank” and “I went to the store” are identical right up until the last token (bank/store) (in a “bidirectional” model, like BERT, this would not be the case.)

This is a toy example. gpt2 is a very small model. Reducing 768 dimensions to 2, for purposes of plotting is obviously a lossy and potentially misleading process.These components represent only ~25% of the total variance, which is suprisingly good given that we started with 768 dimensions! I wouldn’t “read” too much into this plot. It captures a complexity in thinking of meaning as space. If meaning is space, words are not simply points, but regions. And if that’s the case, the sort of questions we might ask change. Are different regions larger? Is shape meaningful? Semantic space can be described not simply in terms of near and far, but with a richer vocabulary.

The Well-Wrought Vector: Uniqueness and Precision

Reflecting on this way of imagining meaning, I was reminded of a passage from Cleanth Brooks’s The Well Wrought Urn (1947). For Brooks, poetry is a mode of language where each meaning becomes as precise and utterly individual as a contextualized vector. Yet,how this uniquness works is rather different. Language, for Brooks, tends to drift into abstraction and imprecision. It is a flawed tool. But this drift, in poetry, can be beaten back and precision achieved. In trying to describe this model of language, and poetry’s place in it, Brooks relies on images of meaning as space. Discussing Wordsworth’s “It is a beauteous evening, calm and free”, Brooks writes:

T. S. Eliot has commented upon “that perpetual slight alteration of language, words perpetually juxtaposed in new and sudden combinations,” which occurs in poetry. It is perpetual; it cannot be kept out of the poem; it can only be directed and controlled. The tendency of science is necessarily to stabilize terms, to freeze them into strict denotations; the poet’s tendency is by contrast disruptive. The terms are continually modifying each other, and thus violating their dictionary meanings. To take a very simple example, consider the adjectives in the first lines of Wordsworth’s evening sonnet: beauteous, calm, free, holy, quiet, breathless. The juxtapositions are hardly startling; and yet notice this: the evening is like a nun breathless with adoration. The adjective “breathless” suggests tremendous excitement; and yet the evening is not only quiet but calm. There is no final contradiction, to be sure; it is that kind of calm and that kind of excitement, and the two states may well occur together. But the poet has no one term. Even if he had a polysyllabic technical term, the term would not provide the solution for his problem. He must work by contradiction and qualification.

We may approach the problem in this way: the poet has to work by analogies. All of the subtler states of emotion, as I. A. Richards has pointed out, necessarily demand metaphor for their expression. The poet must work by analogies, but the metaphors do not lie in the same plane or fit neatly edge to edge. There is a continual tilting of the planes; necessary overlappings, discrepancies, contradictions. Even the most direct and simple poet is forced into paradoxes far more often than we think, if we are sufficiently alive to what he is doing. (9–10)

Brooks’s description helps me understand my own surprise at this element of contextual embeddings’ behavior. “The terms are continually modifying each other, and thus violating their dictionary meanings.” This contrast between the model of meaning as space and the dictionary is clarifying. The dictionary is organized by words. It imagines a meaning as an instance of some general type. Contextual embeddings start with a dictionary-like look up; but the whole process that makes them contextual embeddings involves modifying and altering them in a series of ways such that the vector representing a token may no longer most resemble the token it began as. We turn to the dictionary believing that we can understand a word in isolation. But the fantasy of the self-contained integrity of a word, embodied by the dictionary, is violated by the power of a context, which “continually modifies” the individual word. Science (Brooks suggests) attempts to solidify the mushiness of language, freezing it into a stricter denotation. It intensifies the logic of the dictionary, adding more and more words. It introduces new terms, and uses old terms more strictly. Poetry achieves a greater precision. This is a deliberately audacious claim—outperforming science on its own territory. Poetry does accepts the imprecision language, but overcomes it through paradox. By bracing one imprecise meaning against another. Poetry navigates meaning-space more precisely not by trying to finely indicate a particular point, but by understanding precise meaning as an intersection in meaning space.

Brooks’s view of a meaning is LLM-like insofar as he imagines words as imprecise regions, representable as spatial, geometric forces. A desired meaning and its expression don’t line up; you use a metaphor, but that comparison includes too much or not enough—the meanings “not lie in the same plane or fit neatly edge to edge.” Meaning is “a continual tilting of planes”—a nice enough image for meaning as space. Brooks further elaborates this geometric model of meaning, comparing the poet’s work to a quotation from Hamlet (Polonius’s advice on successful spying).

Brooks explains:

Shakespeare had in mind the game of lawnbowls in which the bowl is distorted, a distortion which allows the skillful player to bowl a curve. To elaborate the figure, science makes use of the perfect sphere and its attack can be direct. The method of art can, I believe, never be direct—is always indirect. But that does not mean that the master of the game cannot place the bowl where he wants it. (10)

Science seeks to reform and rationalize meaning space, to straighten it out, to allow precise statement. The poet instead takes advantage of this curvature through paradox.

LLMs are not poets. They might, as Leif Weatherby suggests in Language Machines (which is undoubtedly the best, and most sophisticated work, on the intersection of structuralist theory and LLMs), represent something like Roman Jakobson’s “general poetic function” ( esp. chapter 5). But I want to end by observing a difference between LLMs and Brooks’s description. They share a geometric understanding of meaning, and focus on individual meanings a unique points within this geometric space. Yet, the fact that each message/context is singularly unique is not the same as Brooks’s assertion that the poet’s meanings are singularly precise.(Crucially Brooks completely separates poetry—and perhaps all of “literature”—from the rest of language and culture. The rejection of this segregation is probably the central fact of academic literary criticism since the late 1960s.) At the crucial moment, Brooks must reach outside language, through a deictic gesture. To follow the reading I’ve quoted above, we must agree with him that the paradox of the feeling Wordsworth describes exists. We must recognize “that kind of calm and that kind of excitement.” Brooks relies on this ostensive, deictic appeal to link the uniqueness of the poetic representation back to the world. And it is this link that allows the uniqueness to indicate not a point in semantic space, but a point in the world. For Brooks, the uniqueness of the representation allows it to overcome the imprecision of language and name something exact in the world. In an LLM, of course, uniqueness reflects nothing except consistency with training data.

Discosure: All the prose here was, for better or worse, written by me. The code I used to generate the plot above was generated by Claude Code.

Works Cited