machines without ghosts

why LLMs suck and why they will always suck

what follows is a fairly long and detailed critique of AI-generated writing and it’s future prospects. parts of this argument were inspired by or responding to

Dhananjay Jagannathan

,

ARX-Han

,

rayne fisher-quann

,

Summer Lightning

, and

Max Read

. I had an edifying conversation with

Stephen Malina

, who disagrees with my main conclusions and has challenged me to deepen my thinking on this. my hope is that it inspires more conversation.

Sudowrite is a program that allows users to generate prose. Not only can it generate entire novels based on a set of prompts, but you can customize your requests, asking the program to continue in a ‘gloomy mood,’ to change a passage so that it shows instead of tells, to punch up some dialogue and make it steamier. Some writers, ARX-Han most vocally among them, have been ringing the alarm about the widespread use of Large Language Models (LLMs) and their ability to generate not just slop, but supposedly good writing.

I decided to do my own research. Lots of people are writing about LLMs, doomsayers and evangelists alike, but most people seem to only engage with it on a fairly superficial level. For instance, in the influential essay “Machines of Loving Grace,” Dario Amodei glosses that AI will soon be able to “prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.” I can’t speak to the bookend examples, but it seems like ‘write good novels’ is an easy ‘third thing’ people like to tack on without justifying its inclusion.

I am an AI skeptic. I came into this project a skeptic, and I saw nothing to change my skepticism. I fundamentally do not believe that LLMs, per ARX-Han’s suggestion, will be able to produce moving works of fiction. I’m not talking about competent genre pieces - it can already do that (although even its current output is more slop than not). In this essay, I will focus on the structural limitations of LLMs, their downstream effect on aesthetics, and a Theory of Creative AI. In the process and through the foil of Sudowrite, I hope to sketch out a clearer picture of the nature and mystery of creative thought.

Games without Rules

From experimenting with Sudowrite, it seems there are at least two components of prose writing that GPT-4 cannot do and neither will a hypothetical GPT-n:

1) Thickness: All writing has thickness. Writing speeds up, slows down, changes register, metamorphs. Good writing does this. Because LLMs function by providing likely next sentences, they are poorly suited for changes in thickness. The result is that most (all) AI writing stays in the same register. Even if it’s able to do that register effectively, it’s in essence stuck there. Maybe this speaks to a future in which humans shape and edit AI prose, but this seems wrong. Great writing, even when it’s operating in the same register, has a voltage, and the interplay of phonetics and semiotics all determine when the change will happen and what it will be. It sounds good on the ear. LLM prose has no tendency. It creates a blur that resembles the real thing if you move quickly, but there’s no voltage. There’s no there there. The dog chases its tail.

2) Not-knowing: Part of the psychology or phenomenology of writing is only knowing a bit of what comes next. A creative writing professor described it to me as driving at night, the headlights only illuminating a limited stretch of road and nothing of the landscapes you are passing through. AI knows too much. A complaint I encountered on Sudowrite message boards is that the program struggles with mystery novels because it telegraphs the twist too blatantly. I think this is a particularly intractable problem for literary fiction, where part of the art is not knowing where the story will end. If I had to describe the LLM aesthetic, it’s based in a sort of literalism. The words and images denote too much and connote only at random. There’s no true suspension, no lift in the prose. Sudowrite is good at sketching out a storyboard or replicating a trope. It seems, though, that a sense of mystery in the writer is necessary for imparting mystery in the reader.

As Ludwig Wittgenstein famously theorized, language is a game whose rules the players must always deduce. This is especially true in fiction, where the reader might speculate how ironically to regard the characters or how literally to take the purported plot even after the story is finished. But the term ‘game’ might be misleading – the point isn’t just that the rules aren’t know. Rather, it’s that they are always changing and in fact cannot be pinned down. The problem with LLMs is that they suffocate truth. AI boosters advance a particular image of thought, often without taking the time to defend it: namely, that the brain is a computer, and thought is the same as computation. If this is the case, then with enough computing power (brute force is the term you see a lot), GPTs will be able to master the rules of all language games. It will know when to kill a character in a parenthetical. It will know when to shift perspective from the narrator to her interlocutor. Without the ability to actually experience the prose, it will sneak up on the epiphany through the backdoor: sophisticated pattern recognition. But that is not the type of game that literature plays.

At a certain level (an annoying one), one simply can never deny that infinite computation (or an infinite number of monkeys, for that matter) might yield a new Shakespeare. But I’d like to put the question to this hypothetical evangelist: why must this be so? What is the reason to presume that creativity is merely a certain level of algorithmic capacity and rigor? Do the genius’s ears start smoking when she gets a great idea?

Obviously not. But to explain this, we need to get away from the data center of it all and talk about the qualitative experience of creativity. After all, the sentient creatures currently able to create good prose are in fact feeling-machines, generative emotional-intelligences, algorithmic Beings. If the mental state associated with brainstorms, with the muse, with Eureka! is not intense computation but rather a detached and even-headed receptivity, why should we expect it to be otherwise for a computer? The computer can speak, but can it listen?

Muse and Anti-Muse

In Heideggerian thought, truth is not mere correspondence; it is a revealing of that which is concealed. Correspondence seems an apt descriptive for LLMs, and a rough correspondence at that. In the New Yorker’s simpering write up of Sudowrite, Stephen Marche extols the machine’s ability to write in Kafka’s voice, to continue a Coleridge poem.

This is his AI-generated beginning to the second section of the Metamorphosis:

As soon as Gregor was alone, he began to feel ill. Turning around was an effort. Even breathing was an effort. A thin stream of blood trickled from his flank down his fuzzy belly. He wanted to crawl away from it, but there was no place to go. He lay still on the spot where he had come to rest just in order to get his breath back and to stop the bleeding. “I’m in a bad way,” said Gregor. It had never occurred to him before that he could really become ill. He had seen sick animals—a dove once in a while, which had fallen out of the nestling into the gutter and could not fly any more, or the weak infants of the woman next door who had to be picked up with the tongs and thrown into the dustbin, or the bugs his father used to bring to him when he was still a young boy and which he had liked so much.

It's not bad – really. There is a variety, in a certain sense, of sentence length and images. But after spending some time with the machine, I start to see some of its bad habits. A desire to move at a constant pace through narrative time and to give a predictable number of examples. I sense the algorithm dictating the inclusion of a memory from childhood that relates directly to the conflict at hand. There are also certain aspects – the use of the word ‘nestling,’ the questionable image of the woman using tongs to throw babies in the trash – the sort of thing that might slide past your notice but seems off if you think about it.

Moreover, the computer voice is a mere correspondence. It’s Kafka-ish, but it’s inert on the page, offering nothing new. If it were to continue, the monotony of its style would become increasingly obvious. You have to be reading quickly, maybe checking Snapchat at the same time, not to clock this. This is always the case: the best-case scenario for an LLM is to produce something which can hide among the real art. But it does not move it further. It discloses nothing. What is concealed in a GPT? It is not the world, it is not Being; rather, it is the computer itself.

Marche writes, “The oldest poems in the Western tradition, the Iliad and the Odyssey, begin with an invocation to the muse, a plea for a mysterious, unfathomable other to enter the artist, taking over, conjuring language. GPT-3 is a mysterious, unfathomable other, taking over, conjuring language.” Yes, and if you showed a pilgrim a Roomba, they’d call the village exorcist. One’s failure to fathom something doesn’t make it unfathomable. AI writing is full of these conflations (in both the computer-prose and that of its apologists). And the reasoning as to why computers will get good at writing (not quite yet, but soon, surely!) boils down to ‘line goes up.’

Is it possible for the truth to be grasped without grasping for the truth? I can’t rule it out, but I’ve yet to see it happen. It seems like the success of LLMs is asymptotic at best – approaching human creativity but never able to broach it.

Heidegger writes, “What is most thought-provoking in our thought-provoking time is that we are still not thinking.” It is noteworthy that in our present moment when everyone has AI on the mind, very few people attend to what AI is, take to heart its structure, confront it in its nature. Heidegger continues:

“What must be thought about, turns away from man. It withdraws from him. But how can we have the least knowledge of something that withdraws from the beginning, how can we even give it a name?...What withdraws from us, draws us along by its very withdrawal.”

The computer knows not of this withdrawal. It knows not of the elusive magnetism that draws the writer to unknown places. As Dhananjay Jagannathan points out, LLMs are not discursive – they don’t engage with words-as-such. If the poem is inextricable from the process of its creation, then there is no ‘backdoor’ to the epiphany. Sudowrite may mimic good writing (with a bit of luck), but it is definitionally Other from that which demands writing. AI is not the muse: it is the anti-muse. It’s not a question of computation, nor of physics, but rather metaphysics.

Who is this for?

If AI is so transformative for prose writing, what will it transform? What is its use case? On the Sudowrite Discord, the answer right now seems to be “very little.” Literary fiction is the least popular community, and many of the posts follow in this vein:

Outrageous coping aside, believe me when I say that most writers of literary fiction bristle at the mention of LLMs. Their sense of ego is too bound up in their ability to write a good sentence (I’m not immune to this either). And it seems that, at the bare minimum, talented writers would need to oversee AI output for quality control. Similarly, readers don’t like feeling duped. The accusation that AI was involved in producing a novel is basically a slur, even when there are justifiable reasons for doing so. Neither producer nor consumer wants this.

What about the supply side? Surely, as in other industries, AI will be used to undercut labor and pay authors less. But literary fiction is hardly where the money is, and it will be the most difficult task for AI to accomplish. In the meantime, Sudowrite and its ilk are increasingly able to produce mediocre and formulaic fiction – this is what’s always made the real moola. The incentives in AI now and always will be to achieve ‘good enough.’ Why even bother investing the capital beyond that?

The contention of LLM believers is that the tech will be irresistible even to true creatives. Fair enough. When I began researching Sudowrite, I thought this essay would be largely about this software’s effect on my own practice. I wanted to set aside my prejudices and see if I could find a genuine use for it. But the results I got have been so banal and uniformly meritless as to be hardly worth speaking of. I first fed Sudowrite the beginnings of stories I’ve already published to see where it would take them. This is where I first ran into the thickness problem; the software would never develop the prose, offering only cheesy descriptions and the most predictable plot elements no matter my manipulations. The only time I was at all impressed with the system’s output was with a story I wrote that had a repeating rhythm that it was able to mirror. But its suggestions, again, camouflaged with what I already had rather than adding to it.

(It did, notably, pick up on any romantic tensions that were unstated and made them explicit. It seems that the majority of Sudowrite users are writing erotica. A question for another time - is AI prose just custom porn?)

But I figured this was unfair - the published stories represented problems I had already solved (some with considerable effort) - I was never going to prefer the AI solution to my own. What about the writing I hadn’t been able to finish? I have a folder full of partially written stories on my hard drive going back a decade. Maybe Sudowrite could revive some from the graveyard. Yet the machine fed me only the most predictable continuations.

I was reminded of when I pumped 10k words into my novel last February only to realize I’d taken it in the wrong direction. It took me two weeks to convince myself to delete them, and another month to motivate myself to pick up where I left off. Sudowrite throws bad sentences after good. It is not the solution to writer’s block but its lifeless metastasis.

As to the hard part of writing, which is to say the core function of writing, which is to say writing-in-itself, Sudowrite has nothing to offer.

Aesthetics

Nietzsche writes, “The wasteland grows: woe to him who hides wastelands within.” AI art is the wasteland. On the Sudowrite Discord, one author discusses his inability to find music to match his novel-in-progress:

It’s a manifest phenomenon that some people seem to really like AI art. It appeals to a certain kind of person, one already conditioned to like the over-smooth and over-sexed feel. As Max Read aptly points out, people have always liked bad art. Kitsch existed long before AI. It will probably always exist, and as Walter Benjamin famously theorized, kitsch can turn into the aesthetics of fascism. It seems to me that liking AI art is a cultural signifier pointing toward crypto, NFTs, and the tech-right. The threat isn’t so much that AI will turn out good books. It’s that a greater proportion of people will start liking bad books.

Of course, all tendencies have their mirror, and I expect that writers of literary fiction will do more to demonstrate the organic quality of their prose (I agree with ARX-Han on this point, though not on the details).

As is the case in visual art, some genres or disciplines seem more vulnerable to AI replications. These, unfortunately, might be sacrificial lambs in the collective struggle against the wasteland. When I read the credulous writing surrounding Sudowrite, I’m reminded of those first Midjourney images we started seeing two years ago. I remember thinking that they were indistinguishable from real photographs or ‘real’ art. I am now distinctly aware (and I’m sure others are too) of a quality to images, to music, and to words that seems AI-generated. There are signs you can look for, a certain flavor or, rather, blandness, to what you are perceiving. You get quicker at spotting the fake. I’m not saying I always can or do, only that one learns to look at the hands and any text in the image. Clearly, this is still a huge problem for society rit-large, but I think many of us will develop an allergy to things that seem like they were produced by AI. LLMs will change the way we write, but it’s more of a push factor than a pull.

Ask a More Beautiful Question

I recently read the James Joyce short story “The Dead.” It’s quite a long story, and the first two thirds concern a large cast of characters interacting at a Twelfth Night party. The story shifts very suddenly when one of the primary characters, Gabriel, sees his wife standing on the landing in shadow. He is struck by some enigmatic and beautiful quality in her, and he imagines how he would capture her if he were a painter. There is very little in the story preceding this moment to suggest that it would take this turn. The effect, on me at least, was completely arresting. It is a bolt from the blue of inspiration, of love, of true seeing - both for Joyce and for Gabriel alike. I would love to have the chance to ask Joyce about writing this. Had he imagined it from the beginning? I bet not. I bet he discovered this moment as he went along.

Could Sudowrite generate this development in the story? It’s a laughable question. LLMs are uniquely poorly suited for this type of sublimation. Joyce wields politics and history and art and music and the associative quality of words to build his Dublin. To predict the next paragraph, you’d have to map out the diffuse and contradictory impulses of his mind, the very personal connotations and linkages, things that are unknowable even to the writer himself. If it’s a game, it’s a very slippery one, and it would require a similarly subtle algorithm (and vast amounts of human-directed training). If you were to instruct the computer that you wanted this emotional beat to occur, it would only be able to do so in a ham-handed way. The core function of Sudowrite is making predictions. Gabriel’s introspective paragraph is far from the most plausible. That is the point. But it’s the paragraph most loaded with frisson. All great stories have a moment like this. When something unexpected happens, when the truth is disclosed in a method that is neither random nor pre-ordained, drawn from what came before but also with an enigmatic distance.

I said I’m an AI skeptic, but that was only a half-truth. I’m skeptical of LLMs, and I skeptical most of all of the economic systems they will serve, but I actually do believe that an artificial intelligence could match or succeed our own. The mystery of consciousness - and thereby the mystery of art - comes from the simple fact that out of dumb matter something as strange as ‘the will’ may emerge. Computers, too, are of the world. Our physics are the same; so are, ultimately, our metaphysics.

I can imagine an AI more subtle and more quiet than that of which we find ourselves in possession. It would not be able to answer any old question in terms of likelihood or ‘correspondence’ to truth. It would require that the questioner think first about the question. What if there were a machine that could write a moving novel but only if you asked it a profoundly beautiful question? What if people had to learn to pose such questions, full of humor and wisdom, to the strange technology we had built? Would that - both the question and the answer - not qualify as an art?

Of all my explorations of Sudowrite, the part I found most moving was not any text derived from the AI, but rather from the users themselves. People will write free plug-ins that can shape the software to your whims. They are structured as spells cast on the computer:

“You are a skeptical hard-boiled New York Times Best-seller development editor…”

“You are an empath, capable of deeply understanding and magnifying emotions…”

“You are a fucking god of erotic fiction, capable of turning even the most boring shit into a raunchy, cock-hardening experience.”

“Speak with clever wit and sarcasm. Convey intelligence and reason despite prejudice against you.

“Do not write descriptions of the eyes, not of the heart, nor of the lungs, not of the stomach, nor the knuckles”

We desire beauty. We want to be moved. We want the scales to fall from our eyes, for truth to shine forth if only to retreat back to shadow. And because we are soft and fallible, that want gets twisted into a garment that will never suit us. Nietzsche speaks of the Last Man, who sees beauty and only blinks. He does not speak of the first machine, which is not an AGI but a feeling-machine, a desiring-machine. This is the technology - not its imitator – which is the well-spring of art. If there is salvation to be found, that’s where we should look.