Prejudiced Against Magic?
On Supposed Machine Intelligence
Contents
You’re sitting comfortably in your seat talking to your friend when the lights go out, the room goes silent and the curtain is raised. The magician walks on stage and the magic show begins.
For his opening act the magician shows an empty hat, then puts it on a table, chants abracadabra and promptly pulls out a rabbit. The audience claps, you clap, but you know that the hat has a false bottom and the rabbit was under the conveniently draped table. Boring.
Then he asks for a volunteer. You raise your hand, and he picks you.
“Think of a number from 1 to 1000,” he says. “Write it down on this paper and hide it.” You see where this is going. No way he can guess the number in my head. So you write down the most unnatural number you can think of: 738.
“The number is 738!” the magician exclaims with a knowing smile.
Your jaw drops. How in the world? You start looking for the trick. Did he have someone with a camera behind you? Or did he watch the motion of your pen? Whatever the case may be, this trick certainly doesn’t convince you he actually used magic.
Now swap the stage for a chat window. You type a question, and an answer appears that looks like it came from a mind. Your jaw drops just the same, but this time many stop looking for the trick.
And it’s easy to see why. It is simply undeniable that AI has progressed enormously and today’s models can do extraordinarily useful things. So people selling AI, but also some intelligent individuals and the general public have started to believe that AI is intelligent, or soon will be, or at least that we cannot rule out such a possibility.
There is no such possibility.
Recall the magic show. When a magician “reads your mind”, no one reasonable should take that as proof of him having magical powers. Why not? Because what matters for magic and intelligence alike is not what you do, it’s how you do it.
So if a trick you can’t yet explain doesn’t convince you of magic, outputs that merely look intelligent shouldn’t convince you of intelligence either. And with LLMs, we can explain the trick: it’s numbers being multiplied and passed along.
The Metaphysical Elephant in the Room
I came up with this analogy a long time ago, and you may have even heard me tell it in person at the introductory lecture held by Professor Šnajder on AI, but what got me writing now is actually an article by my college colleague Karlo Vrančić: To understand AI is to anthropomorphize it, which I recommend you read.
While the article is rhetorically well-written, unfortunately it illustrates how most mainstream discussion about AI being intelligent is superficial, riddled with ad hoc persuasive definitions1, and most of all, never explicitly addresses the elephant in the room.
This topic is more than an intellectual quandary. What people believe about AI can have a big impact on their lives. We already see the trend of rising enrollment in Computer Science reversing at FER (University of Zagreb) in favor of Electrotechnics since 2023 and I consider that merely a precursor of what’s to come.
Furthermore, this debate about AI intelligence, believe it or not, isn’t really about AI. It’s about metaphysical assumptions. So in this article I won’t be diving in-depth into the mechanisms of AI and how it differs in many ways from a human brain, because that’s largely irrelevant. Instead, I will show a positive case for the correct view of human nature, intellect and AI.
Before that I must briefly address the convenient definitions Karlo uses in his article, like the essence of creativity being defined as, of all things, copying:
The essence of creativity is fittingly given in a quote by the great Pablo Picasso: “Good artists copy, great artists steal.”
…the core mechanism can be narrowed down to a form of context switching: essentially picking from everything you already know, giving some of it more weight than the rest, then coming up with something new.
Welp, I think I just described exactly how a neural network works.
Or the dubious definition of emotion:
An emotion is a state that changes your behavior without asking your permission. You cannot decide to stop being afraid, nor can you switch off the warmth of holding a child.
To put it precisely, emotions are some hidden signals that make the system lean toward some actions and away from others.
Welp, I just described how reinforcement learning works.
I have to say, by definitions this reductive and abstract, a kaleidoscope is creative and a thermostat has emotions.
But these are just symptoms. The biggest weakness in Karlo’s article is that he has a hidden assumption beneath all his arguments but never addresses it. So let’s get straight to the point: The possibility of AI intelligence presupposes materialism.2
To be fair to Karlo, he never says outright that materialism is true, or even that AI is intelligent. He leaves those questions open on purpose, but an argument can rest on a premise its author never states. His “welp” moments work only if the mind shares a purely material nature with a machine, implying that differences between humans, machines and rocks are matters of degree, not kind.
How, Not What
Back to magic. No matter how incredible the trick, a materialist will never concede it’s real magic. But not because he’s examined each trick — his worldview settled the question before the curtain went up. Fair enough; we all have such commitments. But when we talk about AI, the materialist suddenly stops asking how it happened and just cares about the what.
Karlo provides an example of what I mean:
“It’s just a probabilistic machine”
…However, when a machine does most of what we can do and plenty of what we can’t, the word “just” stops being useful…
I cannot prove that we are the same, so I think it’s easier to answer what we can do that AI absolutely can’t. The answers mostly fall into three piles: creativity, feelings and goals.
A materialist’s belief that machines can think wasn’t derived from watching ChatGPT build a website; it was settled the moment he decided the mind is just matter. The outputs don’t decide anything; the worldview decides what the outputs mean.
So yes, I’m prejudiced against magical computers. The materialist prejudges in their favor. The only question worth arguing is whose worldview is true.
But when we talk about intelligence as most people understand it, and as it is traditionally defined, the materialists perform a trick of their own, lexical gymnastics: machines are described in human terms (agents are “giddy,” “desperate,” “creative”), while human creativity gets called “weighting” and goals a “system prompt.”
Meet in the middle and the two look identical. But nothing has been discovered; only words have been changed. The honest materialist position would be to say humans don’t really understand anything either. Since almost nobody would accept that, it is far more effective to redefine intelligence until the machine qualifies.
Intelligent Rocks?
Let us examine the materialist claim more closely: “AI/LLMs are intelligent.”
What exactly is the referent here? The model weights? The ones and zeros stored in silicon? The powered-on machine with steel frame, cooling system and all? This is never stated precisely because once you think about it, the whole process of LLM inference can be done by hand and paper, or by a complex system of falling dominoes.
The referent is arbitrary because an artifact has no unity of its own. A tree or a dog is one thing by nature; a machine is many parts that count as one thing only because someone arranged them for a purpose. In Aristotelian terms, an artifact has only an accidental form, imposed from outside by someone capable of assigning purpose and meaning.
So, a couple of questions for the materialists: What do you mean by “AI” when you say it’s intelligent? Why isn’t a simple calculator intelligent also? What if I simulate the AI’s “reasoning” with water and sticks? Is that intelligent?
These are hard questions for the materialist, but not for a Thomist like myself. The calculator is not only not intelligent, it cannot even add. Why? When I add 2 and 2, what matters is not that I get 4, since the calculator gets 4 too; it is that I mean addition: a rule that covers every pair of numbers, including ones I will never compute.
Nothing physical can mean that. A calculator only ever produces finitely many outputs, and any finite set of outputs fits infinitely many different rules. Imagine a function that matches addition on every sum the calculator has ever done, but answers 5 for any numbers larger than those. Saul Kripke called it “quaddition.”3 Nothing in the circuitry decides whether the calculator is adding or quadding. Its voltage changes mean addition only because a person designed it for addition and reads its outputs that way.
The same goes for nature. If two sticks fall into the river and then two more, and they hit a rock and become a pile of four, the river didn’t do addition.
Similarly, an LLM models the structure of language because a person put language in, and it produces language out because a person mapped the number vectors to words. The LLM does not have any concept of what it’s producing.
A Brain Empty of Meaning
At this point a materialist will likely want to avoid these questions and ask: “But the human mind also is just calculations and electrical signals and matter in motion! What about the human mind is different from the AI?”
There are so many differences that the only trouble is where to begin. Let us start with meaning, since we just spoke of it.
Consider a book. A mathematics textbook contains many proofs, yet understands none of them. Suppose when asked a question the book had a mechanism to open the page with the answer. Suppose further it could rearrange the words on the page and construct new proofs — very impressive!
But do these mechanisms make the book understand its contents? Nonsense! And understanding is not some optional extra on top of intelligence; it is what the word means. The Latin intelligere simply means “to understand.”
Edward Feser puts the whole argument in a single sentence:
Thoughts and the like possess inherent meaning or intentionality; brain processes, like ink marks, sound waves, and the like, are utterly devoid of any inherent meaning or intentionality; so thoughts and the like cannot possibly be identified with brain processes.4
The materialist will say the brain is the exception: neurons are the one kind of matter that means something all by itself. But what would make them special? A neuron firing is an electrochemical event, no different in kind from the voltage in a calculator. If the calculator’s voltage means nothing until a mind interprets it, the same goes for the brain’s.
So who interprets the brain? If it’s another brain, that one needs an interpreter too, and so on forever. Somewhere the chain has to end in something that means things on its own, without having meaning assigned from outside. Matter can’t be that something, so the mind is not just more matter. And if the materialist denies that anything means things on its own, he has a bigger problem: then no statement means anything, including his.
Karlo cites studies showing that language models predict brain activity while people listen to sentences.5 But this is exactly what a Thomist would expect. St. Thomas taught that the intellect relies on the brain for the images it thinks with, as sight relies on colors to see. Modeling the brain’s activity is not modeling thought, any more than a weather simulation gets wet.6
But suppose we granted the materialist meaning. He would still have to explain reasoning. J.B.S. Haldane put the problem this way:
If my mental processes are determined wholly by the motions of atoms in my brain, I have no reason to suppose that my beliefs are true. They may be sound chemically, but that does not make them sound logically. And hence I have no reason for supposing my brain to be composed of atoms.7
In other words, under materialism a brain producing a conclusion and a shaken soda fizzing differ only in complexity, not in kind. Both are causes producing effects. Neither is a reason producing a conclusion.
If the materialist is right, his argument is just fizzing, and so is his conviction that he’s right.
From Representation to Understanding
Picture a triangle in your mind. The triangle in your mind is a particular triangle; it could be equilateral or scalene, maybe right-angled, maybe not. In any case, you did not picture triangularity itself, because there is no particular image you could form that represents all triangles. Yet you do understand what a triangle is. You judge whether a shape is triangular based on its correspondence to the concept of triangularity — that concept is called a universal. Each particular triangle has the nature that the universal names.
Descartes provides a similarly useful example with the chiliagon, a polygon with 1,000 sides. You cannot picture it in your mind, nor distinguish it from a myriagon (which has 10,000 sides), yet you understand the difference perfectly well. Therefore, understanding differs from representation.
Representation, mental or otherwise, does not produce understanding. A drawing of a triangle, the word “triangle,” a photograph of one, or weights in an LLM: each is a particular thing standing for something else, and none of them knows what it stands for. Even your own mental image is just a representation, which is why something in you must understand it. Having a representation is one thing; understanding it is another. The power that understands is what we call the intellect.
Now ask: what in the material world is universal? Nothing. Every material thing is particular: this neuron, this pattern of firing, this vector of numbers, here and now, with these specific features. Whatever is received in matter is received as a particular. So if the intellect were a material organ, it could hold only particular images, never the universal itself. Since it does hold the universal, the intellect is not material.
This is St. Thomas’ argument:
For it is clear that whatever is received into something is received according to the condition of the recipient. Now a thing is known in as far as its form is in the knower. But the intellectual soul knows a thing in its nature absolutely… For if the intellectual soul were composed of matter and form, the forms of things would be received into it as individuals, and so it would only know the individual: just as it happens with the sensitive powers which receive forms in a corporeal organ; since matter is the principle by which forms are individualized. It follows, therefore, that the intellectual soul, and every intellectual substance which has knowledge of forms absolutely, is exempt from composition of matter and form.8
The Light of the Sun
Karlo reached for Plato’s allegory of the cave9 in reference to the theory that we could be living in a simulation. Ironically, his thoughts remain in the physical cave, and the argument never ventures into the metaphysical sunlight. Plato himself explains the allegory:
…the prison-house is the world of sight, the light of the fire is the sun, and you will not misapprehend me if you interpret the journey upwards to be the ascent of the soul into the intellectual world according to my poor belief, which, at your desire, I have expressed whether rightly or wrongly God knows. But, whether true or false, my opinion is that in the world of knowledge the idea of good appears last of all, and is seen only with an effort; and, when seen, is also inferred to be the universal author of all things beautiful and right, parent of light and of the lord of light in this visible world, and the immediate source of reason and truth in the intellectual; and that this is the power upon which he who would act rationally, either in public or private life must have his eye fixed.
This ties in perfectly with the argument from universals we just covered. Man is not trapped in the cave, precisely because he can turn around and climb toward the light: the universals, and above them all the Good itself.10 The machine will never reach the sun by grasping at more and more shadows.
There is another interesting, almost prescient, correspondence between the prisoners and the machine. They hold a contest in which those are prized who best predict the movement of the shadows based on patterns already seen, and the man who returns from the light is ridiculed because he no longer sees in the dark. In the same manner, next-token prediction is lauded as intelligence by those whose eyes have wallowed in the dark of the sensible world their whole lives.
The Man in the Mirror
For a year I have been preaching how much AI agents resemble us, because I believe the quickest way to understand one is to hold it up against ourselves like a mirror.
Karlo says to use an AI like a mirror. I find this analogy intriguing. Why would looking at your face in the mirror tell you anything about the mirror? It’s ironic, because he does see a man in the mirror — or does he really? No; he sees a reflection, a representation.
The language model was built to reflect our thoughts. Its words are our words, the language is ours, the relationships between words model the data it was fed. The harness is code designed by humans, the tools also human, and its purpose — human. No wonder it resembles humans.
But the resemblance remains superficial; there is nothing behind the glass.
Follow what I write and do
Footnotes
-
“A ‘persuasive’ definition is one which gives a new conceptual meaning to a familiar word without substantially changing its emotive meaning, and which is used with the conscious or unconscious purpose of changing, by this means, the direction of people’s interests.” — Charles L. Stevenson, “Persuasive Definitions,” Mind 47 (1938). ↩
-
Strictly speaking, the premise is that the right arrangement of matter is sufficient for a mind. A few non-materialist views share it, such as panpsychism or David Chalmers’s property dualism, and the arguments below tell against them too. In practice, though, nearly every argument for AI intelligence is a materialist one. ↩
-
Saul Kripke, Wittgenstein on Rules and Private Language (Harvard University Press, 1982). Kripke himself did not conclude that thought is immaterial. That argument was developed by James Ross, “Immaterial Aspects of Thought,” The Journal of Philosophy 89 (1992), and defended by Edward Feser, “Kripke, Ross, and the Immaterial Aspects of Thought,” American Catholic Philosophical Quarterly 87 (2013). ↩
-
Edward Feser, “Some brief arguments for dualism,” Edward Feser (blog), 2008. ↩
-
Martin Schrimpf et al., “The neural architecture of language: Integrative modeling converges on predictive processing,” PNAS 118 (2021). ↩
-
St. Thomas Aquinas, Summa Theologiae I, q. 75, a. 2, ad 3 and I, q. 84, a. 7. ↩
-
J.B.S. Haldane, “When I Am Dead,” in Possible Worlds and Other Essays (London: Chatto & Windus, 1927), 209. Funnily enough, Haldane retracted this view (“I Repent an Error,” The Literary Guide, 1954) due to nothing other than the fact that computers could be built that process arguments validly. However, this falls under the same determinacy issue as calculators — a computer “processes” in the same way a calculator “adds,” by a mind supplying that interpretation of the process. ↩
-
St. Thomas Aquinas, Summa Theologiae I, q. 75, a. 5 and I, q. 76, a. 1; see also Summa Contra Gentiles II, 50. The thousand-sided figure is Descartes’ example, from his sixth Meditation. For a full modern treatment, see Edward Feser, Immortal Souls: A Treatise on Human Nature (Editiones Scholasticae, 2024). ↩
-
Plato, Republic, Book VII, 514a–520a. The translation quoted is Benjamin Jowett’s. ↩
-
For Plato the universal Forms exist outside the cave, which works fine as an analogy against materialism. The Aristotelian tradition, however, would say the analogy errs insofar as it separates the Forms from material reality: for Aristotle and St. Thomas, forms exist in things themselves, and the intellect abstracts them. ↩
Comments
No comments yet. Start the conversation.