MICHAEL SCHOFIELD

A new and bizarre substance has entered the world. And in using it here, I seem to have opened a portal to hell…

“Nightmarish visions of contorted faces, disembodied body-parts and bones swirls around aimlessly, morphing in and out of frames, and dissolving into a plethora of rotten webs and liquids. As this model struggles to interpret each previous generated moment, for the next one, it makes no discrimination between human beings, inanimate objects and technical features of the image itself. Everything has become part of the same endlessly-evolving sludge.”

This is vectorplasm. An entirely new form of decaying matter, that mutates and denatures everything that it transcribes. Is this rot some kind of new digital decomposition or a digital mould? Is this generation, regeneration or de-generation? Why does it seem somehow demonic? Is this all hyperbole? What are even we looking at here? These are all good questions, and I will attempt to answer them here.

“Today’s deep neural networks mark a rupture in the history of photography… A ghost haunts our AI-generated images… The ghost of photography.” (Elcott, 2024)

For this film, which is called Not A Soul, an AI model was trained on early 20^th century spirit and ectoplasmic photography, these are very old images from the archive that originally claimed to channel the undead. Much like AI they were fakes, but they echoed a spectrality that was real and intrinsic to what a photograph does. Roland Barthes called this ghostly fabric of photography “ectoplasm” (1993). But this new generative matter, that derives from it, behaves in a way that is much closer to a kind of haunted goo. While in the photograph, the recognisable trace of the past is mostly preserved, in generative AI it is totally melted down into something else. What we might find secretly disturbing, frightening and uncanny in this, is not the return of the dead, but the way it seems to echo its finality. The decay of bodies. The rotting and breaking-down of everything. The accelerated disintegration of information and culture and meaning.

This paper tries to break down this breaking-down. In order to understand this new form of decay we need to go back and compare it to others that lead to it, and see how it is both a continuation (and discontinuation) of previous media spectres, and how, like other forms of decay, it is, in fact, a vital prerequisite for the potential creation of the new.

Back in 2018 I was engaged in a heritage project called The Remote Viewer. I worked with special collections at Leeds University Library, who were busy trying to save and preserve a crumbling collection of Victorian photographs by a local industrialist called Godfrey Bingley.

The glass plates were badly damaged, falling apart as we tried to scan them for posterity. I was tasked with developing a new body of work from these digitised artefacts. Taking them back to where they were originally taken, presenting them again using the technique of rephotography. But the decay was so profound the views of the past were all but lost. Should we try and restore them somehow or preserve them in this damaged state? What does the visible decay even signify, if that is all we can now see?

The greater the decay of the photographs, the more their spectrality became visible and foregrounded. The materiality of the image’s decaying substrate revealed its referent as a ghost, before it too succumbed to time. Explaining and working with this became the subject of my PhD research. I became interested in hauntology and the spectrality of media. For Mark Fisher it was important to try and reveal the spectral time of recording devices, show the ghosts as ghosts, the decay as decay – not gloss-over, restore, air-brush out, or otherwise ignore how uncanny all recordings actually are.

“Spectrality is at work everywhere, and more than ever, in an original way, in the reproducible virtuality of photography or cinema”

Jaques Derrida, Paper Machine, 2005

“Modernity was built upon ‘technologies that made us all ghosts’, and postmodernity could be defined as the succumbing of historical time to the spectral time of recording devices.
Postmodernity screens out the spectrality, naturalising the uncanniness of the recording apparatuses.”

Mark Fisher, Phonograph Blues, 2006

Decay in this work symbolised an authenticity, its patina was the realness of the artefact. Its “aura” as a unique time-travelling object, not just representing something else, but a kind of lost memory made solid. I went back to Walter Benjamin for clues… if the decay is “aura”, what do we mean by “aura” in this context? For Benjmain the “aura” of original works formed “a strange tissue of space and time. The unique apparition of a distance, however near it may be” (Benjamin, 1935). I wrote in response, “one of the ways we can really feel that time distance, that intense temporal disjuncture, is through damage to the medium – auratic signs of the unique artefact that has travelled through large quantities of time” (Schofield, 2018). This is the decay that is often fetishised in hauntology’s obsession with obsolete media and formats. From aura as a marker of authenticity, to aura as memory loss, and then finally aura as nostalgic fantasy.  

“The obsolete or semi-obsolete delivery technology, such as the vinyl record, has now acquired an auratic dimension” (Roy, 2015)

Somehow, I persuaded the museum to let me play around with the decay itself, and I called the resulting images “auras”, after Benjamin. Details from the damaged dry plates that contained no historical value or useful information about the past whatsoever, but had become something new and interesting, a different thing, but no less haunting or saturated with the feeling of elapsed time and profound change. The hidden backdrop of photography itself had now become the new subject of the work.

Skip forward several years, and my new research area had become the relationship between photography and AI.^{[1]“AI is a new tool. It makes me think of the earliest forms of photography. So many of the first photographs aped the paintings of the time. Famously, [19th-century French painter] Paul Delaroche said, ‘From today, painting is dead,’ when he first saw a photograph around 1840. At the time, photographers were portraying people in togas and posing them with Grecian columns, copying allegorical paintings, making still lifes, and so on, because it was the only way they knew how to make art. Until artists like Man Ray and Alfred Stieglitz came along and created photography as we know it today. Right now, people are just starting to play with AI. Someone is going to be the Alfred Stieglitz or Man Ray of AI, but right now, they’re aping what we currently think of as art. There’s so much potential. Who knows what’s going to happen.” Marilyn Minter artsy} An artist called Tom Kennedy had seen my work in this field, and approached me with a project idea. He was influenced by an ancient oak tree that stood on his farm in Shropshire, UK. It was a tree that had stood for a 1000 years and had born witness to the very beginnings of the industrial revolution – the first mills and factories on Earth were less than a mile from where the tree stood. That was all gone now, of course, but the tree somehow remained. He wondered whether we could use photography and AI to try and imagine what the tree had seen or just evoke the vast amounts of time and change involved in its lost story. We took photographs of the ancient tree, and we took my rephotographs and my “auras” from The Remote Viewer, and fed them into a learning model, to see what would happen. The result was “/imagine Ronsack” (2024).

This was my first encounter with this new form of decay – this digital melting of the photographic trace, the hallucinatory disintegration of data. This decay was not seemingly auratic, it didn’t signify the authenticity of any artefact or show the marks and passage of any real time. It synthesised time, simulated it, to a degree, and then transmogrified it. Time had now become non-time. We were quite shocked by the results, but we decided to keep them, use them in the work. The way the model liquidised reality and time was mesmerizing and haunting… but were these images still ghosts in the way that photographs were?

In AI frame interpolation, missing time is synthesised using the previous frame. It needs a constant source of real reference frames to maintain its illusion of time and movement. We didn’t give it that. We didn’t have that to give it. We gave it what we had: confusing glimpses back in time, and then, we let its own generations become the reference frames. As the information subsequently decayed, the hallucinatory worlds it created are hard to explain. It does still feel haunted, but these aren’t the ghostly afterimages of real things or real people, like photographs, they are the broken dreams of the ghost in the machine. Uncharted voids in its probability space of the past.

Using some of these same techniques I went on to make a short science fiction film called The Jettison (2024). It was a kind of digital ghost story about grief and machines and the simulation of memories. It was during production of this film that I came up with the idea of Vectorplasm.

The film actually started off life as a remake of Chris Marker’s La Jetée (1962). But Vectorplasm’s melting backdrop and data decay wouldn’t let me faithfully recreate anything. Its memories were always slipping, turning into something else. Made of this same shifting spectral liquid, even my film’s main characters wouldn’t stay looking the same for very long. Rather than get frustrated with this glitch, I decided to work its uncanny amnesia into the new plot. In the story, machines try to bring the protagonist’s dead daughter back to life from memories and photographs, but her ghost keeps changing and slipping, gradually looking increasingly like him, she becomes a kind of projection of his inner turmoil. Eventually she melts back into the machine itself, as the simulated world it created for him, collapses entirely.

This idea was very much in-keeping with Marker’s own ideas on memory and technology. He was fascinated by the ways the latter transforms the former. As he said in Sans Soleil, “we don’t remember, we rewrite memory, much as history is rewritten” (1983). Vectorplasm doesn’t just rewrite memory, it liquifies it. As Shobeiri writes, “these ghostly figures encapsulate the entirety of the past into one haunting body” (2024) – all the past is still there somewhere, stored as cryptic vectors in the AI’s database. So why is it so slippery and mutating and weird? What is going on here? Why does it decay, what is the mechanism behind this new spectral process?

“Thanks to the virtualization of the photograph enabled by AI, we can now actualise a memory that has never existed factually… By feeding AI with existing photographs, one can materialise a memory that is simultaneously authentic and inauthentic, and thus both legitimate and illegitimate in its embodiment of the past… (Shobeiri, 2024)

Ragosti also notices a strange fluidic dimension to image generation, writing “imagine each image as a liquid, with its pixels acting as atoms in the liquid. Now, how do we replicate this diffusion process within these images? The solution lies in noise…” (2023). This is how diffusion works. First, it must systematically destroy the images in its database. It adds random noise to a series of related images until they disappear completely, and then, rather amazingly, it tries to reverse this process using just pure random noise as the new starting point. It then tries to look for things that aren’t there and tease them out. The model is literally hallucinating the images it produces. Using pareidolia, it sees patterns in the chaos and it interprets them, based on what it has learned from its training data. Yet more bizarrely, this same phenomenon is a common scientific explanation for ghost sightings, in human beings. So, this new form of decay leads to the creation of new ghosts, and it isn’t just an accident, some glitch in the machine, it is the very fabric of how all image generation works.

For Mark Fisher, all ghosts emerge from a form of decay – the decay of memory. If anything, generative AI is more spectral than the photograph, in this regard. Diffusion melts memories into noise and then tries to resurrect them. Haunting images emerge, but they can only be beguiling illusions of the past. Liquid re-imaginings. AI, it seems, has a profound memory disorder.

“Hauntology isn’t about hoky atmospherics or ‘spookiness’ but a technological uncanny” (Fisher 2006)

“Hauntology is exercised by the problem of memory and its imperfect recovery” (Fisher 2013)

“it is the very foregrounding of temporality that makes hauntology differ from the typical products of the nostalgia mode… The great sonic-theoretical contribution of The Caretaker to the discourse of hauntology was his understanding that the nostalgia mode has to do not with memories but with a memory disorder.” (Fisher 2018)

“AI-prompted memories are generated rather than retrieved. The result is the creation of a new kind of past that never really existed before… We can no longer believe our own eyes when confronted by that which seems strangely familiar yet unreal: a kind of uncanny memory.” (Hoskins, 2024)

For Gieben, “When we use the weird doppelganger of our artificial intelligences to create the imaginary spaces where we go to be entertained, we do not find ourselves in Baudrillard’s conception of a fantasy that structures hyperreality. Instead, we are in a liminal space, haunted by strange and unfamiliar ghosts” (2024). For Nora Khan, “AI images, with their often weird, goopy, and unsettling aspects, can be compelling in part because of what they don’t directly represent…these processes have, increasingly, emergent effects that can’t be predicted at all” (2024). It is these emergent effects that make AI potentially new and interesting as a medium. If all it could do was regurgitate the past, working flawlessly as the copying machine that it is so often accused of being, there would be nothing new to find here. But that hasn’t been my experience of using it, at all.

The images created in these projects have often been surprising and unsettling, occasionally not resembling anything I’ve ever seen before. The way AI uses stochastic processes, its unique and decaying form of memory, the way it liquidises images into these unstable forms, has some real creative potential – but so often the hallucinations and errors are patched and ironed out by the companies that make and market these models. The vectorplasm underlying image generation is subsequently hidden away, even as it forms the entire basis of the medium.

If we want to embrace what AI does that is genuinely new, we must also embrace the disturbing and decaying digital liquid that it moulds the new from. The last section of Not A Soul addresses this paradox directly. Emerging from these frightening and deathly fluids there appear to be new forms of life evolving. Decay is never just a destructive process; it releases the raw materials needed for the next generation of beings.