Image Capital

Typology

Title

Author

Text

IMAGING

Estelle Blaschke

One of the most important functions of photography is the visualization of objects and processes that cannot be perceived by the human eye: Chronophotographic experiments, for example, conducted by Eadweard Muybridge and Étienne-Jules Marey in the second half of the twentieth century, recorded the sequence of a movement; light-sensitive emulsions made it possible to trace radioactivity; X-ray photography screened internal body parts; and microscopic photography opened up new perspectives on nature and new ways of perceiving the world. As an imaging and recording instrument, photography played an essential role in the development of science, especially the natural sciences, and contributed to the concept of objectivity.

One of the most important functions of photography is the visualization of objects and processes that cannot be perceived by the human eye: Chronophotographic experiments, for example, conducted by Eadweard Muybridge and Étienne-Jules Marey in the second half of the nineteenth century, recorded the sequence of a movement; light-sensitive emulsions made it possible to trace radioactivity; X-ray photography screened internal body parts; and microscopic photography opened up new perspectives on nature and new ways of perceiving the world. As an imaging and recording instrument, photography played an essential role in the development of science, especially the natural sciences, and contributed to the concept of objectivity.

Photography was also used to visualize, document, and reflect work processes of all kinds, making them repeatable and measurable. This was crucial for the optimization of scientific, engineering, and industrial processes as well as in the development of educational material. In this context, photographs were always dependent on empirical data and written annotations to be considered reliable information or visual evidence. Photographs were one element in a larger scheme of knowledge production in which images became operational.

Beyond the epistemic value of a single image, photography, especially digital photography, is the medium of choice when lots of images, or visual big data, are needed to extract information and patterns—or to derive a “bigger picture,” as is the case with aerial photography and cartography or the recording and analysis of particle events in physics. Another, related use of photography’s imaging capabilities is the increasingly sophisticated and widespread practice of object-modeling and rendering, which aims to replicate cultural artifacts and the natural world with ultimate fidelity. These practices tie into the long-held dream of substituting physical objects with their visual representation and the idea of manipulating, comparing, and controlling these objects at the scale and format of a photographic print or computer screen. Architecture and design, the gaming industry, and e-commerce, have all operationalized photorealistic renderings in recent decades. Based on the whole gamut of high-tech scanning and imaging techniques and extensive, mostly commercial, 3D asset libraries, computer-aided renderings have become blueprints for the design and manufacture of physical objects. They are the building blocks of future physical and digital environments whose designers promote the transformation of the human experience.

Eine der wichtigsten Funktionen der Fotografie ist die Visualisierung von Objekten und Abläufen, die vom menschlichen Auge nicht wahrgenommen werden können. Die von Eadweard Muybridge und Étienne-Jules Marey in der zweiten Hälfte des 19. Jahrhunderts durchgeführten chronofotografischen Experimente registrierten beispielsweise die genauen Abläufe einer Bewegung, lichtempfindliche Emulsionen ermöglichten den Nachweis von Radioaktivität. Die Röntgenfotografie durchleuchtete das Körperinnere, und die mikroskopische Fotografie eröffnete neue Perspektiven auf die Natur und ermöglichte eine andere Wahrnehmungsweise der Welt. Als bildgebendes und aufzeichnendes Instrument spielte die Fotografie eine wesentliche Rolle bei der Entwicklung der Wissenschaft, vor allem der Naturwissenschaften, und trug zum Konzept der Objektivität bei.

Mit Hilfe der Fotografie wurden Arbeitsprozesse aller Art visualisiert, dokumentiert und reflektiert und damit wiederholbar und messbar gemacht. Dies war entscheidend für die Optimierung wissenschaftlicher, technischer oder industrieller Prozesse einschließlich der Entwicklung von Lehrmitteln. In solchen Gebrauchszusammenhängen waren Fotografien stets von empirischen Daten und schriftlichen Anmerkungen abhängig, um überhaupt als zuverlässige Informationen oder visuelle Beweise zu gelten. Fotografien waren nur ein, wenn auch wichtiges, Element in einem größeren Schema der Wissensproduktion, in dem Bilder operative Funktionen übernahmen.

Über den Erkenntniswert eines einzelnen Bildes hinaus ist die Fotografie, insbesondere die Digitalfotografie, das Medium der Wahl, wenn es darum geht viele Bilder oder visuelle Big Data zu generieren, die dazu dienen visuelle Informationen und Muster zu extrahieren und daraus ein „größeres Bild” abzuleiten. Dies ist beispielweise in der Luftbild- und Satellitenfotografie der Fall sowie in der Kartografie oder aber bei der Aufzeichnung und Analyse von Teilchenereignissen in der Physik. Eine weitere, damit zusammenhängende Nutzung der Bildgebungsfähigkeit der Fotografie sind die zunehmend ausgefeilten und weit verbreiteten Praktiken der Objektmodellierung und des Renderings, die darauf abzielen, kulturelle und natürliche Artefakte möglichst originalgetreu nachzubilden. Diese Praktiken knüpfen an dem lange gehegten Traum an, physische Objekte durch ihre visuelle Darstellung zu ersetzen. Fotografien sollten dazu dienen, diese Objekte im Maßstab und Format eines fotografischen Abzugs oder eines Computerbildschirms bearbeiten, vergleichen und kontrollieren zu können. In der Architektur und dem Produktdesign, der Gaming-Industrie und dem Online-Handel sind fotorealistische Renderings eine weitverbreitete Praxis. Basierend auf der gesamten Bandbreite von Scan- und Bildgebungsverfahren und umfangreichen, meist kommerziellen 3D Objektbibliotheken sind computergestützte Renderings zu den Blaupausen für die Gestaltung und die Herstellung physischer Objekte geworden. Sie sind die Bausteine zukünftiger digitaler Umgebungen, deren Gestalter:innen die Transformation der menschlichen Erfahrung vorantreiben.

Una delle più importanti funzioni della fotografia è la visualizzazione di oggetti e processi che non possono essere percepiti dall’occhio umano. Per esempio, gli esperimenti cronofotografici realizzati da Eadweard Muybridge e Étienne-Jules Marey nella seconda parte del XIX secolo hanno consentito di registrare le fasi successive di un movimento, le emulsioni fotosensibili hanno potuto rilevare tracce di radioattività, la radiografia ha permesso di scansionare le parti interne del corpo umano e la fotografia al microscopio ha aperto nuove prospettive allo studio della natura e cambiato il nostro modo di osservare il mondo. In quanto strumento di visualizzazione (imaging) e registrazione, la fotografia ha avuto un ruolo essenziale per lo sviluppo scientifico, particolarmente nell’ambito delle scienze naturali, e ha contribuito alla definizione del concetto stesso di obiettività.

La fotografia è stata utilizzata per visualizzare, documentare e investigare i processi di lavoro, rendendoli ripetibili e misurabili. Questo è stato cruciale per l’ottimizzazione di processi scientifici, ingegneristici e industriali e per il conseguente sviluppo di una serie di strumenti di formazione.

In questo contesto, le fotografie sono sempre dipese da dati empirici e informazioni scritte per potere essere considerate affidabili come prove visive. Le fotografie costituivano un singolo elemento all’interno di un più largo modello di produzione di sapere, dove le immagini acquisivano una funzione operativa. Al di là del valore epistemico di una singola immagine, la fotografia, in particolare quella digitale, è lo strumento privilegiato quando sono necessarie grandi quantità di immagini, o big data visivi, per ricavare informazioni e pattern visuali, o per ottenere un’“immagine più grande”, come accade con la fotografia aerea, la cartografia e la registrazione/ analisi degli eventi che coinvolgono le particelle in fisica. Un altro utilizzo delle capacità di visualizzazione della fotografia riguarda le sempre più diffuse e sofisticate pratiche di rendering e modellazione degli oggetti, sviluppate per replicare elementi naturali o artificiali con straordinaria fedeltà. Queste pratiche si riallacciano al sogno lungamente coltivato di sostituire gli oggetti fisici con la loro rappresentazione visiva e all’idea di poterli manipolare, confrontare e controllare come si fa con una stampa fotografica o davanti allo schermo di un computer. Negli ultimi decenni l’architettura, il design, l’industria dei videogiochi e l’e-commerce sono stati dominati dall’utilizzo di rappresentazioni digitali fotorealistiche. Basati su una vasta gamma di avanzate tecniche di scansione e visualizzazione e su estese librerie di modelli 3D (principalmente commerciali), i rendering digitali sono diventati la base di partenza per la progettazione e la produzione degli oggetti reali. Questi costituiscono i mattoni con cui vengono costruiti gli ambienti virtuali del futuro, i cui designer sostengono la trasformazione dell’esperienza umana.

La photographie permet de visualiser des objets et des mouvements qui ne peuvent être perçus par l’œil humain, permettant ainsi leur analyse. Les expériences de chronophotographie menées par Étienne-Jules Marey et Eadweard Muybridge dans la seconde moitié du 19e siècle, par exemple, ont permis de mettre en lumière la logique interne du mouvement humain et animal ; les émulsions photosensibles ont permis de tracer la radioactivité ; la photographie aux rayons X a permis d’examiner les parties internes du corps ; et la photographie microscopique a ouvert de nouvelles perspectives sur la nature et de nouvelles façons de percevoir le monde. La photographie a joué un rôle essentiel dans le développement de la science, en particulier des sciences naturelles, et a contribué de manière significative au concept d’objectivité.

La photographie a également été utilisée pour documenter de manière extrêmement détaillée et précise les méthodes de travail de toutes sortes, les rendant mesurables et reproductibles. Cela a été crucial pour le perfectionnement des méthodes scientifiques, d’ingénierie et industrielles. Dans ce contexte, les photographies n’étaient jamais de “simples” images, mais dépendaient de données empiriques et d’annotations écrites pour être considérées comme des informations fiables ou des preuves visuelles. Elles constituaient un élément clé dans un schéma plus large de production de connaissances. Malgré leur importance, ces photographies n’avaient aucune valeur matérielle. Elles étaient rarement conservées, et encore moins archivées, mais considérées comme un moyen de parvenir à une fin et jetées une fois leur objectif atteint.

Au-delà de cet usage, la photographie, en particulier la photographie numérique, est le support de choix lorsque de nombreuses images, ou big data visuels, sont nécessaires pour recueillir des informations et détecter des schémas visuels, ou pour établir une « vue d’ensemble ». C’est par exemple le cas de la photographie aérienne et de la cartographie, de l’enregistrement et de l’analyse des événements liés aux particules en physique, des examens médicaux ou de la reconstruction du patrimoine culturel à grande échelle.

Les pratiques de plus en plus sophistiquées et répandues de la photogrammétrie, qui permet de mesurer un objet dans l’espace à partir de plusieurs prises de vues, et de la modélisation d’objets constituent une autre utilisation liée aux capacités d’imagerie de la photographie. Chacune des photographies individuelles sont de minuscules composantes assemblées par un logiciel et transformées en de nouvelles formes, visant à créer des environnements artificiels ou à reproduire des artefacts culturels et le monde naturel avec une fidélité totale. Ces pratiques sont liées au vieux rêve de remplacer les objets physiques par leur représentation visuelle et à l’idée de manipuler, comparer et contrôler ces objets à l’échelle et au format d’une impression photographique ou d’un écran d’ordinateur. L’architecture et le design, l’industrie du jeu vidéo et le commerce électronique ont tous utilisé les rendus photoréalistes au cours des dernières décennies. S’appuyant sur toute une gamme de techniques de numérisation et d’imagerie de haute technologie et sur de vastes bibliothèques de ressources 3D, pour la plupart commerciales, les rendus informatiques sont devenus à leur tour des plans pour la conception et la fabrication d’objets physiques. Ils sont les piliers des futurs espaces physiques et numériques dont les concepteurs favorisent la transformation de l’expérience humaine.

Article

Video Games – (Photo)Realism – Photo(Realism)

Selim Krichane

The video game at the intersection of different media

Video games have become a major cultural industry, playing a key role in the circulation of images and the configuration of our collective imagination. Backed by a cultural and economic history that spans more than fifty years, video games have always had an intense relationship with all the other audiovisual media around them. When the first home consoles appeared in 1972, video games relied on TV cathode ray tubes to arrive in people’s living rooms and produce images, as with the Magnavox Odyssey designed by television engineer Ralph H. Baer (1972).

The cinematic/photographic image soon came to embody a model for the nascent video game industry. This was first of all because cinema and the images it produced constituted an invaluable cultural and symbolic point of reference that video game creators and manufacturers could rely on as a lodestar to guide the public’s reception of the games as a new cultural practice. Another reason was that the graphics capabilities of the first generations of video games in some cases meant that decoding and reading them required an external model to refer to. For instance, the first home consoles mostly adapted sports games (tennis, hockey, ping-pong) which typically aired on the same television screens. Adaptations of Hollywood films also appeared over the course of the 1970s – for example, the arcade game Shark Jaws marketed by Atari in 1975: although the company did not own the rights, it still went ahead and adapted the film Jaws (Steven Spielberg), which had its cinema release the same year.

Today, the ‘realism’ – or ‘photorealism’ – of video game images has become a truism in the discourses of promotion and reception surrounding the medium. We are accustomed to hearing that video games are now increasingly realistic, that their incredible precision puts them on a par with cinematic and photographic images and that recent developments in virtual reality mean that they can even compete with the pictures produced by our sense organs. Popular discourse seems to suggest that video games are destined to reproduce a certain way of seeing reality, with the cinematic/photographic image representing an essential point of reference.

In the context of this article, we intend to locate this idea, as well as the relationship between video games and (photo)realism, in a cultural history of the medium, involving, as a key component, the discursive strategies of promotion and reception that have gone hand in hand with it. In this way, we will see that although photorealism is one of the main visualisation methods in the history of video games, in reality it is just one of a variety of options within a wide array of aesthetic possibilities in the area of video games. Even within the prevailing mode of production, photorealism is no longer the one and only means of representation for a field of cultural production that has diversified a great deal over the last twenty years.

Photography and video games

Swiss photographer and film-maker Pascal Greco was due to go to Iceland in April 2020 to proceed with his photographic project documenting the island’s architecture. The first lockdown in March 2020 disrupted his travel plans. So Greco decided to buy a PlayStation 4 console (Sony) to help him through the first lockdown. It was then that he discovered the game Death Stranding by game designer Hideo Kojima. Death Stranding is a game of exploration and adventure that invites players to take on the role of a delivery man – Sam Porter Bridges, played by American actor Norman Reedus – in a post-apocalyptic world in which humans have taken refuge in military bases or insecure villages, while wraithlike creatures haunt the territory. Impressed by the realism of the depictions and the way the game’s fictional world approximated the natural landscapes of Iceland, Greco decided to try his hand at in-game photography. Realised in 2020, his work gave rise to a series of photographs – titled Place(s) and published in book form in 2021 – and a series of prints (10 × 10 cm). The resulting images of rivers and mountains, waterfalls and rock formations generated a peculiar feeling of realism. The photographer rightly points out that he wanted ‘the people who view these pictures not to have the impression that they come from a virtual world, but that they look real.’[1]

More broadly speaking, Greco’s approach is part of a practice of virtual photography manifesting as the kind of in-game photography that has developed over the last ten years, at the intersection of established artistic practices and gaming practices.[2]

Using a Polaroid camera to take pictures of a screen playing images from a video game may be a surprising development. However, adopting an approach of this kind is a response to certain social, technological and cultural developments that are bound up with the history of video games, as we shall see.

Firstly, such a practice is predicated on the pursuit of photorealism as a means to create the visual environment for video games. In its use of photographic textures, light simulation tools and 3D modelling, Death Stranding responds to the ‘cinematic urge’ that marks all of Kojima’s productions. Accordingly, Greco’s work is aligned with the same photographic ambition that is evident in the game, ‘bringing things full circle’, in a manner of speaking. Secondly, this photographic project is based on the presence of a ‘photo mode’ within the game, which allows the player to make their avatar disappear along with the interface elements (icons, gauges, mini-map, etc.) to simulate a situation in which a photograph is being taken. The process of integrating and simulating the photographic act in video games started in the late 1990s in games like Pokemon Snap (Nintendo, 1999) and Beyond Good and Evil (Ubisoft, 2003). In these examples, the option of taking photographs is a key mechanism in the game that replaces, in a sense, the action of aiming and shooting at in-game opponents. During the 2010s, when game consoles also became platforms for socialising and sharing content, the ‘photo mode’ spread within mainstream production to become a feature appreciated by a section of the gaming community.

The act of capturing an image during a gaming session, like a screenshot, feeds the processes of sharing and circulating content created by the communities of practice. This is a constituent element of contemporary media usages, which are characterised by a convergence of technologies and participative approaches.[3]

Given that Greco was able to pursue a photographic practice like this in 2020 and that such a practice is actually central to the regimes of sociability in video gaming, where the practice of taking and circulating gaming images has become standard, what are the historical factors that have led to these kinds of developments?

A historical perspective: game video, cinema, photo

The technological attraction of video games and the devices used to play them has been a framework for the medium’s promotion and reception since the 1970s. A frame of reference of this kind – one that showcases the graphics capabilities of consoles, the power of their processors and the quality of their graphics cards – represents a constant feature in the discourse of ‘techno-industrial glorification’, which has become normalised within the industry’s promotional discourse, fed by the main economic actors in the sector.[4] This discourse has a particular focus on the ‘realism’ of representations. We can find an example of this in the RealSports series of games launched by Atari in 1982. Although the American company’s first home console – the Atari VCS (or Atari 2600 in Europe) – had limited capabilities in terms of graphics, the promotional language used for these games highlighted their ‘realism’ both in terms of what they depicted and their simulation of the complexity of the sports in question. A 1982 advertisement, complete with photos, reads, ‘These pictures show you how incredibly life-like new ATARI RealSports video games look.’[5] Looking at the pixelated images of the 1982 RealSports Baseball game referred to in the ad, we can see the extent to which the issue of ‘realism’ is historically and discursively located. That said, this promotional strategy was picked up by journalists in the specialist press and by communities of practice during the 1980s. The issue of ‘realism’ would become even more important from the late 1980s on, owing to the technological and aesthetic convergences that would affect the video game and film industries.

The late 1980s were a watershed period for video game images as 3D graphics became increasingly widespread, recapitulating cinema’s linear perspective. This fragmented process was a feature of mainstream video game production throughout the 1990s. It depended in part on technical developments, such as enhanced computing power, increased storage space with the advent of optical media, and the development of video compression standards like MPEG-1. To start with, these modes of visualisation affected flight simulators and sports simulations in the late 1980s, before influencing all mainstream production.

The presence of video sequences, which might be pre-calculated 3D models (‘CGI’) or digitised live-action sequences (‘FMV’, which stands for ‘Full Motion Video’), became widespread in video games and was a new locus of technological attraction and a much-vaunted ‘photorealism’, which was focused on in promotional discourses. This period also witnessed the emergence of a new genre, the ‘interactive film’, a hybrid of the video game and cinema which made extensive use of live action material, as exemplified by Night Trap (Digital Pictures, 1993) and The 7th Guest (Tribolyte, 1993). This period of strong convergence between audiovisual media saw the spread of ‘remediation’ processes, illustrated by games like Myst, which, as David Bolter and Richard Grusin put it, combined ‘three-dimensional, static graphics with text, digital video, and sound to refashion illusionistic painting, film, and, somewhat surprisingly, the book as well’.[6]

The aesthetic codes of video games were then shaken up, particularly with the routine appearance of non-interactive sequences (‘cut-scenes’ or ‘trailers’) to punctuate the game activity, and with the spread of the ‘virtual camera’ as a point of view – sometimes controlled by the gamer – directed at the diegetic space of games modelled in 3D, which often made use of photographic textures. The commercial success of Sony’s PlayStation console, which came out in 1995, and Nintendo’s transition to 3D in 1996 (with the release of the Nintendo 64 console), completed this transition to three-dimensional graphics. This was coupled with the solid integration, in the discourse and frames of intelligibility associated with the medium, of evaluation criteria based on the ‘photorealism’ of the images.

From photorealism to diversified modes of representation

Historically speaking, the interactions and crossovers between the video game industry and the film industry, supported by television and TV depictions, helped make ‘photorealism’ a criterion that was used to assess and promote video game images. The increased interaction between these areas in the 1990s, in technical, economic and cultural terms, was of crucial importance in the aesthetic history of the medium. During the 2000s, this quest for photorealism continued with successive Sony console releases, PlayStation 2 (2000) and PlayStation 3 (2006–7): this was spurred on by the arrival of Microsoft on the home console market (Xbox consoles, 2001). Photorealism thus retained its place as a cultural and aesthetic marker, a means to evaluate (and promote!) the technical capabilities of computer hardware that is still used today.

However, the development of games for mobile phones from 2010 on and the emergence of an independent video-game production scene had a major impact, shaking up the norms of game imagery and leading to their diversification.

The success of mobile games was presaged by the fad for the Nokia phone Snake game that started in 1997 and reinforced when apps appeared that could be downloaded to so-called ‘smart’ phones: this success radically changed the way video games were produced and distributed – witness the examples of Angry Birds (2009) and Candy Crush (2012). Over the last ten years, the limited technical capabilities of telephones, as compared with home consoles or computers, have led economic players to focus on other visual modes in games and other ways of marketing them. Here, photorealism is not required; instead, the prevalent form is abstract and stripped-down graphics, coupled with the remediation of the norms of representation used in comics and cartoons. This evolution in game platforms and practices has gone hand in hand with a reorganisation of the ways games are created and distributed. The advent of online distribution platforms like Steam (Valve, 2003) and the PlayStation Store (2006), which were controlled by the economic heavyweights in the sector, also led to the emergence of an independent production scene consisting of small development studios with limited financial resources at their disposal.

Enabled by the Internet, the electronic sale and distribution of games now easily predominate, facilitating the participation of these minor economic players. The independent scene, which profits from and feeds the process that gives cultural legitimacy to the medium, has also had an impact on the way video games are displayed. In many cases, ‘Indie Games’ – to give them their proper name – opt for pared-down graphics and styles that fall within ‘Pixel Art’, i.e. the 2D representation of the graphic elements of the game world using rasters of visible pixels, in reference to the game aesthetics of the 1980s. With limited means and teams of fewer than ten people, it is difficult to compete on the terrain of hyperrealism and emulate the Assassin’s Creed series (Ubisoft, 2007) or Horizon (Guerilla Games, 2017).

Consequently, video games now bring together a huge range of objects, cultural practices and technical platforms. While photorealism is still a dominant mode of visualisation in video games, many producers opt for alternative modes, such as the cartoon quality of Nintendo productions (Mario Odyssey, Switch, 2017), the abstraction of certain games of skill or puzzle games (Super Hexagon, Terry Cavanagh, 2012; Glitchspace, Space Budgie, 2016) or pixelated graphics (Celeste, Maddy Thorson & Noel Berry, 2018).

Video game images

After this brief historical overview, let us return to the contemporary practice of in-game photography. It is striking to note that the artists practising it today, such as Duncan Harris, Leonardo Sang or Pascal Greco, mostly opt for games with a strong sense of photorealism (Fallout 4, 2015, Control, 2017, Ghost of Tsushima, 2020). Besides having a ‘photo mode’ that makes it much easier to capture images, these games also display a density and visual richness that explains the remediation of the photographic act and permits a multitude of choices to be made in selecting the subject, frame, lighting, etc.

The commercial history of the video game now spans half a century. Against this background, the practice of in-game photography indicates the cultural and social importance of the medium. Through it, the captured image – based on the historical model of photography – facilitates interactions and allows memories to be concretised, while making it possible for traces of the game activity to circulate on digital networks, benefiting the communities of practice and the economic players in the industry.

In-game photography also illustrates the complexity of the interactions and influences at work in the present-day world of audiovisual media. Given its symbolic value, the photographic imprint has played a critical role as a model for how video games have been displayed since the 1980s. For the last ten years, photographic practice has been re-establishing a connection with the video game, but this time it has captured it from the outside, as the very subject of its recording, taking advantage of the density and richness of its images, which the games owe, in part, to their distant cousin: photography.

[1] Pascal Greco, Place(s), Geneva: Chambre Noire (2021), n.p.

[2] Nathalie Dassa, The Imaginary Worlds of In-Game Photography, in: Blind Magazine, 5 January 2022, https://www.blind-magazine.com/stories/the-imaginary-worlds-of-in-game-photography/.

[3] Henry Jenkins, Convergence Culture: Where Old and New Media Collide, New York: New York University Press (2006).

[4] Martin Picard and Carl Therrien, ‘Techno-industrial Celebration, Misinformation Echo Chambers, and the Distortion Cycle: An Introduction to the History of Games International Conference Proceedings’, in: Kinephanos: Journal of Media Studies and Popular Culture (January 2014), http://www.kinephanos.ca.

[5] ATARI RealSports advertisement, accessible at www.atarimania.com.

[6] Jay David Bolter and Richard Grusin, Remediation: Understanding Media, Cambridge, MA: MIT Press (1999), p. 94.

Article

Modelling Specific Realities: On the Implications of Simulation Technologies

Raymundo Stadelmann

A virtual drive through the desert landscape in an armoured vehicle reveals a cartographically calibrated model world. An instructor introduces obstacles, weapons and enemy figures on the screen and supplies them with different equipment with a click of the mouse. Harun Farocki’s multichannel installations Ernste Spiele I–IV (Serious Games I–IV) (2009–10) show the training that US soldiers received before being deployed in the Iraq War and scenes from their subsequent treatment for PTSD. Farocki intersperses the material with white titles inserted on a black background that refer to the images that come before and after. ‘The follow-up images – / resemble those that prepare for war.’ The intertitles refer not just to the virtual simulations but also to the real world of military exercises. For the virtual model world that is presented reconstructs memories of the war while also designing the real world after the war.^[1]

^{_{Still from: Harun Farocki, “Serious Games IV: A Sun without a Shadow” (2010), Harun Farocki GbR.}}

Farocki’s footage from various US military facilities shows young soldiers being trained with computer software. The Virtual Iraq program was used to prepare soldiers for war, to rehearse military scenarios and to recreate them after the event.

Shots of a MOUT (Military Operations in Urban Terrain) site takes the idea of ‘indistinguishability’ to the extreme. Not only is it almost impossible to tell the constructed training facilities apart from Virtual Iraq but we can also see how the real world strives to emulate the simulation.

_{^{Esko Townell, “A view of, from left, an office building, homes and shops in the combat village at the Military Operations in Urban Terrain (MOUT) CollectiveTraining Facility”, Photograph of a MOUT site from 12.9.1990 near North Carolina (Camp Lejeune), The U.S. National Archive, signature: 646395.}}

These feedback loops – a focus of Farocki’s study – can be seen as an extension of something he has termed ‘operational images’. These are images that are not, in the first instance, intended for contemplation but are instead crucial to the processing of information within technical (for example, military or medical) visual operations. They increasingly redefine what an image is in a society that is now permeated by visual media. Farocki’s research confirms the fact that images no longer seek to represent the world but rather actively intervene in it. The image is no longer a reproduction but has become an autonomous reality, an intervention in visual worlds that are constantly being refreshed. This leads to a reversal of the logical and temporal relationship between image and reality. Images are now their own models, uncoupled from reality – an anti-mimetic condition that Jean Baudrillard called ‘the divine irreference of images’.^[2] But how does Baudrillard arrive at the radical conclusion that it is ‘the reference principle of images which must be doubted, this strategy by means of which they always appear to refer to a real world, to real objects, and to reproduce something which is logically and chronologically anterior to themselves’?

Simulation or Virtualisation?

The tragic fire in Paris’s Notre Dame cathedral is a pertinent example of how the virtualisation of a cultural building with – more than just – national significance can enter reality through the back door. French President Emmanuel Macron’s emotionally charged tweet (‘I am sad to see this part of us burn tonight’^[3]) helped galvanise a sense of identity: it was followed by various pledges of aid and donations from billionaire companies like L’Oréal and Louis Vuitton (LVMH), which immediately triggered a debate in France about donations in general. In the public discourse, the offering of the French video game company Ubisoft got drowned out by the debate over the most peculiar restoration proposals and the criticism of PR campaigns worth millions in the form of donations. For a short time, the video game developer and publisher augmented its financial contribution to the restoration of Notre Dame by making its virtual models of the building – which are the setting for the video game Assassin’s Creed Unity (2014) – available for free download. The accuracy with which Ubisoft brings historical scenes back to life is well-known among gaming fans and can be characterised very much as a love of detail. There has been some speculation about whether the video game producer might be of assistance in the reconstruction process in another capacity: by making the data compiled for the construction of the virtual model available for use in reconstructing the original building. Responding to an enquiry from the Guardian newspaper, a Ubisoft spokesperson reported that this was not currently the case, ‘but we would be more than happy to lend our expertise in any way that we can to help with these efforts’.^[4]

_{^{Notre-Dame in Assassin’s Creed Unity (2014), Ubisoft}}.

Ubisoft’s model was recreated by senior level artist Caroline Miousse using various documents, photographs, scans and drawings. She spent fourteen months reconstructing the cathedral, which had to be transposed to the condition it was in, in around 1790; this applied not only to the architecture but also to the paintings, the furnishings, the fabrics and banners. A reconstruction of this kind – which turns out to be a stumbling block, even for art historians – would seem to be an insurmountable challenge for a level developer who had never visited the cathedral in person prior to the game’s release and impossible to accomplish flawlessly.^[5] For Ubisoft, the idea was for the world-famous cathedral to establish itself as a new benchmark for photorealistic video game worlds. As if this were not a big enough challenge, Miousse had to take a number of liberties in order to make Notre Dame ‘playable’, which meant creating new routes and scaling or shifting certain objects. In addition to this, some of the works of art in the cathedral are protected by copyright – including the monumental stained-glass windows: this meant the developers got ‘to put a little bit of [their] own identity on an iconic monument’.^[6] Evidently the producers did not seek to scientifically reconstruct the cathedral in the game: this was not their ambition, nor was it within their power to do so. ‘Since I had never been formally taught, I had to figure things out on my own to recreate Notre Dame’s structure in the most realistic way possible.’^[7]

It is hard to imagine what a sacred building bearing the Ubisoft developers’ ‘own identity’ might have looked like, and what identity (‘this part of us’) the reconstruction would have espoused had a commercial video game company played a major role in rebuilding the cathedral. But more interesting than speculations about how the architecture might have been realised is the fact that these virtual models can be consulted at any time and are only one decision away from being turned into reality. This brings up questions about the status of these kinds of models.

In game studies there has been lively discussion about how virtual worlds should be characterised and what we should expect of them. The word ‘simulation’ should be treated with particular caution, because ‘simulations cannot stand alone, for their conceptual status is dependent on the something of which they are imitations’.^[8] Veli-Matti Karhulahti’s suggestion is to use the term virtualisation for computer-generated game worlds, as it is not based on an empirical reference system (real – virtual). In light of the example above, however, I would like to explore the question of what status the virtual model takes on when it goes a step further and serves as a model for the real. For if we remain with the computer game industry, while abandoning the ludo-scientific perspective, it soon becomes apparent that virtualisations are not in any sense models in an insulated virtual environment. What status would the real, reconstructed Notre Dame have if it had been built by Ubisoft on the basis of a virtualisation? What if the game (or the virtual environment) is not based on empirically compiled data (virtualisation), and reality is modified as per the virtualisation? Do reality and simulation then change places in terms of their conceptual status?

Unreal Engine – ‘Let’s get serious’

For the video game community and the commercial music landscape one of the highlights of the nearly two-year period during which major events were disrupted by the pandemic was Travis Scott’s live performance on the virtual stage of the popular video game Fortnite (2017). Over 12.3 million people congregated for this historic moment, coming together to watch the world’s largest live music performance to date on the video game platform created by US software company Epic Games. The company, which started doing business in 1991, used virtual galaxies and underwater landscapes to transform the setting of the shooter-based online game into a monumentally orchestrated attraction with seductive allure. There is, first up, no evident connection between the shooter game and the US rapper, but it is clear that their commercial tie-up calls into question previous conventions (be it the idea of the stage or a hermetic computer game community) and generate new spaces for themselves. And like any space that relies on infrastructure, the virtual space invariably has a political dimension.

One of the engine’s customers and users is Cubic Global Defense, a subsidiary of Cubic Corporation. The company, which specialises in virtual combat simulators and training systems, supplies technical support to the US military and allied armies. This includes using the Unreal Engine 4 to help develop systems and services for ‘performance-based training, scalable training architectures, air combat training, and high-fidelity combat training’.^[10] For example, the SWOS (Surface Warfare Officers School) crew is not trained on board but directly on a virtual clone of the ship. Besides being economically expedient, virtual training also has the advantage of using media technologies that the crew are already familiar with anyway.

^{_{Young sailors play Call of Duty at the SWOS training centre. From: “SWOSVirtual Reality Training: LCS Engineering PlantTechnicians”, 27.06.2017, https://www.youtube.com/watch?v=k-aT90ehH8Y(as of 27.06.2023), TC: 2:03.}}

‘Younger sailors today, as they come on board, you know, they’re so familiar with video games that it helps them so much when they come into a classroom environment where they can go through a virtual reality which is similar to video games and they remember a lot more of what they do through the virtual reality than the normal legacy teaching that we used to do.’^[11]

Although there may be a number of perfectly obvious reasons – including risk mitigation and economic or ecological concerns – to explain the general prioritisation and shift in the direction of visual media technologies, they still require a new balance to be struck. If the process of training a ship’s crew is now a purely virtual affair, reality has become the exception. When reality is supposed to operate as much as possible like the simulation, convergence strategies must ultimately be applied to bring it more and more into line with the simulation. This involves minimising the factors that generate too many discrepancies. In an ideal situation, there would be no distinction between simulation and reality. This is not a new issue that is only now being flagged up by the entertainment industry – it has long been known about in business informatics. For one thing, there is a considerable cost advantage to using simulation technologies. ‘At the same time, it is hard to ensure the necessary isomorphism between the system and the model to be created that will yield valid knowledge about the real system.’^[12] Because the simulation models can only ever be as good as the operating system and the availability of data will allow (and cannot permit any significant variances as a result), the – disruptive – factors are reduced to a level that can be computed by the system. I have tried to trace how these kinds of convergence strategies occur outside the realm of business informatics and to map the effects they can have on the aesthetics of the media. The final step is to demonstrate why the study of simulation technologies is particularly fruitful from the point of view of photography or, more precisely, image theory and thus align myself with the tradition of Farocki and Vilém Flusser, who have never underestimated the interventions of nondescript image operations within complex systems relating to the technology and ecology of media.

‘Photography, the mother of all things’

I t may be that in post-history, photography (and the technical images that are heir to it) performs the same function in society that war did: that of rupturing processes. In the dawning age of virtual spaces, it is possible that photography and not war is at the origin of all things.^[13]

When Flusser talks about photography (and the technical images that are heir to it), he goes far beyond the supposedly representative nature of the visual medium. He introduces the idea of the techno-imagination, which he regards as distinct from the imagination of traditional images: he uses it to show that reality itself is a computation – with photography the first ‘step toward the synthetic production of objects from grains’ – while the ‘distinction between image and object, between fiction and reality becomes increasingly non-operational’.^[14] Flusser describes photography as a force that is at once creative and destructive, making fiction and reality indistinguishable in history and leading to the ‘unstoppable disintegration of the world of objects and its subject’.^[15] While propositions of this kind are generally located in discourses around digitisation – and the accompanying displacement of concrete lifeworlds into digital environments – this essay is designed to examine, à la Flusser, how current imaging processes are transforming our present. However, the fact that today’s radical imaging processes are worlds away from the historical narrative of photography does not make them any less a subject of photographic theory. For just as Flusser argued the case for a philosophy of photography to study the overlap between fiction and reality, Joanna Zylinska makes a stand for an expanded concept of photography when she describes it as a ‘technology of life’.^[16] Photography does not just represent life, it actively moulds, regulates and documents our everyday world: ‘Photography has become pervasive and ubiquitous: we could go so far as to say that our very sense of existence is now shaped by it.’^[17] This approach is also associated with the critique stating that prior discourses in the humanities on the subject of photography are insufficient if we want to get a proper understanding of photography – and thus of image operations too.

In line with this, my proposal is to include simulations in the ‘historical totality of photographic forms’.^[18] The subject of this work, then, was visual products that lay claim to reality but are not necessarily based on the processes of photographic optics. For a simulation is not meant to record a moment of reality – although it is questionable whether photography has ever done that – but rather serves to create a model, a ‘pre-imaging’, of a future reality (whether or not it actually occurs) from real data: it is the emulation, the ‘post-imaging’, of real scenarios for a variety of purposes. Accordingly, this essay sets out to delineate various interactions between reality and virtuality by presenting current simulation technologies and theories. Simulation is a particularly good locus for reflecting on this apparent dichotomy, as inevitably it is itself an object of both poles: it uses data – that has been harvested from the empirical world – to create new scenarios, which can be described first up as visualisations. However, if the data were not obtained from empirical reality, the simulation would be nothing more than a fiction. The reverse is also true: if the simulation is not embedded in the virtual, it is already reality.

It is still important to consider whether the ‘dawning age of virtual spaces’ has already arrived and ‘photography and not war is at the origin of all things’.^[19] The discovery that technological advances in the realm of ‘virtual spaces’ benefit from the dialogue and close collaboration between the military machine and simulation developers is indicative of the proximity of the two phenomena (war and photography). We have focused on simulation technologies because, unlike other imaging processes, they make no pretence of capturing (after-)images of familiar realities; rather, they are used to generate (pre-)images of specific realities. Works like those of Harun Farocki or the cutting-edge technologies of Epic Games show that model worlds situated in a purely virtual realm do indeed extend beyond the designated programs. However, the theoretical implication of this is that the status of what is simulative and what is real is being destabilised. A thought experiment of this kind has been carried out in respect of Notre Dame. If we downgrade simulations or virtualisations as representations of objects from the real world, we underestimate the power they have to affect us. But if we view these imaging processes as interventions in real imagery and the real world we live in, it is possible to map out their aesthetic effects as media phenomena – as Farocki’s work serves to remind us. The final step in fully appreciating these constructed image worlds involves examining strategies for bringing the concrete world into convergence with simulated reality and looking at where such practices occur. In the future, will MOUTs or SWOS fleets be built to align with the capacities of simulation programmes to allow performance-based training to be optimised? Or, to put it another way, at what point will it become easier, economically and environmentally speaking, to realise the simulation instead of reconstructing the reality? These are questions that can perhaps only be answered when the technological advances have manifested and would seem to remain, therefore, in the realm of speculation. However, these new worlds of images, along with the new lifeworlds they give rise to, also create a new relationship between image and reality. In this respect, we are also faced with the emancipation of the image from any referential constraints, as we move in a direction that Flusser astutely foresaw when he maintained that photography and not war seems to be at the origin of all things.

^[1] For a more in-depth analysis of Farocki’s work, see Christa Blümlinger, ‘War as a Challenge to Digital Realism’, in: Harun Farocki: Soft Montages, ed. Yilmaz Dziewior, Bregenz: Kunsthaus Bregenz (2011), pp. 34–47.

^[2] Jean Baudrillard, ‘Simulacra and Simulations’ in: Selected Writings, ed. Mark Poster, trans. Paul Foss, Paul Patton and Philip Beitchman, Stanford, CA: Stanford University Press (2001), p. 170.

^[3] Tweet from 15 April 2019, https://twitter.com/emmanuelmacron/status/1117889484358426625?lang=en (accessed 15 June 2022).

^[4] From the article by Keza MacDonald, ‘Assassin’s Creed creators pledge €500,000 to Notre Dame’, in: The Guardian, 17 April 2019, https://www.theguardian.com/games/2019/apr/17/assassins-creed-creators-pledge-500000-notre-dame-restoration.

^[5] Interview with Caroline Miousse, conducted by Tara Long, https://www.dailymotion.com/video/x280dne (accessed 15 June 2022).

^[6] Brett Makedonski, ‘One Dev Spent Two Years Making the Notre Dame in Assassin’s Creed Unity’, in: Destructoid, 16 April 2019, https://www.destructoid.com/one-dev-spent-two-years-making-the-notre-dame-in-assassins-creed-unity/.

^[7] From the article by Keza MacDonald, ‘Assassin’s Creed creators pledge €500,000 to Notre Dame’, in: The Guardian, 17 April 2019, https://www.theguardian.com/games/2019/apr/17/assassins-creed-creators-pledge-500000-notre-dame-restoration.

^[8] Veli-Matti Karhulahti, ‘Do Computer Games Simulate, After All? Reconsidering Virtuality’, The Philosophy of Computer Games Conference, Istanbul (2014), n.p.

^[9] ‘Let’s Get Serious’, Unreal Engine, https://www.unrealengine.com/en-US/solutions/training-simulation (accessed 5 March 2020).

^[10] ‘Training’, Cubic, https://www.cubic.com/solutions/training (accessed 15 June 2022).

^[11] Rasheed Brown, in ‘SWOS Virtual Reality Training: LCS Engineering Plant Technicians’, Surface Warfare Officers School, 27 June 2017, YouTube video, 2:04–2:21 https://www.youtube.com/watch?v=k-aT90ehH8Y.

^[12] Wilhelm Dangelmaier and Christoph Laroque, ‘Simulation’, Beiträge von A–Z, Enzyklopädie der Wirtschaftsinformatik, 8 April 2019, https://wi-lex.de/index.php/lexikon/technologische-und-methodische-grundlagen/simulation/.

^[13] Vilém Flusser,‘Fotografie, die Mutter aller Dinge’, in: Fotogeschichte: Beiträge zur Geschichte und Ästhetik der Fotografie 12/43 (1992), pp. 3–4, here: p. 4.

^[14] Vilém Flusser, Vom Subjekt zum Projekt: Menschwerdung, Frankfurt am Main: Fischer (1998), p. 19.

^[15] Vilém Flusser, Vom Subjekt zum Projekt: Menschwerdung, Frankfurt am Main: Fischer (1998), p. 21.

^[16] Joanna Zylinska, Nonhuman Photography, Cambridge, MA: The MIT Press (2017), p. 2.

^[17] Joanna Zylinska, Nonhuman Photography, Cambridge, MA: The MIT Press (2017), p. 2.

^[18] On this, see Peter Osborne, Infinite Exchange: The Social Ontology of the Photographic Image, in: Philosophy of Photography 1/1 (2010), pp. 59–68. ‘Photography, like art, is a historical concept, subject to the interacting developments of technologies and cultural forms (that is to say, forms of recognition); increasingly, developments within photography, along with digital-based image production more generally, are driving the historical development of art’ (p. 61).

^[19] Vilém Flusser,‘Fotografie, die Mutter aller Dinge’, in: Fotogeschichte: Beiträge zur Geschichte und Ästhetik der Fotografie 12/43 (1992), pp. 3–4, here: p. 4.

Talk

The Map and Photography: A very Florentine Story

Franco Farinelli

Watch video here

Quote

In thirty years, from a few crude experiments in the laboratory of a private chemist and artist, photography has extended its various applications and uses throughout the length and breadth of every quarter of our globe.

Albert S. Southworth, 1870

Innerhalb von dreißig Jahren hat die Fotografie, ausgehend von einigen rudimentären Experimenten im Labor eines privaten Chemikers und Künstlers, ihre verschiedenen Anwendungen und Verwendungszwecke auf alle Teile der Erde ausgedehnt.

Source: Albert S. Southwork, An Address to the National Photographic Association, 1870. Accessible through the Roy Rosenzweig Center for History and New Media / https://chnm.gmu.edu/aq/photos/texts/8pp315.htm [last accessed: 15.08.2022]

Quote

Photography proliferated, becoming reproducible and accessible in the modern sense, during the late nineteenth-century period of transition from competitive capitalism to the financially and industrially consolidated monopoly form of capitalist organization. By the turn of the century, then, photography stood ready to play a central role in the development of a culture centered on the mass marketing of mass-produced commodities.

Allan Sekula, 1981

Verbreitet hat sich die Fotografie während des späten 19. Jahrhunderts, in der Periode des Übergangs vom Konkurrenzkapitalismus zur finanziell und industriell konsolidierten monopolistischen Organisationsform des Kapitals, als sie im modernen Sinn reproduzierbar und zugänglich wurde. Zur Jahrhundertwende stand also die Fotografie bereit, eine zentrale Rolle in der Entwicklung einer Kultur einzunehmen, in deren Zentrum die massenhafte Vermarktung massenhaft produzierter Waren stand.

Source: Allan Sekula, “The Traffic in Photographs,” Art Journal, vol. 41, no. 1, 15–25, p. 21.

Typology

Title

Author

Artwork

DIF_000846_136

Armin Linke

CERN, Large Ion Collider Experiment (ALICE), model of the event sensor for public presentations, Geneva, Switzerland, 2021
In experiments such as ALICE, different types of sensors, here in form of a tilted rectangle, are positioned around the colliding event to detect and “photograph” the traces and sequences of the particle.

Artwork

DIF_000645_87

Armin Linke

CERN, Large Hadron Collider (LHC), control room, Geneva, Switzerland, 2019

Archival

KHI original

Gustav Ludwig’s reconstruction of Carpaccio’s Saint Ursula Cycle, wooden model with photomontage, ca. 1904, Gustav Ludwig collection, Photothek of the Kunsthistorisches Institut – Max-Planck-Institute, Florence, Italy
The photomontage, functioning like a pre-digital rendering, combines photography´s richness of detail with the pictorial composition capabilities offered by drawing and painting.

Archival

CERN original

Invitation card, entitled “Two neutrino reactions on the same photograph at CERN, 28 January 1971”, CERN, Geneva, Switzerland

Archival

REF_000910_2

Estelle Blaschke

Ralph P. Shutt, Nick Samios, and Robert Palmer presenting photographic evidence of the omega minus particle discovery, Brookhaven National Laboratory, Upton (NY), USA, 1992 . *Brookhaven Bulletin*, vol 47, no. 2 (1993)

Archival

CERN original

A scanning and measuring table (IEP) for bubble chamber photographs, CERN, Geneva, Switzerland, 08.03.1963
The Milady scan-table is used for taking rough measurements of bubble chamber photographs in preparation for automatic analysis.

Archival

CERN original

Events of particle tracks in experiment LEBC, LExan Bubble Chamber, installed in the North Area of the Super Proton Synchrotron accelerator, CERN, Geneva, Switzerland, 09.12.1981
The photographic recordings form the very basis for the visualization and creation of evidence of the movement of particles.

Archival

CERN original

Archival

CERN original

Archival

CERN original

Video

Interview

Maria Fidecaro

Maria Fidecaro, experimental physicist specialized in the phenomenological aspect of particle physics, CERN, Geneva, Switzerland, 20.10.2021, 10 min

Video

Interview

Peter Jenni

Peter Jenni, physicist and former spokesperson for the ATLAS experiment at the Large Hadron Collider (LHC), CERN, Geneva, Switzerland, 21.10.2021, 9 min

Video

Interview

Estela Suarez

Jülich Supercomputing Center, DEEP Projects, Estela Suarez, Project Coordinator, Jülich, 04.11.2021, 13 min

Video

Interview

Rolf Heuer

Rolf-Dieter Heuer, particle-detector systematist and physics professor, who succeeded Robert Aymar as Director-General of CERN, CERN (European Organization for Nuclear Research), Geneva, Switzerland, 20.10.2021, 9 min

Video

Silicon Graphics

Excerpt of promotional video for Silicon Graphics computer systems, ca. 1987
Published on YouTube, https://www.youtube.com/watch?v=Oy-kE0dq1cE [accessed: 11-10-2021]
Computer-assisted design with Silicon Graphics, an early visualisation and simulation software that promotes the synchonisation of visual modelling and dynamic data processing.