click! Photography Changes Everything

Jos Stam, computer scientist and 3-D graphics specialist, wonders whether photography is, in fact, the best way to depict reality.

Three depictions of reality: photography, painting and computer graphics.

"They now have movies which use computer-made landscapes, and they say look real! They are only real in the context of the perspective picture we are used to. We must use the computer another way." David Hockney

Look at the three pictures that accompany this short essay. Each one attempts to convey the experience of looking at a real-life scene. The first is a photograph I took on the coast of Faro, in southern Portugal. The second is a photograph of a painting by Jacob van Ruysdael. The third is computer generated, a set of pixels depicting an ocean created with Autodesk MAYA software.

How are these pictures related? Which is the most “realistic?” The photograph from Portugal is a reminder of how ubiquitous photography is, and the big role it plays in defining how we share our visual experiences with others. Photography is convenient, but is it always the best way to depict reality? A photograph is actually a very limited way of sharing our experience of the real as it represents only a single vantage point in space and a brief moment in time. Shutting one eye and looking around with the other gives you more information than a camera would; your head moves and visual data entering that open eye is directly wired to the brain where it is filtered and prioritized. Crucially, there is no static or single picture in your mind of what you see. Your brain, wired to your eyes, is always affecting what you see as well as the way you see things.

Now look at the photograph of a painting by Jacob van Ruysdael. Most people would say that it is not as realistic as a photograph of a similar scene. However, I’d argue that Ruysdael’s painting conveys more temporal and sensory information than any single photograph can. Through exaggeration, it provides a more vivid visual experience—a natural scene captured, not by a piece of hardware like a camera, but filtered over time by the subjective mind and skills of a talented painter. This experiential “effect” is even truer in portraiture; we often relate more to caricatures than to photographs, because caricatures, if done well, capture in a single image an aggregate of typical expressions and facial features.

When photography was in its infancy at the turn of the last century, the French sculptor Auguste Rodin criticized it as being unrealistic, claiming that his sculptures were more realistic. To understand this, try to mimic, for example, the pose in “The Thinker,” one of Rodin’s most famous sculptures. You’ll be surprised how hard it is to achieve the actual pose. Rodin’s art is exaggerated to achieve higher realism. One of my favorite forms of pictorial art—what people in the art world call, and some dismiss as Photo Realism—is another and more contemporary exaggeration of photography. Hyperreal artworks—not copies of photographs but caricatures of photographs—are, to me, more evocative and appealing than photography.

With the spread of digital imaging, a more recent fallacy concerning the relationship of photography and realism is that more megapixels inevitably yields more realism in any given image. But we only have to look at Rembrandt’s portraits to see that this is not true. He famously said that people shouldn’t look too closely at his paintings as the smell of oil paint would sicken them. As in the case of caricatures, Rembrandt’s paintings, and especially his self-portraits, give you the impression of looking at a real person. Consciously or not, Rembrandt knew how our brains would interpret his blobs of paint.

As a computer graphics researcher, while photography has a huge influence on what I do, the major part of my work is devoted to achieving convincingly realistic imagery through non-photographic means. Interestingly, we incorporate defects of the photographic medium, like lens flare and depth of field, in order to achieve depictions that are close to photographic. But the distinctive look of computer graphics is largely based on physics and a theory called Radiative Transfer, which gives us a great framework to ponder how light bounces around in an environment. In short, this is how it works: we figure out how light interacts with objects within a scene, and then render that process numerically before we project that data, as an image, onto a two-dimensional planar surface. The last two steps are performed by computer.

The bottom line in our industry is to get people to watch the special effects that are created using our software. And to achieve that goal, we are always questioning our basic methodologies. There is a lot of room here for innovative research that proposes different mappings from a three-dimensional virtual world to a two-dimensional array of pixels. I don’t think a single photographic, mathematical or perceptual model can achieve this, and we need to keep providing filmmakers and artists with new and better sets of tools.

To strive for photographic realism is just one of many ways to convey a real life perceptual experience to other people. My point is that it is not always the most compelling one. Our interest in the field of computer graphics is to better understand how our brains work, and then go beyond the specifics of photography to find ways of achieving higher realism by using non-camera centric models.