We invited Dr Javier Suarez to tell us more about the psychology behind comics. The following is an edited transcript of a talk he gave to his fellow neurology residents at Mass General Brigham
---
The human visual system is astonishingly well-designed to create a unified percept based on a myriad of visual primitives such as orientation of lines, color, and contrast. Far from the atomistic, component by component addition model popularized by such British empiricist philosophers as John Locke and David Hume, the modern view first developed by the Gestalt psychologists holds that perception is an active creative process. Through the interaction between bottom-up integration of information gathered by the first visual receptive organ, the retina, and top-down modulation based on expectations and context, we end up with the ability to recognize objects and scenes despite radically different external conditions, track movement in space with detail, and use these perceptions to guide actions. The brain imposes patterns on perception and does so fluidly, quickly, and almost completely unconsciously. There is internal logic to assume borders and the spatial relationships between elements, which helps to separate foreground from background and separate one object from another, but can also create illusions such as the Kanizsa triangle. Furthermore, object identification involves hypothesis testing in which internal representations of objects are compared with information arriving from the retina. The inferior temporal cortex (ITC) and its interplay with prefrontal cortex (PFC) has been implicated as the neural correlate for object recognition. These functions are leveraged by artists to create both clearly recognizable figures in realistic representations (e.g. Leonardo da Vinci) as well as mind-bending abstract works in which the brain imposes meaning and form where there may be none (e.g. Jackson Pollock). One particular art (or is it writing?) form uniquely blends the visual world of objects and language to tell stories: comics. By leveraging the brain’s infallible ability to recognize with its penchant for inferring, comics contain a wealth of information and present it to engage the reader in a way that cannot be done by any other medium.
The definition of a comic is difficult to state without invoking the possibility that another medium may be included. I favor the one proposed by Scott McCloud in his book “Understanding Comics: the Invisible Art” because it is detailed and specific, if lengthy. Indeed, much of what will be presented in terms of the structure of comics is derived from this book and it is an excellent work for anyone interested in the technique and history of comics. He defines a comic as: “juxtaposed pictorial and other images in deliberate sequence intended to convey information and/or to produce an aesthetic response in the viewer.” Note that this does not include any mention of style, genre, or materials. Stained glass windows in many churches, for example, are comics. The style and medium are different from the comic strips or superhero books the mind might conjure in association with the word “comics,” but they are comics nonetheless. This is important to recognize because comics can be seen in cultures around the world, from pre-Columbian Central America to Egypt (I do not refer here to hieroglyphics, which are representations of entirely separate concepts than what they depict and are the precursor to modern written language). They are a compelling narrative tool because they often exploit the brain’s ability to recognize depictions of images as what they are. This sounds simple but is powerful; when we look at an image of a person, we recognize it as a person even if there is no human being on Earth that looks anything like what is depicted. Comics leverage our pattern-seeking brains to tell stories, and the use of iconic over realistic depictions is one of their great tools.
Cartoons and otherwise less realistic images are used often in comics because they strip the depiction down to its base form and, in doing so, can convey a message cleanly and clearly. Take, for example, the perennially recognizable Charlie Brown, one of the main characters of the popular comic Peanuts by Charles Schulz. His design has evolved a bit over time, but his face has never bore resemblance to a real human, save perhaps one with holoprosencephaly and a proboscis squarely in between the two eyes. Yet instead of seeing him as a strange disfigured being, the PFC and ITC allow us to see him as a normal human child, and in fact prevent us from seeing him as anything but. This iconic representation likewise allows for easier connection between creator and reader; when there are fewer essential details provided, there are more opportunities to insert them with your own experiences and qualities. Cartoons, then, seem to be full of life and personality precisely because they are not a depiction of what we know as the real world. As McCloud further points out, these nuances can be used to manipulate perception by creating a lifelike, hyper-realistic backdrop for iconic characters to live in, an effect known as “masking.” It can also be used to heighten the “otherness” of characters the creators want to portray as villains. This is used more often by Japanese creators than American or European ones, the former often creating a stark dichotomy between the cartoonish, highly expressive characters whose emotions and intentions we can easily identify, and the pragmatic background in which they exist. The latter, on the other hand, tend to use a similar style for both characters and background. A notable European exception is The Adventures of Tintin by French writer/artist Hergé, though even this is a far cry from many Japanese works such as the popular Dragon Ball Z by Akira Toriyama and One Punch Man by ONE and Yusuke Murata. These differences highlight that drawings in general and icons in particular are not merely a recreation of what the artist sees or even a purely isolated representation of an individual’s ideas, but are cultural, learned schematic patterns. In addition to our recognition of iconic representations, comic creators take advantage of the brain’s ability to impose patterns and make inferences when there is a lack of information. We perceive the passage of time and the motion of objects based on the way two-dimensional images are placed with respect to each other. This is exemplified in the gutter, perhaps comics’ most special tool.
The gutter in its simplest form refers to the space between two images. It is where the reader must use their powers of inference to connect the images temporally and spatially with regards to the world being depicted. ITC and PFC have also been shown to be capable of storing information in working memory, which in turn aids in the perceptual constancy required to make sense of a comic. ITC stores primarily visual information, while PFC carries additional multimodal information that adds context to the visual scene and updates perception as a whole as new information comes in (this is likely an oversimplification of a process that involves a wider network in the brain, but is sufficient for illustrative purposes here, no pun intended). This allows for vibrancy to develop from a scene as the visual information triggers multimodal associations that have developed through experience. By adding richness to the visual scene, creators add richness to all senses; by reducing the complexity of the scene, creators can instead force the reader to create their own world that allows them to make sense of what is being portrayed. Creators use the gutter, and its absence, in fantastically creative ways to depict events, with the reader constantly updating their conception of the world being presented and comparing it to the new information provided. This happens with any form of storytelling, but comics have a unique flexibility in their use of representative images and language together. One need only read such beautiful works as The Sandman Overture #4 by Neil Gaiman, JH Williams III, Dave Stewart, and Todd Klein to see that the directionality and conventions of reading in Western cultures (i.e. left-to-right, top-to-bottom in a Z-shaped pattern) can be played with to highlight smaller aspects of a scene and emphasize the strangeness of mystical worlds while maintaining a relatively coherent syntactic path. The reader must infer a great deal more than, for example, a movie or TV show while also being provided with a relatively large amount of information compared to a purely printed language work. The magic is in the gutter, and the interplay between panel and gutter is expertly crafted to convey the passage of time and the actions of characters.
Time always moves forward at one second per second, but can be manipulated in comics due to the fact that it takes some amount of time to physically process the information in front of you. In other words, it takes time to move your attention from one panel to the next. McCloud astutely points out that even within a single panel, text takes time to read and words take time to be spoken, so any given panel rarely represents a single instance. By adding multiple panels (or a single larger panel) that do not contain much new information in between two events, creators can convey the passage of time by forcing the reader to perceive more between said events. In the Invincible Iron Man #3, Brian Michael Bendis and David Marquez use this technique to accentuate the awkwardness of a long pause, while Robert Kirkman and Ryan Ottley use it to ironically portray their commentary on the mainstream comic book industry in Invincible #10. Even if you skip directly to what might be considered the next scene, the brain cannot help but acknowledge the existence of the other panels. Time has passed, and the reader must perceive it. As time moves forward, so too do characters and objects tend to move in space. Via a variety of motion cues, creators can use two-dimensional images to convey motion in three.
Perhaps the most straightforward way to portray motion is to use the gutter. That is, show two static-appearing scenes in two panels and force the reader to infer that motion must have taken place in order to make sense of the panels. Not satisfied with this approach, artists have created multiple techniques that can indicate motion within a panel, most of which take advantage of cerebral motion perception. Sometimes subtle, others exaggerated, lines are easily processed and can be leveraged to explicitly trace the trajectory of moving bodies. Other artists mimic the blurring effect of an object that moves across the visual field without being kept squarely on the fovea. McCloud again comments on the differences between Western and Eastern, specifically Japanese, styles. The latter pioneered a “first-person” mode where lines are used to exaggerate motion explicitly from a character’s perspective. Creators use these techniques to make “moving pictures” much differently than anything you see at the cinema.
Through both iconic and realistic imagery, artists allow readers to insert themselves into the story in one moment and convey the abject otherness of characters and backgrounds in another. In panels and in the magical gutter, time and motion are demonstrated and all of these techniques are used in different ways across cultures. The interplay between PFC and ITC, along with the language networks necessary to understand semantics and syntax, are expertly manipulated to portray everything from the mundane to the fantastical in ways that cannot be mimicked by other media. Well beyond a small strip in the newspaper or spandex-clad superheroes in a stapled collection, comics are a rich medium wherein our highest-order cognitive functions are tested and exploited to magnificent effect.
References:
Kandel ER, Schwartz JH, Jessell TM, Siegelbaum SA, Hudspeth AJ. Principles of Neural Science. Fifth edition. 2013.
McCloud, S. Understanding Comics: The Invisible Art. HarperCollins Publishers, Inc. 1993.
Comments