72

Iconic memory Immediately after the presentation of a stimulus, reverberating retinal activity especially in the rods and also activity in V1 grounds perceptual representation of low- level properties, what is often called “visual persistence” (Coltheart, 1980). As a result, for a few hundred ms, there is a genuine form of memory, “iconic memory,” that also represents higher level properties (Pratte, 2018). The classic experiment demonstrating iconic memory was done by George Sperling. In the Sperling experiment, there is a brief flash of an array of letters separated into rows, e.g., 3 rows of 4 letters each (Sperling, 1960). Subjects report seeing all or almost all the letters but can recall only three or four of them once the display has gone off. However, if one row is cued by a tone within a few hundred milliseconds after the stimulus disappears (a high tone for the top row, low tone for the bottom row, etc.) subjects can recall three or four from any given row, suggesting that they did have a brief visual representation of all the letters. The ratio of total capacity (roughly 3.5 in each of three rows) to capacity without a cue is called the “partial report superiority.” Fragile visual short- term memory Victor Lamme’s laboratory at the University of Amsterdam demonstrated fragile visual short- term memory in a series of articles (starting with Landman et al., 2003). The experimental paradigm combines the “iconic memory” paradigm of the Sperling experiment with “change blindness.” This paradigm shows a greater capacity in fragile visual short- term memory than in working memory but a smaller capacity than in iconic memory. A recent experiment (Pratte, 2018) suggests that representations in iconic memory undergo a “sudden death” decay, in which the surviving representations maintain the same level of precision rather than decaying in precision as “pool of resources” models would predict. Since the memory capacity found by Pratte decays smoothly from 33 ms to 1000 ms, and since iconic memory does not last more than a few hundred milliseconds, both iconic and fragile visual short- term memory would appear to be involved in this experiment. Since working memory does fit the pool of resources model, it would appear to be of a different kind than the earlier stores, as would be predicted by the claim of a format difference. This is one item of information that suggests that while perception is iconic, working memory is discursive (though it can contain iconic remnants in a discursive envelope). How Should We Understand the Distinction 0005464032.INDD 503 503 09-10-2022 11:32:10Working memory Working memory is a kind of cognitive scratch pad that can be used to manipulate information for cognitive purposes. For example, if you want to reason from the proposition that p and the proposition that if p, then q, you must hold the premises in working memory in order to make the deduction. There can be cognition without working memory, but working memory is necessary for reasoning in which a premise is retained for later use. Presence of a representation in working memory is not “storage” but rather active maintenance. Working memory is far more robust than either iconic or fragile visual short- term memory. Ilja Sligte found that a white screen (a so- called light mask) obliterated iconic memory but not fragile visual short- term memory or working memory. A pattern mask obliterated fragile visual short- term memory but not working memory (Sligte et al., 2008). Working memory is generally taken to be controlled by prefrontal cortex on the outside mid- level surfaces (dorsolateral prefrontal cortex). Transcranial magnetic stimulation (TMS) is the application of an electromagnetic pulse to a brain area, creating neural noise. Transcranial magnetic stimulation to visual areas (notably V4) impaired fragile visual short- term memory, and TMS to a cognitive area, the dorsolateral prefrontal cortex, impaired working memory but not fragile visual short- term memory (Sligte et al., 2008; Sligte et al., 2010; Sligte et al., 2011). So these different forms of memory are distinct both at the psychological and neural levels. There have been many proposals for further fractionating working memory. For example, Justin Wood has argued that working memory can be divided into a view- dependent store with a capacity of roughly four items and a more abstract view- independent store of about two items (Wood, 2009). However, it is unclear whether the view- dependent store might involve fragile visual short- term memory. For the kinds of stimuli discussed here, working memory has a limit of three to four items. Typically, the three- to four- item limit is observed with small closed- class groups of stimuli. For large open classes of stimuli, many more items can be represented in working memory with diminished precision. See (Endress and Potter, 2015; Endress and Siddique, 2016) for an explanation of the difference between cases in which the three- to four- item limit is observed and cases where it is not. When working memory representations do not show iconicity, one cannot be sure whether the iconicity was lost in the conceptualization process, but when they do show iconicity, the iconicity derives from perceptual remnants that are contained in the working memory cognitive envelope. I now move to the evidence that the object representations of working memory are fundamentally different from those of perception even when the object representations of working memory involve perceptual materials. I will mention three points of difference: 1 Perceptual object representations have a higher capacity than working memory object representations. 2 Working memory object representations do not show fundamental computations of perception. 3 Working memory object representations are task specific in ways that perceptual object representations are not. 504 0005464032.INDD 504 Ned Block 09-10-2022 11:32:10Capacity As just mentioned, the Sperling experiment showed that the more perceptual representations of iconic memory have roughly three times the capacity of working memory. And fragile visual short- term memory has around double the capacity of working memory. Fundamental computations Perception exhibits a canonical computation, divisive normalization. One manifestation of this computation is center–surround suppression, in which perception of a central disk is suppressed by similar properties in a doughnut surrounding it. This is illustrated in Figure 27.7. When the disk and the doughnut were presented one at a time, with the first stimulus maintained in working memory, there was no center– surround suppression (Bloem et al., 2018). This result suggests that a basic computational feature of perception is absent in perceptual working memory. Task specificity A recent experiment showed how two quite different perceptual representations can be converted into the same working memory representation if the subjects’ tasks are appropriately similar. Yuna Kwak and Clay Curtis (2022) used two kinds of stimuli on different trials, oriented gratings (Gabor patches) and clouds of moving dots. Subjects’ task was to indicate the orientation of the grating or the direction of the moving dots after a delay period. They scanned the subjects using fMRI during the delay period prior to doing the tasks. The first result was that decoding trained on the grating task also worked on the dot task and vice versa. This fact shows that the working memory representation was sufficiently abstract as to be common between the two perceptions. The second result homed in on what the actual shared representations were. They developed a visualization technique that allowed them to transform the brain representations into a display on a screen that would have produced that brain activation. And the result was that both the representations of the grid and the dot motion transformed to an oriented stripe. The representation of the cloud of dots abstracted away from the representations of the individual dots and the representation of the Orthogonal Collinear FIGURE 27.7 ILLUSTRATION OF THE EFFECT OF DIVISIVE NORMALIZATION. THE CENTER DISK IS THE SAME ON BOTH SIDES BUT LOOKS LOWER IN CONTRAST ON THE RIGHT BECAUSE OF SURROUND SUPPRESSION THAT DEPENDS ON SIMILAR ORIENTATION OF THE DISK AND ITS SURROUND. THANKS TO SAM LING FOR THIS FIGURE. How Should We Understand the Distinction 0005464032.INDD 505 505 09-10-2022 11:32:11grating abstracted away from the spatial frequency and contrast of the grating. What this experiment shows is that working memory representations depend not only on the stimulus but also on the task. A similarity in task can lead to a similarity in working memory representation even if the percepts differ. This experiment also shows that there is a sense in which the perceptual materials in a working memory representation can “misrepresent” the stimulus for the sake of usability of other information. In the case of the moving cloud of dots, there is no “stripe” in the stimulus. I mentioned earlier that the “object files” of working memory may not even be grounded in perceptual object representations as is often claimed (Recanati, 2012). As just noted, the task specificity of working memory object representations can transform the perceptual information in a perceptual object representation. The working memory representation that is derived from perception does not just discard some information— it transforms the information for task- specific use. In the Kwak and Curtis experiment, the information about the dots and the spatial frequencies are transformed into a quite different representation. I have described three important differences between the perceptual information in “object files” of perception and of working memory. I have omitted others that would have required more extensive discussion, e.g., fineness of grain. I now move to a discussion of a concrete example of the difference keyed to the issue of iconicity. Arguments against iconic object representations that are based on perception and memory of objects The evidence provided by E. J. Green and Jake Quilty- Dunn (Green and Quilty- Dunn, 2021; Quilty- Dunn, 2016, 2020a, 2020b) for discursive object files representations is based on the “object- specific preview benefit” or OSPB (Kahneman et al., 1992). They use the OSPB to argue that the format of object files is discursive rather than iconic. In one version of the OSPB, two boxes are on the screen containing pictures, for example, pictures of an apple or a loaf of bread. The pictures disappear and the boxes move. Then a picture appears in one of the boxes, either of an apple, a loaf of bread, or something else. The subject’s task is to name the object. Subjects are faster in naming an apple if a picture of an apple was in either one of the boxes. (So far, that is just “priming,” a phenomenon whereby something just seen or appropriately related to something just seen is easier to recognize.) However, and this is the OSPB, subjects are faster still if the apple is in the very box that it started in, even if that box has changed sides. Another version of the OSPB is illustrated in Figure 27.8. Words are presented in boxes. Then the words disappear and the boxes move as indicated for 1.5 seconds. Then a picture appears in one of the boxes which the subject is supposed to name. The result is that the subject is faster to name the apple if the box the apple is in was the one in which the word “apple” had appeared. Green and Quilty- Dunn take this result to indicate that the perceptual representation— the “object file” that underlies this ability— is a symbol that has the content apple and is bound to semantically linked information in a separable, nonholistic fashion. For example, the object file might simply be a discursive list of linked properties. 506 0005464032.INDD 506 Ned Block 09-10-2022 11:32:11Preview display apple bread Linking display Target display FIGURE 27.8 VERSION OF THE OBJECT- SPECIFIC PREVIEW BENEFIT THAT SHOWS THAT OBJECT FILES CONTAIN BOTH LINGUISTIC AND PICTORIAL INFORMATION. FROM QUILTY- DUNN (2016), BASED ON GORDON AND IRWIN (2000). THANKS TO JAKE QUILTY- DUNN FOR THE FIGURE. Congruent Match Trial Incongruent Match Trial No-Match Trial [ring] [bang] 1500ms [whistle] 1000ms time Until response FIGURE 27.9 VERSION OF THE OBJECT- SPECIFIC PREVIEW BENEFIT. FROM JORDAN ET AL. (2010), P. 495, WITH PERMISSION OF TAYLOR AND FRANCIS, HTTP://WWW. TANDFONLINE.COM As Green and Quilty- Dunn note: There is also an OSPB from lowercase words to uppercase versions of the same word, for instance, from “bread” to “BREAD.” That, they say, shows that the representation of the word abstracts from shape properties and so cannot be iconic. This is part of the abstractness argument against iconicity. In another variant, pictured in Figure 27.9 (Jordan et al., 2010), two boxes are presented with pictures in them, say a hammer and a whistle. The pictures disappear and the frames then move so that the boxes can end up on a different part of the screen from which they started a second later. Then the subject hears a sound and has to say whether the sound matches one of the pictured items. Subjects are faster if the sound matches the object that was in the box that is now on the side that the sound is coming How Should We Understand the Distinction 0005464032.INDD 507 507 09-10-2022 11:32:13from. For example, in the top row of Figure 27.9, the sound of ringing matches the picture of a telephone. Subjects are fastest for the “congruent” situation in the top row. The sound of banging in the second row does not match but was present (bringing with it the speed increment of priming.). That row comes in second. The slowest is the bottom row in which the sound— a whistle— does not match either of the pictures. Green and Quilty- Dunn conclude that object files involve discursive symbols that abstract away from modality- specific information in an amodal format. How do we know that the representations involved in the OSPB are working memory representations? In the experiments pictured in Figures 27.8 and 27.9, there is a delay between the first stimulus and the last stimulus. In the experiment with the word “apple” and the picture of the apple, the delay is 1.5 seconds. In the experiment with the sounds matched to objects, the delay is 1 second. A further experiment showed that the OSPB was preserved even if the blank period lasted as long as 8 seconds (Noles et al., 2005). However, iconic memory of the perceptual kind exhibited in the classic Sperling experiment lasts only a few hundred milliseconds, so the OSPB representations cannot be representations of iconic memory. As I mentioned earlier, there is another kind of perceptual memory, “fragile visual short- term memory” (Lamme, 2016). Fragile visual short- term memory has been shown to last up to 4–5 seconds, but never longer. In addition, fragile visual short- term memory has been shown in static displays but not to my knowledge in moving displays. Further, fragile visual short- term memory is, well, fragile, and easily overwritten. The motion in these displays may be enough to damage fragile visual short- term memory representations. These considerations strongly suggest that the kind of memory involved in the OSPB is working memory, the least perceptual of the three kinds of visual short- term memory. I think the OSPB concerns working memory representations that have conceptualized remnants of perception in a cognitive envelope and that there is no evidence that the abstractness shown in the OSPB can be ascribed to perception as opposed to the cognitive aspects introduced by the conceptualization and the cognitive envelope.1 So the crucial issue concerns whether the OSPB involves perceptual representations of the sort that are involved in perception itself. The first thing to note about the OSPB is that after the picture or word disappears, the subject is no longer in a state that seems like seeing them. They see the boxes that are rotating, not what was originally in the boxes. I have looked at OSPB displays. Once the letters disappear one just sees the boxes moving with no awareness of the letters. The fact that the subject does not see the picture or word by itself shows that we should be suspicious of any claim that in the blank period the subjects have perceptual representations of the items that were originally in the boxes. There is no reason to think that the subjects in this experiment have any visual phenomenology of the items in the boxes during the blank period. The iconic memory and fragile visual short- term memory mentioned above are said by subjects to be phenomenal, but I don’t know of any reports of phenomenology of working memory in experiments that contrast iconic memory, fragile visual short- term memory, and working memory, such as the experiments by Victor Lamme’s group in Amsterdam (Lamme, 20 03, 2004, 2006, 2016, 2018; Landman et al., 2003; Pinto et al., 2013; Pinto et al., 2015; Sligte, 2011; Sligte et al., 2008, 2009, 2010, 2011). 508 0005464032.INDD 508 Ned Block 09-10-2022 11:32:13Further, it takes 1.5 seconds for a subject to generate a mental image. In the 1 second that the boxes are rotating as depicted in Figure 27.9 there would be no time to generate a mental image of the hammer or telephone. Both of these points suggest a difference in kind between the “object files” of working memory and the “object files” of perception. Consider the top row of Figure 27.9. On the left we see a box with a telephone on the top and a hammer on the bottom. Then the pictures disappear and the boxes move. They move for 1 second as depicted in Figure 27.9, but as I mentioned the time lag can be as long as 8 seconds. Then a sound plays. As I mentioned, the subjects are not seeing the telephone or the hammer. They just see the empty boxes moving. If the representations of the telephone and the hammer are real perceptual representations, perhaps they would be unconscious perceptual representations. Now I happen to be a fan of full perceptual representations in unconscious perception (Peters et al., 2017; Phillips and Block, 2016). But one lesson of recent work on unconscious perception is that it is harder to produce than was earlier thought. Megan Peters and Hakwan Lau (Peters and Lau, 2015) did an informal survey of people who work on perception and found that though most thought unconscious perception exists, most also thought that unconscious perception had not been demonstrated to exist. Note the contrast with the evidence presented earlier for iconic object representations in perception. Recall the apparent motion case, in which a bird is seen to be moving and then changing into a rabbit. The trajectory and bird/rabbit shapes are consciously experienced even though they are not on the screen. And in the Nakayama experiment, the moving object is seen to rotate even when nothing is rotating on the screen. More illumination on the difference between perceptual object representations and OSPB representations can be found in a phenomenon known as the tunnel effect, in which an object disappears behind a narrow occluder (the “tunnel”) and an object emerges from the other side of the tunnel. The second object may differ in color, shape, and kind from the first (e.g., a lemon goes in and a kiwi goes out). If the tunnel is narrow enough relative to the size of the object moving through it (best results are achieved when the occluder is the width of the object) and the motion is fast and smooth enough, subjects see a single object going behind, changing shape and color, and emerging from the other side. An early article on the effect from the days in which first- person descriptions were routinely used in perception journals, says that “an absolutely compelling impression of continuous and uniform movement can be produced … all the observers agree that the movement behind the tunnel is as ‘real’ as” motion without the occluder (Burke, 1952, p. 124). As the relative length of the tunnel increases and the speed decreases, subjects can still track the moving object using a working memory representation, but they no longer experience motion. My point is that when the representation becomes a working memory object representation rather than a perceptual representation is when consciousness fades. I have seen no report of awareness of the objects in the OSPB. In the OSPB, perceptual representations are conceptualized in working memory. As we saw in the Kwak and Curtis experiment described earlier, we can expect that conceptualization in working memory will produce a format difference that is keyed to the task. Kwak and Curtis describe a format change in the direction of abstraction.