68

Box 2. Why double representation is problematic Trends in Cognitive Sciences As mentioned in the main text, standard versions of the higher-order approach have postulated two representations of a conscious perceptual content. For consciousness of a first-order representation of motion, the higher-order thought would be something like, ‘I myself am experiencing visual motion.’ The main problem with this ‘double representation’ aspect of higher-order views has to do with conflicts between the higher-order thought content and the perceptual content [53,54]. Suppose I have the thought that I am seeing an entire surface as pure red and pure green at the same time. I can have that thought, but there will be no corresponding experience or first-order representation with that content without a specialized apparatus that allows both ends of the red/green opponent channel to be activated at once [55]. (You need a device that tracks the direction of eye gaze and projects an imagetothesameplaceontheretina,nomatterhowtheeyemoves.)Perhapsthethoughthastobeofsomespecialtype, but, as we will see in the next paragraph, no special kind of thought will work. A second issue is that if there really are two representations of every conscious content, we would expect there to be cases in which these representations conflict, but no one has ever given convincing evidence of such a thing. Of course, there are cases in which we have conflicting perceptions; for example, in perceptual crowding, subjects report confused and changing perceptions [56]. But, as noted in [57], these reports can be explained entirely in terms of conflicting representations in perceptual areas. Onemightwonderwhyhigher-order theorists did not abandon the doublerepresentation by getting rid of the requirement of any first-order state at all. In fact, some theorists did adopt the view that because of the possibility of an ‘absent’ firstorder state, ‘we aresometimesconsciousofourselvesasbeinginvariousmentalstatesthatwearenotactuallyin’[58].Or, alternatively, ‘one can be in a conscious state even if that state does not exist, so long as one is suitably awareof oneselfas being in that state’ [59]. But no kind of thought that I am visually experiencing a surface as pure red and pure green all over at the same time makes it so, given that special apparatus is needed for that experience. can be based in the PFC without representation of perceptual contents in the PFC and thus without the possibility of decoding conscious contents from the PFC. Let me explain. According to the higher-order thought account, a conscious perception of, say, motionbecomes consciousbyvirtue ofacognitivestateaboutthatperceptionofmotion.Olderversionsoftheview required the thought to rerepresent the perceptual content [34]. This double representation was problematic for reasons discussed in Box 2. However, newer versions of the higher-order approach (Figure 1) are hybrid, requiring both first- and higher-order states, with the perceptual content in the first-order states (see [35] for a brief summary of scientific advantages). According to this approach, the cognitive representations in the PFC do not rerepresent but rather are pointers to or indexes of perceptual activations [10,35–38]. As Hakwan Lau puts it ([10], p. 134), Representation This is real Explicit belief Dereferencing Reality monitor Conscious Imagery Conscious Perception This is self-generated This is noise Unconscious Trends Trends in in Cognitive Cognitive Sciences Science Figure 1. Cartoon depiction of the pointer version of the higher order theory. Perceptual monitoring requires a ‘discriminator’ that decides how perceptual contents are to be represented in cognition, as imagery (‘This is self-generated’), as reliable perception (‘This is real’), or unconsciously represented (‘This is noise’). ‘Dereferencing’ is a computer science term for retrieving the content that a pointer points to. See [10,35]. 4 TrendsinCognitive Sciences, Month 2024, Vol. xx, No. xxTrends in Cognitive Sciences ‘the role of the prefrontal cortex may not be to “duplicate” the sensory information. Rather, it may just monitor and redirect information in the sensory cortices, using something akin to indexing mechanisms.’ The concept of a pointer is taken from computer science, where a register can contain the address of another register, sending the processing to that other register. In pointer theories, a PFC pointer is a link to a first-order representation, typically a perceptual representation of the outside world. The pointers can have contents involving the monitoring of probability or reliability (in some versions, subpersonal contents), but not perceptual contents. Pointers function to recruit first-order contents in the service of cognition concerning that content [35,39] (Figure 1). Once one sees that monitoring pointers can have this role, it is natural to suggest that global workspace theories make the same modification [39,40]. The key concept of the global workspace viewpoint is ‘ignition.’ The idea of ignition is that there is a competition among neural coalitions in posterior visual areas. Only one or two of these can win out, and these winners can ‘ignite’ larger coalitions via long-range projections to the PFC and to parietal areas. These ignited activations are mutually reinforcing and allow the perceptual information to be accessed by any of the ‘consumer’ systems of reporting, decision-making, reasoning, and the like. The important point is that although previous versions have emphasized ignited rerepresentations in the PFC, what is ignited need only be a pointer to perceptual contents. The perceptual contents can be accessed by consumer systems via ‘dereferencing’ the activated pointers, a process via which the contents are retrieved. As Mashour et al. describe it, the global workspace can act asarouter‘throughwhichinformationcanbeamplified,sustained,andmadeavailabletospecialized sensory processors’ [12]. It has commonlybeenthoughtthatchangesinconsciousperceptionduringbinocular rivalry have been driven solely by a winner-takes-all competition between activations in perceptual areas in the occipital and temporal cortex, but a recent result suggests that these changes are driven instead by signals in the PFC [4,41]. These researchers show that one kind of PFC activity promoted binocular change and another promoted stability. It may be that these PFC signals are related to pointers. My point has been that failure to decode from the PFC does not challenge the pointer view. The PFC can be pivotal even without decoding perceptual contents from the PFC. But what about the converse: does the success in decoding from the PFC described earlier challenge the pointer view? I think not, for several reasons. First and foremost, I think there is a good case that decoding fromthePFCisduetopostperceptualprocessing.Thatis the topic ofthenextsection.But whatifI amwrongabout that? If we exclude postperceptual processing, do the decoding results from the PFC already mentioned challenge the pointer view? There are several ways rerepresentation and pointer views can coexist. Some decoding results might be explained by rerepresentation in the PFC for generic or abstract contents (e.g., red or rectangle). Results that suggest conscious representation in perceptual areas could be accommodated by combining this view with pointers for the full details of consciousness (the shade of red, the shape of the rectangle). One well-known criticism of rerepresentation views has been that they are incompatible with what we take to be the f ine grain of perceptual phenomenology [42]. Combining rerepresentation for abstract contents with pointers to details defuses that objection. Alternatively, low-level contents such as orientation might berepresented by PFCpointers, at least briefly, then replaced by a pointer to face representations in the temporal lobe. This might besuggestedbytheresults of[7,43,44], but another explanation of those results, suggested by what I report in the next section, is that faces are simply of more interest to the subjects than orientations. (Another possibility is that– as in every case in which decoding fails– more fine-grained techniques could detect orientations.) Trends in Cognitive Sciences, Month 2024, Vol. xx, No. xx 5Trends in Cognitive Sciences In sum, the pointer option for both higher-order and global workspace theories faces no problem of a lack of PFC decoding. See Box 3, however, for a challenge to PFC pointer views. Whydecoding from the PFC may not support cognitive theories of consciousness I have been arguing that decoding of perceptual contents from the PFC shows too much (because cognitive theories do not require decoding); I shift now to pointing out that PFC decoding also shows too little (because it may reflect postperceptual processing rather than conscious perception). Bored monkey problem There is a persistent problem with the positive evidence for PFC decoding. Even no-report methodologies do not preclude subjects noticing or thinking about or otherwise cognizing the stimulus. I argued that this ‘bored monkey’ problem is especially pressing in the aforementioned binocular rivalry experiments in which monkeys see gratings moving one way, then another way, repeating over and over [45]. I said monkeys that are sitting in a primate chair with no task and nothing else to do might be noticing or otherwise cognizing the stimulus direction, contaminating the perceptual decoding from the PFC [15]. Other researchers [39,46] objected, arguing that reproducible stimulus-specific thoughts would be unlikely to occur as a result of boredom. However, what these replies ignore is that the cognitive categorization might be a kind of automatic cognition– or, as suggested in one of the replies [39], it could be due to a cognitive representation of confidence in the perceptual content. Perceptual contents were decoded from PFC 60 milliseconds after the stimulus in a paradigm presenting ten pictures per second, each masking the previous picture [26]. The authors argue that masking precludes post-perceptual processing and since it has been found [47] that even at 12 pictures per second, subjects are above chance at post-perceptual matching of pictures with descriptions, the study is supposed to show conscious representation based in the PFC. There are two errors in this reasoning. First, the PFC representations 60 milliseconds after the stimulus that they find are not conscious. They are likely to be a result of the low spatial frequency Box 3. Problems with pointer theories Although pointer views do not predict PFC decoding, they cannot use one of the advantages claimed for prefrontalist theories concerning rich versus sparse perception. I mentioned the ‘inattentional blindness’ results in which the periphery of the visual field loses color and form but subjects do not notice this. Rerepresentation versions of higher-order theories purport to explain this by postulating PFC representation of a periphery without loss of color and form, but pointer views cannotusethisidea,becausepointersdonothaveperceptualcontents.(SeeChapter4of[10])anda reply in[1], pp. 436–442.) However, inattentional blindness results can be accommodated by pointer theories if they result from the failure to apply concepts, resulting in a failure to notice the peripheral features (see Chapter 6 of [1]). Furthermore, some recent anti-PFC results would apply to pointer views. My colleagues and I [60] noted that a survey of intracranial electrical stimulation work failed to find reproducible perceptual perturbations of ongoing experience with PFC stimulation. Others [61] responded by noting that PFC representation is highly distributed, so one would not expect that electrical stimulation would yield a coherent percept and that stimulation that works for perceptual areas might work less well for the more distributed representations of PFC. This is true. However, the key finding is that ongoing perception (e.g., of the doctor’s face) is not perturbed by intracranial electrical stimulation. If consciousness is based in PFC pointers, it would not be unreasonable to expect that electrical stimulation might cause a pointer to be degraded to the extent of perturbing the patient’s conscious experience or might cause the pointer to point to a different perceptual representation, despite the differences between the PFC and perceptual areas. But no indication of such perturbations appears in the literature. With some exceptions (e.g., olfactory effects) as detailed in [60]andsection3.10of[10], from the subjects’point of view, intracranial stimulations to the PFC are treated the same as shamstimulations in which the noise of a stimulation appears without the stimulation. Of course, this is a null result that should be interpreted with caution, and, as [46] noted, the stimulation could conceivably have affected subjects’ ability to report the perturbations. Michel [39] discusses this issue, noting that progress will require putting together stimulation with behavioral tasks. 6 TrendsinCognitive Sciences, Month 2024, Vol. xx, No. xxTrends in Cognitive Sciences ‘fast feed-forward sweep’ that is knowntoberelativelyimmunetobackwardmasking[49,50]and in which unconscious representations are used to focus attention signals in perceptual areas. Earlier work on these early signals in PFC acknowledgedthatthesesignals precede scenesegmentation and selection of attentional targets [50]. Second, the conscious representations occur after the feed forward sweep. As [47] notes, we should think of these experiments in terms of the ‘carwash’ model of masking in which multiple stimuli can be processed in parallel so long as they are not in the same stage of the ‘carwash’ [48] showed that pictures were poor masks of previous pictures at these high rates of processing, so it is likely that there is extensive futher processing of the pictures. At this point in the debate, it seemed that we had a clash of intuitions that required further evidence to resolve. Fortunately, that further evidence was available. Bifurcation dynamics Sergent et al. [51] used a paradigm in which the target stimulus was task-relevant in some trials but notothers. Theydidnotusebinocularrivalry; rather, they presented two French vowels, /a/ or /ə/, embedded in noise with varying signal-to-noise ratios. In ‘active’ (task-relevant) sessions, subjects were askedtoreportthe identity and audibility of the vowels. In ‘passive’ (task-irrelevant) sessions, the vowels were present, but the tasks were unrelated to the vowels. Subjects’ brain activity was recorded using electroencephalogram (EEG). There were two key findings, one related to the no-report methodology, the other related to the bored monkey problem. The no-report result was that at threshold (behaviorally determined), there was a bimodal distribution of activation: Sometimes there was widespread activation between 250 ms and 700mspost–vowelstimulusandsometimesnot,withfewintermediatecases,whattheycalled ‘bifurcation dynamics.’ Widespread activation accurately predicted subjects’ reports in the active version of the study. That is, when there was widespread activation linked to the vowel onset, subjects reported the identity and audibility of the vowels, and when there was no widespread activation, they did not. Here is the methodological result: the authors [51]wereabletouseaversionof bifurcation dynamics as a substitute for report in the passive (no vowel report) sessions. That is, even when subjects were not reporting the vowels, widespread activation yoked to the sounds predicted awareness of them, as shown by a ‘mind-wandering’ method to be described later. The widespread activation in the passive (no-report) condition was not as widespread as global workspace activation, leaving out PFC areas involved in executive function. Sergent et al. [51] term the widespread activation in the passive condition ‘global playground’ activation, where global playground activation is a subset of global workspace activation. Global playground activations are denuded of the decision and executive processes underlying report, and this makes them a better candidate for the neural basis of access-consciousness than global workspace activations (Figure 2). Trends Trends in in Cognitive Cognitive Sciences Science Figure 2. Reconstruction from electroencephalogram (EEG) data of activations 330 ms from stimulus onset. The leftmost figure depicts global workspace activation, the middle depicts global playground activation, and the rightmost figure represents the difference. EEG data are not very locationally precise, so these reconstructions must be regarded as very approximate, as emphasized by Sergent and colleagues [51]. Trends in Cognitive Sciences, Month 2024, Vol. xx, No. xx 7Mind wandering Trends in Cognitive Sciences This result appears to support the prefrontalists since the global playground activation included a PFC component. However, there was another key result. Recall that half the sessions were ‘active’ in which subjects were asked to identify and report the audibility of the vowel. In the ‘passive’ sessions, there were four different tasks, one or another of which was substituted for the vowel task, although the vowels were still presented. The four tasks were (i) a visual task (detecting a large green circle); (ii) a multiple-choice task on arithmetic, general information, and other topics; (iii) pressing a ‘click to continue’ button; and (iv) finally, the task of interest to us, a mind wandering probe. In the mind wandering probe, subjects were asked (in French), ‘What is on your mind just now?’ with four options: (i) ‘the sound,’ (ii) ‘my thoughts,’ (iii) ‘the task,’ and (iv) ‘nothing/I feel sleepy.’ Most of the subjects’ responses in the mind-wandering task reflected the visual task or ‘my thoughts.’ Only 19% of the responses were ‘the sound,’ with 16% responding ‘the sound’ at the lowest intensity (signal-to-noise ratio) and 33% at the highest. So, even at the highest audibility level, most reports were not ‘the sound.’ Here is the result of interest: when there was global playground activation linked to the sound, subjects tended to give ‘the sound’ answer to the mind-wandering probe. The authors conclude (p. 11), ‘…with the bifurcation model of conscious access, we could use neural activity to predict whether participants were spontaneously aware of sounds in a passive listening condition.’ This result shows that global playground activation predicted awareness of the sound because participants wouldnot bethinking aboutthe sound if they had notheardit.But thisresult also suggests that it may be cognition of the sound that was responsible for the global playground activation rather than the consciousness of the sound itself. Global playground activation predicts mindwandering reports of the sound on their mind. Reports most directly reflect cognition, whereas conscious perception of the sound is inferred from cognition of the sound. Given that the reports most directly reflect cognition of the sound, and we know that the PFC is the home of cognition, we must accept the possibility that the global playground activations reflect, entirely or in part, cognition rather than conscious perception. Some cognitions– for example, automatic cognitions perhaps including thoughts about the sounds– may be regarded by cognitive theorists as a component of conscious experience, but the bored monkey problem does challenge them to defend that view. This result is enough to show that the bored monkey problem is real, but we can go further. Note that subjects could also have been aware of the sound when they reported ‘other things’ as on their minds, such as the arithmetic or the green circle. Those of us who teach in New York City have had the experience of teaching a class or attending a lecture while right outside the window there is a loud garbage truck. Of course, some may be able to gate out the sounds, but others of us have had the experience that we would report as being painfully aware of the garbage truck while thinking about how to answer a question. If I choose to report thinking about how to answer a question instead of the sound of the garbage truck, it may be because the answering task seems more respectable or is more salient to me at the moment, not because I am unaware of the garbage truck. Needless to say (but I will say it anyway), we can all think of many ways in which the reports could be wrong or could mislead us. But we should take reports seriously. If we do, we have a further reason for thinking the bored monkey problem is real. Importantly, the signal-to-noise ratio of the sounds was varied in the experiment, and,whenit was very low, subjects were only 50% accurate (i.e., at chance) in the active condition. However, in a 8 TrendsinCognitive Sciences, Month 2024, Vol. xx, No. xxTrends in Cognitive Sciences crucial band of signal-to-noise ratios, subjects were roughly 95% accurate in identifying the vowels in the active condition, even though at this signal-to-noise range, they were mostly not giving ‘the sound’ response. So, subjects were reporting that arithmetic or the green circle or the answers to questions were on their minds, even when the sounds were clearly identifiable at least when they were task-relevant. Thus, reports of arithmetic or the green circle as being ‘on my mind’ in the mind-wandering task may be accompanied by background awareness of the sounds, just as awareness of the garbage truck can remain part of the conscious background of my thoughts about answering the question. So, there may have been awareness of the sound even without global playground activation. Concluding remarks Returning to the topic of decoding perceptual content from the PFC, we cannot rule out the possibility that the impressive intracranial decoding from the PFC might reflect in part the contents of postperceptual cognition rather than conscious contents. This fact indicates the ‘bored monkey’ problem is real, suggesting caution about overinterpreting decoding binocular rivalry contents in monkeys (but see Outstanding questions). Decoding from the PFC has played an outsize role in debates about the neural basis of consciousness, but it is doubly mistaken. First, failures of decoding from the PFC are harmless for theories of consciousness to the extent that pointer versions of those theories work. Second, successes in decoding from the PFC can reflect thought processes based on the stimuli rather than conscious perception of them.