Here’s a transcription of an excellent talk by Benjamin H. Bratton at Postopolis LA. Though it doesn’t take music or cognitive science as its primary subject, it identifies some cultural currents important to the social and political positions of both fields.

And here’s coverage of some interesting but deeply problematic research regarding the universality of musical emotion. The researchers had subjects who claim never to have heard Western music classify the emotionality of songs by pointing at pictures of faces expressing one of three emotions: happiness, sadness, and fear. I have not yet read the paper, but the media coverage is characteristically enthusiastic and uncritical, using such misleading and awful headlines as Feel-Good Music Feels Good Around the World and Language Of Music Really Is Universal, Study Finds.

Update: Decent coverage of the Mafa music study at Cognitive Daily.

Simulation theories of emotional understanding posit the recognition of emotion necessarily involves some kind of internal simulation of that emotion. However, these internal simulations don’t necessarily have the same phenomenology as feeling (you can’t see it, but right here I am making a hand gesture to buttress my point!) elicited in a more direct way. The musical examples from Viellard’s library are a good example; they are (deliberately, rightly) cliché, boring, stripped of their expressive qualities. For the most part, I find myself recognizing the emotion I believe the music attempts to convey (with some exceptions, but as a composer I’m not an ideal candidate for this sort of thing), but experiencing nothing like emotion at all. Provisionally, I think this is a good thing. Music which elicits emotion rather than referencing it is slippery, different for different people, dependent upon context, maybe impossible to isolate and bottle. It’s nice to have a collection of musical examples which have been empirically assessed as accurately conveying a basic set of emotions, evocative sterility notwithstanding.

One fact which leaves me a little unsettled, mostly because these clips don’t cause me to experience any emotion, is that subjects performed better on the emotion identification task when they were told to focus on emotional experience instead of recognition:

“A significant effect of Instruction [i.e. the instruction to focus on experience versus recognition], F(1,37)=4.97; p=.032; h=.118, was observed, with a higher rating for the intended emotion in the experience condition (from .82 to .91 across emotional categories) than in the recognition condition (from .76 to .84 across emotional categories).”

I find that result very strange.

Another issue, somewhat less troubling, but still a bit problematic: A forced choice paradigm was used for categorization of the stimuli. For each stimulus, subjects were told to apply as many of the labels “happy”, “sad”, “scary”, and “peaceful” as they wanted, and the best label of those four was determined. Each of the stimuli were composed with one of those labels in mind, and so the intention of the composer was validated by the categorization task, but in a pretty weak way. What if, for example, an additional label had been allowed: “angry”? Would the categorization task then have clustered the stimuli into five groups, despite the compositional intention of conveying one of four emotions? It’d be nice to see a freer labeling procedure, it would make the library much more powerful.

References:

• Vieillard, S., Peretz, I., Gosselin, N., Khalfa, S., Gagnon, L. & Bouchard, B. (2008) Happy, sad, scary and peaceful musical excerpts for research on emotions. Cognition and Emotion, Vol. 22, Issue 4. 720-752.

Melissa Saenz and Christof Koch at Caltech offer up the first lab-tested reports of visual-audio synesthesia. They’ve found four people who hear sound when they see motion or flashes of light. The sounds they hear are “simple” (I’m not sure what that means) beeps, taps, and whirrs. No sound is perceived due to eye movement, so it really does seem to be triggered by the perception of stuff-tagged-as-motion, and not something lower-level, right off the retina.

To validate the claims of synesthesia, pairs of short rhythms were played to subjects, either as visual flashes or auditory beeps. Subjects reported whether they thought the rhythms in each pair were the same or different. Typically people are quite good at identifying auditory rhythms and bad with visual ones. As expected, everybody did well on the auditory task, but only the synesthetes could accurately compare the visual rhythms, presumably because they could hear them. Interestingly, over the course of the experiment, the synesthetes reported that the synesthetic sounds they heard along with the visuals changed to match the real sounds played during the auditory tests.

A neat direction to go with this would be to play various visual stimuli for the synesthetes and collect phenomenological descriptions of the sounds heard, with the aim of mapping the visual-audio mapping. Are the sounds similar in dynamic profile to the visuals which trigger them? What causes the sounds to change even as the visuals stay the same, as they did during the experiment? What are the intersubjective differences in the synesthetic sounds?

References:

• Saenz, M.; Koch, C. The sound of change: visually-induced auditory synesthesia. Current Biology (2008) vol. 18 (15) pp. R650-R651

Fitch begins by providing his interpretation of Dennett’s thought in The Intentional Stance. He sees Dennett’s work as a full-force attack on the possibility of intrinsic intentionality (all citations Fitch 2007):

The thrust of Dennett’s argument is that this “something” (which he terms variously “original” and “intrinsic” intentionality) is an illusion, a deep philosophical mistake that derives from our unwillingness to fully bite the bullet and accept natural selection (and its products, especially ourselves) for the blind and goal-less process that it is. Dennett criticizes intrinsic intentionality with arguments that need to be taken seriously, arguments which seem to force us into a corner where we must choose between the following propositions: either accept that all intentionality, including our own, is derivative (and then admit that a thermostat has a little bit of intentionality, too), or we retreat from this unpalatable option into a mysterian, outmoded belief in “original/intrinsic intentionality”, a concept that under close scrutiny leads to contradiction and paradox at every turn.

At Neukom, Dennett says Fitch misinterpreted his work on intentionality, but doesn’t explain exactly how. Regardless, he thinks Fitch is right, and he’s on board with nano-intentionality and its implications.

From the beginning of his paper, Fitch focuses on a key difference between living things and machines: the ability of living things, especially small ones like cells and neurons, to adapt to local circumstances by changing their shape. Formal changes act as both a kind of memory and a way in which the cell is “about” its environment. This inherent, formal aboutness is nano-intentionality. Fitch argues that the kind of inherent intentionality we attribute to minds is a consequence of the nano-intentionality of cells and neurons. “When combined properly into large interconnected systems, this combined mass-action of cellular nano-intentionality yields intrinsic intentionality in the typical philosopher’s sense, as well as both consciousness and the efficacy of our subjectively sensed self to move the body and perform other acts of will (‘intentionality’ in its general English sense), just as the mass and velocity of gas molecules contained in a volume are constitutive of its temperature, pressure and weight.”

Here is how the argument evolves: Start with an amoeba. We understand much of how amoebas work on a mechanical level, and we know they came to behave this way because of natural selection. This is Fitch’s “prototypical example of [...] nano-intentionality”. The “purpose” of an ameoba is “realized via its physical form: it is a complex arrangement of matter serving to do useful things like find food and avoid toxins”, etc. Critically, an ameoba can deal with novelty using trial and error, “remembering” successful strategies by modifying its form. In contrast, a thermostat can only interact with what it was designed to interact with; it is only “about” what it was designed to be “about”. But an ameoba is about its whole environment, a world of things.

In multi-celluar organisms, groups of specialized nano-intentional cells cooperate, creating intentional subsystems which are “about” different things. Fitch’s example is the sponge. Each subsystem in the sponge is “about” something different, like creating efficient water flows or digesting. Fitch argues that the intentionality of these groups allows the sponge to be intentional itself, independent of an observer taking Dennett’s intentional stance. A sponge is “about” filter-feeding because its constituant parts allow it to filter feed, because the sponge evolved that way. “The aboutness of the sponge is reflected in its structure, but is constituted by its evolutionary history: the story of sponge-kind.” I.e., the way in which individual cells are “about” their environment changes when their environment contains other, collaborating cells: a kind of group intentionality emerges. The same is true on the next level of abstraction: collaborating groups of groups of cells like a sponge.

Next Fitch looks at jellyfish. They have a basic nervous system, a nano-intentional subsystem which is about information processing. In the nervous system, neurons don’t just collaborate with other neurons to perform a simple task like flaggellation, they interact with each other, such that structural changes in one cell are changes in the environment of the others. Indeed, a change in the nervous system of an organism is a change in the environment of many other cells in that organism, neurons or not. Neurons are “‘about’ amplification of information into locally adaptive patterns of action”. Neuronal events, however, are about whatever environmental change triggered the event. The organism as a whole can now react to the world around it. Fitch calls systems with this capacity “proto-mental.”

Fitch goes all the way up. The nervous system evolves from simply reacting to environmental changes, to representing those changes, and eventually to acting as a generative model, a source of ideas about “possible worlds”. Neuronal assemblages which are successful, i.e. whose representations correspond to actions actually taken by the organism in the world, are “tagged” by other neuronal assemblages, and from this emerges the serial experience of consciousness. Here we have intentionality in the philosopher’s sense, and a theory about how it arises. Fitch concludes that any system which has this kind of possible-world-tagging system must have some subjective experience, some awareness:

“To the extent that this argument is correct, awareness is the intrinsic subjective side of an objectively-verifiable capability of some types of nervous system to both entertain multiple hypothesis at a given time, and to later learn from their mistakes and successes. This leads to a rather strong and surprising thesis: that any nervous system objectively capable of considering and choosing among mental options, and of learning from its past decisions, will have at least a little bit of awareness as it does so: a little bit of (serial) consciousness.”

Fitch is, essentially, biting the natural-selection bullet offered by Dennett. But rather than looking for ways in which some internal mind could interface with an external world, he shows us how mind and world are inextricably related, and how directedness emerges from this relation.

The paper also includes an interesting discussion of how this way of thinking will affect artificial intelligence research, among other things.

References:

Dennett, D. How Mindless Algorithms Build Minds. May 9, 2008. Online video clip, accessed on July 21, 2008. http://neukominstitute.com/index.php/site/feature/dennett_talk/symposia/49/symposium08

Dennett, D. (1987) The Intentional Stance. MIT Press.

Fitch, W.T. (2007) Nano-Intentionality: A Defense of Intrinsic Intentionality. Biology & Philosophy, Vol. 23: 157-177.

Dennett’s talk takes as its starting point the claim that the brain is a computer, and to support this he introduces a distinction between cooperative and competitive computation—corresponding (funnily enough) to a distinction between brains and personal computers of the sort we use every day. Within a personal computer there are a number of processes which are systematically assigned resources used to accomplish goals. The brain is different: we don’t have the same kind of resource allocation protocol. Inside of us numerous, massively parallelized processes engage in heated, vicious, life-or-death competition for resources with which they may or may not accomplish their goals. According to Dennett the resource allocation paradigm of the personal computer underpins our intuitive, “default model” of computation—but the “serious, deadly” competition of cells, neurons, and higher level processes for resources in the brain is computational as well, and of great importance to the understanding of consciousness.

This represents a significant update to Dennett’s multiple-drafts/fame-in-the-brain model of consciousness. Very, very briefly—I can not do this idea justice in a single sentence—there is no one place in the brain where consciousness arises (e.g. Descartes’s pineal gland), but we become conscious of things when useful information about them is shared between various, independently operating cognitive modules (which are somewhat chaotic themselves, a la Selfridge’s pandemonium). The latest addition to this model is that all of these different units, or their component parts, are competing for resources, essentially working to starve competing processes out of existence. Much of Dennett’s talk focuses on competition between high-level, abstract units, like metaphors, turns of phrase, or memes. According to this theory, ideas compete for resources, and the ideas we express are the ones which “win”, starving their competitors into oblivion. This is where I diverge from his talk, which expands upon the computer metaphor, discussing the mind as software, ideas as virtual machines, and the implications of competition in the brain for normal and abnormal psychology, sociology, etc. I’m going to go in the opposite direction, into the low-level, micro-scale, really dirty stuff.

How might this competition in the brain really play out on lower levels, such as that of the cell, the synapse, or the neuron? Dennett refers us to Fitch’s paper on nano-intentionality. In order to make sense of nano-intentionality, we’ve got to understand normal-sized intentionality first, an area in which Dennett has been a major player.

Intentionality is a tricky subject. It is the “aboutness” of things; their directedness. One of the hot areas of contention in modern philosophy is the means by which things acquire directedness, especially when those things are minds or thoughts. There is some consensus that there are at least two basic types of intentionality: derived and intrinsic. For example, there is clearly a difference between how a word acquires its directedness—by definition and consensus, from the intentions of the word’s designers—and the way a thought seems to have a kind of intrinsic, original intentionality. We don’t need to define the meaning of a thought about something, in fact the very idea sounds ridiculous. The target of a thought seems somehow inextricably wrapped up in the form of the thought itself, long before the definitive power of language can get inside and fiddle with it. Another canonical example is the difference between a thermostat and the human heart. The thermostat is designed; it is “about” the ambient temperature by virtue of the intentions of its designer. What is the human heart about? Pumping blood? How? Where does this aboutness come from? Is it derived, or intrinsic? And, further, if there is intrinsic intentionality, how exactly does it arise? That last one is a killer.

In The Intentional Stance Dennett suggests that the whole discussion of derived/intrinsic intentionality is pretty confused, and what we are actually doing when we assess intentionality is using one of three tools: the physical, design, or intentional stances. When we take the physical stance, we are evaluating how something operates in a reductive, low-level way. When we take the design stance, we determine the possible function of an item and the reason for its design. Finally, when we take the intentional stance we are speculating about the conscious state of an item; its goals, desires, emotions, what it knows or does not know. The design stance roughly corresponds to assessment of derived intentionality. But where does intrinsic intentionality go? We can adopt the intentional stance toward a thermostat just as well as we can adopt it toward a person. It turns out we do this all the time, referring to inanimate objects as if they have belief and desire. The thermostat “knows” the ambient temperature and “wants” to keep it within a certain range.

The apparent intrinsic intentionality of mental phenomena, then, is just a useful folk-psychological metaphor we use to understand each other, and doesn’t correspond to a real, extant thing. In Fitch’s words, Dennett wants us to “bite the bullet and accept natural selection (and its products, especially ourselves) for the blind and goal-less process that it is.” (Fitch 2007) If we have some sort of intentionality, it is derived from the process of natural selection, the same kind of intentionality possessed by a thermostat. This doesn’t sit right with people, it either seems to ascribe too little to the mind or too much to the thermostat.

Fitch tries, I think successfully, to resurrect intrinsic intentionality. Dennett says he’s on board with Fitch’s approach. Details later this week.

References:

Dennett, D. How Mindless Algorithms Build Minds. May 9, 2008. Online video clip, accessed on July 21, 2008. http://neukominstitute.com/index.php/site/feature/dennett_talk/symposia/49/symposium08

Dennett, D. (1987) The Intentional Stance. MIT Press.

Fitch, W.T. (2007) Nano-Intentionality: A Defense of Intrinsic Intentionality. Biology & Philosophy, Vol. 23: 157-177.

• Multiple drafts model of consciousness
• Sine-wave speech
• Synesthesia
• Cybernetics
• Models of emotion
• Magnetic resonance imaging
• Functional magnetic resonance imaging

The fMRI article makes no mention of recent pattern classification/MVPA techniques… but neither does the fMRI Wikipedia article. Still room to grow!

About the application of pattern-classification techniques (e.g. MVPA) to fMRI data, Logothetis says the following:

In humans, fMRI is used routinely not just to study sensory processing or control of action, but also to draw provocative conclusions about the neural mechanisms of cognitive capacities, ranging from recognition and memory to pondering ethical dilemmas. Its popular fascination is reflected in countless articles in the press speculating on potential applications, and seeming to indicate that with fMRI we can read minds better than direct tests of behaviour itself. Unsurprisingly, criticism has been just as vigorous, both among scientists and the public. In fact, fMRI is not and will never be a mind reader, as some of the proponents of decoding-based methods suggest, nor is it a worthless and non-informative ‘neophrenology’ that is condemned to fail, as has been occasionally argued.

A lot of this has to do with how one defines “mind-reading,” and whether or not what’s being read is low-level (e.g. the orientation of a visual stimulus) or high level (e.g. the presence of a mental state). There are numerous examples of papers which exploit fMRI data to erroneously support what I would call very high-level conclusions about human behavior, such as whether or not subjects have anxiety regarding Mitt Romney. (This and many other such studies are handily refuted by The Neurocritic, who also had some interesting commentary on Logothetis’s paper.) However, I think the study by Kay et al. on the identification of novel, real-world images suggests that, at lest for low-level, sensory information, fMRI might be quite the mind-reader indeed. There’s a good editorial in Science by Greg Miller which has, I think, a pretty balanced view of things, resorting to neither abject pessimism nor fanciful speculation.

The risk I think Logothetis and others are pointing out is that inferring a mental state from brain imaging data puts one on pretty shakey ground. I think this is especially true for studies which don’t take into account distributed patterns of brain activation, but even pattern-classification based studies need to be very careful about making the leap to state attribution. Just because we can say, in a sense, that there is a hierarchy inside a rhesus macaque’s brain that differentiates between cats and dogs, we can’t necessarily say that the monkey knows what a dog is and that it has a degree of similarity to a cat equal to X. It’s possible that the observed hierarchical organization might be completely unconscious.

Logothetis gets specific about his reservations regarding pattern classification techniques:

Such multivariate analyses or pattern-classification-based techniques (decoding techniques) can often detect small differences between two task or stimulus conditions—differences that are not picked up by conventional univariate methods. However, this is not equivalent to saying that they unequivocally reveal the neural mechanisms underlying the activation patterns.

And he’s right. But the strength of pattern classification of fMRI data is not that it can reveal underlying neural mechanisms, nor that it can simply detect small differences between patterns of activation (cheating past the spatial resolution limit, basically), but that it allows us to evaluate the similarity of patterns of activiation based on a variety of arbitrary criterea. It’s not cool because it gets us closer to some underlying machinery, it’s cool because it lets us talk about similarity and difference of patterns of activation in a quantitative way. This has surprising power.

References:

• Logothetis, N.K. (2008) What we can do and what we cannot do with fMRI. Nature 453:869-78.
• Kay, K.N.; Naselaris, T; Prenger, R.J.; Gallant, J.L. (2008) Identifying natural images from human brain activity. Nature 452: 352-356.

Aniruddh Patel got on the case, did some experiments and wrote a paper on the cockatoo. He concludes that, while it’s confined to a relatively small range of tempos and isn’t particularly reliable, the dancing bird is basically for real. The paper is a great read. Its final sentence is oddly touching:

It will be interesting to determine whether the range of tempi to which Snowball can synchronize is expanded when dancing with a partner.

Here’s the paper:

• Patel, A.D., Iversen, J.R., Bregman, M.B., Schulz, I., & Schulz, C. (in press). Investigating the human-specificity of synchronization to music. In: Proceedings of the 10th International Conference on Music Perception & Cognition (ICMPC10), August 2008, Sapporo, Japan. M. Adachi et al. (Eds.), Adelaide: Causal Productions.

More videos on Patel’s publications page.

Some of my thoughts and questions about MVPA follow.

We can use MVPA to hierarchically cluster patterns of neural activation into groups which seem to match up pretty well with our common-sense understanding of categories in the world. Does the MVPA-based discovery of category structure imply knowledge on the part of the subject? Because we can see that a rhesus macaque’s brain responds with a distinct and recognizable pattern of activation when shown various pictures of cars, can we say that it has some knowledge of what a car is? How well correlated are MVPA-derived categorical structures with the reports of subjects regarding their own ideas about categories?

It may be possible to learn something about the acquisition of category discrimination from MVPA. One can learn to perceive things which initially appear imperceptible. It’d be fascinating to see how the categorical perception of different species evolved in terms of patterns of neural activation in a biologist-in-training; or to watch the ability to discriminate chords and their functional associations develop in a music student. Similarly, MVPA may tell us a lot about neural representation and memory.

Kay et al. suggest that it might be possible to reconstruct what a person is seeing from brain activity data alone (Kay et al. 2008). This may shake up debate about the nature of representation within the brain. All of these MVPA studies seem to vindicate a connectionist model of representation, where distributed patterns of activation across a neural network account for the content of mental states.

Most of the MVPA work I’ve discussed focuses on visual experiences as they occur to the subject in the moment. Presumably the brain contains some ephemeral, basically lossless representation of data being transmitted from the retina. Kay et al. used spatial location, orientation, and spatial frequency of as the dimensions of the analysis space in which they compared the similarity of patterns of activation (Kay et al. 2008). That these variables are useful in reconstructing information about the sensory experience of subjects does not necessarily mean that the subjects have conscious access to them. However, if the project of complete reconstruction of visual input from brain activity data is successful, it would seem likely that the parameterization used was at least a good approximation of the terms in which the part of the brain in question utilizes or perhaps makes available relevant information. If a successful model of the perception of syntax in natural language were created using MVPA techniques, it might provide some significant insight into how the brain acts as an information processor.

I think MVPA may also make some progress towards filling the “explanatory gap” claimed by some philosophers as an obstruction to the scientific understanding of the phenomenal nature of experience. For example, if we can use MVPA to examine the relative similarity of massively distributed—perhaps even whole-brain—activation patterns, and classify them in terms of qualitative, phenomenal reports from subjects, we may find a convincing neural correlate for “what it is like” to experience some fairly tricky things, e.g. what it is like to be in love, or to tell a lie. Having a love detector doesn’t necessarily explain why being in love feels the way it does, but comparing that pattern of activation to others, both similar and drastically dissimilar, might have some more explanatory power. Quite fancifully, for example, perhaps the pattern of activation which correlates with being in love is somewhere on a continuum between being hungry and looking at flowers. This is a little better than “it feels that way because it does.”

References:

• Norman, K.A.; Polyn, S.M.; Detre, G.J.; Haxby, J.V. (2006) Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Science Vol. 10, No. 9: 424-430.
• Kay, K.N.; Naselaris, T; Prenger, R.J.; Gallant, J.L. (2008) Identifying natural images from human brain activity. Nature 452: 352-356.