Curve and lattice

A fantastic YouTube video has been making the rounds. It’s a cockatoo named Snowball dancing to The Backstreet Boys. It’s all kinds of awesome, but it’s also one of, if not the first, totally conclusive demonstration of an animal enthusiastically entraining its movements to rhythmic sound. Here’s the bird:

Aniruddh Patel got on the case, did some experiments and wrote a paper on the cockatoo. He concludes that, while it’s confined to a relatively small range of tempos and isn’t particularly reliable, the dancing bird is basically for real. The paper is a great read. Its final sentence is oddly touching:

It will be interesting to determine whether the range of tempi to which Snowball can synchronize is expanded when dancing with a partner.

Here’s the paper:

• Patel, A.D., Iversen, J.R., Bregman, M.B., Schulz, I., & Schulz, C. (in press). Investigating the human-specificity of synchronization to music. In: Proceedings of the 10th International Conference on Music Perception & Cognition (ICMPC10), August 2008, Sapporo, Japan. M. Adachi et al. (Eds.), Adelaide: Causal Productions.

More videos on Patel’s publications page.

Obviously MVPA has a really interesting future. As neuroimaging technology improves spatial and temporal resolution, we’ll get finer distinctions and more detailed pictures of neural representation. Norman et al. have a fantastic overview of MVPA, including numerous interesting questions and speculations about its future, called Beyond mind-reading: multi-voxel pattern analysis of fMRI data.

Some of my thoughts and questions about MVPA follow.

We can use MVPA to hierarchically cluster patterns of neural activation into groups which seem to match up pretty well with our common-sense understanding of categories in the world. Does the MVPA-based discovery of category structure imply knowledge on the part of the subject? Because we can see that a rhesus macaque’s brain responds with a distinct and recognizable pattern of activation when shown various pictures of cars, can we say that it has some knowledge of what a car is? How well correlated are MVPA-derived categorical structures with the reports of subjects regarding their own ideas about categories?

It may be possible to learn something about the acquisition of category discrimination from MVPA. One can learn to perceive things which initially appear imperceptible. It’d be fascinating to see how the categorical perception of different species evolved in terms of patterns of neural activation in a biologist-in-training; or to watch the ability to discriminate chords and their functional associations develop in a music student. Similarly, MVPA may tell us a lot about neural representation and memory.

Kay et al. suggest that it might be possible to reconstruct what a person is seeing from brain activity data alone (Kay et al. 2008). This may shake up debate about the nature of representation within the brain. All of these MVPA studies seem to vindicate a connectionist model of representation, where distributed patterns of activation across a neural network account for the content of mental states.

Most of the MVPA work I’ve discussed focuses on visual experiences as they occur to the subject in the moment. Presumably the brain contains some ephemeral, basically lossless representation of data being transmitted from the retina. Kay et al. used spatial location, orientation, and spatial frequency of as the dimensions of the analysis space in which they compared the similarity of patterns of activation (Kay et al. 2008). That these variables are useful in reconstructing information about the sensory experience of subjects does not necessarily mean that the subjects have conscious access to them. However, if the project of complete reconstruction of visual input from brain activity data is successful, it would seem likely that the parameterization used was at least a good approximation of the terms in which the part of the brain in question utilizes or perhaps makes available relevant information. If a successful model of the perception of syntax in natural language were created using MVPA techniques, it might provide some significant insight into how the brain acts as an information processor.

I think MVPA may also make some progress towards filling the “explanatory gap” claimed by some philosophers as an obstruction to the scientific understanding of the phenomenal nature of experience. For example, if we can use MVPA to examine the relative similarity of massively distributed—perhaps even whole-brain—activation patterns, and classify them in terms of qualitative, phenomenal reports from subjects, we may find a convincing neural correlate for “what it is like” to experience some fairly tricky things, e.g. what it is like to be in love, or to tell a lie. Having a love detector doesn’t necessarily explain why being in love feels the way it does, but comparing that pattern of activation to others, both similar and drastically dissimilar, might have some more explanatory power. Quite fancifully, for example, perhaps the pattern of activation which correlates with being in love is somewhere on a continuum between being hungry and looking at flowers. This is a little better than “it feels that way because it does.”

References:

• Norman, K.A.; Polyn, S.M.; Detre, G.J.; Haxby, J.V. (2006) Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Science Vol. 10, No. 9: 424-430.
• Kay, K.N.; Naselaris, T; Prenger, R.J.; Gallant, J.L. (2008) Identifying natural images from human brain activity. Nature 452: 352-356.

I had the pleasure of attending the Neukom Institute’s 2008 symposium at Dartmouth College, “The Human Algorithm.” Videos of the symposium are now online, so I’m going to pull out my notes and do some follow-up on on what I thought were the most interesting presentations. The sleeper hit of the conference was James Haxby’s “The Minority Report: Update 2008″, a summary of recent developments in multi-voxel pattern analysis (MVPA), which is essentially the application of pattern classification algorithms to fMRI data.

The use of MVPA represents a bit of a paradigm shift in neuroimaging research. We can do some pretty serious mind-reading. This gets a little geeky, but bear with me, it’s worth it.

With MVPA, rather than just looking at hot spots in the data, like single activated voxels or contiguous groups of voxels, distributed patterns of activation in a region are considered. Haxby begins the talk by discussing an experiment of his where subjects in the scanner were shown images of faces, houses, shoes, and other objects. The resultant dataset was split in half, and a pattern classification algorithm was run on each subset. Haxby et al. then demonstrated that the patterns of activity for each category in both subsets matched the other, e.g. shoe-related patterns from one half matched shoe-related patterns from the other half better than patterns representing different categories. Given a pattern without a label, they were able to guess its category with 90% accuracy (Haxby et al. 2001).

This is really neat. It’s basically mind-reading. If you’re in an fMRI scanner, and you’re shown a picture of a shoe, and they’ve mapped out your brain a little bit, they can confidently say “You’re looking at a shoe.”

In 2005, Kamitani & Tong pushed it even further. They looked at V1, the primary visual cortex, and showed subjects images of parallel lines with various angles of orientation. Using MVPA, they identified patterns of activation associated with with each given angle (Kamitani et al. 2005). The voxel resolution was 3 cubic millimeters, and multiple orientation-sensitive cells exist within any given voxel. The success of this experiment shows that with MVPA we can cheat and examine patterns of neural activity in finer detail than the fMRI voxel resolution. Researchers at UC Berkeley used a similar model of orientation-sensitive areas in V1, but showed subjects photographs of real-world scenes. They were able to predict what images people were looking at with 80% accuracy (Kay et al. 2008). When these models make mistakes, they make pretty smart mistakes, never choosing, for example, an angle orthagonal to the correct answer.

Further, MVPA allows for hierarchical clustering of similar patterns of activation. Analysis of the data from Haxby’s 2001 study showed that patterns associated with faces, for example, are all more similar to each other than patterns associated with houses. They built a similarity tree, showing human faces were represented similarly to cat faces, inanimate objects were represented dissimilarly from animate objects, and so on (Hanson et al. 2004). This hierarchy of cognitive representational similarity was replicated in a test of 600+ individual neurons of a macaque, finding that the monkey made similar categorical distinctions between animate and inanimate, mammal and reptile, etc (Kiani et al. 2007).

(image from Kiani et al. 2007)

That’s it for now. I’m going to discuss some of the implications MVPA and mind-reading research further in a future post.

References and papers mentioned in Haxby’s talk:

• Haxby, J.V.; Gobbini, M.I.; Furey, M.L.; Ishai, A.; Schouten J.L.; Pietrini, P. (2001). Distributed and Overlapping Representations of Faces and Objects In Ventral Temporal Cortex. Science 293: 2425-2430.
• Kamitani, Y.; Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience 8: 679-85.
• Haynes, J.; Rees, G. (2005) Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nature Neuroscience 8: 686-691.
• G.M. Boynton. (2005) Imaging orientation selectivity: decoding conscious perception in V1. Nature Neuroscience 8: 541-542.
• Kay, K.N.; Naselaris, T; Prenger, R.J.; Gallant, J.L. (2008) Identifying natural images from human brain activity. Nature 452: 352-356.
• Hanson, S.J.; Matsuka, T.; Haxby, J.V. (2004) Combinatorial codes in ventral temporal lobe for object recognition: Haxby (2001) revisited: is there a “face” area? NeuroImage 23: 156-166.
• O’Toole, A.J.; Jiang, F.; Abdi, H.; Haxby, J.V. (2005) Partially Distributed Representations of Objects and Faces in Ventral Temporal Cortex. Journal of Cognitive Neuroscience 17: 580-590
• Kiani, R.; Esteky, H.; Mirpour, K.; Tanaka, K. (2007) Object Category Structure in Response Patterns of Neuronal Population in Monkey Inferior Temporal Cortex. Journal of Neurophysiology 97: 4296-4309.
• Norman, K.A.; Polyn, S.M.; Detre, G.J.; Haxby, J.V. (2006) Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Science Vol. 10, No. 9: 424-430.