Neural Decoding of Visual Imagery during Sleep

In April 2013, a paper about decoding dreams from the neural activity patterns during sleep, has been published on Science Magazine^*1. The research group of Dr. Horikawa has presented that it is possible to read out what the experiment participants saw in their dreams using the neural signals. These neural signals are gathered by utilizing fMRI (functional magnetic resonance imaging) method, which observes the activities of the participants’ brains. And this time, we would like to post on NAIST Edge the content of the interview with the first author of the paper, Dr. Horikawa Tomoyasu.

^*1) T. Horikawa, M. Tamaki1, Y. Miyawaki, and Y. Kamitani. Neural Decoding of Visual Imagery During Sleep. Science 3 May 2013: Vol. 340 no. 6132 pp. 639-642.
http://www.sciencemag.org/content/340/6132/639.abstract

Dr. HORIKAWA Tomoyasu is a researcher at Department of Neuroinformatics, ATR Computational Neuroscience Laboratories. After graduating from the Department of Psychology, Faculty of Letters, Arts and Sciences, in Waseda University, he continued his study at Computational Neuroscience Laboratory (Co-operating laboratory in Nara Institute of Science and Technology). Now his major is to develop neural information decoding.

 

Question: At first, please tell us about the reason why you pursued the research field that studies our brains.

Dr. Horikawa: In my undergraduate school, I studied about psychological statistics in the Faculty of letters. From that time, I had interests in the research topics of Cognitive psychology, such as memory, vision, etc.  In addition, I read some books written by Oliver Sacks^*2, Ramachandran^*3, Steven Pinker^*4 and got excited about the relations of consciousness, mind and brain hence, I decided to study about neuroscience.

I realized that it’s difficult to know about minds no matter how hard I try to research on neurophysiology. While I was really confused, I heard about the idea of neural computational theory and the neural information decoding. Neural information decoding is a method to read out neural information based on statistical analysis of neural activities. This is intensively researched by Professor Kamitani Yukiyasu (the 4th author in the paper, supervisor). The idea of using statistical method to scientifically study our brain matched both my interests and what I studied before, and I found it really interesting hence I decided to choose this field.

^*2) Oliver Sacks: Physician and professor of neurology at the NYU School of Medicine. The author of “Awakenings,” “The Man Who Mistook His Wife for a Hat,” “An Anthropologist on Mars,” etc.
^*3) Vilayanur S. Ramachandran: Director of the Center for Brain and Cognition and Distinguished Professor with the Psychology Department and Neurosciences Program at the University of California, San Diego. The author of “Phantoms in the Brain’’ etc.
^*4) Steven Pinker: Harvard College Professor and Johnstone Family Professor in the Department of Psychology at Harvard University. The author of “The Stuff of Thought: Language as a Window into Human Nature,” etc.

Q: I see. So, you have found a match between your interests and research work during your undergraduate in the field of computational neural science. By the way, could you share some overview of the research results?

Horikawa: Okay. In this paper, we measured activities of brains during sleep, then, analyzed the patterns and based on that, we succeeded in reading out dreams. Dream is a subjective phenomenon which happens during sleep. Since a dream is only visible for the person who dreams and can be forgotten quickly, it is thought to be very difficult to do any objective analysis. However, by the method we used in this research, it turned out to be possible to make objective estimation on such a subjective thing like dream. I think it is extremely useful to dive deeply into functional roles of dream, such as the relation between dream and learning.

In our experiment, we repeated the following procedure. We used fMRI to measure the neural activities during sleep and then woke up the participants. We asked them to tell about the dream that they had. Next, we built a pattern recognition model to predict about the things that are seen during awakening using neural activity patterns. These neural activity patterns are derived from showing different things to the participants. Applying this model to the neural activities during sleep, we have succeeded to predict visual contents of dream with high precision.

Furthermore, based on this research result, we also know that there are common neural activity patterns generated by seeing a dream and by seeing a real image.

“The video shows decoded contents during sleep, followed by a report from the subject describing what they saw.” (ATR)

Q: In Advanced Telecommunications Research Institute International (ATR), where you are working now, has made some achievements in reconstructing images based on human neural activity patterns^*5. Is your current research related to this?

^*5) Y. Miyawaki, H. Uchida, O. Yamashita, M. Sato, Y. Morito, H. C. Tanabe, N. Sadato, Y. Kamitani. Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron 10 December 2008: Vol. 60, Issue 5, pp. 915-929.

Horikawa: It’s a little bit different. In the previous research, they used neural activity patterns of seeing real pictures to reconstruct black and white contrast pattern image of size 10×10 pixels from the participants’ visual cortical activities. Before this research, only more than 10 categories of information can be decoded, however, through the research of reconstruction based on visual images, theoretically, it can achieve to read out about 2 to the power of 100 patterns. This was an innovative point of the previous research.

In fact, this research about dream decoding still has not reached to being able to reconstruct the picture that a person sees in his dream. We tried to reconstruct visual image from neural activities during sleep, but we only got random black and white images of size 10×10 pixels, and that was not enough to know about any specific feature of a dream. Therefore, similar to the researches so far, we study to see if it’s possible to know from neural activities during sleep about the categories of dreams, which is chosen and decided in advance. At the beginning, we tried to recognize very rough categories, living things and not living things, which are shared among the testers. But finally, we are able to recognize about 20 categories of objects and scenes for each participant, such as “car,” “male,” “street,” based on reports about dreams of the participants.

Q: I see. We know that the method used in this research is useful for reading out dreams, is it possible to use it in other fields?

Horikawa: I think this method is not only useful for reading out dreams, but also can work well for studying about spontaneous neural activities, or diagnosing neural diseases like hallucination.

Q: I see. It really can be applied widely. By the way, when did you start this research?

Horikawa: I was in my first year in the master course, about 5 years ago. My supervisor, professor Kamitani, has thought of doing research about reading dreams. Also my lab-mate, Ohnuki, who entered the same time with me in NAIST was interested in studying about dreams decoding. Given all of these, the research has started.

I still remember that at first stage, data collection was not as we expected, Ohnuki and researcher Dr. Miyawaki (the 3rd author in the paper) have worked very hard for building basics, such as, effective dream reports collecting method, participants’ sleep status deciding method, so on.

Ohnuki graduated in his 2nd year of the master course, so as a substitution, I joined the research project.

I was mainly responsible for analyzing data from fMRI. Dr. Tamaki (the 2nd author), who joined from the end of the 2nd year did sleep experiments and collected the data. Dr. Miyawaki did both data collection and analysis. That was how we casted. From the 3rd year, thanks to Dr. Tamaki, we can know whether a participant is already sleep or still awake from the collected data of brain waves as well as the time that a dream starts. Then we were able to get data effectively. So the data related to sleeps have been collected from 3rd year to 4th year. After that, in the end of try and fail cycles, we got results which can guarantee our capability of reading out dreams from the data. And in the spring of the 4th year, the result converged hence we wrote the paper about it.

Q: So this paper appeared in Science Magazine. What was the most difficult thing you faced during your research?

Horikawa: From the point of view of the research project, one of the biggest problems was to collect brain measured data during sleeps. I did not have such deep relation to this problem. The main contribution were from the co-researchers, Dr. Tamaki and Dr. Miyawaki. Thanks to Dr. Tamaki, we were able to confirm sleep status of the participants and effectively collect much information of dream reports. Consequently, I think to get appropriate data was really an important thing in this research.

I was deeply responsible for verifying whether or not the collected data relates to the dream, and I worked very hard for that.

Generally, I think the difficulty of analyzing data about living things is there is no guarantee to know if the collected data is really appropriate. From the beginning, I and Dr. Miyawaki were responsible for the analysis, because we did not know if we got useful result. Analysis was like searching in the dark, and to know whether there is appropriate date included in the collected data is an almost endless chain of trial and errors.

The research result was only something quite obvious. There are common neural activity patterns from a dream and when a participant sees a real image, however to achieve the result perfectly, we have tried very hard without knowing whether the analysis method or the collected data is bad. When we try to do various analyses, maybe we can get some useful statistical data. It is believable that one can get very useful result by chance when tried various analysis methods. Hence, it took much effort to confirm if it is really useful result.

Q: I see. So, what did you care about the most in doing research?

Horikawa: It is related to what I said before, the thing that I specially cared about is to not tell “lies” which seem to be good result made randomly. It is quite obvious, for instance, when we try 100 analysis methods, even there is no useful information, there may be still about 5 ones with statistical good results. However, this important research got extensive cares, and we cannot report lies. While we analyze data, there are many effects, which can lose the generality, to the results from outsides. We tried to avoid things like that, and tried to always do analysis in various conditions to confirm the significance of the results. It was really troublesome to not be fooled by the data.

Q: Lastly, could you talk about your future research goals and directions?

Horikawa: After publishing the research result, we got more feedbacks from general people than we thought. Through this research work, I found the importance of doing the research, which can be appreciated and said “How come such a result can be done,” by the people who even don’t care about science. I’m not sure if it is good to always do research on general level that everyone can understand, but I would like to do research that everybody, even don’t have interest in science can find it interesting.

This time, thanks to many people, especially my co-researchers, Dr. Tamaki, Dr. Miyawaki, Professor Kamitani, the paper was published in Science Magazine. However, it was also the chance for me to see my lack of experience. Hence, in the future, my goal is to do a research that is more interesting than the research about dreams this time, so that, if someone hears my name, will not say “Ah, that is the person who can read out dreams.”

Q: We are looking forward to your researches in the future. Today, thank you very much for spending time for the interview.

—————————————
How was that? Maybe in the near future we can create “dream observing device,” which was described in SF’s writings such as “Galaxy railway 999.” In the interview this time, besides the research, we also know how the research was started. I think it was very interesting interview for not only the people who are aiming at becoming a researcher in neurosciences but also others. Please wait to see NAIST Edge next time.