Movie reconstruction from human brain activity

This is a much cooler example [VIDEO] If I'm understanding the description correctly, they are just training it to recognize what the image is closest to and taking an slice of a youtube video that most closely matches it. I imagine if they used a more efficient method or trained it more, they could do way better. It seems like most of the data to build an accurate picture of what they are seeing is already there.

This is one year and something old and returns better results, in my opinion. [VIDEO] On the other hand, they presented only the best results in that video.

According to the abstract: Here, we present a neural decoding approach in which machine learning models predict the contents of visual imagery during the sleep onset period given measured brain activity, by discovering links between human fMRI patterns and verbal reports with the assistance of lexical and image databases. Decoding models trained on stimulus-induced brain activity in visual cortical areas showed accurate classification, detection, and identification of contents. So it sounds like the fMRI patterns are associated with words via verbal reports from humans, and the video is simply generated from image databases by using those words. In other words, they classify the fMRI patterns based on how well they correspond to various words (measured while the subjects are awake and giving verbal descriptions of images they're looking at), and then the program measures the subjects' fMRI patterns while they're dreaming, determines the strongest 'word signals' in the patterns, and selects images from a photo database corresponding to those words and assembles them into a video based on how strong the similarity score for each word is at each moment in time. It's interesting, but the video seems like more of a "reenactment" of a dream than a prediction. It doesn't look like you can infer actions and occurrences within a dream from this. But this is all just my interpretation. edit: This is a fundamentally different approach from the one used in other research (e.g. [VIDEO] me

Although it's hard to tell from the images presented with the article, the face generation looks like it could be similar to the techniques used in Nishimoto et al., 2011, which used a similar library of learned brain responses, though for movie trailers: [VIDEO] Their particular process is described in the YouTube caption: The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI. The procedure is as follows: [1] Record brain activity while the subject watches several hours of movie trailers. [2] Build dictionaries (i.e., regression models) that translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points at which brain activity was measured. (For experts: The real advance of this study was the construction of a movie-to-brain activity encoding model that accurately predicts brain activity evoked by arbitrary novel movies.) [3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions. [4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Se

MRI technology is improving over time, but I don't know how practical it would be in everyday usage. fNIR can potentially do the same thing much cheaper. I believe it's restricted to only the first few cm of brain tissue but that might be plenty. With fMRI you can even reconstruct entire videos of what a person is seeing (e.g. [VIDEO] ). Controlling something simpler should be easier.

Extremely interesting to compare with Jack Gallant's reconstruction of video from the brain MRI of a viewer: [VIDEO]

Reminds me of this project from 2011: [VIDEO] If I were a billionaire, one of the projects I'd do would be to apply this sort of technology at scale, upon volunteers who were witness to poorly recorded historic events. WWII, Beatles concerts, JFK assassination, 9/11... all of them have been recorded from angles/perspectives that exist only in peoples' memories, and the longer we wait, the more of those memories will be lost.

I feel like we are so close to wirelessly monitoring all brain waves (next 50 years) we will probably be able to, in real time, recognize and diagnose abnormal brain patterns. I think it would be promising if we get this sensor tech up and then perhaps deploy nanotech to interfere with maladaptive brain patterns. The ethics of such a device is considerably sticky. TED talk by Jack Gallant [LINK] recreating vision: [VIDEO] another ted talk by John-Dylan Haynes [LINK] paraphrased: "wireless non-contact mind reading tech may be cheap and mass producible within the next 50 years." the only rate limiting step with his method of decoding thoughts is that it takes time under a high energy device to pair associations in the brain. Think of how much screen time we currently use. If we could pair a wireless 'headset' like headphones while looking at pictures on a screen it would be possible to build a brain bank of children->tenenagers. Currently we just need a lot of processing power and reserved bandwidth on the internet to do some cool stuff.