The Visual Microphone: Passive Recovery of Sound from Video
Вставка
- Опубліковано 7 вер 2024
- The Visual Microphone: Passive Recovery of Sound from Video
by:
Abe Davis
Michael Rubinstein
Neal Wadhwa
Gautham J. Mysore
Fredo Durand
Wialliam T. Freeman
Project webpage:
people.csail.mi...
A similar process was invented in the 1940's by Léon Theremin(a russian scientist best-known for his invention of the theremin, an electronic musical instrument). Instead of a high speed camera, he used use a low-power infrared beam against a glass window, thus measuring the subtle distance variations between the emitter of the infrared beam and the window, causing to vibrate by the surrounding noises, and then converting the distance variations into a sound wave.
Thanks for this iinfo
God. I saw a light beam against my window the other month.
@@butterflymoon6368 😄 you can not see infrared beams. No worries
@@kekthecookwe cannot but camera’s do,o,o🤣
Wow! This is like a fictional superpower brought to life!
weird coincidence huh
Heyyy so funny to see you here! I listen to you almost every night before going to bed! Thanks for what you do!
"If only the walls could talk" 😢🤔🤯
At 2:25 - am I wrong, or did they fucking SHAZAM the song from the sound that they extracted from a video? What a time to be alive...
No, that was just a mid clip UA-cam ad
Does anyone else remember the scene in Eagle Eye when the evil robot thing could "hear" someone by analyzing the ripples in a nearby glass of water? Yeah, we just crossed that line
That rolling shutter bit though is just true genius
Yes I remember that part quite well. Thats a thing yes. Because the principles has been done for ages.
Its no different than how a old record works. You could even make a clay version as well.
Congratulations. You have simultaneously amazed me with these techniques, and horrified me with the thought of the real-world application for further erosion of our first and fourth amendment rights by the NSA and their ilk.
You guys are brilliant. This is truly amazing.
Learn Japanese From Zero! Hi hai bhi xx xx
In movie "Eagle eye" there is a seen that a computer listen to sound using reflection wave pattern of a tea cup.
I remembered this movie too, and I did not remember the name thank you. I'm going to make a relationship on my facebook of science with cinematographic art, sorry my english I'm using google translator.
This is literally the first thing that popped in my head when I saw the title lol
@@ZachHixsonTutorials i searched this thing after watching the movie
This means you can make street sounds from old video recordings. That is brilliant!
This is pretty awesome! Does anyone else think the sounds recovered from the video sounded like something from a horror game?
Also, in before CSI uses this to catch a criminal or something.
May sound like a joke, but if they can perfect this technique, in the future, maybe the police will be able to use it (or whatever comes out of this study by then) to solve crimes with greater ease. It could revolutionize forensics for real!
Hardly. If a video was filmed by a camera capable of capturing minute vibrations on the surface of objects, it would almost certainly be recording audio as well.
Lewis Herd But you're not thinking of long distances. Like across the street through a couple of windows. A good enough camera could look at the vibrations of a piece of paper and figure out what people are saying inside.
TheUnluckyEverydude That kind of camera would cost hundreds of thousands of dollars, if it even exists.
A) It does exist B) High def cameras are about 600 dollars.
This it the kind of forgotten online video that should be brought to light more often.
This is so mind boggling and cool! As a musician, I wonder what sounds could be created by taking videos without sound from around the internet, and artificially ripping the (non existent) audio! I can’t wait to set up experiments of my own!
I wouldn't...
Keep in mind this is done with a high speed camera, videos on youtube have a low framerate, not so sure it would be possible to recover anything from a 24 or 60fps video.
Very interesting stuff but one thing that stood out to me is that they mentioned "Loudspeaker" several times in the video. I checked out the PDF on the link provided for the project webpage and according to that, the sounds produced were in the 80 - 110 dB range! I did not find in the PDF a distance measurement from the source speaker to the medium being recorded but by the example photo on page 4, it appears to be 1.5 - 2 feet away. The setup noted also required a 5,000+ FPS camera and professional lighting to get it to work.
This is not to say that down the road the techniques to recovers sound would not be improved but these were by far best possible case controlled tests. The limiting factor of recovered audio would be the quality of the video and the actual volume of the audio to be recovered. In short, if you aren't yelling at the top of your lungs and can notice someone sneaking into your room to setup some pro lighting and a camera then you can consider yourself safe. It stands to reason that this sort of approach is only viable in an indoor controlled environment because the slightest shift in the wind would throw off your results unless you had enough other objects in the frame that were also affected equally from the wind so the processing could compensate for the shifts.
And I guess regular video compression destroys such subtle data. Nice, but no miracle.
Bruno Mailly It's a start and I am sure that if this little project gets some backing, more efficient compression algorithms will be used to. At a guess, you could provide a region of interest on the picture, ie. a 20x20 pixel section of the image because as you might probably come to realise, 80% of the image isn't going to respond to the sound.
We got 400 pixels times 5000 frames per second, maybe using 10bit grey scale image sensor, that is 20 mbps, or only 2.5MB per second and only an hour of this will consume 9 gigabytes. That is without compression and is easily stored on a microSD card.
well, they also included a segment of the camera recording ear buds, also..
In other news, loud sinister laughing and cheering was heard coming from the NSA headquarters.
I'm sure they've been using this for years, sadly.
Why in their right mind would they use a visual approach that only works on select targets and requires manual setup of a (high-speed) camera when they can just tap into a smartphone's microphone and get way better recordings that way?
Recovered from video recordings of their potato chip bags
Nah, spy tech came first. For decades, we've been able to listen to conversations inside rooms by aiming lasers at the window and measuring the vibrations. Very effective.
If we're hearing about it, it's behind what they have.
Amazing, I thought about extracting sound from silent films almost 40 years ago. If man can imagine it, then he can build it.
+Stew Atkin Just ANY old movies where the camera isn't moving. You might be able to hear the director shouting orders, or something.
Unfortunately old movies are not even close to 24 fps
Even new movies where the audio gets added later. There would be loads of scenes like that. Assuming everything isn't just added later using CGI.
First saw a variation of this technique on 'Fringe' where they recovered a conversation between two people from small vibrations etched onto glass of a nearby window. You guys deserve a Nobel! Genius.
Yeah, but this is a completely different technique and I'm pretty sure that something like in Fringe can't work to reconstruct any useful sounds. In Fringe the sounds where "etched" in the surfaces, as you wrote. This would make the surfaces "storage mediums" for information, which isn't possible in my opinion. It would require a combination of a space, orientation and time to create a storage system, which the material simply isn't capable to do. In this video the chips bag doesn't act as a storage medium, but simply as a medium to transform kinetic energy (sound waves) into an oscillation so light is reflected slightly differently. It's a kind of a very simple "live feed", nothing complex.
I did some DSP analysis on sound waves (specifically on speech) through layered medium for my master's thesis, but this takes it to a whole other level. Neat stuff you guys are doing.
This is amazing. This may mean we will be able to recover audio from silent films taken over 100 years ago that no one has heard yet.
I doubt there would be enough detail in the recordings to do that. This was done using a modern high speed camera which was able to take in tiny bits of movement, with an old b&w film camera recording at 15fps, the information that could be gained would be very small.
Couldnt we still use the same method to "zoom in to the picture" to find its subtler movements that it radiate outwards? though I have to agree with the sound it would possibly reveal would make no sense.. but, just maybe
Another TEMPEST technology...and Big Brother still watcing you!
Amazing. Although that "Mary had a little lamb" tune is incredibly low in the audio frequency spectrum. in which case, our hearing being up to the limit of about 20,000 Hz, does that mean that if you record something at 40,000 frames per second (due to the nyquist theorem) that we'd have a much more accurate and audibly clear audio sample with much higher end frequencies?
I don't know if people realize how amazing this really is! In ancient times Vibrations were known as Akasha or the language of the universe... this just goes to show how all matter is affected by any and all vibrations, even sound waves which effects the time space position and action of these objects, whether alive or not.
I can imagine some poetic applications to this tech. Somehow, Mary Had a Little Lamb touched a nerve. Probably naturally so. Imagine a life gleaned archaeologically through the passive responses of our environment. I was so moved by this idea. I'm a musician and your video with Mary had a Little Lamb touched the sentimental artist in me. Keep it up :')
I'm not usually a violent man but I think the 215 people who disliked this incredible video of human ingenuity should be rounded up and 'dealt' with.
You know that joke about how, when someone gets really pumped on caffeine or high on acid and claims to be able to see sound?
Maybe they could...
Very nice. Appreciation for the clarity of subtlety. What wonders may lie hidden. Like the microwave signatures of ET ships traversing our atmosphere.
Maybe old silent movies can be partially recovered...
No it cannot be done due it’s low frame rate and low-res video but maybe trough a red blue purple pixel process but again not reliable.
@@johneygd old film use film roll, it has high quality, since it not using pixels
I’ve seen old movies that used slow motion.
@@fzr850 that is even worse. Im not an expert but just from common sense, u aint gonna get that much information out of a small film roll. You just wont, the clarity will be mush.
@@prumchhangsreng979 well no old tape video that use tape has been restored and turned to 4k videos
Why would anyone down vote this, its amazing... (I mean other then maybe the understandable loss of even more security lost.)
This is just...I had to check the date to make sure it wasn't April 1st. Totally amazing.
So funny. I was researching this story again and I again thought that this might be an April Fools Day joke.
I would imagine that this only works with uncompressed video. The video compression would wreak havoc on the sound extraction process.
might sound good though
i hope
I think the power of this is that you can use this to subtract sound from what you capture with your microphone, thus decreasing unwanted noise. Very useful when filming a war-reporter in the field!
Let me explain: film as usual with the microphone recording the sound and the video-sensor recording the images. The - future - software will analyse the image for vibrations and compare these to the recorded sound. It will eventually show us a reporter shouting and us asking why (s)he is speaking that loud! (lol)
Amazing, with a high tech satellite, you could capture conversations through a video feed from space!
This is absolutely amazing and terrifying.
Fantastic work!
The only thing that could make this better is using it to listen to space!
In the TV Series "FRINGE" the enemy agents(Observers) did a deeper version of this by recovering the memory of sound vibrations stored in the various surfaces in an empty room. They were able to recover the conversation of the previous occupants.
Can also be used to send fancy encrypted messages. You use faint directional sound (to prevent interception), aim it toward a designated surface or item. Then use this algorithm to decode the seemingly innocuous vibrations to reproduce the sound encoding the message or noise encoding an encrypted message.
Next step: using the vibrations of sound and light scattering bouncing off an object to eventually build a preliminary visual representation of the immediate surroundings to that object which may be hidden or not in view. Using it kind of like a mirror even though it might not have good reflective properties.
it is so special when we watch the future unfolds in front of your (eyes) screen !
congrats on extra-ordinary application of imaginative auditory perception...
those are the 'bullet-time' ideas that make civilizations move forward !
cheers from Rio de Janeiro.
9gag brought me here. Nice to see 9gag at the forefront of cutting edge technology.
They're always watching you. Michael Jackson and Rockwell said it right!!!
Amazing. What next? So music really does affect the mind, body and our physical space. Music and Math create the world as we know it. Remarkable!
Putting us one step closer to the 1984 "telescreen"
I love this technology!!! Rolling shutter for the win!!!
Cool stuff I have to admit! You better switch on your fan in order to avoid being secretly listened to by someone filming the pot plant on your window sill :)
This left me in total shock. I mean, loot at this, it's simply.. Wow. I'm in awe. wow
This is outstanding, to say the least. Thank you so much for sharing your research.
This is awesome, I would like to extract actual sound in green screen movies, it should work. We could hear sounds like "cut", "action" or general chit chat.
I think the implications and potential of this technology is pretty crazy. Insane discovery, this could be used for a lot of things good and bad.
I remember watching a movie (Hotel Lux) that took place in soviet russia shortly before or during WW2 where everyone was expecting the KGB to be listening in on conversations. So whenever anyone was talking about a sensitive subject they turned on the water so the sound of the splashing water would overwhelm any listening devices.
Nowadays that wouldn't work because there's tons of software to filter out something as monotone as that but I imagine having music play might still work.
At the very least it makes it more difficult to filter out the actual conversation.
Given enough paranoia you'd probably have to go into the cellar, write whatever you want to say in complete darkness, so the other person can read it with a flashlight below some form of cover. Afterwards the paper is burned (still below the cover) and the ashes are mixed with a big amount of other ashes, from wood or whatever and then flushed down the toilet.
Come to think of it, I kinda want to do that once!
Although there's probably still a ton of flaws there, I'd just expect *them* to not be prepared for such unusual forms of communication. And so long as you come up with something new for every conversation you should be good :)
I CAN FINALLY HEAR LIGHT!!!!!! Or see sound? What am I doing exactly? Either way this is ridiculous and awesome.
+Matthew “MadMatt” Younce ...and a little scary if someone wants to spy on you. :-D
Zaragil von Slangenwald Wow you're right. Someone can now just film through my window and know what I'm saying.
Bio Hazard yep, and that’s the work from the devil,not from god.
This research is absolutely amazing!
This is jaw droppingly impressive..Such amazing work, well done guys! :)
please do use this great idea on old movies that dont have sounds in them i wan to hear their beautiful voices
A Russian engineer by the name or Theremin, the same inventer/engineer that devoloped the electronic instrument that bears his name used this exact technology to bug some us state department offices during the cold war. He perfected a technique where a diaphram and antenna was embedded into a picture, or statue.
Microwaves were transmitted into the building from outside, and the resonating antanna that was vibrating with the sounds inside the room could be received outside back at the receiver. this was before the laser beam technique.
Oh my, guys, this is incredible. Just imagine the opportunity of this!!!
The last video at 60fps is impressive, I wonder if they will ever develop it to a stage where we can crudely hear the sounds of silent ww1 films
This is amazing! You guys are geniouses!
I knew I hated rolling Shutter, but never so much.
It should be fun running that Analysis on old Video, wonder how well it works with a lot of movement in the Scene (not caused by the sound being searched for).
And come get those Lambs ! They don't just go where Mary went, they go everywhere !
USE THIS TECHNOLOGY ON THE LUNAR MISSIONS!!! WE DESERVE THE TRUTH!!
Just great. I hope you fellas are satisfied to bring more tools for the all too innocent folks at NSA or FSB or whatever. So nice to know great intelligent people like you are bringing on the Big Brother as fast as you possibly can to all of us.
j'adore, c'est révolutionnaire!
Gotta love this. Reminded me of HAL from 2001 he used the same technology
Each example was cooler than the last for me. Amazing work!
sounds logical, if sound is nothing but vibrations, then when the sound hits the objects, they tune together .... although I didn't know that such subtle sounds can actually produce vibrations in the objects to be extracted! awesome :D
Admit it -- when we first saw this idea on Eagle Eye we were all like "HUR DUR DASS SO STOOPID CAMERAS AND COMPUTERS WOULDN'T BE ABLE TO DO THAT IN REAL LIFE" but boy how wrong we were.
actually its been a well known technique, but i bet its eye opening for alot of people
fucking Kek man
Awesome!!! I guess with this algorithm (or another even simpler), it can also be determined the position of the source of sound relative to the observed object.
In space, no one can hear you scream...unless they are recording your space helmet!
I posted a clip in my channel replying to this one. It's from the movie Eagle Eye, that came in 2008. It seems, once again that fiction was ahead of this time.
this is super COOL! what will they come up with next?? thanks UA-cam!!
amazing! I look forward to seeing how far you guys can take this technology
This is amazing! Way to go with your research.
I knew you could do this with lasers, but not video recorders. This is awesome technology!
That is amazing!
I'd love to see this run over some old movies
Amazing!! Presumably the next logical step is enhancing recorded microphone audio with audio obtained from this method? Totally secret squirrel stuff!
That rolling shutter technique is INSANE
Technology of making sound from vibration is not new, the main respect for researchers for opening their method to everybody. For a long time there are a lot of features for anti vibration glasses and so on.
Imagine old silent movies. I wonder if you can capture it from that. Maybe you can hear people talk, Chaplin movies maybe? That'd be amazing.
Amazing. Then mind blow when it showed a basic film recording that we all use today allwing sound to be herd. Imagine this refined. no one is safe
Very cool, especially using the consumer DSLR. Excellent work, guys.
Amazing. It makes me wonder, could the same technology and algorithms someday be applied on a much larger scale to the observation of extra solar planets? In other words, do humans on Earth produce enough sound to make the planet vibrate in a similar way on a planetary scale? If so, might an extra solar planet vibrate slightly due to the noise of an alien civilization across its continents?
And some thought that surveillance cameras in the city were bad enough lol. Imagine quantum processing of video frames used to capture sound.
IT'S INSANE!
Genial...
What if two or more of this kind of camera's used? Could it increase the fidelity of recovered sound compared to the original audio??
It could but the software required to merge the two is not something I would want to work on!
So this is how they do it, highly interesting research.
this is exactly what the world is missing ..
what an awesome share, love this!
everybody check your vibes, the nsa is watching :D
Incredible work! Congratulations.
It's time to have a listen to the Zapruder film again.
I read a book in 1997 about spies in the seventies that were doing something similar with lasers on windows. The author was Tom Clancy, so I'm sure somebody could point to one of his titles.
Saw it on CSI. Can't believe this is actually true!
There are so many crimes that can be closed by the use of this new technology to screen old captured video. Because of this, there is nowhere for a liar to hide anymore (I hope politicians and lawyers read this part!)
so interesting. researchers are genius. I cannot imagine how they convert visual images to sound. They needs to reconstruct 3D space and then map video to virtual space and then extract sound from vibration. right?
Wow.. So next time you see an empty bag of potato chips, be aware that someone might be listening.
In 1983, the movie "Blue Thunder" used laser microphones to listen in real time to conversations inside a room. I wonder how long before the technology of this video is converted to a real time police/military device?
All earbuds are basically antennae which you could send signals to locally to overwrite whatever the main signal is
Pretty covert, I noticed that
So that’s one way to covertly brief people
Just looks like they’re listening to music on a iPod or something
Wow! This is pretty impressive. Audio recovered from video of "still" objects...?
MIND BLOWN
any signal of any type can now be monitored, tracked, logged and potentially used against you ... my what big ears you have Big Brother
cool, now we can finally record sound in space... :D
Ahahaha and maybe we can now also record sound from alians that way.
Nice recording trough sound proof glass, a new way to stalk
Wow, that is rather cool. Nice project/study they did!!
Guys you HAVE to try this on an old silent movie, that's headline material!
"mary had a little lamb music". No, that's "London Bridge is falling down"... other than that, amazing video and great work!! :o
Day B joke lost on Wario. They have the same tune.
Sheppo & Garthis No they don't..
Java Lu dude it's sound system 009 dreamscape
Sheppo & Garthis
Yeah, sorry, but they don't have the same melody. :-(