This is really cool stuff, even though I may have had slightly different expectations - and one big question is open: What happens between the cochlear signal generation and the partially observable (fMRI) activity in the auditory cortex? Or more to the point: What happened to all the processing between cochlear signals going to the auditory nerve which goes to the cochlear nuclei in the medulla, and so forth until finally we get to the celebrated auditory cortex? Apparently, in this work, emulating the whole shebang you're trying to use big deep networks, which perform well, but seem to have very little structure in common with the the actual biological auditory pathway. Essentially, in this work everything between the cochlea and the auditory cortex is modeled as a big black box, making little use of what is known about the neural anatomy. Shouldn't these nice DNN's be made a little more constrained by at least trying to representing the known pathways from the cochlea and the auditory cortex? (Perhaps you're doing that but didn't mention it, and so I suppose I'll have to read all these papers first before I make a fuss ;-)
The spectograms shown here are extraordinary primitive. Using a 120 "Gauss windows of 33msec" per second gives a 10x better accuracy that what is demonstrated here.
Papers referenced in the talk:
3:11 DOI:10.1016/j.heares.2010.02.001
10:25 DOI:10.1121/1.2816572
11:15 DOI:10.1121/1.1945807
12:03 DOI:10.1073/pnas.97.22.11793
12:40 DOI:10.1016/s0896-6273(03)00669-x
20:01 DOI:10.1073/pnas.1403112111 DOI:10.1371/journal.pcbi.1003963 DOI:10.1371/journal.pcbi.1003915 DOI:10.1523/JNEUROSCI.5023-14.2015 DOI:10.1038/srep27755
22:38 DOI:10.1016/j.neuron.2018.03.044
22:54 DOI:10.1038/s41467-019-11710-y
27:30 DOI:10.1101/2020.07.21.214486
28:30 DOI:10.1121/1.4799803
29:37 DOI:10.1121/1.1471898
31:20 DOI:10.1038/1633
33:51 DOI:10.1101/2020.11.19.389999
35:17 DOI:10.1121/1.1904268 DOI:10.1073/pnas.0306958101 DOI:10.1121/1.1536631 DOI:10.1121/1.391936 DOI:10.1121/1.409970
46:33 DOI:10.1016/j.neuron.2018.03.044
47:08 DOI:10.1016/j.neuron.2015.11.035
49:44 DOI:10.1073/pnas.1403112111
50:56 DOI:10.1016/j.neuroimage.2016.10.001 (other refs are same as 20:01)
52:53 DOI:10.1038/s41467-019-11710-y
53:51 DOI:10.1109/TIT.1962.1057698 DOI:10.1093/oxfordhb/9780199686858.013.058 DOI:10.1038/nn.2889
54:37 J. Feather, A. Durango, R. Gonzalez, and J. McDermott, “Metamers of neural networks reveal divergence from human perceptual systems,” in Proc. NeurIPS, 2019, vol. 32, pp. 10078-10089.
57:06 DOI:10.1109/CVPR.2015.7299155
58:21 arXiv:2011.10706
Awesome. Very interesting. Thx
37:30 time/place information
This is really cool stuff, even though I may have had slightly different expectations - and one big question is open: What happens between the cochlear signal generation and the partially observable (fMRI) activity in the auditory cortex? Or more to the point: What happened to all the processing between cochlear signals going to the auditory nerve which goes to the cochlear nuclei in the medulla, and so forth until finally we get to the celebrated auditory cortex? Apparently, in this work, emulating the whole shebang you're trying to use big deep networks, which perform well, but seem to have very little structure in common with the the actual biological auditory pathway. Essentially, in this work everything between the cochlea and the auditory cortex is modeled as a big black box, making little use of what is known about the neural anatomy. Shouldn't these nice DNN's be made a little more constrained by at least trying to representing the known pathways from the cochlea and the auditory cortex? (Perhaps you're doing that but didn't mention it, and so I suppose I'll have to read all these papers first before I make a fuss ;-)
Nice...
The spectograms shown here are extraordinary primitive.
Using a 120 "Gauss windows of 33msec" per second gives a 10x better accuracy that what is demonstrated here.
Very interesting!
Thank you.