Відео

And thus most of them are not generalized.

what you have described with "feature encoders" sound very similar to convolutional maps in convolutional neural network models. if you already have not, you might enjoy learning about them

@@MASQUALER0 Convolutions are a computer algorithm designed for processing pixels in computer images to produce certain effects, such as gaussian blur. The problem with them is the are expensive and do not work like the "feature encoders" I am describing, because they operate after the fact. Encoding in computer vision starts in the imaging hardware such as the CCD or CMOs which often use Bayer encoding to produce an R, G, B image. Computers have no way of knowing what an individual RGB pixel value represents in an image file as it is just a set of byte values. So most neural networks have to do extra work to 'figure out' what each RGB value "represents" in an image but that often means using convolution layers, which are expensive. And no matter whether or not you are using neural networks and machine learning or traditional computer programs, this same algorithm serves the same purpose regardless. Yes, you can visualize these convolution layers but how you got there is the issue. When I talk about "feature encoding" in the human brain, I am talking about the efficient process of converting light signals into networks (plural) of neural signals that are part of the low level process of vision found in most animals with eyes. This task is basically analogous to the analog to digital conversion done by computer imaging chips. The difference here is that the vision systems in biology output neural signals and therefore don't need a separate step to "convert" the data which is what is happening in convolution layers. Because biological vision systems have evolved over millions of years, this process is very efficient, whereas convolution algorithms are very expensive. Not to mention the "conversion" step and "thinking" step are all combined in a single model, which means any learned "features" are embedded in a single model not as a network of separate sets of models that are independent features unto themselves.

@@willd1mindmind639 "Computers have no way of knowing what an individual RGB pixel value represents" Why I can't I say "brains have no way of knowing what one activation of retinal neuron represents (neurons have 3 opsin types for different wavelengths of light approximately corresponding to RGB)"? In terms of the rest of your argument, people have tried learning from raw image data before DSP (digital signal processing). All that happens is that the lower level convolution layers learn to do DSP. My question to you is, why do you see signal processing as fundamentally different than feature processing? They both take an input signal and give you an output signal that contains most of the same information as the input but encoded differently so that it is more useful for the next task. You can do learn the parameters of your feature processors (ex. convolutional parameters). Or you can have fixed, predetermined ones like those used for DSP.

@@brooklyna007 At base level, all computing is based on the movements of electrons and therefore there are only two discrete "types" in that architecture at a physical level. Biology is based on molecules which have a much more diverse range of "types" that can be discretely encoded physically. That is the main distinction here that I am calling out. And that physical discreteness in biology is what allows for intrinsic distinction between things on a fundamental level. Everything in computing is a calculation based on mathematical models that have no true capability to distinguish types because there is no discreteness in the data itself on a fundamental level. Meaning all pixels in a file look the same because they are all the same numeric data type and there is nothing physically that distinguishes one from another. Same with a set of text tokens in a language model and so forth. Therefore the difference between how neuron networks operate over data is that those distributions are based on already discrete values. And those discrete values give every item encoded by biological vision a distinct and discrete signature automatically based purely on the inherent properties of how biology works. No such thing exists in computing and this is why you require labels or human defined examples as the initial kernel to use in training. So in my example about pixels, to understand a specific color, humans have to give an example of that color value in pixels to use in training. Whereas "green" as a color is going to already be a discrete element in the brain based on the molecules used to represent any color in the visual spectrum. So there is already a physically discrete distinction between that color and other colors. And that will automatically trigger network activation, learning and understanding without any 'pre training' or labels. A good example of this biological discreteness can be seen in the camouflage systems of certain animals such as octopus and cuttlefish.

@@willd1mindmind639 Your first sentence was a massive and false assumption. There are many-valued logic computers that have been researched. It is just that simulating multi-valued systems with binary logic is just as powerful. That has been proven mathematically and confirmed with experiments. Then you mention physical discreteness as some driving factor to allow distinction. But digital computers are physically discrete as well. "Everything in computing is a calculation based on mathematical models that have no true capability to distinguish types because there is no discreteness in the data itself on a fundamental level." Please learn some statistics, information theory and harmonic analysis. This statement is just ignorant and bold. "Meaning all pixels in a file look the same because they are all the same numeric data type and there is nothing physically that distinguishes one from another. " Pixels have positions (relative position is what matters, i.e. their context) that distinguish them from other pixels. Pixels simply share structure but they are distinct. "Same with a set of text tokens in a language model and so forth. " Text components also have a context that distinguishes them from either seemingly identical text components. "Therefore the difference between how neuron networks operate over data is that those distributions are based on already discrete values." False. Not only were none of your premises true but this conclusion is directly provably false. Digital computers are discrete systems. That is literally what the word "Digital" means in the context of computing. It is discrete computing.

Totally agree with you and I’ve been thinking the same. Do you know such approaches exist in the literature or somewhere?

do you have some pointers for me on that?, it is quite an interesting topic but whenever I (as a computer scientist) have a look into some studies they say 'its more genetically than we've thought' (e.g. personality of persons)

@@martinschulze5399 There is a huge literature on the molecular mechanisms that wire up brains. The brain isn't all that different from a face in some ways. We all have faces but identical twins have the same face and the reason is 99% due to their genes. The brain is the same. The fact that brains wire up almost identically in all humans is a remarkable testimony to the information contained within our genomes.

@@edwardruthazer1849 Yea, as a computer scientist with some experience in genomics, I was shocked by his comments on that. Further yet he made the most elementary mistake about evolution possible. He suggested that learned connections in the brain can somehow be passed on. There is no known biological mechanism for anything close to encoding the connections of trillions of neurons into the genome. All that can be encoded is a bias for a set of connections. If that bias is reproductively successful, it gets passed on. The only question is how specific that bias can get. Taking the number of neurons in the brain, their variance of connections and comparing that to the size of the genome (3 billion nucleotides, 6 Gigabits) you see a limitation on how much can be encoded genetically. This implies that there is a large learned component and that maybe the exact connections don't matter that much. But it is far from implying that these connections are learned.

The rest of the talk is fascinating as are all the rest of Hinton's similar talks on this subject. The transpose redundancy is downright enigmatic.

I think his claim was more along the lines of, innate skills are by definition innate, but we are not explicitly told what is innate and what isn't innate. In order to make that distinction, we must go through an evolutionary learning process. An analogy would be that suppose we are given a set of Legos that forms a ship and the instruction manual with an incomplete, unsorted set of pictures that shows the building progression of the Lego set. Since the pictures are unsorted and incomplete, this operates as a "fog of war" property like in video games. The only thing we are given and know for certain is the "initial value", where to start. Thus the only way to build the Lego set is to learn by trial and error: which pieces go together and which pieces don't. By learning which pieces go together (validated by the fact that the Lego pieces start looking more and more like a ship), we start to validate the instruction manual and gain a better understanding of it.

@@evankim4096 I've been seriously thinking about this for about ten years or so, No disrespect intended but the question is quite empiric, there is no way to explain why human languages are all so similar, it is like as nonsensical as reaching an alien planet and where everybody talks different versions of Java and you would claim, yes, they learn Java because Java is easiest to learn. Its ridiculous. There are things that guide human cognition, some of them might be at the neural level, but in most interesting problems and especially in terms that are uniquely human, they are more to be found in modular structure or inter modular connectivity, and I don't think Hinton would in any case disagree with this point if the question was put to him. In the point I related to he seems quite uncharacteristically (but quite in pace with the times) disrespectful towards Chomsky's work and towards linguists in general and as a linguist who does NLP and thus is an admirer I must say that he might very well be mistaken on this point, I mean it could be possible to do both the work of evolution as that of the language acquiring baby in a single search but it isn't trivial to see whether that is the case and in any case evolution doesn't seem to have found it that easy to do since language only happened once.

BTW, just to elaborate innate, innate is at least as innate as the facial structure and feature recognition module in the brain which is quite known and has been somewhat elaborated by now.

​@@veltzerdoron I agree, he did seem quite dismissive of linguists in general and of any possible insight they could contribute towards NLP models in computer science. My knowledge in NLP is limited, but in general, it's hard to accept points of views which are dismissive towards the other side (maybe he just has a sarcastic personality). When he was asked about which side to work on (computer science vs neuroscience), he seemed quite dismissive of other computer scientists in general when he indicated that they were working in the opposite direction as he was (his direction was towards neuroscience)