How are memories stored in neural networks? | The Hopfield Network

Поділитися
Вставка
  • Опубліковано 18 кві 2024
  • Can we measure memories in networks of neurons in bytes? Or should we think of our memory differently?
    Submission to the Summer of Math Exposition 2022 (#SoME2). More information: summerofmathexposition.substa...
    Time stamps:
    0:00 - Where is your memory?
    1:41 - Computer memory in a nutshell
    2:58 - Modeling neural networks
    4:42 - Memories in dynamical systems
    9:54 - Learning
    13:36 - Memory capacity and conclusion
    Animations largely made using the manim community edition:
    www.manim.community/
    Original Paper on Hopfield Networks:
    Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8), 2554-2558.
    Neuron image by Santiago Ramón y Cajal, The pyramidal neuron of the cerebral cortex, 1904 Ink and pencil on paper, 8 5/8 x 6 7/8 in. Credit: Cajal Institute (CSIC), Madrid
    Music: Aakash Gandhi - "Dreamland"

КОМЕНТАРІ • 986

  • @aegontargaryen597
    @aegontargaryen597 7 місяців тому +357

    As a neural network, I confirm that this is how we store memories

    •  5 місяців тому

      early

    • @ugestacoolie5998
      @ugestacoolie5998 5 місяців тому +6

      Lmao

    • @claudiamanta1943
      @claudiamanta1943 3 місяці тому

      All the screws of those who are OBSESSED to think that people are nothing but cybernetic machines are rather loose…

    • @aegontargaryen597
      @aegontargaryen597 3 місяці тому

      @@claudiamanta1943 I haven't seen anyone thinking that people are cybernetic machines lol

    • @merozemory
      @merozemory 3 місяці тому

      LMAO

  • @murilospineli2241
    @murilospineli2241 Рік тому +1080

    is this a first video? it's surprisingly good

    • @jasoneel76
      @jasoneel76 Рік тому +33

      My god I need to subscribe immediately

    • @jaumeivars
      @jaumeivars Рік тому +10

      Oh my god i didn't figures it out until you commented!

    • @ryanpmcguire
      @ryanpmcguire Рік тому +30

      When something is made by someone who cares, you get a good result. Caring is the first ingredient in the recipe for success.

    • @fadonoso
      @fadonoso Рік тому +2

      I thought the same!!! This guy is really good. Thanks!!

    • @NickSklias
      @NickSklias Рік тому +22

      The first video uploaded to a channel does not mean the person/people editing/producing the video had no prior experience. That being said, it's a fantastic first upload and this channel is off to a tremendous start!

  • @luke2642
    @luke2642 Рік тому +309

    You nailed the 3blue1brown feeling whilst being ligther on the maths. I wish you the best of luck with future videos! 🙂

    • @ianthehunter3532
      @ianthehunter3532 11 місяців тому +10

      He likely uses Manim, the software made and used by 3b1b.

    • @poultrypants
      @poultrypants 9 днів тому +1

      Are you guys also watching the 3B1B videos about GPTs?

  • @piggy9539
    @piggy9539 Рік тому +772

    This guy explained such a complex topic so well, great job.

    • @alexanderhandjiev2845
      @alexanderhandjiev2845 Рік тому +3

      It's going viral soooo ye

    • @mihailmilev9909
      @mihailmilev9909 Рік тому

      @Yatish38 yeah and the subs for one video is crazy but deserved too

    • @RoseOnFire
      @RoseOnFire Рік тому

      @Yatish38 Really good, yes, deserves more, also yes

    • @TheRainHarvester
      @TheRainHarvester Рік тому +1

      Yeah i had trouble with the super/subscripts in all the math. So i just made a video easily explaining multilayer back prop with only 1 subscript. It's so easy now!

    • @pizzainc.1465
      @pizzainc.1465 Місяць тому

      If humans know how it works, it’s a simple subject that is explained badly. If they don’t, it’s too complicated anyway. For example, we know nothing about how the brain works because it is simple too complicated. However, we know plenty about gravity, other areas of anatomy, computers, and more.

  • @keithmiller3770
    @keithmiller3770 Рік тому +114

    This is remarkable. How the heck does this channel only have 1 video?!

    • @rabiaedaylmaz1198
      @rabiaedaylmaz1198 11 місяців тому +8

      Still waiting :( ...

    • @alexb3617
      @alexb3617 9 місяців тому +2

      yeah i knew this video would be good. i noticed it few months back when i was beginning to study neural networks, just stored it, knew wasn't ready yet. now finely watched, as expected, it turned out good. nothing like clickbat stuff that usually out there on youtube

    • @Mikee512
      @Mikee512 День тому

      Everyone must start somewhere :)

  • @Sazazezer
    @Sazazezer Рік тому +304

    I've been idly musing about the brain's memory in terms of binary and bytes for years and always thought it was such a limited way of looking at the subject but didn't know any better. I asked around in the past about alternative ways of looking at the subject but never came away with a decent answer. This video has been such an eye opener and i think i'm going to have to take a deeper dive into the subject. Thanks!

    • @Daniel-ih4zh
      @Daniel-ih4zh Рік тому +4

      You should haken's book on synergetics

    • @andrewhancock2451
      @andrewhancock2451 Рік тому +2

      You mean, like, deep into neural nets? ;P

    • @SpringySpring04
      @SpringySpring04 11 місяців тому

      I heard in an scifi anime (which is surprisingly more accurately scientific than it may seem despite being scifi) called Steins;Gate that said the brain can hold a little over 4 terabytes of memories, but that was just a claim, I'm not sure if that's true.

    • @pyropulseIXXI
      @pyropulseIXXI 11 місяців тому +3

      Memory doesn't just exist in the brain; there is evidence it is dispersed throughout the entire body. People have had large chunks of their brain removed and their memory was unaffected. People born with basically no brain, just a small layer around the skull and the 'cavity' where the brain should be was filled with cerebral-spinal fluid. And yet their IQ and memory was well above average.
      I know it is hard for purely materialistic to understand, but consciousness is more fundamental than physical matter, with the brain merely being an interface for the physical world
      A true ghost in the machine; consciousness does not arise from the machine being complex; it is controlling the biological machine

    • @goldnutter412
      @goldnutter412 11 місяців тому

      @@pyropulseIXXI Exactly, I posted a heap of times about this already, this video really inspired.. brain caching, permanent information, rinse and repeat with new context.
      Optimization is often overlooked. It will be obvious sooner or later, there is no other way to efficiently battle entropy. We get sense data and need a nervous system and brain in general, but uncertainty is the ultimate optimization. Probabilistic by design.. part of why the speed of light is so high. We see the very very distant past, and are disconnected from it.. we get data and interpret it and wonder how the heck physical matter organized itself a certain way and why the math doesn't work
      0 to 1 is a logical impossibility.. the Big Bang is a misinterpretation, but a plausible back story for the universe and adds to the immersion. We process things bound by our brain capacity (see Einstein's freak brain!) and easily fall into materialist traps of our own design. Logically the universe and biology is simple, but obfuscated by so many LAYERS of background processes ! we call it subconscious.. yeah it's busy !

  • @gianlucazinarof8617
    @gianlucazinarof8617 Рік тому +13

    You cannot fathom my disappointment when I looked at your channel and discovered that this is your only video. What do you mean there aren't any others? I need more! Seriously, interesting topic, crystal clear presentation, keep up the great work!

    • @Mangputt
      @Mangputt Рік тому

      Do you know, what software able to produce and animate these animation? adobe after effects or any software else?

    • @layerwiselectures
      @layerwiselectures  Рік тому +1

      It is a custom toolbox written in Python, called Manim. Here it is the community edition, but Manim was originally devised and open-sourced by 3blue1brown. There is a link in the description if you want to learn more.

    • @Mangputt
      @Mangputt Рік тому

      @@layerwiselectures thankyou... I still waiting for your next videos. Keep it going Bro 🔥

  • @mmarrotte101
    @mmarrotte101 Рік тому +61

    The amount of effort required to not only curate such complex information in an intuitive way, but also to animate the concepts in the way you've done, is clearly tremendous - thank you for taking the time to produce this! I'm inspired by your work, really looking forward to more!

  • @HansLemurson
    @HansLemurson Рік тому +115

    Different "seed" states will call up different memories. I wonder if this could be analogous to how certain ideas link together, and maybe some of the underlying information of related ideas is actually a correlated section of their memory patterns.

    • @MrNatsuDragneel
      @MrNatsuDragneel Рік тому

      This is wrong, our brain can excite and depress certain neurons with neurotransmitters, hormones, data from sensors of the environment and our own body etc...

    • @hufficag
      @hufficag Рік тому +51

      This explains why I have an idea at my desk, get up to walk across the room and forget what I was thinking about. Get back to sitting at my desk seed state, and remember it, get up and forget it again.

    • @mihailmilev9909
      @mihailmilev9909 Рік тому +4

      @@hufficag omg we figured it our also lmfao

    • @floriantesto4925
      @floriantesto4925 Рік тому +1

      @@hufficag just take a look on en.wikipedia.org/wiki/Encoding_specificity_principle

    • @woolfel
      @woolfel Рік тому +18

      the video is meant to explain how modern artificial neural networks work. biological brains don't work that way. Take for example a deer. Within an hour of being born, a deer can walk and within a few hours it can run. Clearly the baby deer didn't learn how to run in the womb. The evidence suggests some kinds of memory are genetic, while other skills are learned. Even if there was seed states, keep in mind memories are stored redundantly in biological brains. the obvious example is stroke patients that temporarily loose skills, but then recover them. Assuming there something like a seed, it would be seeds or parts of a seed stored in multiple places. then there's the fact that brains are analog, not digital. In classic computer science, associative memory is two elements linked with an address that points to both elements. In neural networks like GPT3, the association is captured by many weights. One could consider those weights as a seed, but the analogy breaks down.
      A common problem with neurons in a neural network is that a single neuron may participate in multiple patterns. If you change 1 weight, it can cause the network to perform better on one type of task and worse on a different task. The problem is called polysemantic neurons. There are people working on fixing polysemantic neurons, but there's no general purpose fix today.

  • @NajibElMokhtari
    @NajibElMokhtari Рік тому +9

    This is amazing! I finished the video and immediately went looking for the next one where you'd explain the more accurate models. Then realized this is the first and only video on this channel.
    Can't wait for the upcoming stuff. Congrats for the launch!

  • @macchiato_1881
    @macchiato_1881 Рік тому +20

    It's really amazing how something as simple as a neuron has highly emergent properties that can do very complex tasks.

  • @sciencelab4225
    @sciencelab4225 Рік тому +25

    This is one of the most important pieces of media I have yet consumed to better understand this world. It is incredible to realize how one's brain works in principle. Thank you so much!

  • @martingisser273
    @martingisser273 Рік тому +6

    Brilliant exposition for both mathematicians and nonmathematicians. It took a few decades to get me (mathematician) interested in neural networks. Now I'm hooked. Musing about connections to Markov processes and PDE...

  • @frenk3928
    @frenk3928 Рік тому +18

    I studied Hopfield networks last year in the framework of spin glasses and all assumptions were dropped seemingly out of nowhere in a big confusing mess (as usually happens in lectures). This video really tied up all together and now makes so much sense beyond memorising proofs. Excellent video and brilliant demonstration.

  • @Wonders_of_Reality
    @Wonders_of_Reality Рік тому +13

    Looking forward to watching the second part! And many thanks for choosing dark text on a light background. It looks like a calm creamy ocean of fascination!

  • @konradd8545
    @konradd8545 Рік тому +2

    Please continue making videos! This was not only super interesting but also very well explained and animated. I hope to see more videos on this channel soon!

  • @TR-kn3sn
    @TR-kn3sn Рік тому +3

    This is absolutely incredible. Just immense, wow. I was initially skeptical of neural networks because they lacked the "logic" side of the problem (as I think a lot of people were at first) but I knew they were unreasonably effective, so I assumed they had their place. But now, I think they have more than a place, they are officially critical to our understanding of the world. In short, they were always cool, but now we've proven it. Thank you for making this video! It is so well made by the way!

  • @BrandonWong55
    @BrandonWong55 Рік тому +4

    Wow, this video is really eye-opening. As someone who has been interested in both memory sports and math, it really intrigues me how these false memories arise. Whenever I make a mistake memorizing a deck of cards, I'm always trying to figure out what when wrong, and it's cool to see a mathematical model also make similar mistakes in your video.

  • @pafloxyq
    @pafloxyq Рік тому

    Hopfield nets were the first thing that got me interested into studying neural networks some years back, but I hadn't seen such an eloquent presentation on it since now. Will wait for the subsequent videos of this series.

  • @brandontylerburt
    @brandontylerburt Рік тому +1

    An excellent explanation. I'm starting to realize what a profound influence this type of neural network model has had in recent decades on all kinds of disciplines ranging from machine learning to the development on new theories on PTSD and how traumatic experiences alter brain function. Thank you for providing such a succinct and illuminating video on a complex and vitally important subject.

  • @YasheshBharti
    @YasheshBharti Рік тому +3

    Such a unique topic, so beautifully explained! Thank you so much for this piece of art!

  • @tf3confirmedbuthv54
    @tf3confirmedbuthv54 Рік тому +40

    Love it! Hopefully this video will get big, this question is wonderful and really makes you think outside the box, and I think you explained it perfeclty.

  • @sebastianmuller4649
    @sebastianmuller4649 Рік тому +1

    "Contrast this with what we believe about the brain..." I like how careful you are with how you make your statements.

  • @exinangai2216
    @exinangai2216 Рік тому

    The best illustrated explanation on Memory modelling of inter-connected neuron (switch), bit and quirks of Neural Networks , formation and significance of multiple "stable states", and how they are presented as row & col vectors, update function as Matrices. Your video is both clear and excellent visual intro to the topics. I am really looking forward to your next video. Helping more ppl to visualize how transformers, recurrent net, back propagation, main idea of ML papers, etc are being modeled.

  • @cmilkau
    @cmilkau Рік тому +49

    I guess you have to make a distinction between short term memory (stored on activity patterns) and long term memory (this video? stored in the network weights). The latter is much simpler and in theory there could be a single neuron for a single memory, although in practice there are more efficient encodings.

  • @redaabakhti768
    @redaabakhti768 Рік тому +25

    Perfect video. I didn't know you could look at neural networks from this neat mathematical vantage point. I thought about eigenvectors and how the eigenvector with the largest eigenvalue would influence the network sort of as a dominant weight that increase exponentially , this would need a slightly different f the function you operate on the resulting weighted sum this would work for storing one memory maybe with increase possible states of single neuron and the right activtion function you will be ablse to store many patterns. Amazing video once again.

  • @eldertom
    @eldertom Рік тому +1

    This is… ridiculously good. Thank you very much for making this, and really looking forward to the next one!

  • @alexharvey9721
    @alexharvey9721 Рік тому

    Super interesting and fantastically presented! I wish there were more videos like this! Love that you're not afraid of diving into deep concepts and details. Instant sub.

  • @br.3250
    @br.3250 Рік тому +4

    This is just an outstanding explanation and it was a very interesting subject.
    And it's even more amazing that even though I know very little about this subject I could understand almost everything
    Keep up with the great work :)

  • @dererzherzog
    @dererzherzog Рік тому +11

    Even a reductionist model of neural network (disregarding the dynamic spatio-temporal interplay of a plethora of neurotransmitters, neuromodulators, receptors, secondary messengers and downward pathways) leaves one awe-struck by hinting toward the tremendous scale of capabilities in a biological neural system.
    Thank you for this wonderful presentation and thanks to the YT algorithm for suggesting it! May your channel and its viewership grow exponentially!

    • @LuaanTi
      @LuaanTi Рік тому +7

      And don't forget, the _in_ capabilities. Even from this extremely simplified model, you can see why it's hard to write memories, and keep them stable against change (which of course happens even when you're just _reading_ the memory). The computer on your desk can easily store terabytes of information without a glitch just by "remembering" the information one by one, while the computer in your skull needs to keep reinforcing what you want to remember, and keeps overwriting related memories as others are read or changed.
      Mind, this is most likely a critical component of learning and intelligence in the first place - your memories aren't just static data stored elsewhere, they're ultimately a part of the computational network itself. Every experience has _some_ effect on your decision-making, and memories interact with each other (and with other inputs from all over the brain and the outside world).

    • @goldnutter412
      @goldnutter412 11 місяців тому

      @@LuaanTi apparently the brain throws out TONS of data all the time.. anyway won't repeat the other posts.. maybe read those first above.. but it's a good filter/cache for information processing.. short, long and permanent information. Only what is really significant to us, lasts.. and we come back to evolve past our problem with entropy. The unknown is enticing.. fear of the unknown is like getting stuck in loops on tangents.. or outright stuck and unable to deal with whatever patterns we're experiencing.
      The optimal configuration for experience is of course permanent state, temporary brain and universe. Next time, the patterns will have a completely different basis, while still understandable ! almost every random set of DNA and environment will be the same pattern. Choose, which gives result (quantum mechanics delivering data on demand), JIT type of interpretation with the brain cache, more choices, more life experience, on and on we go.
      Because we are self aware, we are aware of entropy.. time advances, we collect more data. More entropy unless we evolve away from "fear" or.. I guess you could call it over management of the data structure. That's the nature of information ?
      "Let it be" playing in the background..

  • @rafaeladivapinho2399
    @rafaeladivapinho2399 Рік тому

    You are by far the best teacher for soft soft . It's very complicated at first - overwhelming, actually - but, you make it doable for

  • @hcky6054
    @hcky6054 Рік тому

    I really like how you explain neural networks with necessary elements only which helps me focus on the concept itself!
    Nice work!

  • @anon_y_mousse
    @anon_y_mousse 11 місяців тому +3

    I don't know why you haven't made another video in 9 months, but I'm hoping you continue this series. Maybe go into the math a little bit more, but keep it simple so even a simpleton like me can understand it.

  • @schnitzelsemmel
    @schnitzelsemmel Рік тому +3

    Yes yes yes this is exactly the video i didnt even know i needed. incredibly good explanation and animation

  • @hengchunsong301
    @hengchunsong301 Рік тому +1

    Very rarely UA-cam has video with this much deep insight. Treasure.

  • @Tigrou7777
    @Tigrou7777 Рік тому

    By looking at this video, you just updated the neural network in your brain to be able to remember stuff about neural networks. Brilliant.

  • @venamaybe
    @venamaybe Рік тому +3

    This was super interesting and awesome!! I'm excited for the next video :)

    • @user-dp1od3gd3u
      @user-dp1od3gd3u 4 місяці тому +1

      looking forward to the next video too

  • @nulled7888
    @nulled7888 Рік тому +3

    Super interesting video, got me really thinking about the correlations to real neurons.. very well made as well :)

  • @histeward
    @histeward Рік тому

    It’s rare to find a concise video about a learning topic. This is awesome! Hope your channel grows huge!!🎉

  • @LiquidMasti
    @LiquidMasti Рік тому

    can't believe it's your first video. it's very well explained and the background sound makes it even more compiling.

  • @somedudeok1451
    @somedudeok1451 11 місяців тому +4

    Love the presentation. I think that the mistakes you were frustrated and fascinated with are maybe a chance to generate something new from something stored. It reminds me of our ability to be creative by morphing our memories over time. It's like when you have memory A and memory be and over time they fade and become memory AB which then turn into memory C. And sometimes memory C might actually be something useful. It also reminds me of generative AI. Of course, this is all just layman talk and I have no idea if this has anything to do with how human memory actually works.

  • @brandonzhang5808
    @brandonzhang5808 Рік тому +7

    What I'm getting out of this is that there is still a scalar memory parameter in the number of neurons or size of the weight matrix, and storing patterns in it is similar to file compression, allowing the same network to represent multiple patterns, but with a compression ratio that is actually disadvantageous, requiring O(n^2) weights to store O(n) patterns. Something of a Rube-Goldberg for memory systems.

    • @dominikwitczak1655
      @dominikwitczak1655 Рік тому +1

      Eactly, this does not prove that our memory can't be measured in Gb of information.

  • @bakdiabderrahmane8009
    @bakdiabderrahmane8009 Рік тому

    the IDEA explained in this video is something a was looking for for a looong time, thank you so much you have no idea how helpful you have been.

  • @hardronix
    @hardronix 11 місяців тому

    Thanks for opening my eyes. I already knew about recurrent networks and stuff, but now I think about them with an existential crisis. Thanks for such an awesome explanation!

  • @ofrigal9557
    @ofrigal9557 Рік тому +3

    Amazing video! Great job, i would love you to explain autoencoders

  • @alzario
    @alzario Рік тому +4

    Awesome video, comparatively digestible explanations and really good visualisation of what you're describing.

  • @matthiasdenhamer3925
    @matthiasdenhamer3925 11 місяців тому

    Absolutely mindbogglingly well done. Can't wait to see more videos from you!

  • @JoelGreenyer
    @JoelGreenyer Рік тому

    Amazing video. Very insightful. I cannot imagine how much work went into the animations. Very well done!

  • @boopsnoot7180
    @boopsnoot7180 Рік тому +4

    Woah how did this end up in my recommended lol, this is so well made but it has low stats and appears to be their first video

  • @Kaepsele337
    @Kaepsele337 Рік тому +3

    That's really neat. I wonder what happens if you add "eyes" to the network, i.e. some subset of neurons whose activation is driven by an external signal instead of the update function.

    • @layerwiselectures
      @layerwiselectures  Рік тому +2

      Good point. The original paper actually mentions an input current to the neurons. This would then be combined with the summed input of all other neurons, so it would modify the update function rather than replacing it. I left this out for simplicity. However, it's actually an important detail once you start thinking about it. It sort of biases the network to a certain state and would explain where the starting state of the network comes from, if it's not random. The story would then go like this: the "eyes" of the network (love the metaphor btw) would cue the network with a partial state and the network would then autocomplete to its memory state. Kinda-sort-of-almost-like seeing a visual cue of some sort and then being reminded of a scene from your past - if we're being a little bit fanciful. Thanks for the comment.

  • @jaumefabregaroda
    @jaumefabregaroda Рік тому

    Mindblowingly good! You left me in awe, please keep uploading videos!!

  • @TenderBug
    @TenderBug Рік тому

    I find this explanation is fascinating. Shines light on neural networks from an uncommon direction. Great Video 👍

  • @theClaytron
    @theClaytron Рік тому +3

    If you change the procedure to use a synchronous update, and make the activation function the identity, then you can analyze the dynamics of the weights matrix W. That is, v_{n+1} = W*v_{n}. In this case, a memory which you converge to is simply a stable eigenvector of the matrix W. When you "train" the network with multiple memories, you are performing multiple rank-one updates to the matrix. This also explains why the maximum number of memories you can store in W varies linearly with the size of W, since a matrix of size NxN cannot have a rank greater than N. In other words, if you tried to train W on N+1 memories, you are guaranteed that at least one of those memories is a linear combination of the other memories. Not only that, if you add a memory with a rank-one update to W, that does not necessarily mean that memory will be an eigenvector of W. The memories can combine such that an eigenvector emerges which is not equal to any of the memories.
    This approach also gives another perspective on why you can converge to the "flipped" version of the memory. An eigenvector defines a 1-dimensional space which is simply a scalar times a vector, where the scalar can be positive or negative.
    Also, the properties of the W that emerges from Hebbian Learning are very interesting. That is, W is symmetric and has a zero-diagonal. It is a property of symmetric matrices that their eigenvalues are all real. So we would not expect the network to oscillate around some solution. Also, the trace of a symmetric matrix is equal to the sum of the eigenvalues, so that means if we have negative eigenvalues, we must also have positive eigenvalues.
    Again, this is just some conclusions for the case of synchronous update and identity activation function. It may not be so relevant to the case you discuss.

    • @layerwiselectures
      @layerwiselectures  Рік тому +3

      That's perfect, thank you. The original paper does not use this reasoning to arrive at the linear memory capacity, but the similarity of the conclusions is striking. Thank you for this addition!

    • @theClaytron
      @theClaytron Рік тому

      @@layerwiselectures Thank you for your excellent video! I am looking forward to the sequel.

    • @isaacaraya3848
      @isaacaraya3848 Рік тому

      @@layerwiselectures Have you heard of adaptive resonance theory? Could the stable states in the network be harmonics? Like if each pattern is stored not just spatially but at a specific frequency of oscillation then you could maximize the number of memories a single neuron could contribute to. Kind of like the tv's that play multiple images on the same screen at once

    • @isaacaraya3848
      @isaacaraya3848 Рік тому

      Great insight to complement video. Could you expand on why symmetric matrices can't oscillate around a solution? Might save me some googling

  • @dominicbaumer6038
    @dominicbaumer6038 Рік тому +6

    Thanks for this great lecture! My idea for a next video topic is that you can talk about differences between artificial perceptrons and biological neurons

  • @arvid_music
    @arvid_music Рік тому

    Thank you for this very high quality and interesting video! The visuals and examples really help in making the topic easier to understand.
    God bless you and keep up the great work!

  • @Neptoid
    @Neptoid Рік тому +1

    This had amazing animations and insights I have been wondering about for a long time, and although I didn't understand much of the math I still felt grounded by insight after insight to the end

  • @drdca8263
    @drdca8263 Рік тому +4

    Hm,
    I don’t know that this really shows that the memory capacity can’t be appropriately described in terms of “how many bits”.
    That there aren’t addresses for physical locations for different chunks of memory, sure.
    But, information theory isn’t fundamentally inapplicable to the brain’s ability to remember things?
    Like, if there is some random distribution of possible observations (e.g. a sequences of symbols, or other combinations of e.g. color, shape, texture, whatever) at one time, and then at a later time the person says their guess about all the information needed to describe which of the observations were randomly selected,
    then, ((the entropy of the distribution of observations) - (the conditional entropy of the distribution of observations given the person’s recollection of the observation)) should give a lower bound of how much information about the observation the person remembers on average. This can be expressed in bits.
    Of course, this isn’t describing the person’s entire memory capacity! Just how much they remember from these observations in such tests.
    This could, I think, be used to measure an estimate of a lower bound on how much information (measured in bits) a person can rapidly memorize for a short amount of time.
    Presumably how much info one could memorize would depend on the way the information is expressed. For example, I would expect a random sequence of words would be easier to memorize than a random sequence of literal bits with approximately the same entropy. And that’s part of why I say it would only be an (approximation of) a lower bound. (Approximation because there’d be measurement error due to the random variation.)
    We can also probably put an extremely weak upper bound on total capacity based on the amount of information theoretically needed to describe the physical state of the brain. If nothing else, there is at least the Bekenstein bound based on just the size of the brain along with the fact that it isn’t part of a black hole. This is so weak of an upper bound that it is barely worth mentioning.
    I don’t know how to get a better estimate of “the entire memory capacity of the brain” than these though.
    Well, of course, you can look at how much data people have memorized over longer periods of time.
    Oh, maybe some estimates could be made based on people with eidetic memory.
    Like, if we query facts about randomly selected dates and check how accurate what is recalled is, then, we could extrapolate that and assume it would be equally accurate for all the many dates in the same range which we didn’t ask about, then we could estimate from that how much they remember overall from that range of time.
    This could probably give an estimate which reveals a rather large amount of information stored.
    And, presumably, people with an eidetic memory don’t have just, an inherently larger memory capacity, so much as, just more stuff gets committed to memory? (I could be wrong about that.) So I think this would still give a good estimated bound on how much memory capacity a brain in general has.

    • @layerwiselectures
      @layerwiselectures  Рік тому +2

      Wow, thanks for the comment. There's a lot to think about. My favorite part is the black hole information bound. No you're right, information theory isn't fundamentally inapplicable here and I didn't mean to imply it was.
      Personally I'm not clear on whether the "bits" in a hard disk are information theoretic bits. Are they? Maybe in the sense that, if I didn't know the state of the hard disk, and then I looked to find a certain pattern there, I would have gained that many bits of information? But then on the other hand I wouldn't expect every pattern to be equally likely to appear on a hard disk or in RAM.
      What I'm getting at is that, yes, there must be a way to compare the information in both memory systems (computer and neural) in bits, but no, it isn't usually being done when we talk about computer memory.
      My goal was to show that casually speaking of bits of memory has a tendency to obscure real differences between the two memory systems. If in the process I appeared to discount information theory ... oh my, not my intention.
      Thanks for the comment again!

    • @drdca8263
      @drdca8263 Рік тому +1

      @@layerwiselectures I think I phrased my comment a bit too critically given how much I liked the video. I quite liked the video, and subscribed (before your reply).
      I think computer bits can be related to bits in information theory by, if you sample from a distribution many times, the optimal way to store such a sequence on disk, would have its length be on average, the number of samples times the entropy of the distribution. And, I think if the number of samples is large, then the variance in how much storage is needed overall should go down, eventually to the point where you can regard it as nearly always the same. ...uh, assuming the variance of the number of bits each sample contributes, is finite (which it will be if the number of possible outcomes is finite).
      So that’s at least *a* connection. It perhaps isn’t quite as close a correspondence as one might hope? Well, the amount of information in an information theory sense that can be stored by a hard drive, is bounded by the storage capacity, and at least for some distributions of a random variable, this bound is exact. But for other random variables where the outcomes aren’t just 2^n equally likely outcomes, then I guess there can be some overhead compared to the ideal, in storing a finite number of samples. But in the limit of many samples, I think the two notions of “amount of information” become the essentially the same? In some sense anyway.

    • @sullivan3503
      @sullivan3503 Рік тому

      @@layerwiselectures You're correct in saying that information theoretic bits don't always correspond exactly to hard drive bits. In order to calculate the amount of information stored, you need to also understand the encoding. For instance, ASCII text uses 7-bits of information per character to store text, but these 7 bits of information are often stored at byte boundaries, resulting in each character taking up 8 bits of space on the disk. So the leading bit is wasted. Compression can also reduce the amount of bits needed to store a certain piece of text drastically. So, as long as your data is not maximally compressed (which is hard to tell), then your hard drive bits is not the best measure of the amount of information stored. However, in practice, we typically use file size as a good proxy for information.

  • @d0tz_
    @d0tz_ Рік тому +9

    This is the best SoME video I’ve seen so far. Such an intuitive explanation for a niche yet mindblowing idea.
    Maybe it’s my lack of intuition for high level math or my affinity to computer science, but most of the other videos had me closing out of it mid way, while this one was interesting in its entirety.

  • @awritrojitbanerjee1055
    @awritrojitbanerjee1055 Рік тому +1

    you knocked it out of the park dude! need more of the good stuff!

  • @Prince-xp9vk
    @Prince-xp9vk Рік тому +2

    had an idea to write about this, "is the brain really a RAM or ROM?", but seeing you do justice to it, I can safely procrastinate for another day.
    it's kinda sad too that you only have one video. please do more.

  • @user-of9sr8bm9i
    @user-of9sr8bm9i Рік тому +3

    Watched

  • @okopyl
    @okopyl Рік тому +3

    Too complicated for somebody with 0 knowledge in math

  • @Gnevnyj
    @Gnevnyj Рік тому

    Instant subscription. This video was extremely clear and well-made. Looking forward to your next lectures!

  • @FelipeCampelo0
    @FelipeCampelo0 3 місяці тому +1

    When a video catches your curiosity in the first two minutes that’s because it’s going to be a piece of art !

  • @Ryush806
    @Ryush806 Рік тому +1

    I just learned your video was for 3Blue1Brown's contest by watching his video about the contest. Makes sense why you only have 1 video. I originally watched it weeks ago and was dumbfounded by it being your debut. PPLLLEEEAASSSEE make more! You have a knack for explaining a very complex topic in terms us commoners can understand!

  • @fabiospinelli4179
    @fabiospinelli4179 Рік тому

    Wow this video is Gold, I'm impressed by the fact that this is your first!
    I will look at your career with great interest

  • @sinaomidvar2807
    @sinaomidvar2807 Місяць тому

    The content you provide is greatly appreciated. We are looking forward to your new contents. ❤

  • @htfx11
    @htfx11 Рік тому

    This video REALLY helped me make sense of Hopfield Networks. Outstanding work

  • @clementej03
    @clementej03 Рік тому

    Amazing video, I tried my self to model a neuron network on my own but barely even knew where to start. Fascinating subject and an excellent video.

  • @donewithwork
    @donewithwork 11 місяців тому

    Nice video!! Really like how you broke down complicated terms into more manageable discussions.

  • @satadrudas3675
    @satadrudas3675 Місяць тому

    This was a fantastic explanation. Waiting for more from you!!

  • @symbolspangaea
    @symbolspangaea Рік тому +2

    One of the best explanations I've seen

  • @meteorinc4259
    @meteorinc4259 Рік тому

    What a great quality video! Please keep releasing more like this!

  • @acyned8079
    @acyned8079 Рік тому +1

    The conclusion really was what I was here for. I didn't get most of it but I got the point of it!

  • @artysanmobile
    @artysanmobile 3 місяці тому

    The thumbnail image makes a very important point on the subject of human thought. We don’t look for the room, shelf, book, page. We build a complete path to every experience we have ever had since birth. If we contextualize it by re-use, or by interconnecting it with another path, we improve its clarity, its importance, and remember that address. The classic explanation for having no idea what you ate this morning. Breakfast has not yet been made important.

  • @LamNguyen-zc8xs
    @LamNguyen-zc8xs Рік тому

    This is brilliant. I've never seen any video about Hopfield network as interesting as this video. All they show are just math. Man you should do more video on brain and neuron networks like this :D

  • @hanswoast7
    @hanswoast7 Рік тому

    I really like your definition of memory! This way we can use fix point analysis and non-linear dynamics to model memory. Great!

  • @chanyan2024
    @chanyan2024 Рік тому

    Congratulations on winning the honorable mention prize for the Some2 competition. Your entry is my favorite

  • @jlevenson1745
    @jlevenson1745 Рік тому

    Pieces of images and thoughts put back together in our brains. Makes a clearer picture.

  • @AK-vx4dy
    @AK-vx4dy 3 місяці тому

    Excelent explanation in quite concise form. Great job !

  • @ariondorin4480
    @ariondorin4480 11 місяців тому

    This is awsome ! Man, you really understand things deep and what is more, you really explain the essential!

  • @loganpowell
    @loganpowell Місяць тому

    Dude, this is the best video on the topic I've found (and I've looked). When can we expect a follow-up? Would love to watch it.

  • @mauroabidalcarrer4083
    @mauroabidalcarrer4083 5 місяців тому

    Can't wait for the follow up video!

  • @spin6720
    @spin6720 Рік тому +2

    I don't usually comment on youtube but.. I don't know what to say. This is by far one of the most enlightening videos I've watched ever. I just subscribed and I'm looking for your next video

  • @Mangputt
    @Mangputt Рік тому

    I can't imagine, how great you put effort to this Informative Video. The way you explain, animate and Compositing evidence, awesome..
    (Sorry for my bad English)

  • @asanzramaxi
    @asanzramaxi Рік тому +1

    didnt realize the views and subscribers until casually saw comments. keep it up, such a good video!

  • @spicysciency8428
    @spicysciency8428 Рік тому +2

    great video can't wait for the follow-up
    this channel will be huge soon

  • @michaeljames5936
    @michaeljames5936 11 місяців тому

    Magnificent! Thank you.

  • @fabioteixeira868
    @fabioteixeira868 Рік тому

    Wow! Awesome first video! Subbed and eager for more!

  • @toni8632
    @toni8632 Рік тому

    Thanks for the tutorial, it's much faster than any other method I came across.

  • @noahcordier7002
    @noahcordier7002 Рік тому

    This video was clearly made by a perfectionist, and it's very enjoyable. I can't imagine the time taken to produce it

    • @HubertRozmarynowski
      @HubertRozmarynowski 11 місяців тому +1

      You're absolutely right about him being a perfectionist.. This video is literally among the best I've watched on youtube and he's still not happy with it (judging by his replies to criticism)

  • @TheMan84466
    @TheMan84466 Рік тому

    just stumble into this video on my feed absolute awesome please make more videos like this !!!!

  • @manofculture69
    @manofculture69 Рік тому

    Astonished that this video is the first video by you. Kudos 👌

  • @alexanderk5835
    @alexanderk5835 Рік тому

    Nice way of presenting and storytelling, got me motivated to do some experiments.
    Thank you for the video.

  • @loeschbart
    @loeschbart Рік тому +1

    Sehr gutes und aufschlussreiches Video, gerne mehr!

  • @generativeresearch
    @generativeresearch Рік тому

    This video is really well made for a first video. Keep up the good work man

  • @pumkin7222
    @pumkin7222 Рік тому

    great first video! keep it going I'll be here to support you!

  • @fr3zer677
    @fr3zer677 Рік тому

    What a fantastic video! I'm eager to see what comes next!

  • @user-pd4bb8tx2n
    @user-pd4bb8tx2n 5 місяців тому

    What a fantastic explanation. I'm looking towards more content from you!

  • @revimfadli4666
    @revimfadli4666 Рік тому

    Looking forward to the modern Hopfield network video! Very clear explanation