This is a clip from a conversation with Francois Chollet from Sep 2019. New full episodes every Mon & Thu and 1-2 new clips or a new non-podcast video on all other days. If you enjoy it, subscribe, comment, and share. You can watch the full conversation here: ua-cam.com/video/Bo8MY4JpiXE/v-deo.html (more links below) Podcast full episodes playlist: ua-cam.com/play/PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4.html Podcasts clips playlist: ua-cam.com/play/PLrAXtmErZgOeciFP3CBCIEElOJeitOr41.html Podcast website: lexfridman.com/ai Podcast on iTunes: apple.co/2lwqZIr Podcast on Spotify: spoti.fi/2nEwCF8 Podcast RSS: lexfridman.com/category/ai/feed/
One of THE deepest and most well tought through arguments I have ever heard. Mind boggling, must say this changed my view of the trajectory of ideas heavily...
It's been many years that i've written about this randomly. It has major political implications: societies are not getting more productive because we have mined all the easy nuggets of knowledge and now our production rate is essentially constant - in science and all areas. In many places it's actually decreasing because groups of humans lose motivation, of course this is accompanied by unrest and violence sometimes. I am surprised that François Chollet does not mention here complexity theory: stable dynamics lead to computational models abiding the TH. Then we can consider human societies as theorem provers, computers trying to prove increasingly difficult theorems. A new technology is a new theorem speeding some search process. This analogy is actually very real in many fields: for instance scientific research is about proving theorems, in a generalized sense. This is easy to see for mathematicians, philosophers, theoretical physicists; for experimental sciences we consider the Earth as a oracle on our Turing-limited model. When we consider artisan or industrial production, physical production in general, we can see that the issue reduces to applying limited forces and limited precision to create increasingly fine goods. The limitations on forces and precision come from physics, and then making better use of them requires combining them in increasingly complex ways and we can reduce this question to a scientific problem, by the above (sketch) a task of theorem-proving - or equivalently theorem-generation. Next question is: do increasingly pleasing goods grow linearly with their amount ? In normal conditions twice an amount of material is valued by society twice the amount of money. Utility theory is a complicated topic, and there are many nonlinearities but usually relations are linear. And with twice the amount of energy one can make twice as complex or "fine" goods, in many circumstances, though one adds nonlinearities and that depends on the definition of "complex" or "fine". Now we can apply complexity theory: the P != NP assumption tells us that human societies, as computers, cannot produce a very fine good without putting in it an amount of energy (time) that is superpolynomial, and probably something like close to exponential. An additional factor making difficult understanding this intuition (that is pushed back against) is that during a long time there were many springs of energy/complexity in nature that we could tap: for instance we could find gold nuggets when we began searching for them massively in the US, but now we cannot and gold mining is much more difficult. So while there were many hard limits to reach we could make the illusion of exponential growth last, but because our world is finite and its fineness (low level complexity) is finite too, we are bound to dry out the springs of complexity. And we have now. The low-level limit on fineness is of course fundamental physics, which tell us that fundamental particles are all equal (thus they do not contain complexity, there is no complexity below their size). There should be computer scientists working to upper-bound the limits of human society in terms of production. We would see that we have reached most of our capacity in the West - and we overestimate it. And most of the progress still to come would be from politics, social sciences. But that is another topic.
I think there is a default argument that we already got all the low hanging fruit, but that doesn't totally make sense to me, there should always be very new and barely figured out things to discover that are lower complexity, I think an argument could be made that the scientists we are training now are taught to re-work the existing knowledge base because doing the new things are too risky
There are so many simple ideas that I haven't seen get more funding, project starshot, mass drivers, using animals to train AI, really powerful desktop physics GPUs to do virtual experiments, garage genetic engineering kits, these all deserve much more attention and work than what they are getting right now and I suspect because most funding goes to these thousands of papers writing about that one minor discovery that is probably repetitive but is more credible
I think the statements around exponential friction are important. Relief of that friction through 'radical' changes in process will lead to bursts in scientific and technological progress. Francois even brings up one of the most important ones my generation and that's communication overhead among researchers. I also think as utilities like Paperspace Gradient, Sagemaker, GCN, Kaggle Kernels, start reducing time to experiment in ML, we'll start seeing more productive open discussions about shortcomings of existing methods and applications - including, fingers-crossed, abstraction of input vectorization. On the heels of these process problems is the slow, but steady, growth of automated crafting. Robotic construction is still not fast enough, or at scale, to significantly increase general access to specialized mechanical and testing devices. You can 3d print a case for custom silicon, but try doing lithography in your bedroom. There are still some major advances required in streamlining learning to mechanical reality I think we need to be focusing on.
this guy sounds a lot like Stephen Wolfram ... would love Lex for you (if possible) to get François Chollet and Stephen Wolfram together on a broadcast ... would really enjoy hearing their thoughts on physics ..
There's also the tendency of scientists to get things wrong or to publish banal papers that don't add any knowledge. The more money we spend, the more fluff and lies are produced.
Well there is nothing wrong with getting "things wrong". It's not a desirable output one would want but you can't make progress if you force people to never make mistakes. The second thing wholeheartedly agree; we have too much research being done for the sake of writing research papers rather than making real scientific progress.
Kurzweil says information processing increase is exponential. That is what makes everything has to do with information processing exponential. Exponentials are very powerful.
Kurzweil said that about 15 years ago. It's no longer true, and Moore's law hasn't held for a few years now. Maybe another information processing technology will come along, but until then Moore's law isn't true at the moment.
It's harder and harder to make breakthroughs. Just look at how much time it took Wiles to solve Fermat's last theorem or perelman solving the poincare conjecture. There is no low hanging fruit anymore. That being said, it doesn't really make sense to look at the number of papers published because there are a lot of low quality journals out there publishing results that may be interesting but not super noteworthy. For instance, you can't really get a job at a decent school by publishing in mediocre journals. You also can't really compare to the past because of the explosion of Chinese and Indian researchers who, on average, aren't that good.
Maybe You have to measure over a period of say 1000 or 2000 years to get a exponential curve. In you measure in a period of 100 years which is exponential for bigger time period then in that period you will a get flat curve
Ehh... The question was - Does heavy investment of manpower and time in research yield the same output in research impact? This is actually pretty hard to measure unambigiously, so you would want a baseline and they picked the Nobel Prize-winninf discoveries impact judged by a panel of expert scientists. The Nobel Prize started in 1901. Hence why the period dates approx. 100 years back. So whether we had a baseline that dates a thousand years back, or not, if it was exponential, it would clearly be evident in this case still. Unless you argue that it somehow was exponential in the past and suddenly started to stagnate in the past 100 years.. but I agree with you in the sense that their experiment was not perfect. In fact, it does not even measure anything past the year 2000, only between 1910-1990.
We can teach better. We can teach more broadly an in a more equitable way. A very important part of scientific progress is reasonably well informed pressure from below. Unfortunately majority of modern science lives in a paradigm of simulacrum.
This is a clip from a conversation with Francois Chollet from Sep 2019. New full episodes every Mon & Thu and 1-2 new clips or a new non-podcast video on all other days. If you enjoy it, subscribe, comment, and share. You can watch the full conversation here: ua-cam.com/video/Bo8MY4JpiXE/v-deo.html
(more links below)
Podcast full episodes playlist:
ua-cam.com/play/PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4.html
Podcasts clips playlist:
ua-cam.com/play/PLrAXtmErZgOeciFP3CBCIEElOJeitOr41.html
Podcast website:
lexfridman.com/ai
Podcast on iTunes:
apple.co/2lwqZIr
Podcast on Spotify:
spoti.fi/2nEwCF8
Podcast RSS:
lexfridman.com/category/ai/feed/
One of THE deepest and most well tought through arguments I have ever heard. Mind boggling, must say this changed my view of the trajectory of ideas heavily...
This was such a good explanation, I am sure some people will agree/disagree but it's clarity was superb!
In summary, the problems grow at exponential rates as well as the effort, so the overall progress remains near linear.
This guy is seriously raining on my parade lol
It's like coming out with a new product. At first there are high returns. Eventually, the margin gets smaller as the market catches up.
exponentionally more difficult to understand french people at 2x speed
It's been many years that i've written about this randomly. It has major political implications: societies are not getting more productive because we have mined all the easy nuggets of knowledge and now our production rate is essentially constant - in science and all areas. In many places it's actually decreasing because groups of humans lose motivation, of course this is accompanied by unrest and violence sometimes.
I am surprised that François Chollet does not mention here complexity theory: stable dynamics lead to computational models abiding the TH. Then we can consider human societies as theorem provers, computers trying to prove increasingly difficult theorems. A new technology is a new theorem speeding some search process. This analogy is actually very real in many fields: for instance scientific research is about proving theorems, in a generalized sense. This is easy to see for mathematicians, philosophers, theoretical physicists; for experimental sciences we consider the Earth as a oracle on our Turing-limited model. When we consider artisan or industrial production, physical production in general, we can see that the issue reduces to applying limited forces and limited precision to create increasingly fine goods. The limitations on forces and precision come from physics, and then making better use of them requires combining them in increasingly complex ways and we can reduce this question to a scientific problem, by the above (sketch) a task of theorem-proving - or equivalently theorem-generation.
Next question is: do increasingly pleasing goods grow linearly with their amount ? In normal conditions twice an amount of material is valued by society twice the amount of money. Utility theory is a complicated topic, and there are many nonlinearities but usually relations are linear. And with twice the amount of energy one can make twice as complex or "fine" goods, in many circumstances, though one adds nonlinearities and that depends on the definition of "complex" or "fine".
Now we can apply complexity theory: the P != NP assumption tells us that human societies, as computers, cannot produce a very fine good without putting in it an amount of energy (time) that is superpolynomial, and probably something like close to exponential. An additional factor making difficult understanding this intuition (that is pushed back against) is that during a long time there were many springs of energy/complexity in nature that we could tap: for instance we could find gold nuggets when we began searching for them massively in the US, but now we cannot and gold mining is much more difficult. So while there were many hard limits to reach we could make the illusion of exponential growth last, but because our world is finite and its fineness (low level complexity) is finite too, we are bound to dry out the springs of complexity. And we have now. The low-level limit on fineness is of course fundamental physics, which tell us that fundamental particles are all equal (thus they do not contain complexity, there is no complexity below their size).
There should be computer scientists working to upper-bound the limits of human society in terms of production. We would see that we have reached most of our capacity in the West - and we overestimate it. And most of the progress still to come would be from politics, social sciences. But that is another topic.
I think there is a default argument that we already got all the low hanging fruit, but that doesn't totally make sense to me, there should always be very new and barely figured out things to discover that are lower complexity, I think an argument could be made that the scientists we are training now are taught to re-work the existing knowledge base because doing the new things are too risky
There are so many simple ideas that I haven't seen get more funding, project starshot, mass drivers, using animals to train AI, really powerful desktop physics GPUs to do virtual experiments, garage genetic engineering kits, these all deserve much more attention and work than what they are getting right now and I suspect because most funding goes to these thousands of papers writing about that one minor discovery that is probably repetitive but is more credible
Scientific advance is like expanding a balloon of knowledge; as its size increase, it becomes harder to expand.
I think the statements around exponential friction are important. Relief of that friction through 'radical' changes in process will lead to bursts in scientific and technological progress. Francois even brings up one of the most important ones my generation and that's communication overhead among researchers. I also think as utilities like Paperspace Gradient, Sagemaker, GCN, Kaggle Kernels, start reducing time to experiment in ML, we'll start seeing more productive open discussions about shortcomings of existing methods and applications - including, fingers-crossed, abstraction of input vectorization. On the heels of these process problems is the slow, but steady, growth of automated crafting. Robotic construction is still not fast enough, or at scale, to significantly increase general access to specialized mechanical and testing devices. You can 3d print a case for custom silicon, but try doing lithography in your bedroom. There are still some major advances required in streamlining learning to mechanical reality I think we need to be focusing on.
this guy sounds a lot like Stephen Wolfram ... would love Lex for you (if possible) to get François Chollet and Stephen Wolfram together on a broadcast ... would really enjoy hearing their thoughts on physics ..
Expert's subjective experience of significance is a meaningless measurement for the objective progress of science.
Ish
There's also the tendency of scientists to get things wrong or to publish banal papers that don't add any knowledge. The more money we spend, the more fluff and lies are produced.
Well there is nothing wrong with getting "things wrong". It's not a desirable output one would want but you can't make progress if you force people to never make mistakes.
The second thing wholeheartedly agree; we have too much research being done for the sake of writing research papers rather than making real scientific progress.
Kurzweil says information processing increase is exponential. That is what makes everything has to do with information processing exponential. Exponentials are very powerful.
Kurzweil said that about 15 years ago. It's no longer true, and Moore's law hasn't held for a few years now. Maybe another information processing technology will come along, but until then Moore's law isn't true at the moment.
It's harder and harder to make breakthroughs. Just look at how much time it took Wiles to solve Fermat's last theorem or perelman solving the poincare conjecture. There is no low hanging fruit anymore. That being said, it doesn't really make sense to look at the number of papers published because there are a lot of low quality journals out there publishing results that may be interesting but not super noteworthy. For instance, you can't really get a job at a decent school by publishing in mediocre journals. You also can't really compare to the past because of the explosion of Chinese and Indian researchers who, on average, aren't that good.
Lol the chinese and indian students are better educated than our students. The US is 'ahead' in terms of technology because of the H-1B visa
Maybe You have to measure over a period of say 1000 or 2000 years to get a exponential curve. In you measure in a period of 100 years which is exponential for bigger time period then in that period you will a get flat curve
Ehh... The question was - Does heavy investment of manpower and time in research yield the same output in research impact? This is actually pretty hard to measure unambigiously, so you would want a baseline and they picked the Nobel Prize-winninf discoveries impact judged by a panel of expert scientists. The Nobel Prize started in 1901. Hence why the period dates approx. 100 years back. So whether we had a baseline that dates a thousand years back, or not, if it was exponential, it would clearly be evident in this case still. Unless you argue that it somehow was exponential in the past and suddenly started to stagnate in the past 100 years.. but I agree with you in the sense that their experiment was not perfect. In fact, it does not even measure anything past the year 2000, only between 1910-1990.
Because half of the time is spent on finding which person in the lab stole your scissors.
Have an exam in about 8 hours and i'm watching this at 1am :(
Yes, it is logistic.
We can teach better. We can teach more broadly an in a more equitable way. A very important part of scientific progress is reasonably well informed pressure from below. Unfortunately majority of modern science lives in a paradigm of simulacrum.