Can ChatGPT Handle Infinite Possibilities?
Вставка
- Опубліковано 14 кві 2023
- Patreon: / mlst
Discord: / discord
Twitter: / mlstreettalk
In this video, Dr. Tim Scarfe and Dr. Walid Saba discus the wonders of Turing Proof and the distinction between recurrent neural networks and recursion. He explains that recursion is a computing model with clear semantics for stopping criteria and base cases, while recurrent neural networks simply involve a repetitive process without the same constraints. Dr. Saba also emphasizes the power of representing infinite objects with finite specifications, as seen in the practical infinity of compilers like Python, which must be ready for an infinite set of valid programs.
He then addresses the question of whether natural language models, like transformers, can approximate recursion to a level of complexity that covers most of the language for practical purposes. While these models have done a good job of approximating recursion so far, Dr. Saba cautions that the debate is not over and that more testing is needed when these models are used in real-world situations.
TOC
The magic of Turing Proof
Recurrent Neural Networks vs. Recursion
Recursion as a computing model
The power of representing infinite objects
Practical infinity in compilers
Approximating infinity in natural language
References:
1. Turing, A. (1936). On Computable Numbers, with an Application to the Entscheidungsproblem. Proceedings of the London Mathematical Society, 42(1), 230-265.
www.cs.virginia.edu/~robins/T...
2. Chomsky, N. (1956). Three models for the description of language. IRE Transactions on Information Theory, 2(3), 113-124.
ieeexplore.ieee.org/document/...
3. DeepMind (2023). Mapping the Chomsky Hierarchy to Different Neural Network Models. (ICLR 2023)
arxiv.org/pdf/2207.02098.pdf - Фільми й анімація
One can tell that Walid is not just a great Academic, but an experienced Teacher. Just sooo talented in grounding abstract noise in the real world 🤩
Infinity is not the antithesis of probability. Consider a function that first internally generates a uniformly random real number n between 0 and 1 and then returns n if n < 0.5 or 1 if n >= 0.5. We can say that the probability of the output being 1 is 0.5 even though there are infinite possibilities.
When speaking of language prediction, it's not nonsensical to say that there's a p>0 probability that the next sentence is X even if there is an infinite number of possible valid sentences.
These discussions go on just due to a lack of formality. Probability is very clear: you cannot define probability to a point on R (because there are infinite), but you can still define a density function from which you obtain probabilities over any non singleton subset. Now are there infinite sentences? Definitely yes. But There still are fundamental characters, out alphabet, which make a sentence a non singleton point. So obviously you can at least estimate the probability of any sentence. I don't really get the overall discussion here as I am lacking background on most things they are discussing, so hope this turns useful
I especially loved the Turing with six fingers on his right hand. ;-)
It didn't occur to me to check for that, but your post made me re-watch to check!😂
Walid is picking and choosing his practical infinities. I would argue that it is actually extremely possible to predict in the general case what a person is going to say, or type or write next. This is precisely why LLMs have such utility in an autocomplete scenario.
Eye, that's the rub. "Extremely possible" is not "absolutely certain". It is well known in physics singular limits and divergent series occur naturally all over the place, you cannot use approximants. So whenever a phenomenon depends upon a singular limit (you must actually sum all the way to infinity, (eg., as we do with Ramanujan or Cesàro summation) the machine algorithms are going to fail/break/be irrelevant, unless they actually are conscious so can think about summing the divergent series. I mean, you can go far in _behaviourism_ with dumb approximants and zombiehood, but it's never going to be subjectively conscious, so will never have subjective semantics. Major Ai philosopher-nerd mistake is thinking consciousness and semantics can boil down to behaviour.
We OLMs (Organic Language Models) have a very sophisticated language prediction program. Being approached by someone signaling intent to speak we immediately begin predicting the topic based on environmental data:
Time of day.
Any remembered prior communication from the person.
Gender, age, and state of dress of person.
Proximity of business establishments or opportunities.
Etc...
We even have a near universal error handling routine for those rare occasions where the incoming conversation stream has evaded ALL of our situational prediction routines.
We Bust Out Laughing.
My god what an amazing and valuable interview. I appreciate how you were able to ask the questions I was thinking, but also asked questions I hadn't considered that were extremely interesting. Thanks so much for this interview, PLEASE continue the good work!
The algorithm of a compiler can handle infinite number of inputs, but an actual implementation can't. If you limit the program's length, then there's only a finite set of possibilities, because there are finite number of symbols to choose from and a finite string you are allowed to construct from them. So to have infinite number of programs, you have to allow infinite length, and that's where any practical implementation will run into problems. The machine it runs on have finite memory and disk space, the CPU has finite address space and finite registers, so eventually something will break. Today it's not a practical issue, because the limits are so big, but a few decades ago programmers had a hard time staying within them.
Technically you can write a compiler that can overcome all of the limitations except disk space, but no actually existing compiler does that because it's unnecessary in practice, and would drastically decrease speed.
If we ignore practical implementations, and consider the largest theoretically possible computer, it's still finite. Even if the universe itself is infinite, any computer you can construct in a finite amount of time will necessarily be finite, because the building blocks have finite size and finite possible states, and nothing can go faster than the speed of light.
I personally like Stephen Wolfram's theory, because it works way too well to be wrong, and it guarantees a finite universe.
Also, allowing infinity to exist creates contradictions, which generally considered a proof of your assumptions being wrong.
Although I agree with the practical side of your comment, I think the point of this debate was about RNNs "approximating" natural language i.e. trying to predict an infinite set vs. a grammar being able to digest anything from that infinite set (even the power set). In practice, any model's training data can only be finite, otherwise we're trying to compress an infinite set into a finite latent space which is impossible. There'll always be test text (pieces of natural language) that are outside the finite set a given RNN can approximate. Anyways, I highly recommend reading Deepmind's paper on the topic, this may ignite new NLP-Recursive machines hybrids
@@pieriskalligeros5403
Yes, but it's a theoretical problem, not a practical one. Just like how the Halting Problem never had any effect on software development, or the Incompleteness Theorem never prevented the solution of any practical problem.
In practice we don't need infinite intelligence, neither a perfect one. We just need something comparable to a human mind, which is incredibly stupid and has dozens of well documented unnecessary flaws. In practice, GPT-4 is there in many practical ways. I use it in practice to help me with my day job, and it already saved me countless hours of frustration.
The philosophical part is interesting and worth studying too, but just because you can prove that something isn't perfect, that doesn't mean it's useless.
It's ironic how he calls out AI researchers for being biased, when it's much worse in philosophy. We all are inherently biased because it's our brain. And even worse, it's not just the home team bias, but it's also the thing we use to think about the limits of thinking. So of course we conclude that we are the smartest in the universe and we don't have any limitation. If we think about all the things we are able to think about, then we find that we can think about all of those things, therefore our thinking is unlimited. But that's obvious circular reasoning. Obviously we can't think about things we can't think about. Through science we discovered things that we can't think about (like more than 3 dimensions), and many ways our thinking is flawed (cognitive biases). So it's not just a philosophical problem, but a very practical one.
@@andrasbiro3007 agreed, it's not about one or the other. I don't think humanity is thinking (or should) we're the smartest out there, it's just a matter of exploring. Most times we destroy everything in our way of exploring our capabilities, theories or even the universe (e.g. space exploration, parallel to lives destroying our only active life-supporting environment). We are biased and will remain to be probably forever, people were biased outside and within science since dawn of time. It's an ongoing thing, science, it debunks itself, I highly recommend reading 'The structure of scientific Revolutions"
Wow. This helps me understand better the critics of LLMs. I love the tone Dr. Saba uses at the end like: "Approximating infinity?".
Omfg that guy is a genius! Simplic explanation of infinity
In its simplest sense, which doubles as the metaphysical “base case” of a minimal/maximal model *theory*, recursion is self-infinitization.
Very insightful, glad I watched! Thanks for putting it for public.
Loved this. Very insightful. Inspires also an important perspective to alignment; among the infinite possibilties of inferences, decisions and behaviors of AI that we in the worst case can't predict or prevent, are such that produce suffering or termination of humanity. The question and issue is of limiting the occurrence of those.
Wish that had gone on for 3+ hours; that was very cool
random variables can be defined on infinite sets, eg gaussian distribution. interviewee several times hallucinates that probabilitY over infinite sets makes no sense, that “anything over infinity is zero” eg @17:00
Fair enough he makes a valid argument, and the parsers in modern compilers do actually accept arbitrarily large valid programs (in theory infinitely large, but they would then need infinite amounts of time and memory). The question is if we arbitrarily restrict the size of the program to say 1TB, are neural networks/transformer based systems unable to build an internal representation which you can ask interesting questions (e.g. what is the output of this program). Or if the program had an infinite loop in it, would a transformer recognize it.
To have an infinite large program the universe would have to expand infinitely fast, or have infinitely devideable parts or infinite dimensions on other planes all of these thing would not work quite out quit easily with the symmetries we see in physics I guess.
@@norbertfeurle7905my comment wasn't clear. What I was asking was, first could we design a neural network that accepts a program and generates a parse tree for a language that has recursion and if so how would the size of the network scale based on the size of the program/grammar. If the network would need to be so large that no computer is large enough in the universe to do it, the. Walid has won
The Halting problem sounds a lot like MMs pulling the rug on a stock after enough people predict it will go up and buy it. This happens because they watch retail positions on their L2. Similarly, the act of observing and reacting breaks this Turing program. Ultimately, the more you react on predictions, the more the system reacts to change them.
So if I understand this correctly... If I have a python function that calls itself to parse through every node in a Json, or every folder on a hard disk, that is a recursive function right? But in an RNN, it only repeats the creation of the token, but doesn't understand the "content" of the token (e.g. it doesn't see the potential "nodes" that exist in its data and thus can't recurse into them and parse all the sub structures inside it? Or something like this?
Nice conversation.
Good stuff! Very interesting question repeating vs recursion. Actually it seems like recursion implies preserving a context (on the stack) in addition to having the stopping condition of the base case. I'm guessing their might be a way to define a RNN that preserves some defined context on the stack, but that is the issue, you don't know exactly what to preserve and then you have a problem with the deepness of the stack and deepness of the neural network layers. Usually in a computer algorithm the context to be preserved is small compared to available stack space.
Excellent point (recursion implies preserving a context (on the stack)) and for RNNs to do that they have to admit symbolic computation.
@Machine Learning Street Talk
Just because an element in an infinite set has a probability of 0 doesn't mean that it is impossible. In the continuous space between 0 and 1 each specific number has 0 probability but what matters is the probability of a range and the limit as that range decreases. This gives rise to the probability density function. In a similar sense, we may define a pdf for python programs where the area of the pdf is the probability of a range of programs. I'm not sure however what would be the most sensible way to order python programs
You can define a pdf for Python programs by lifting the sintactic rules of python to probabilistic sintactic rules.
Since the set of all possible programs is countable, there is no need to define a continuous pdf. It is possible to define a descrete probability distribution with a non zero probability for each program.
@@juanalvarado6943 In that case, how do you define a range of programs analogous to p(0.75 > x > 0.5) for x in [0,1]?
you got fantastic perspective
@UA-cam- Sucks- the problem is that we don't have a continuum since programs are made up of discrete tokens. So it is really not clear how to define a range of programs.
One way around it may be to embed the program in a continuous space
How do you test with certainty that a Python compiler can handle an infinite number of valid programs? Isn’t that a unrealizable and therefore forever an abstract infinity?
Thanks!
A program that can recognise if a bit string is even or not (the parity problem) can be applied any element of the infinite set of bit strings. With a simple (and finite, duh!) recurrent function, I can define an infinite set of bit strings that will be classified as even or odd. If I implement the parity function, I have to keep in mind that it has to work for an infinite number of possible bit strings. This can also be done by an RNN. What is the argument at 14:00 even about?
Is the RAM or disk space that contains the string bits magically infinite or something? Your program will not handle all the possible strings that could exist and it's just a fact.
“Insanity is doing the same thing over and over again and expecting different results.”
QED
Amazing channel
Introducing: bed time stories for computer scientists!
How does this compare to autoGPT that continuously runs to complete a given goal, it provide himself with instructions how to complete the task? But how would he detect that at some point his intructions are part of a infinite loop of the repetitive instructions or not?
13:10 What does it mean to be ready?
I wonder if it is truly about division by infinity, or could it be that the space of possible questions contracts significantly when a person formulates a query while another assesses the size of this question space? Certainly, not all questions have an equal likelihood of being posed. The probability of a particular question emerging is influenced by the person's recent experiences, interests, and upbringing.
excellent explanations
I’m not 100 sure if the number of pythons programs is infinite or ‘just’ enumerable ;
The set of all valid python programs is likely bounded, not infinite - you need to make them out of matter, there's only so much of it within reach. If you put too much of it close together it collapses into a black hole. You can't compute faster than the speed of light. There was a start to the universe, there will be a point when matter ceases to be, there's a smallest length in both space and time, and all the things within it seem to be discrete.
The evidence suggests is that the cardinality of all possibilities is extremely large but bounded, and infinities and continua are artefacts of our models - it's the map, not the territory. Someone smarter than me could probably put a number on it, but it's likely to be less than 10^60 bits in length or instructions. This is infinitely smaller than the cardinality of the natural numbers, which, like zero and negative numbers, are also just concepts.
Goedel's incompleteness theorem is our curse and our blessing forever.
The set of (python) programs is not infinitely large, but combinatorially large, for a finite code file
its infinite large, you would have to freeze compiler upgrades, freeze ram size, freeze the ability to push state to other machines, disk etc for your statement to be correct.
@@TheReferrer72 at the moment of compile on a given machine all those things are in fact fixed, kind of paradoxical.
@@jessedaly7847 You are just adding more constraints to prove a point that is not representive of the real world.
ChatGPT is infinte because you will never be able to input enougth symbols to test all output without coming against the heat death of the universe.
Thats why hes talking about "practical" infinity
It's infinitely large in the same sense that the cardinal of N (the set of natural numbers) is. We can always make a different program, bigger, adding a new line of python code. In the real world, of course, we couldn't write such programs. But we also can't write some very big numbers because there isn't enough paper to print them in the Earth.
Infinity, program, compiler... these are mathematical concepts, they are "fully real" only in the Platonic sense. Implementations are finite by necessity, not by nature. Think about floats, for example, and the numbers they represent. The real cardinal of the real numbers between 0 and 1 is infinite, but the number of possible float values is not.
This is, unfortunately, a naïve view of infinity. I can't predict what he's going to say next, that doesn't mean it's not predictable. It can *approach* infinity, but it *is not infinity* ... to use an example, that example even.. Ignoring all possible analytics for actually predicting what he's likely to say - for example researching what languages he's capable of speaking, or making some (valid, as it happens) assumption that he will continue making his point in some way, thus limiting the context of the words chosen to grant the lowest possible likelihood of successfully predicting the next word, or string of words given. If we simply base the probably on the absolute limits that we, as far as we can, know to be true (and to assume otherwise would also disprove this argument of infinity, but hopefully that'll be self evident) then we could theoretically apply a hard number, and thus not infinity, to the potential limits of "sounds / communications" Dr. Walid Saba is physiologically capable of making. Again, offering the greatest advantage to the attempt at reaching infinity, let's include all potential nonverbal communication as well. There is still a real number of possible outcomes. Were he to continue asking his question from here, til the end of his life, there would still be a finite number of possible "actions" (actions meaning sounds, communications, etc.). Even if we continued asking this seemingly never ending question from here, til the end of humanity, and extended the question to all living organisms capable of communication through any form of any kind... *It would still be finite*
*This is the nature of infinity. It defies understanding, it is beyond explanation. It is the ineffable. The void before time, and the void after time, and all the time in between*
All of the "practical infinities" are not even close to infinity... they are very much finite, compilers have limits imposed by underlying structures, thinks like 32/64 bit integers, there is only a set of CPU instructions and PCs have a limited amounts of ram and storage, so even if you took all of the computers that exists (and even all of the computers that will be made) that number is still finite.
ua-cam.com/video/4KIQH1VEwBI/v-deo.html
That's the heart of the issue. Transformers can "approximate" recursively enumerable languages. I'm glad Dr. Walid Saba thinks that it is still an open question if this is enough. A question, that could be soon answered perhaps empirically.
Certain streets of San Francisco are definitely infinitely wild.
There are an infinite number of real values between 0 and 1, and yet we have probability distributions that cover all these values. Since we can't talk about the probability of a single point we instead talk about probability density functions, and then we can talk about the probability of ranges by integrating the PDF over that range.
For any specific sentence the probability would be 0 on account of belonging to an infinite set, instead we would have to look at the probability density in a distribution. Then we can certainly talk about the probability of collections of sentences.
But sentences are discrete, hence you cannot use densities, they require the distribution to be absolutely continuous. You would have to define a discrete probability mass function for the infinite set of possible sentences.
Polydactylism at 2:22 :]
Spotted the 6 finger hand
Taking Walid’s point on board, it seems the descriptive power is the thing, not the exercise of it over infinite sets. Nearly everything is better optimized with a bounded heuristic approach- approximation.
It’s the ability to create compact representations and operate over *those* to extrapolate outcomes- reasoning -that really moves the needle.
But if you do the same operations over and over within a space of typical activity (LLM), then approximation certainly suffices for a large variety and extent of semantic IO.
The question as a lot are posing in one way another in the comments is, when is this not just an academic distinction? It seems difficult to formulate that question exactly in the case of chatGPT given our poor understanding of its use of finite resources, i.e. what is it doing with all those network weights and layers exactly? (Space and time)
@@oncedidactic
I mean from a calculus perspective infinity is just a symbol representing the hypothetical result of a generator.
The generators themselves are the practical infinity, because of the unending possibility. Whether or not those possibilities are realised is beside the point.
If the box has literally no limit to it's potential output, for all intents and purposes that box is a representation of infinity.
Can it be proved that generative AI can always produce something it has never done or seen before ? If so, it is a practical infinity.
@@SjS_blue yeah, roughly agree with this. Stating it like such even if someone may raise a counter argument, still points out that essentially we need better science of NNs and LLMs to have a theory that can explain whether they do or don’t cover the infinite possibility space.
Since the context length is finite, there are only so many input states, but a very large number. Would be nice to have a rough measure of the sensitivity to boring vs productive IO mappings. i.e. a sole token repeated sufficient times would expectedly produce the same output, and such progressively more intricate still much collapses the IO space. In other words, a lot of inputs are incoherent and should generate nearly null behavior.
This is a great explanation of why we find GPT output so banal and recognizable. Statistically probable text is inherently uncreative. GPT is only ever as creative as its prompts, and those necessary come from human minds.
I suppose that implies our minds are (at least) turing machines.
That's why self prompting and longer outputs is vital to get more interesting and novel results, and / or allowing more random choices when selecting the token, but that also leads to more dravel. A lot of it is not inherent to the function of the GPT structure but how we use them and train them. For example, just allowing/training them to plan and reason "out loud" allows them to grasp and work with more complex and novel issues since they don't have an "inner monologue" that stays with them between tokens.
Any input into a transformer in A LLM is a query based on the context provided in the form of tokens, where the token order matters, and assuming the LLM is deterministic and nothing is added to insert randomness into it, we are really doing a search into the solution space with the results bounded by the creativity of our queries.
Aren't you two grasping at singular limits and asymptotics, divergent series, they arise in physical reality - cannot be evaded. So that seems to answer Saba's question. Approximation is not good enough with many singular limits, or whenever you need a Ramanajan or Cesàro summation that is not analytically discoverable. You have to actually sum to infinity, and it'll always blow up. So how do you even know to renormalize or apply Ramanujan methods etc? Only the most trivial divergent series can be Ramanujan or Borel or Cesàro summed. If your (zombie Ai system) does not "know" (LOL) it has to solve a divergent sum problem it is not going to know to Cesàro sum, so will fail.
The sharp question no one can answer is whether subjectivity and semantics might require such "non-computable" summations. Philosophical. But fun to think about anyhow, if I am right then it's another one of those reasons to suppose consciousness or even just "physics" might not be computable. And guys, no one knows whether quantum gravity is computable, it well might not be, and it has ramification we now know for ordinary everyday matter if the ER=EPR conjecture is correct (quantum gravity is around us everywhere, wherever there is entanglement). I mean I cannot prove subjectivity and semantics depends irreducibly upon such putative non-computable physics, but nor can Ai philosopher-nerds prove it doesn't.
Didn't know Turing had 6 fingers 😅 @2:13
We might have to update his wikipedia article now bro, the secret is out!!
Human's can't handle infinity (or even a large number of possibilities) either.
Is infinity a good example of symbolic representation ?
Does approximating infinity really sound silly? Don't we already do that with cat/dog classifiers? There is an infinite set of cats and dogs, yet it is possible to correctly slot them into categories with some degree of accuracy based on visual or behavioral cues. Interesting conversation, but not sure what conclusion I can draw from it.
Great video, but there is a big caveat, captured by Chaitin's constant, a real number you can calculate the first n digits of, that gives you the probability that an arbitrary length program will halt. Impossible infinite tapes don't affect it, and the failure of the oracle in one abstract case doesn't prove that an oracle can't exist that calculates the halting probability of some X% of the program space, or the space that represents the problems we're interested in right now. This might include natural language, as walid suggests! But I have to disagree with walid, compilers can only accept finite programs of length N, where N is your computer.
He was talking about a theoretical infinity of possible programs. Practical limitations such as memory etc. are not related to that argument.
@@Michal_Wlodarczyk2100Is 2^n not infinitely smaller than infinity? It's definitely not a Hilbert hotel scenario, the logic is very different ;-)
I don't understand why they say an RNN or similar auto-regressive model cannot implement a recursive mechanism. Of course it can given that it has enough memory for a particular problem and if it is trained to do so. Just like any other finite turning machine is capable given enough memory. I can just generate traces of a recursive algorithm and train an RNN to emulate those. Of course this is possible?!
Yes, recurrence is not recursion. But a recurrent function can implement recursion. Recursion is just a more constraint version or a subset of recurrence. If your recurrent function is properly implemented it can do recursion. Your turning machine is recurrent. And your turning machine can do recursion too.
I really don't understand what this video is about.
The term "practical infinity" is an oxymoron to me. Yes, a compiler can compile all valid programs, but in *practice* you can not _represent_ all programs.
I am surprised too, by people who struggle to conceive or accept infinities, and will say, OK, maybe as a theoretical thing, it is possible, but never physically or in practical terms. Why not? Why should that be impossible? To me, it is perfectly conceivable and sometimes even more intuitive than finity.
There's no such thing as infinity. You don't even conceive of it. No matter how pretentiously you sound off "perfectly conceivable." You simply replay the notion from a set point to a finite end, and create a virtual simulation of the idea of infinity.
@@dunebuggy1292 , I disagree.
Infinity can be pictured as a perpetual addition of finite parts, if you decide to look at things in terms of discrete fragments of reality, but you can as well picture things differently, in terms of a boundless continuity of whatever you want.
Whichever way you choose to picture things, I feel very comfortable with the idea of infinity.
The fact that it is, in the first place, an abstraction of ours, does not prevent it from capturing a possible reality, and capturing it essentially well.
When you think of finite objects, these will be abstractions in your mind exactly the same, not implying anything in terms of them being essentially right or not in their representation of something real or possible vs impossible to exist or apprehend.
Let's talk Euclidean geometry, let's imagine a straight line, OK? We can perfectly imagine any segment of it, same way we can imagine any real number interval, right? And we can just as well imagine the whole straight line, in its infinity. What problem do you have with that? We needn't have infinite minds to capture infinite objects.
@UA-cam- Sucks- , I see your point now, but still disagree, though not from your perspective. It's just that there's more than one. Same language can sometimes refer to different things or express more than one point of view.
For example, I can take your sentence and have it mean the opposite of what you intended. When you say that infinity is materially impossible by definition, being defined as a procedure that never stops... You've just said "defined as a procedure that never stops". If it never stops by definition, there you have it: an infinity, and one just proven not impossible by yourself, at least in abstract terms, by virtue of the fact that it was possible to define it in a clear, finite, consistent, complete manner.
Of course, if you look at infinity exclusively in terms of an open limit to a function, it won't exist in terms of ever being possible to reach. Then yes, "by definition", it will amount to a non existant end to your open interval. It will always lie, not one step, but an infinite (😂) number of steps away from your object, from your reach, "just" outside your function.
But this only regards infinity when considered as such a mathematical limit. It doesn't regard other things, like objects themselves, other than limits. BTW, infinity as a limit won't be reached and cannot be located, it won't have coordinates, but we can define and express it, we can operate with it, and it therefore definitely exists as an abstraction, same way the void set exists as an abstraction.
But consider the function itself, or the straight line itself: the object, as an object, is infinite. Real numbers are infinite. The impossibility to count them all doesn't matter, it doesn't mean it is impossible for them to be infinite.
Now let's move, philosophically, to the realm of physics. What argument is there to disprove the possibility of infinity? Why would infinite objects or dynamics (phenomena or procedures applying to objects) be impossible? Do we have any theoretical knowledge to disprove the possibility of such objects or dynamics? If you have it, please try to explain it to me. Until then, I have no problem conceiving infinite physical realities. If they are impossible, I am failing to see how.
@UA-cam- Sucks- , you seem to be telling, basically, that infinity, as a number, doesn't exist, which is obviously true since it is not a number. Which doesn't mean it doesn't exist. Not existing as a number, not being possible to calculate or compute, doesn't mean something doesn't exist.
I may not exist myself, from more than one point of view, but not from every point of view. There are points of view from which I do exist. I cannot be calculated (unless you find my wave-function, maybe) and I am mostly an empty space, sprinkled with an insignificant amount of physical entities that seem to be, for the most part, an illusion, in an illusory spacetime. Yet, here I am talking nonsense with you, unstoppably. 😅At least apparently, which, for a second, is a valid point of view.
@UA-cam- Sucks- , are you sure that mathematical objects can only be defined through limits?
What limit do I need to define a straight line? Or a function? Take the straight line. To picture it, I don't need to define it. Not much of a mathematical definition at least, just a dictionary definition or even a memory of physical representations of it. In my mind, I will be assuming and picturing it as never ending, without need of calculating it or going through its definition. A child can watch a circle for the first time, and just the memory of it will work as its definition. Its only definition will be "that funny object which I once saw (different from every other object that I also happen to know of)". I can point a laser up the sky and tell you: a line is any trajectory, either side of us, where this laser beam can sit. Where is the limit? (The sky is the limit ☺.)
Take y = ax + b: you know the line without need to calculate. (This is a oneliner 😝.)
Take the straight line again. I only need a couple points to define it, and don't tell us that will amount to three definitions for three non existing entities, a couple of which could be defined as the intersections of other straight lines (infinitely many couples of straight lines, actually).
To a point, you could probably always assign a number to a point, with the number being a perfectly, finitely defined entity, while the corresponding point would seem to be a "non existing" entity at the non existing end of zooming in for ever. You can take the view that a point just can't exist, by definition, but what's the point of saying that, in "reality", in "physical" space, an abstract point can't possibly exist? It seems pretty straightforward that abstractions being abstractions, mental representations of things, they don't live in "physical" "reality". Or maybe better: a representation of a thing is a thing, but one different, separate, from the thing it intends to represent or apply to. So what? Shall the represented thing not exist for that reason?
They should have spent more time describing this supposed "infinity" problem. I suspect that it doesn't exist or is not relevant to AI. As Walid Saba says, we can deal with infinity easily without actually having to represent an actual infinity. Turing Completeness is a quality that specifies the abilities of a machine. The brain easily has enough complexity to compute anything a Turing Machine can compute. Another non-issue. Formal languages allow infinite recursion but that doesn't present any problem for real human languages which always run up against practical limits. Same with programming language compilers. Although their languages allow infinite programs, they will run into physical and practical limits first so there's no reason to worry about it. Unless I'm missing something, which is possible, there's no "infinity" issue here.
just remember about fiction film when I watched from childhood but that reality-fictions will not far from now 2023, human should take it seriously please
If you like this kind of content, I would also recommend this channel: www.youtube.com/@rscgtns
Uhhh…. Did you ever finish asking the question?
This is ridiculous. All sequences are finite. A compiler can handle the finite number of programs expressible in its finite input sequence.
Arent auto-gpt-like models making this irrelevant? The model can simply define and intepret it's own recursion semantics, right?
It's certainly true that adding recursion gives them a missing capability. There are still massive challenges with auto-GPT style models, most important of which is the "specification problem" - you can't clearly define what you want it to do and how it should evaluate its results. Also there is a difference with the model not being in recursive mode at the time it was trained. So these models will only be able to do trivial tasks.
But it can become infinitely better at chasing infinity
Next sentence is *practically* infinite, but is it actually mathematically infinite? We have finite words and finite sentence lengths (barring run on sentences or ridiculousness) -- let's say maximum sentence lengths of 100 words to be conservative.
The permutations of words generating sentences with correct grammar, while enormous, remains finite, not infinite. Furthermore, in context, there would be a probability distribution over all possible sentences, with the vast majority of sentences having a negligible probability.
I'm not saying Dr. Saba is wrong, just that I don't understand.
Two things- we don’t actually have finite words u less you choose this for sake of argument, as any human group will constantly invent new words.
The, semantics has infinite shades of meaning “underneath” the words. So practically speaking there is a combinatorical but finite amount of utterances, but the context of any utterance sets its meaning. I realize this is “escaping” the bounding of the problem but if practicality is at stake then the contexts of language usage and semantic relevance must be included.
The systemeticity of language seems to me much more about having a dependable way of packing/unpacking as a recourse and a theory, not the action mode. So a formal ruleset for doing this is helpful as a backstop, a catchall, but in practice probably brains and putative AGI indeed just approximate with layers of heuristics and pattern matches and semantic embeddings. They have the recursive descriptive power but don’t often employ it.
If you don't explicitly constrain a sentence to a finite length, there are, in fact, an infinite number of valid sentences: just start listing an infinite number of items like one, two, three...
@@strictnonconformist7369 sure... I guess the starting point is to define "language", "sentence", etc.
@@literailly the fun part about human languages is they cannot be defined for all valid words other than recursively within themselves.
Sorry couldn't get American pie out of my head. Now Im Sure that's infinite throughout mankind. Yo, what if the math we devlope in the future could only solve our problems till the precipice of our truth. That would make us believe it is perfect, no?
It can't even handle a Dockerfile
The conversation is quite frustrating, Walid does not allow the Tim to finish his argumentation, disappointing.
Yes, I felt exactly the same thing, he seems so upset by someone thinking that an approximation that has been learned from examples could have the same meaning as a platonically deducted meaning, that he refuses to dwell on it and so he is not playing devils advocate to his own point of view. He may be quite right but he should provide deeper insight in what are exactly the differences between the inducted approximation and the deducted platonic concept, and also reflect on the question wether an LLM that could ask itself questions before returning answers (like already is happening now with libraries like LangChain) could such an LLM perhaps then understand recursively defined concepts? And if not so, can we do any assertion about what extra mechanisms do we need to introduce? Can an LLM follow a “reason stepwise instruction” to add 2 numbers with a very large number of digits or is the fact that the LLM has only 12 layers somehow limiting its ability to follow a recursive definition? That would have been interesting topics to discuss but the way Walid discusses it here is a waste of time of all UA-cam viewers watching this, in my opinion.
But the number of finite possible programs is finite, not infinite.
And the number of finite possible things to say next is also finite, not infinite.
So no, I disagree. What's wrong with my logic?
I really hate python
I really really hate python
I really really really hate python
I really really really really hate python
...
@@MachineLearningStreetTalk that's still finite no matter how far you stretch it
Recursion is way more than just induction-recursion, e.g. codata/coalgebra, the Y combinator and other fixpoints, etc.. Induction-recursion is a (very useful) subset of recursion, but his definitions are wrong.
@DanielYokomizo
My point was that even "primitive recursive functions" (that are related to inductive data types - numbers, trees, lists, etc.) are more expressive than NNs. You are right, general recursion is a bigger set, but recurrence in NNs is not even as expressive as the class of primitive recursive functions (PRFs).
A compiler does not support an infinite set of instructions, yes the number is stupidly large but not infinite.
Programs. not instructions. Like the infinite variation of books, there is no end to the number of programs that could be written that a compiler can handle. Python is a language - LANGUAGE. Like saying there is a finite number of stories to be written.
@UA-cam- Sucks- This is pretty much the only argument to dispel all this infinity bullshit. We don't live in a world in which infinite things exist and I'm not really sure why anyone is wasting their time arguing about something that is physically impossible.
🌈🧩🐬💎🪷
Halting problem is irrelevant ..
Why do you say that? It is an instantiation of incompleteness
@Sad Face no computer programmer cares about it. Of course, some things are not computible. Ai is not affected in any way. My brain is not affected in any way. It's an interesting anomaly about computability and matters to a person trying to break German codes in a finite period of time. But in general, it is irrelevant.
@@farmerjohn6526 Knowing if programs are finite and deterministic is really important.
• The haulting problem come up frequently in computer science and is obscured by abstraction. Your opperating, your compiler, etc are all affected by the haulting problem. Programmers should care about it.
• The brain is not necessarily a turing machine so might be constrained by the haulting problem in the same way. If the brain is not affected by the haulting problem that would be significant.
• It is relavent to Ai if want to know which questions a system could answer. And in the Lukas-penrose arguement suggests it will limit the generalization of artifical intelegences.
@@farmerjohn6526 Deadlock detection is the same as the halting problem.
@@sadface7457 I don't like calling intelligence Artificial. So, I am trying to not use the term AI anymore, except in this sentence. Human Intelligence, assuming it takes place in our brain, and IMO, that is where it takes place doesn't have a halting problem. Our type of intelligence, like you said is not a computer, it is not computed, and the halting problem is irrelevant. So, assuming that general intelligence is human intelligence and assuming that we can create a brain or a brain like circuit either in a computer or in a new hardware device, it too, will not have a halting problem (unless the programmers make a mistake). I don't want to call this new type of intelligence Artificial or Machine. Most people don't think of it as a machine, but for the moment that is ok. So, I will refer to the man-made intelligence as Machine Inteligence to seperate it from Human Intellignece, while acknowleding it may be different, yet still not Artificial. The LP argument claims that Gödel’s first incompleteness theorem shows that the human mind is not a Turing machine, that is, a computer. The argument has generated a great deal of discussion since then. The influential Computational Theory of Mind, which claims that the human mind is a computer, is false if Lucas’s argument succeeds. I agree - the mind is not a computer. But a computer can simulate the mind.
Dude is in denial. Real world is not random, neither it is infinite in practical contexts.
I’m enjoying a well known chat AI app. I asked it a very easy question. It gave a disappointing reply that could only have been derived from “conventional” wisdom, rather than stark logic.
First of all, these systems are probabilistic, even if there is an answer that would satisfy you, novel connections between the concepts or syntax within it's network, it's more probable it's going to select the most probable ones from its training. Actual thinking in humans is a continuous iterative process both conscious and unconscious. These "brains" are locked. If you want to simulate "thinking" you have to achieve that with recurrent loops, feeding it's output back to it self. Giving it a system of self prompting, reflecting and questioning. Then you can for your simple prompt generate a much more complex "memory" or prompt that can result in more complex and unique outputs. That's where it starts to get interesting. Otherwise it's very improbable any output is going to be more novel than your input. Especially after they have been trained to align with human values and conversational output.
@@samandoria I very much appreciate your excellent reply. Many thanks from over here in Australia.