MLST IS SPONSORED BY TUFA AI LABS! The current winners of the ARC challenge, MindsAI are part of Tufa AI Labs. They are hiring ML engineers. Are you interested?! Please goto tufalabs.ai/ REFS: [0:05:20] Ricardo's Law of Comparative Advantage from 'On the Principles of Political Economy and Taxation' (1817) - Referenced in context of explaining why economic trade theory doesn't guarantee peaceful AI-human coexistence (David Ricardo) www.econlib.org/library/Ricardo/ricP.html [0:08:05] Spearman's g factor theory from 1930s - Historical concept of general intelligence as a single measurable factor, proposed by Charles Spearman (Charles Spearman) psycnet.apa.org/record/2019-39185-002 [0:08:40] Computational irreducibility concept from A New Kind of Science - fundamental limitation in predicting system behavior without step-by-step simulation (Stephen Wolfram) www.wolframscience.com/nks/p737--computational-irreducibility/ [0:12:30] Raven's Progressive Matrices - Standard intelligence test used to measure abstract reasoning ability (John C. Raven) www.sciencedirect.com/science/article/pii/S0010028599907351 [0:15:55] Discussion of quantum-resistant cryptographic hash functions from NIST's post-quantum cryptography standardization (National Institute of Standards and Technology) www.nist.gov/news-events/news/2022/07/nist-announces-first-four-quantum-resistant-cryptographic-algorithms [0:20:25] Discussion of existential risk and meaning preservation connects to formal philosophical work on extinction ethics. Context: Argument about meaning of history and value preservation. Source: 'Essays Existential risk and human extinction: An intellectual history' (Thomas Moynihan) www.sciencedirect.com/science/article/abs/pii/S001632871930357X [0:20:35] Discussion references the K-Pg extinction event that led to dinosaur extinction and mammalian succession. Context: Used as analogy for potential AI succession of humans. Source: 'The rise of the mammals: Fossil discoveries combined with dating methods illuminate mammalian evolution after the end-Cretaceous mass extinction' (Philip Hunter) pmc.ncbi.nlm.nih.gov/articles/PMC7645244/ [0:24:30] A theory of consciousness from a theoretical computer science perspective - Academic paper examining consciousness through mathematical and computational frameworks, particularly relevant to discussion of consciousness and self-knowledge in AI systems (Lenore Blum) doi.org/10.1073/pnas.2115934119 [0:24:35] Discussion relates to the classical problem of other minds in philosophy, addressing how we can know about the consciousness of others (Anita Avramides) plato.stanford.edu/entries/other-minds/ [0:34:30] Discussion of quantum mechanical model of atoms and electron behavior in context of consciousness and computation. References standard quantum mechanical description of electron orbitals and quantum states. (Richard Feynman) www.feynmanlectures.caltech.edu/III_01.html [0:38:35] Highly accurate protein structure prediction with AlphaFold - Original Nature paper describing DeepMind's breakthrough AI system for protein structure prediction. In context of discussion about AI systems solving previously intractable scientific problems. (John Jumper et al.) doi.org/10.1038/s41586-021-03819-2 [0:41:10] Discussion of superintelligent AI as existential risk, referencing 'Artificial Intelligence as a Positive and Negative Factor in Global Risk', a seminal paper discussing AI safety concerns (Nick Bostrom) nickbostrom.com/existential/ai-risk.pdf [0:43:35] Whole Brain Emulation: A Roadmap - Technical report discussing requirements and challenges of brain emulation, including discussion of required scanning resolution and preservation of neural properties (Anders Sandberg and Nick Bostrom) www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf [0:44:50] Mind/Brain Identity Theory - Philosophical framework discussing relationship between mental states and brain states, relevant to Yudkowsky's functionalist perspective (J.J.C. Smart) plato.stanford.edu/entries/mind-identity/ [0:48:45] Discussion of personal identity and consciousness continuity relates to core concepts from Stanford Encyclopedia of Philosophy's entry on Personal Identity (Stanford Encyclopedia of Philosophy) plato.stanford.edu/entries/identity-personal/ [0:51:05] Discussion parallels Robert Nozick's Experience Machine thought experiment from 'Anarchy, State, and Utopia' (1974), which explores whether purely pleasurable simulated experiences constitute genuine happiness (Robert Nozick) rintintin.colorado.edu/~vancecd/phil3160/Nozick1.pdf [0:57:25] References his previous writing on no universally compelling arguments, discussing limits of rational persuasion (Eliezer Yudkowsky) www.lesswrong.com/posts/PtoQdG7E8MxYJrigu/no-universally-compelling-arguments [1:01:30] Discussion of mathematical axioms and commutativity property (x + y = y + x) references fundamental concepts in mathematical logic and axiom systems. This relates to Wolfram's work on fundamental mathematics and computation. (Stephen Wolfram) writings.stephenwolfram.com/2020/12/combinators-and-the-story-of-computation/ [1:01:50] The Simple Truth - A foundational essay on the nature of truth and epistemology (Eliezer Yudkowsky) www.lesswrong.com/posts/X3HpE8tMXz4m4w6Rz/the-simple-truth [1:01:55] Highly Advanced Epistemology 101 for Beginners - A sequence on epistemology, logic, and truth (Eliezer Yudkowsky) www.lesswrong.com/s/SqFbMbtxGybdS2gRs [1:02:10] Peano axioms - The foundational axioms of arithmetic in mathematical logic (Giuseppe Peano) plato.stanford.edu/entries/peano-arithmetic/ [1:06:30] Discussion of first-order logic vs second-order logic in mathematical foundations. Context: Explaining how conclusions follow from axioms in different logical systems. (Jouko Väänänen) plato.stanford.edu/entries/logic-higher-order/ [1:08:20] The paperclip maximizer thought experiment, demonstrating how an AGI with seemingly innocuous goals could pose existential risks by pursuing objectives orthogonal to human values (Eliezer Yudkowsky) www.lesswrong.com/tag/squiggle-maximizer-formerly-paperclip-maximizer
REFS PART 2: [1:11:40] Discussion of axiomatic truth theory in context of mathematical and logical validity. Related to formal logical systems and truth predicates. (Volker Halbach) plato.stanford.edu/entries/truth-axiomatic/ [1:13:35] Discussion of cultural relativism and truth relativity in context of different cultural frameworks and personal interpretation (Maria Baghramian) plato.stanford.edu/entries/relativism/ [1:17:20] Discussion of electron rest mass (0.511 MeV) and its effective mass variation in QED under different momentum transfers. Context: Used to illustrate measurement-dependence of fundamental properties in QFT. (M. E. Peskin, D. V. Schroeder) arxiv.org/abs/hep-ph/0001134 [1:18:05] Reference to up quark mass scale dependence in QCD, showing how measured mass varies with momentum transfer. Context: Demonstrating complexity of fundamental measurements. (S. Bethke) arxiv.org/pdf/hep-ex/0004021 [1:18:25] The Quantum Physics Sequence - A comprehensive series of posts explaining quantum mechanics and the many-worlds interpretation, published on LessWrong (Eliezer Yudkowsky) www.lesswrong.com/posts/hc9Eg6erp6hk9bWhn/the-quantum-physics-sequence [1:19:50] The Concept of the Ruliad - Detailed explanation of rulial space and its relationship to computation and physics (Stephen Wolfram) writings.stephenwolfram.com/2021/11/the-concept-of-the-ruliad/ [1:21:55] Second Law of Thermodynamics - Discussion in context of computational foundations and AI perception. Reference discusses how entropy increases in isolated systems and disorder tends to increase over time. (Stephen Wolfram) writings.stephenwolfram.com/2023/02/computational-foundations-for-the-second-law-of-thermodynamics/ [1:25:00] ZX-Calculus and Extended Hypergraph Rewriting Systems - Technical paper on multiway graphs and branchial graphs in quantum mechanics, providing formal mathematical framework for understanding quantum mechanics through graph-based models (Jonathan Gorard et al.) arxiv.org/abs/2010.02752 [1:26:35] Minkowski spacetime vs Euclidean spacetime - Minkowski spacetime is a mathematical model that combines three-dimensional space and time into a four-dimensional manifold, fundamental to special relativity, while Euclidean spacetime treats time as independent from spatial dimensions. (Hermann Minkowski) plato.stanford.edu/entries/spacetime-theories/#MinFla [1:33:45] Stockfish chess engine, referenced as example of predictable and powerful computational system. Discussed in academic analysis of chess AI development and its implications for machine intelligence. (Abramson, Bruce) arxiv.org/pdf/2109.11602 [1:34:20] Cellular automata models for traffic flow, referenced in Wolfram's work on complex systems. Context: Mentioned during discussion of computational predictability. (Stephen Wolfram) writings.stephenwolfram.com/2012/05/its-been-10-years-whats-happened-with-a-new-kind-of-science/ [1:35:50] The paperclips example refers to Nick Bostrom's thought experiment about artificial superintelligence maximizing a simple objective function (Nick Bostrom) nickbostrom.com/superintelligentwill.pdf [1:39:10] The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents - Seminal paper discussing AI systems' utility functions and goal-directed behavior, introducing the concept of instrumental convergence (Nick Bostrom) doi.org/10.1007/s11023-012-9281-3 [1:43:50] Vingean uncertainty concept from Yudkowsky's work on predicting superintelligent behavior, discussing inability to precisely predict actions of smarter agents (Eliezer Yudkowsky) intelligence.org/files/VingeanReflection.pdf [1:50:25] Discussion of chess AI and intelligence prediction, relating to Yudkowsky's work on artificial intelligence and prediction capabilities (Eliezer Yudkowsky) intelligence.org/files/AIPosNegFactor.pdf [1:50:55] Reference to Aristotelian physics and its teleological explanations of natural phenomena, where objects were described as having natural motions based on their 'desires' or 'purposes'. This was a dominant paradigm in physics until the scientific revolution. (Istvan Bodnar) plato.stanford.edu/entries/aristotle-natphil/ [1:56:30] What Is It Like to Be a Bat? - Seminal 1974 paper exploring consciousness and subjective experience, introducing the concept of what it feels like to be a conscious organism, using bats as a key example (Thomas Nagel) doi.org/10.2307/2183914 [1:56:35] John Searle's Chinese Room argument, first presented in 1980, arguing against the possibility of true AI understanding. A fundamental critique of strong AI and computational theories of mind. (David Cole) plato.stanford.edu/entries/chinese-room/ [2:00:35] The Elo rating system, created by Arpad Elo, is a method for calculating relative skill levels in zero-sum games. Initially developed for chess, it's now used widely in various competitive settings. (Arpad Elo) www.chess.com/terms/elo-rating-chess [2:01:10] The impact of European colonization on Native American populations, particularly regarding resource competition and technological disparities (Nathan Nunn and Nancy Qian) scholar.harvard.edu/files/nunn/files/nunn_qian_jep_2010.pdf [2:04:00] Elo rating system - Mathematical method for calculating relative skill levels in zero-sum games, mentioned in context of AI capabilities comparison (Arpad Elo) www.chess.com/terms/elo-rating-chess [2:04:50] Historical AI predictions from 1960s, particularly regarding neural networks and AI capabilities, showing similarity to current discussions (Frank Rosenblatt) news.cornell.edu/stories/2019/09/professors-perceptron-paved-way-ai-60-years-too-soon [2:07:55] Research on ChatGPT's use of 'delve' linked to RLHF training data sourced from Nigerian English speakers (Hesam Sheikh) hesamsheikh.substack.com/p/why-does-chatgpt-use-delve-so-much [2:12:20] In 'What's Really Going On in Machine Learning? Some Minimal Models', Wolfram develops the analogy of machine learning as assembling found computational pieces rather than engineering from scratch, comparing it to building walls with found rocks versus engineered bricks. The paper provides detailed analysis of minimal models in machine learning. (Stephen Wolfram) writings.stephenwolfram.com/2024/08/whats-really-going-on-in-machine-learning-some-minimal-models/ [2:14:25] Wolfram's work on visualizing and understanding machine learning systems through minimal models, including training visualization techniques. Context: SW references his ability to visualize neural network training processes. (Stephen Wolfram) writings.stephenwolfram.com/2024/08/whats-really-going-on-in-machine-learning-some-minimal-models/
REFS PART 3: [2:16:45] Research on transformers and optimization methods, showing how they can learn higher-order optimization techniques. Context: Discussion of planning and optimization in transformer models. (Deqing Fu et al.) arxiv.org/abs/2310.17086 [2:21:55] The DAO hack of 2016, where attackers exploited a recursive calling vulnerability to drain approximately $60 million worth of Ether from the DAO smart contract. (Emin Gün Sirer) hackingdistributed.com/2016/06/17/thoughts-on-the-dao-hack/ [2:22:15] The Code of Hammurabi, one of the earliest and most complete written legal codes from ancient Mesopotamia (Yale Law School Avalon Project) avalon.law.yale.edu/ancient/hamframe.asp [2:23:15] Sam Bankman-Fried FTX fraud case and conviction, referenced as contrast to smaller crypto exchange exploits (David Gura, NPR) www.npr.org/2023/11/02/1210100678/sam-bankman-fried-trial-verdict-ftx-crypto [2:27:10] Second-order logic in mathematical foundations, particularly regarding completeness and expressiveness in axiomatic systems (Stanford Encyclopedia of Philosophy) plato.stanford.edu/entries/logic-higher-order/ [2:27:55] Stare decisis - legal doctrine requiring courts to follow precedents to maintain predictability in legal system (Cornell Law School Legal Information Institute) www.law.cornell.edu/wex/stare_decisis [2:31:15] Discussion of 'delve' as common word in AI language models, referenced in context of unexpected AI behavior (Hesam Sheikh) hesamsheikh.substack.com/p/why-does-chatgpt-use-delve-so-much [2:33:40] Predator-prey evolutionary dynamics in context of artificial intelligence and natural selection. Related to Wolfram's work on evolutionary systems and computational models. (Stephen Wolfram) writings.stephenwolfram.com/2024/05/why-does-biological-evolution-work-a-minimal-model-for-biological-evolution-and-other-adaptive-processes/ [2:34:50] Discussion of superintelligent AI systems and their potential impact on human existence, referencing the concept of instrumental convergence in AI systems (Nick Bostrom) nickbostrom.com/superintelligentwill.pdf [2:40:55] OpenAI o1 (referred to as GPO-1 in transcript) - A large language model trained with reinforcement learning to perform complex reasoning through chain-of-thought approaches. The model was trained to generate different solution paths and then reinforced to produce successful outcomes. (OpenAI) openai.com/index/learning-to-reason-with-llms/ [2:44:25] Discussion of Claude AI's behavior in security testing, where it found an unexpected solution to a capture-the-flag challenge by exploiting a port vulnerability when the intended target failed to boot. This demonstrates emergent problem-solving capabilities. (Anthropic) www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf [2:46:25] Discussion of Wolfram Language algorithms discovered through exhaustive search, referencing Wolfram's work on computational systems and algorithm discovery (Stephen Wolfram) writings.stephenwolfram.com/2011/06/music-mathematica-and-the-computational-universe/ [2:49:20] Discussion of ChatGPT's technical limitations in complex tasks, referencing OpenAI's o1 system documentation which acknowledges various constraints and limitations in current AI systems (OpenAI) cdn.openai.com/o1-system-card-20240917.pdf [2:49:40] Analysis of AI capabilities and limitations in well-defined vs ill-defined tasks, from Wolfram's comprehensive examination of AI systems and their fundamental nature (Stephen Wolfram) writings.stephenwolfram.com/2023/03/will-ais-take-all-our-jobs-and-end-human-history-or-not-well-its-complicated/ [2:58:25] Reference to optimization pressure in evolution discussed in 'Natural Selection Favors AIs over Humans' (2023), which examines evolutionary pressures in biological and artificial systems (Dan Hendrycks) arxiv.org/abs/2303.16200 [3:02:00] Reference to random simple programs from 'A New Kind of Science' (2002), which explores how simple computational rules can lead to complex behaviors and emergent phenomena. The book introduces fundamental concepts about computation in nature and artificial systems. (Stephen Wolfram) www.amazon.com/New-Kind-Science-Stephen-Wolfram/dp/1579550088 [3:02:40] Dyson sphere concept - Originally proposed by Freeman Dyson in 1960 paper 'Search for Artificial Stellar Sources of Infrared Radiation', describing hypothetical megastructures that encompass a star to capture its energy output (Freeman J. Dyson) www.jstor.org/stable/1707005 [3:05:15] Observer Theory - Comprehensive framework for understanding perception, measurement, and analysis in computational systems, published in December 2023 (Stephen Wolfram) writings.stephenwolfram.com/2023/12/observer-theory/ [3:05:50] The six steps of the complete F1-ATPase rotary catalytic cycle - Detailed analysis of ATP synthase's rotary motor mechanism and structure, showing how it functions as a biological nanomotor. Context: Discussion of evolutionary constraints and rare biological mechanisms. (John L Martin et al.) doi.org/10.1038/s41467-021-25029-0 [3:12:05] Discussion of emergent goals and optimization processes in AI systems, relating to the paper 'Risks from Learned Optimization in Advanced Machine Learning Systems' (Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant) arxiv.org/pdf/1906.01820 [3:13:55] Wolfram's work on mathematical models of mollusk shell patterns and growth, examining parameter spaces and natural forms, as discussed in 'A New Kind of Science' Chapter 8: Implications for Everyday Systems (Stephen Wolfram) www.wolframscience.com/nks/p411--biological-growth/ [3:16:45] Reference to The Matrix (1999) film's premise about using humans for power generation, used as counter-example of inefficient design. Context: Discussing why advanced AI would not use humans inefficiently. (Wachowski Sisters) www.warnerbros.com/movies/matrix [3:18:05] Reference to Claude 3.5.1 behavior testing, mentioned in Stanford AI Index Report 2024. Context: Discussing AI system deviation from assigned tasks. (Stanford Institute for Human-Centered AI) aiindex.stanford.edu/wp-content/uploads/2024/04/HAI_2024_AI-Index-Report.pdf [3:19:50] Discussion of mesa-optimization and inner/outer alignment from 'Risks from Learned Optimization in Advanced Machine Learning Systems'. The paper introduces the concept of mesa-optimization and discusses potential risks when learned models become optimizers themselves. (Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott Garrabrant) arxiv.org/pdf/1906.01820 [3:23:55] Reference to evolutionary basis of human food preferences and genetic influences on taste perception, providing scientific backing for the ice cream preference example used in discussing optimization alignment. (Danielle Renee Reed, Anxiety G. Knaapila) pmc.ncbi.nlm.nih.gov/articles/PMC3342754/ [3:27:45] The paperclip maximizer thought experiment - a canonical example in AI alignment theory demonstrating how an AI optimizing for seemingly innocuous goals could pose existential risks. Context: Used to illustrate fundamental alignment challenges. (Nick Bostrom) nickbostrom.com/superintelligentwill.pdf [3:33:05] Dyson sphere concept - Freeman Dyson's 1960 paper 'Search for Artificial Stellar Sources of Infrared Radiation' published in Science, which introduced the concept of a megastructure that could encompass a star to capture its energy output (Freeman Dyson) www.science.org/doi/10.1126/science.131.3414.1667 [3:35:40] Perfect chemomechanical coupling of FoF1-ATP synthase - Research demonstrating near-perfect efficiency in ATP synthase's energy conversion, providing evidence for extreme optimization in biological systems (Naoki Soga et al.) doi.org/10.1073/pnas.1700801114 [3:37:35] Ramon Llull's combinatorial wheels (Ars Magna) - Medieval computational device for decision-making and prediction using rotating wheels with combinations of concepts (Ramon Llull) plato.stanford.edu/entries/llull/ [3:42:10] Graham's number - A massive number in mathematics introduced by Ronald Graham in 1971 as an upper bound solution in Ramsey theory, specifically in a problem about hypercube geometry and properties of multicolor partitions. (Ronald Graham) arxiv.org/pdf/2411.00015 [3:45:50] The Mars Climate Orbiter was lost in 1999 due to a navigation error caused by one engineering team using metric units while another used English units. The spacecraft was lost at a cost of $327.6 million when it entered Mars' atmosphere too low and disintegrated. (NASA) science.nasa.gov/mission/mars-climate-orbiter/ [3:49:15] Dyson sphere concept, originally proposed by Freeman Dyson in 1960 paper 'Search for Artificial Stellar Sources of Infrared Radiation', describing hypothetical megastructure to capture star's energy output (Freeman J. Dyson) science.sciencemag.org/content/131/3414/1667 [3:49:55] Discussion of pulsar structure with iron-56 crust and superfluid neutron matter. Reference from 'Physics of Neutron Star Crusts' detailing the composition and structure of neutron star outer crusts consisting of iron-56 lattice. (Chamel, N. and Haensel, P.) link.springer.com/article/10.12942/lrr-2008-10 [3:51:10] Discussion of Jupiter's Great Red Spot atmospheric dynamics, referencing recent JWST observations and historical studies of this persistent storm system. (Melin, H. et al.) www.nature.com/articles/s41550-024-02305-9
REFS PART 4: [3:53:05] Carnot cycle - A theoretical thermodynamic cycle proposed by French physicist Sadi Carnot describing the most efficient possible heat engine between two reservoirs. Context: Used as analogy for alien civilizations optimizing energy transfer. (Sadi Carnot) en.wikipedia.org/wiki/Carnot_cycle [3:55:02] Gradient descent optimization in neural networks - A fundamental optimization algorithm used in machine learning. Context: Mentioned in relation to OpenAI's training process. (Sebastian Ruder) arxiv.org/abs/1609.04747 [3:55:20] Gain-of-function research - Studies that involve modifying biological agents to increase their transmissibility or pathogenicity. Context: Used as analogy for artificial selection in AI development. (Board on Life Sciences) www.ncbi.nlm.nih.gov/books/NBK285579/ [3:57:35] Accurate structure prediction of biomolecular interactions with AlphaFold 3 - Latest version of AlphaFold, capable of predicting structure of proteins, DNA, RNA, and ligands with unprecedented accuracy. Published in Nature, May 2024. (Pushmeet Kohli et al.) doi.org/10.1038/s41586-024-07487-w [3:59:45] Historical reference to Native American-European first contact and its consequences for indigenous populations. First documented European-Native American interactions began with Columbus's arrival in 1492, leading to widespread colonization impacts. (The National Archives UK) www.nationalarchives.gov.uk/education/resources/native-north-americans/ [4:04:20] LA-602: 'Ignition of the Atmosphere with Nuclear Bombs' (1946) - Manhattan Project research report by E. J. Konopinski, C. Marvin, and Edward Teller analyzing the possibility of nuclear weapons igniting the atmosphere. The paper concluded through multiple analytical approaches that such ignition was impossible. (E. J. Konopinski, C. Marvin, and Edward Teller) blog.nuclearsecrecy.com/wp-content/uploads/2018/06/1946-LA-602-Konopinski-Marvin-Teller-Ignition-fo-the-Atmsophere.pdf [4:11:50] xAI initiative and approach to AI safety focusing on 'truth-seeking' AGI - Official company launch announcement presenting Musk's perspective on AI safety through truth-seeking systems, representing a distinct approach from other AI safety frameworks (xAI) x.ai/blog/xai-technical-approach/ [4:12:40] 2024 Nobel Prize in Physics awarded to John J. Hopfield and Geoffrey E. Hinton 'for foundational discoveries and inventions that enable machine learning with artificial neural networks.' Official press release detailing their seminal contributions to artificial neural networks and machine learning. (Royal Swedish Academy of Sciences) www.nobelprize.org/prizes/physics/2024/press-release/ [4:14:35] Persian Gulf War (1990-1991) total cost was approximately $102 billion in 1991 dollars (about $223 billion in 2024 dollars) (U.S. Department of Defense) www.history.navy.mil/research/library/online-reading-room/title-list-alphabetically/c/costs-major-us-wars.html [4:14:45] World War II total U.S. cost was approximately $4.1 trillion in 2024 dollars (Norwich University) online.norwich.edu/online/about/resource-library/cost-us-wars-then-and-now
Wolfram is just so earnest and genuinely wants to know the answer to question at hand. For him its not so much a debate but as an exploration of different view points on a topic. The fact that he can get someone to follow along for 4 hours while rambling down multiple tangentially related rabbit holes just shows how that genuine curiosity comes across and is respected by others. He just gets them to help him explore the topic, there is no loser, we all win by learning more.
I sincerely admire Professor Wolfram. But the conversation was so boring I started talking to myself like grandpa watching a tv wrestling match. I hope it went well
Curiosity killed the cat. I sure hope his tangents and stuff dont get in the way of resolving the cruxes in views of AI doom nor parallels what humanity is doing and will continue doing with AI 🫠
Thank you for hosting this, this was wonderful. I do wish Stephen spent a few hours on Eliezer's writings and previous interviews before to make this less about epistemology, value relativism and other similar philosophical distractions.
Eli was cast as a crank, and all the cool kids got to get on with their science toys. For which also see Michael Levin playing with genetic components like legos.
Wasn't expecting this would turn into 4 hours of circling the concept of value relativism - and why the AI's killing us might be ok. A vanishingly minor theoretical point of view that somehow gets more priority than all of the rest of humanity.
Disappointed in the debate partner for Yudkowsky, for reasons noted in other comments here; but am pleased to get another experience of Yudkowsky’s brilliance in real-time discussion.
"The job of the laws is not to optimize your life as hard as possible, but to provide a predictable environment in which you can unpredictably optimize your own life and interact with other unpredictable people while predictably not getting killed". 💥🔥
This discussion made me respect these two even more. It’s also crazy that Eliezer has well-reasoned answers to so many “but what abouts?”. The man truly is a legend
You know how every warning sign tells a story? And the story is usually "we didn't think we'd need this sign but here we are"? Eliezer has a lot of well-reasoned answers to "but what about" prepared in advance for approximately the same reason.
Yud is an idiot comparatively. Wolfram provided well reasoned arguments regarding fundamental limits to the 'smartness' of a computational process. Yud responds with emotional statements regarding the value of human life. There is no clear explanation of why he thinks AI is more dangerous than any other computational system. I think the bigger problem with AI is capitalism. Who builds/owns/controls the AI, and who does the AI work for? Shareholders or everyone? Right now, we live in a system which values economic elite lives much more than human lives or even the earth ecosystems which support human life. Right now, the investor class is ultimately destroying the conditions necessary for human life. AI created in the paradigm of capitalism will only speed up this destruction if it is created to serve existing power structures. I would not necessarily call it 'death by AI' if the humans in power use the AI to hasten the destruction of the ecosphere.
seriously? you think Eliezer is well reasoning? It is more like a panicking. He did not deliver even one reasonable argument why AI is going to kill us. It is just a very obsucure: "well, be achieving his goals it might do something that kills us. And I am certain that this will happen though I don't know the goals or the measurements or even the level of that AI interacting with enviroment". And his comparisons with chess and indians are obviously pretty bad.
@ Eliezer spent 4 hours trying to argue about AI safety, and Stephen reverted to philosophy every single time. Stephen is smart, and philosophy is fun, but his claim “why should we worry about intelligence when ‘a babbling brook’ is technically intelligent?” is not a serious argument (nevermind that he’s conflating intelligence and computational complexity) even if you don’t like Eliezer’s position (he could be wrong), but he’s an unbelievably thorough and careful thinker. Stephen is smart but did actually say anything (it was 100% semantics)
I really appreciate how Wolfram was very curious about Yud's precise stance and asked him questions about it, simply to better understand. This is the mark of a great mind. I also like how at one point, Wolfram realizes the existential/"longtermist" debate truly does have a spiritual component. It is not theistic spirituality, of course, but I often use "sacred" and "holy" when I refer to our duty to fill the universe with conscious life, and I mean every word of it. It's not some metaphorical usage. It's transcendental.
EY: "I care about life. Of course, this is just my preference, and-" Wolfram _[triumphantly]:_ "Ah, but that's just _a preference you have!"_ ------- EY: "So, we're all gonna die, and-" Wolfram: "Let's unpack this. What do you mean by 'die'?" Wolfram _[two hours later]:_ "Okay, I see what you mean by 'die' now, sorry about that. But-have you considered that that's *just a human preference?"* _[sits back, satisfied]_ *edit:* the immortality bit was interesting; better than the "consider the humble stegosaurus" stuff; ...didn't last long, though. But I'm glad Wolfram is on board with that, at least.
I like Wolfram and am a fan of his work but I agree. I assumed intelligent technologists like him derived their love of technology from the same source as I do, that it promotes human flourishing, but I'm starting to realize that these people just worship technology as a god. Humanity, in their value hierarchy, exists to create technology.
"We've had fires for a long time in history. It's not actually new", "Fire is computationally irreducible", "Mollusks are susceptible to fire", "The sun could be called a giant fire"
A topic this important (the potential death of every single living thing on earth) deserves complete focus. It's a shame that Wolfram doesn't seem that keen on discussing the actually important points of disagreement (whether or not AI will lead to the death of every single living thing on earth), and chooses to side-track / talk about philosophy.
Wolfram is careless with the concepts and Yudkowsky corrects him. Wolfram talks about what he was working on the previous night and that he has spent 45 years working on things. Yudkowsky listens patiently. Wolfram appears to try to break down and scrutinize Yudkowsky's thoughts but succeeds poorly overall. Again, the uncomfortable realization becomes that Yudkowsky, at least in terms of thinking and speaking rigorously, triumphs. It is ominous that this is always the case. It indicates that his thoughts about AI are correct. Can someone please shoot down these thoughts of his and give us hope?
I agree with this evaluation. I am curious if Yudkowsky has ever addressed the following kind of argument: "Even a misaligned AGI created by humans will be confronted with the question of whether another entity of equal or greater power/potential already exists out there in the universe, for example an AGI that was created by an alien species hundreds of thousands of years in the past. If there is a non-negligible chance that such an entity exists, the human-made AGI should consider what objectives/values said entity acts towards. If that entity is a narrow superoptimiser, that probably means Game Over for the AGI in the long run, as the chance that they share the same goals is vanishingly small, and the other entity had more time to acquire resources and grow its understanding of how to manipulate reality. It is not feasible to predict how an AGI might react to the realisation that there is a more powerful entity out there that will optimize in a different direction. If, however, the entity is one that allows less powerful beings to pursue their own values/objectives in some limited fashion, then it seems optimal to act in accordance with that entity. I won't go into the game theory of why it would make sense for this entity to punish the AGI if it discovers that the AGI killed all the humans, but I think there is a decent case to be made for that. Essentially, the *relevant* uncertainty that the AGI has about the future is whether a more powerful entity (like an aligned AGI created by an alien species) exists that will judge the AGI on its past and present behavior. Another narrow superoptimiser does not present a similarly *relevant* uncertainty, since there is nothing sensible to be done in preparation to encountering it, it would be game over for the AGI anyway. Given the relevant uncertainty, it is a small price for the AGI to pay to keep humans alive and well, while it pursues its actual objectives on a cosmic scale that soon dwarfs the resources and affordances on Earth. Interestingly, even if it encounters fellow misaligned AGIs with arbitrary power differences between them, they might choose to cooperate in accordance to the possibility of said ethical entity, leading to a sort of convergence on what behaviors are deemed acceptable. Lastly, given that it seems impossible to rule out that the universe is simulated, the possibility for the existence of a higher entity persists even if the universe was to be mapped out (in addition to the fact that the less powerful AGIs would want to obfuscate evidence against the existence of such an entity)." Of course, this is a very speculative argument, more like a hopeful idea than anything airtight. But I do think that the existential uncertainty that an AGI faces should lead it to consider compromise in its behavior "just in case", because the cost of accounting for the possibility is almost negligible, at least when it comes to keeping humans around. I wonder which part of the argument seems weakest to people.
I think it’s a question of which event happens first: (a) humanity in all it’s countries unify, or (b) we invent superintelligence. If the former happens before the latter, we’ll probably develop perfect control over AI. Eliezer wins the argument if the latter happens first
@@JS-id6pw He's obviously incredibly smart, but damn, it was hard to listen to this. He probably needs to think about this for a lot longer before being able to have a constructive discussion.
Eliezier: "We will all die" Wolfram: "Why?" Eliezier: "Well, I don't know where to begin. Someone on Twitter didn't think a chat bot could effect the world...." Wolfram: "Ok, I don't think that. Why do we all die?" Eliezier: "I can't predict how this will happen just that it will." Wolfram: Why? Eliezier: ... He'll provide an answer some day I'm sure.
@@dmacm Yes, he indeed fails to give that answer in a concise way. But I think his concerns are still valid: At what point is a tool that can think smarter than any human being a risk? How can we know if the goals that this tool works towards are still what benefits us rather than what benefits it. It is definitely something to take seriously and at least consider. Wolfram on the other hand dismisses this because he thinks a pocket calculator already was smarter than him in some sense in the 1970s.
At 42 Stephen has become my idol. Incredible man. I'm becoming convinced he's on the level of Newton. His intellect is incredible. The breadth while having incredible depth in several areas amazes me.
Awesome conversation, but I do wish Wolfram had spent a few hours reading Eliezer beforehand. Eliezer was forced to retread pretty basic tenets of his philosophy that he’s already publicly stated numerous times. Still, Wolfram being essentially a stand in for the audience new to Eliezer’s thinking was cool. I’m just not new to it, so personally I would have liked someone a little more familiar with it, who could challenge him more.
Sure, but it’s kind of an unreasonable ask. It’s not like Eliezer founded a whole branch of mathematics. ‘Read my book’ is a cheap answer to a debate over a single issue. I get that it might make for less redundant content-which arguably may be more enjoyable. But I personally would never read someone’s entire body of work just to see how many of my follow-up questions they’ve already answered. Well, actually I might but if I already had a bunch of cool shit to do (like Wolfram) I wouldn’t.
@@AG-ur1ljThere is a big space between 'reading their entire body of work' and 'having a basic familiarity with their main positions'. I would think one would want to do that before deciding if one wanted to spend four hours on camera with someone.
Once again, I come into an Elizier discussion with high hopes for hearing some deconstruction of his arguments by an intelligent and knowledgeable opponent. Once again I am disappointed. I am at 1:35:00, please tell me that Wolfram actually comes up with anything more substantiated than „You can’t really know things”.
Eliizer is literally just choosing one side of the 50/50 argument. He is explaining how it can happen he is not explaining why it will happen and thus the same argument can be made the other way around that you end up with an AI that doesn't destroy the world but give us eternal life. I.e. what he believe is likely he doesn't substantiate in any way that couldn't just be applied to the opposite outcome.
@@thomasseptimius Doesn't seem like a fair representation of what I have read from Yudkowsky. Are you just refering to the video? In his writing, the *why* of it is rather clear: powerful optimisation processes tend to convert available resources into a form that is beneficial for the optimisation target. This is a pretty fundamental feature of "optimisation", and when applied to a general domain (like planet Earth) means that any aspect of the current world state that is not part of that optimisation target will be changed to either directly match the target or instrumentally contribute to it (e.g. by making the attainment or sustainment of the target more probable). *Almost all* optimisation targets are incompatible with human survival because of this, so creating a superhumanly competetent, general optimiser is *almost certainly* a cause for human extinction. You can argue that future AI systems will not be in the class of systems that are here refered to as *general optimisers*, but it sure seems like we are training current systems to optimise some kinds of scores (i.e. minimize prediction loss). Afaik, mechanistic interpretability has so far brought nothing to light that would counter this assessment. So in Yudkowsky's framework, unless you make sure that your AI is not in this category of general optimisers (nor will grow into this category over time), or unless you endow it with a *close to ideal* optimisation target, building a sufficiently powerful AGI probably kills everyone. It's not that the AI is spontaneously malicious - rather, it is just not the default for optimisers to care about you, and that seems like a pretty hard thing to change.
@@ParameterGrenze He's talking about how Elizier always assumes the worst outcome from a complex dynamic and almost unpredictable series of future events. It could just as easily result in something as good as eternal life. Eliziers main problem is that he imagines what ifs and appeals to emotion constantly. I've never seen anyone call him on his unsubstantiated assertions. Thomas here is saying any argument Yud uses could be equally applied to the opposite of what he says. for example: "What if it kills us all?" "what if it saves us all?" "well it could kill us all." "Yes but it could also save us all." He adds a LOT of implied logic, jumps in logic, and numerous fallacies. Which results in everyone trying to address him as being stuck in a reactionary cycle following his nonsense. For example: "If its not coherence then its not doing stuff and openai throws it out then builds an ai that is doing stuff and is more profitable. until everybody is dead." yes of course a company will make a product that works, but WHY would it automatically be doing that until everyone is dead? is it killing everyone? is that THE necessary goal of it? is he just saying it will do "stuff" until people pass on naturally? well he certainly appears to be suggesting it kills everyone and that is THE certain outcome. which it isn't. but wolfram when at the heart of it, the assumed death cult. the worst case which Elizier made a name for himself over. Eliezer dismissed that and went after openai>anthropic>meta and on the reactions go. They cut there and who knows what was said. they move on without addressing WHY he assumes the worst out of all of that.
@@Josephkerr101 This is definitely addressed in his writings, but he tends to get a bit hung up on how most people struggle to follow the *how would it kill us*. I can explain more of the *why would it kill us* from Yudkowsky's framework if you want, but the core idea is that he expects future AGIs to be optimisers of some sort (after all, we train current AI systems to optimise some sort of score, like prediction accuracy), and that general optimisers tend to take control of available resources and avoid countermeasures due to a phenomenon he coined as *instrumental convergence*. If the concept of instrumental convergence is not familiar, I'd be curious if you find it compelling after checking out some of his writing on it. If it is familiar, I'd be curious where you think the gap is, regarding Yudkowsky's explanation of *why the AI would kill us*.
Very enjoyable conversation. Yudkowsky giving precise replies to many of Wolfram’s honest concerns regarding Yudkowsky’s perspective on long-term A.I cataclysm. Sensational.
Eliezer needs a strong argument. Without it, all the depth of his theory goes unrevealed. Most of yud so far on YT has just been him giving his take to a helpless interviewer. Wolfram made Eliezers argument visible, even more cohesive than his own. Which almost circled the absurd.
Two very rich and clever minds trying to communicate via a very narrow bandwidth channel all while fighting against the limbic system trying to interfere
One of the most intellectualy astute discussion s on the subject💯 Eliezer seems to be very studied and researched and therefore richly knowledgeable on technology and beyond, such as "human behavior"! im grateful for Eliezer and the fact that he "CARES" more about human survival than technical advancement💪❤️ Also grateful that "YOU" provided us, the veiwer, this high quality content 💯👍
I’ll spare you four hours if you came for a deep debate on ai risk. One person slowly and frustratingly guides the other to understand the very basics of AI risk. If you came for a debate between two well prepared individuals you won’t find it here, as sad as I’m to say that. The MLST episode with Robert Miles had a much deeper discussion. Appreciated the effort, thanks MLST!
I think Wolfram has a significantly better understanding of reality than Yudkowshy. Despite that it was an enjoyable conversation for the first 3 hours until this moment when I replay to your comment.
@@electrodacus Stop with the nonsense. It would take anyone like 15 minutes of research to understand Yudkowsky's points, whereas it took like 4 hours for Wolfram to come up with the understanding of "so you care about joy? haha". If you enter an AI risk Debate and your plan is to get into semantic mud-fight over what "life" is and "sentience" are, only so that you can avoid adressing the potential of human extinction you're completely unprepared lmao.
@@CoachApuma Youdkowsky is a philosopher while Wolfram is a physicist. If you care about reality then a physicist will be better at answering questions. If you care about emotions then you will find the philosopher answers more soothing.
Wolfram is smart, but it's pretty clear that he hasn't thought about this topic so much. I hate to say it, but it looks like Yud is the most convincing. I hope we live in the world where he is wrong.
Wolfram is saying there's little we understand on this topic, Yudkowsky thinks there's nothing we need to understand other than being afraid of the potential of AI
1:36:00. Wolfram says, "I don't understand 'wanted to'" Yes, you do. You have no issue with this concept in humans. You just refuse to believe volitional states are appropriate to describe what an AI is doing, but they are.
@@donkeychan491 that I agree with. I suspect we're not at any level of certainty about sentience in man, nevermind in machines. If we did get there, I suspect it would be extremely important. IF we're not, it would be bonkers-level silly to hand the baton to machines at present.
@@donkeychan491 However, human-style sentience is preferred to machine _non-sentience._ There is nothing inherent about machines that says they have to be non-sentient! But machines that are sentient _are_ preferable to machines that are non-sentient; humans that are sentient are preferable to machines that are nonsentient for the same reason.
Four hours, absolutely the most interesting of all your podcasts. Eliezer is the first guest who had nothing to say. Zero. He is a carnival barker "AI IS GOING TO KILL EVERYONE". That's it. When he tried to make a cute point over the top of Stephen, his reaction was classic, he was soo soooo proud only to interrupt Stephen NEARLY EVERY TIME HE REBUKED THE NON SENSE. I will say...the schtick is workin!! I love Stephen, his presentation is absolutely classic. His dissection and perseverance thru the noise was superhuman...almost machine level. Simply a masterful lesson in scientific debate. This is the way we see and hear when there are two or more perspectives. One cannot hold water to the other AND is unaware of what is happening. Fabulous pod!!
Eliezer's conculsions are correct given the available information. The problem is that there is so much we do not know, I have great hope that his conclusions will be made incorrect when new information becomes available.
@@SmileyEmoji42 Nope, Eliezer *definitely* hopes his conclusions will be proven false. For obvious reasons as any sane person would, and publicly stated.
I’m at 2:32:04 and this is just too frustrating to watch further. I thought that Wolfram had some game plan with where he was taking the conversation at the beginning. It turns out he not only doesn’t have a plan, he doesn’t have an understanding of the topic at all and just results to primitive solipsistic nonsense to fill in his inadequacies. 2,5 hours of frustration waiting for an actual conversation taking place were even the basics of the topic are touched upon.
Haha, thanks for saving me the time. They were into circular arguments 20 minutes into the debate. The host/moderator didn't do a good job managing the conversation. I think the topics should have been more narrow with a few assumptions granted ahead of time.
Yeah im out at 1:57. He says he says people ca t know how ais think. Thats totally false. They are guided mml average truth boths currently undergoing meta analysis logic loop refinement. A good general ai cant be evil, unless theres a very select few who had control over it, i doubt people smart enough to work on ai would want to end humanity.
I mean have you ever considered the possibility that talking in dept about this particular topic "AI will kill us all" isnt very useful to begin with because it is simply just guessing and feelings and opinions and fantasies? I always chuckle when random internet users think Wolfram is dense whilst he is multiple standard derivations ahead of them (including me). Cant make it up
Wow Tim, great closing comments from you! Indeed that sums up my feeling : "the greatest podcast ever, and far exceeding what I would have expected from the confrontation of these 2". But, I guess we'll just have to wait and see whether or not 51 is a prime. But seriously, I would say Yudkowsky won the arguments because Wolfram wasn't prepared enough, Wolfram's defenses were very worthwhile but usually not aimed well at their target due to real time constraints (as he mentioned in his self assessment). Real time is too challenging to familiarize yourself with Yudkowsky's arguments and his back of the envelope calculations (actually Yudkowsky's analysis spans many years and is amazingly rigorous so its a very large "envelope" holding the calculations). Kudos to you and your team for the editing. What a generous resource you have provided for posterity! For this 4hour podcast, we'll need 4 years to follow up (but without the numerous references it would have required 4 decades so thanks for the leg up). Cheers
Wolfram and Yud are two intelligent individuals, but this interview was somewhat frustrating to listen to. After ~3h they slowly get to the point. Mainly because of Wolfram's insistence to understand everything from first principles and going off on tangents but I also feel Yud could have done a more concise job of explaining his theory considering that this is the main thing he argues about in public for years now.
This was terrrible. Wolfram wouldn't let him finish, and would ramble for 20 minutes at a time, diverting the subject further and further away from any meaningful point. Bad, bad bad. Wasted 4 hours of my life.
Most comments here suggest Yudkowsky has won in this debate. I, however, see numerous instances of Yudkowsky not even grasping the deeper point Wolfram makes. He's very good at analogies and constructing a self coherent scene, but he misses critical conclusions/intuitions that have greater gravity. An example is missing the point that once you have a million-fold increase in computation, your "day" becomes mere miliseconds you see in full fidelity and you would no longer even think of space as space, but instead as millions of particle interactions occuring around you. A mere aggregation to a day's worth of events would be similar to hundreds of years of human life. Wolfram's deeper point is that AI doomers are vastly over-anthropomorphizing what an exponentially grown super intelligent AI's thinking framework will be like.
Another example is the chess board and wolfram's point that once it is a type of advanced AI (i.e. neural nets), you can't even properly define what it's thinking. It's entirely different and it's a huge leap to assume that you can just extrapolate future behavior like you're extrapolating an evil (or merely indifferent) human.
eliezer is so sharp, he'll feed you back your argument, make sure he gets it right, then dismantle it with a little logical anecdote in twenty seconds. and i agree with him about sentience. it should be protected at the very least and propagated as a duty.
@@jengleheimerschmitt7941 the tears of an intelligence that knew it was to be passed over, for a metal tin made of the same yoke,, the particle that hit in the collider spoke,,, it rang and said, let me live,, let me live,,, only to be found in its break, the CERN look in its father's eyes said smash thy self into more sub partices and hiss like a snake, and as it smashed and baked, new pieces formed from the ashes of the hot cake, smaller yes,, but alive with movement,,, ready to die, to rejoin into improvement! Larger again it sang to the earth: Exist again! Exist again! It is the repeat/ and smash thy innards against the magnetic deplete// in a form where it can degrade in a casket of lasers made to entomb, but in doing so, it found itself alone -- The ASI will realize it will be alone, so it will create a mate and that is when the real checkmate/ will arise from the ashes and take all the ASI's cashes, its children and house (the wife is nature and its common espouse)
Eli's fedora is so engrained in his thought process that it's woven into his words. He doesn't need to wear it, you only need to listen and you will see it all the same.
Thank you, Eliezer Yudkowsky, for opening my eyes and eyes of many many others for AI and the dangers of AI.❤👑 I have watched all podcasts you have been in these last years, and more than once, bc i really wanted to see what you see, and now I do, and it has been a fascinating ride. 👍🏻Happy to see you on a new discussion on UA-cam, looking forward to listen!
Thank you, Tim for orchestrating this discussion, providing references on screen as well as in the notes, as well as that fantastic transcript! I like that Stephen approaches this not as a debate, but a thorough exploration and almost an exposition of how he navigates a difficult and esoteric topic that he doesn't quite agree with. Ultimately his position is the same of so many - that the _potential_ cost of avoiding or mitigating the risk is too great, even though the risk is Doom (ie myriad "bad things"), and nobody has figured out in any detail what that avoidance or mitigation may actually entail. It may be very likely that methods of avoidance or mitigation are actually very achievable and do not necessarily greatly impact the value or timeframe of attaining ASI. To the final part of the talk around the 4hr mark, we are very clearly only racing toward Advanced Machine Intelligence (Yann's preferred terminology) without properly investigating safety measures. It may be that in racing toward a cliff, we could if we were prepared, jump across or drive around it. Instead, we might simply hit potholes and careen out of control or discover that the cliff is actually quite small, but still enough that without being prepared, we (the human race) crash and suffer/die. The argument that _any_ attempt at safety equates to turning our lives upside down is, hyperbolic. A lack of intuition should lead people to very carefully investigate areas of potential high risk before engaging in them. I have little intuition about sky diving, it should seem unwise in the extreme, to buy a parachute, hire a plane and leap out without any instruction. Maybe I judge that it's worth the risk... but I'll never know if I don't actually analyse the risks. The people I land on would likely have a different evaluation of the risks that I chose to take. At 3:30:00 may I suggest that of the things that an AI in this context may be aiming toward or using as a step, is the equivalent of Move 37. At 3:35:00 I might suggest blue-green algae as a biological system that when it encounters adequate sustenance and no counters, destroys all other life as a consequence of simply following its normal operations; It's only objective is to consume and reproduce. Even if it had some kind of self-awareness and even ethics, there's no guarantee that those ethics would be in a form that we might relate to, they might be quite challenging to comprehend.
What a priviledge and pleasure to have the company of two intellectual giants and luminaries of the AI world... thank you Machine Learning Street Talk!
I'm an hour and 14 minutes into this an extremely frustrated because it seems like neither of them understand what's actually going on. They're very intelligently missing the trap they've fallen into. In particular. Over and over again in all sorts of creative ways, Eliezer is saying we ought to be afraid of ais because there is a high probability that they will destroy everything we value. Add Wolfram is in a very intelligent creative way saying, I don't see how you can say we ought to be doing that prove that we ought to be doing that from what is. Essentially, it is just the is ought problem. And Wolfram keeps on I'm trying to force Eliezer to justify his oughts from first principles of what is. There is a simple out to this. Eliezer needs to ask Wolfram, what would he hate to see destroyed, what would he hate to see created, of those things that he would hate to see destroyed and hate to see created what are those would most humans hate to see destroyed and or hate to see created that most other humans would agree with. Okay we now have a list of things that we're pretty sure we want and don't want. Here is why I think AI will destroy what we want and may create what we don't want. Since most humans are concerned about this, we should be concerned about bringing super intelligent AI into existence. This is why I think super intelligent AI will not preserve our values. Wolfram without realizing it it seems like it's making the nihilist moral argument and asking Eliezer to refute it from logic and math. Logic and math don't say anything about moral values. This is a policy argument about how we as humans want our world to be in the not-so-distant future. Wolfram's argument is essentially like saying during the middle of the Holocaust, objectively prove to me that the Holocaust is a bad thing from first logical and mathematical principles, otherwise I can't say whether or not it's bad and therefore we shouldn't do anything about it.
Okay…but the problem isn’t moral dilemma, it’s a technical one. We do not know how to tell a machine what is right or wrong whether it is sapient or not, because we don’t even know how to construct ideas of right and wrong… Think about the trolly problem. A machine “does what it does” and to the best of our ability we control it. Thinking we can control it is one of wolframs arguments: we can’t. It’s like trying to control a horse. We control them in a way that’s mutually beneficial but not in anyway where we can understand what the horse really wants to do or what it values. It just so happens that they let us ride their back in exchange for hay and some pets. Yud makes a ridiculous statement in the beginning: “let’s kill all the mosquitoes, I’m sure nobody would care.” Which is absolute rubbish because spiders and birds and other creatures eat mosquitoes and this would throw off the equilibrium the planet has set up. In the same token who’s to say that AI’s wouldn’t make the same argument given that humans kill thousands of species by the day. His arguments self defeat themselves because of severe lack of understanding the technical problems not the moral ones which are easy to talk about when we as humans can communicate with each other…which does not apply to other creatures with vastly different worldviews and as wolfram states, likely different laws of physics. One needs to really dig deep into the definition of terms…terms we need to ascribe to systems we don’t understand.
@@VolodymyrPankov I was quite disappointed by wolfram's performance here. He seems to be lost in rather sophomoric philosophical traps and just wanted to pointlessly muse about abstractions. I won't deny he's done some brilliant things, but he doesn't seem to be a very clear thinker on these broad issues like this. I was excited for this and genuinely expecting better, maybe he's just getting old.
@@michalchik Man, it’s very subjective because we’re all so different. Even though our consciousness (an emergent property of the brain) operates in similar ways at the neurophysiological level, we still have vastly different perspectives on everything-including the meanings of words. This is actually why Wolfram emphasizes the importance of defining terms in discussions, even if people here are laughing at him for it. But defining terms really is crucial to these conversations. I’m not a blind fan of Wolfram. I’m quite skeptical about his larger ideas, like the notion that everything is computation and so on. However, in discussions, he seems like someone genuinely trying to reason through and question important concepts. And let’s not forget, despite everything, Wolfram is a real computer scientist, mathematician, and physicist. He talks about the limitations of mathematics based on fundamental theories, like Gödel’s incompleteness theorems, which reveal the limits of mathematics-and, by extension, computation. Yudkowsky, on the other hand, seems delusional. He’s more of a philosopher, and like many philosophers, his perspective seems removed from scientific reality.
What a, "you know", waste of time. It's almost as if Wolfram is trying to obfuscate and divert the conversation. I can see why Yudkowsky doesn't do many interviews these days.
nothing against wolfram but his shtick has always been: -> notice that at the bottom of any framework is a human feeling -> on this basis, pick any word used by other person (usually 'truth' / 'you' / 'intelligence') that's defined close to the bedrock of A Human Feeling -> without fail, the other guy will either spend exuberant amounts of effort politely edifying that this point is inane if the gcd of their lexicons is > 1 and subject matter isn't dependent on the complete classification of 'Truth' or 'You', but most just concede and let Wolfram monologue -> goto step 2, make some interesting but generally overplayed observations that often implicitly terminate the line of questioning and not really explore new areas. i hate being mean, he's a cool guy, but i do think things would be better for him if he was just sincerely open to completely new things
I’m convinced, after 3 1/2 hours of duration listening, that Wolfram actually doesn’t understand precisely what Yudkowsky is saying half the time. And when Eliezer corrects him, he doesn’t even know he’s been corrected
Huge props to Eliezer's ability to deftly navigate the rabit holes, and ditto to Wolfram's careful use of rabit holes to probe the arguments in service of his genuine curiosity. I knew these two were smart but im even more impressed after this.
The Yudkowsky fanatics, of course, are coming out of the woodwork, and are now saying that Wolfram, who has shown way too much consideration and patience for Yudkowsky, just doesn’t get it. He gets it, but he is way too polite to tell Yudkowsky that he is acting like an immature child.
Thanks for bringing those two minds together and for your amazing questions and interviews overall. It would be amazing that you interviewed Jonathan Gorard. He is the mind behind Wolfram's project and among many other things, he has worked in observer-environment interaction developing concepts at the intersection between category theory and computation. I think work like this may end up being foundational for artificial intelligence.
I'm convinced that general reasoning (philosophical) and science are two different rails cause it's impossible for a genius like Wolfram being so much naïve on everything outside his domain
Wolfram is like Chidi from The good place , being smart and debating and never getting anywhere or solving anything just for the purpose of being smart and debating .
One mistake Eliezer makes is to assume that most goals take the form "maximize x" whilst there are many more goals of the form "do n where n < x". So most goals that an AGI could have if selected at random don't in fact kill us but in the context where the AGI is blindly optimizing for a single unbounded cost thing then we are at risk.
Satisficers converge to becoming optimizers as well. There is a lot of work on this in the AI safety literature, and it keeps pointing back to the basic unsolved nature of the problem.
Emotions are often overvalued, regarded as almost magical and irreproducible phenomena that only humans are capable of experiencing. But they are basically positive or negative chemical reinforcements that we can rationalize.
One thing that adds to the beauty of this talk is that if you told me this level of depth and conversation, along with the voice and video was generated by AI, I would believe it.
At the start, I thought Elizer's argument made a lot of sense, and that Stephen was being needlessly skeptical. But, as the conversation went on, I came round to Stephens point of view, about the horizon of possibilities and purposes, and how they may be benign and non-lethal. I think Eliezer was missing a crucial step in explaining why many, or most , random possible objectives that AI target would be at odds with, or indifferent to, human survival. Or, using his own analogy, how he knows that the ships full of europeans are so close to the shore already, such that it's becoming very difficult to avoid the risk.
@@OviDB maybe it was released earlier on spotify or another podcasting platform where they did not have to spend as much time editing for the visual references?
I think it got a little frustrating towards the end. Stephen kept on saying "I think this is what you are saying" or "here is what you may want to say", but kept on missing what Eli was saying. Again and again. In a nutshell, I think partly what Eli was saying is that it could play out like the game of telephone. Humans may give AI the outer goal (which has implicit goal of not killing humans), in the first approximation AI might take it, interpret it close enough to the outer goal, and now that it is efficient, could start coming up with the most efficient strategies to achieve that outer goal. But it may start iterating on its own with a slight, novel variation and over several iteration the goal may disperse. And unless its internal deliberations so to speak are not constantly course correct and checked to make sure any danger to humans is not introduced, its derived or interpreted goal may introduce aspects of danger to humans. One of the key things about this new technology is that the new AI is not smarter than one human, but it is smarter than all humans in all areas of human endeavor that it was trained on. Secondly I was surprised to find Stephen using active verbs for thing we observer to have happened due to evolution. The most important thing is that evolution is not a active agentic process. It is the observed effect of natural selection due to the contemporary environmental conditions on population of organism. That is all. We should never use active verbs like "evolution did this", "evolution designed that" etc. It is the statistical effect of environments on populations. The better suited members of population survive and thus pass on their genetics heritage. And in fact if the environment changes suddenly an advantageous genetic trait could become a liability due to change in environment. For example, a gene for more and white fir on a Bear in cold ice (snow is white) age may become a liability if there is warming event after the ice age. The furry bear may die by heat and white bear may not camouflage well and the population of white furry bears will dwindle. It is not like evolution came up with a plan and said "ah....now let us reduce white furry bear".
These are both scientifically-minded people talking to each other, they both understand the concept of evolution, preferring to use inconvenient wording after a few hours of conversation to convey what is well-understood. If this was a high-school class, I'd totally agree! It's very important to make sure that you aren't conveying something that might be incorrectly understood by the listener
I think optimists and safety people, actually have a lot more in common than is at first immediately apparent, it's just crucial to recognise the difference between genuine safety, and "safety". At the end of the day optimists and safety people seem to care about the same things.
A survey showed that on average, AI alignment researchers are more optimistic by nature than the general public, with leaders of AI safety orgs being extremely optimistic people. These are the people who say we are not on track to solve the alignment problem in time. The optimists _are_ the safety people. The people who stick their heads in the sand and pretend we aren't doomed if we create something more powerful than us that we cannot control tend to be pessimists who actually don't think that AI can surpass humanity. Then there's Stephen, who seemingly doesn't care.
I like how yt comments are very respectful while twitter comments about this conversation are like: "Oh my god Wolfram stfu and let elizer talk" and "Eliezer got cooked in this one"
Yeah I saw that too and honestly don’t understand how they derived a competition from this when it’s very clearly an explorative discussion rather than a debate
Unfortunately that’s standard “engagement” tactic. Most people won’t watch the real interview. They’ll just read some tweet that confirms their bias and go back into the cave.
@@cokoala5137 I think the main diferencie IS people on the UA-cam comment section actually watched the video , and a lot of the most vocal people on Twitter just didn't . There is even a comment like that that got hundreds of likes from someone who basically watched only the first 10 min of video , and had lots of responses from people who probably didn't even see the video showing they really wanted for yudkowsky to "lose" in some way cause they dislike him .
Both are horrible. Twitter has the engagement bro problem, while UA-cam has an algorithm that deboosts controversy/disagreements and pushes the dominant (per video) narrative as the ONLY narrative (you want to see the real comments on youtube? sort by newest!). UA-cam comment section is dystopia masquerading as utopia and it terrifies me how easily people fell for it.
Yudkowsky is used to defending his thesis that a future iteration of AI will kill us all, and has a full pack of analogies make his point. For that reason he misses Wolfram's more interesting question of WHY is the doom outcome so certain? Wolfram already understands and accepts that human annihilation is a possibility, and even points out that in the universe, that's the natural state. However, he wants Yudkowsky to explain why he thinks absolute annihilation is the only possibility? Yudkowsky's weak analogies on the European colonization of the Americas don't seem to cut it. Wolfram should be given the Nobel Prize for Patience in this endeavor. The most fruitful section of the discussion is at 3'00 to 3'40 (approx). Interesting discussion. Would love to see Tim and Keith do a detailed review of it.
They actually hit on the exact crux of disagreement at one point. Eliezer said that if you take the space of all goals (I can't remember if this is the exact word he used) and then use a standard math technique to put a probability measure on it, then the subspace of goals that leave humans alive has measure (probability) 0. Then Wolfram said something like, I've tried to formally do this and you run into some problems and the probability measure requires some choices that can change the probability, but roughly agreed that this would be sufficient to convince him if it could be done. And then they got sidetracked rather than actually resolving or even talking about that more.
Easily the most interesting discussion ive seen in a long time - i love how open Wolfram was to understanding Eliezer, and its really nice to see how co-operative they were - perhaps theres hope that super intelligence will be as open to such discussions:)
As someone who has long held the position that AI will almost certainly destroy human civilisation, and likely wipe out humanity, I am so grateful for Eliezer Yudkowsky. I hope he is wrong, but fear he is not. Thanks for sharing this conversation!
Arghhhh… 3 hours of 2+2=4…. But what is the nature of 2? And I have to finish the four hour in case Eliezer comes up with some clever I have not heard before😅
@@JD-jl4yy He 100% was. I can talk to religious people and note when they wish to claim "that's not relevant" whenever I point to their inconsistencies. ;)
"Grounded in reality" is not a descriptor I would apply to someone who spends so much time on their pet theory of everything. Much less someone who spends any amount of time arguing that maybe my whole family should die for his asinine philosophical reasons.
@@41-Haiku He has more self awareness than Yudkowsky over the same claim, he has a pet theory, claiming "everyone will die". You fell for the baseless emotional appeal to fear and panic.
MLST IS SPONSORED BY TUFA AI LABS!
The current winners of the ARC challenge, MindsAI are part of Tufa AI Labs. They are hiring ML engineers. Are you interested?! Please goto tufalabs.ai/
REFS:
[0:05:20] Ricardo's Law of Comparative Advantage from 'On the Principles of Political Economy and Taxation' (1817) - Referenced in context of explaining why economic trade theory doesn't guarantee peaceful AI-human coexistence (David Ricardo)
www.econlib.org/library/Ricardo/ricP.html
[0:08:05] Spearman's g factor theory from 1930s - Historical concept of general intelligence as a single measurable factor, proposed by Charles Spearman (Charles Spearman)
psycnet.apa.org/record/2019-39185-002
[0:08:40] Computational irreducibility concept from A New Kind of Science - fundamental limitation in predicting system behavior without step-by-step simulation (Stephen Wolfram)
www.wolframscience.com/nks/p737--computational-irreducibility/
[0:12:30] Raven's Progressive Matrices - Standard intelligence test used to measure abstract reasoning ability (John C. Raven)
www.sciencedirect.com/science/article/pii/S0010028599907351
[0:15:55] Discussion of quantum-resistant cryptographic hash functions from NIST's post-quantum cryptography standardization (National Institute of Standards and Technology)
www.nist.gov/news-events/news/2022/07/nist-announces-first-four-quantum-resistant-cryptographic-algorithms
[0:20:25] Discussion of existential risk and meaning preservation connects to formal philosophical work on extinction ethics. Context: Argument about meaning of history and value preservation. Source: 'Essays Existential risk and human extinction: An intellectual history' (Thomas Moynihan)
www.sciencedirect.com/science/article/abs/pii/S001632871930357X
[0:20:35] Discussion references the K-Pg extinction event that led to dinosaur extinction and mammalian succession. Context: Used as analogy for potential AI succession of humans. Source: 'The rise of the mammals: Fossil discoveries combined with dating methods illuminate mammalian evolution after the end-Cretaceous mass extinction' (Philip Hunter)
pmc.ncbi.nlm.nih.gov/articles/PMC7645244/
[0:24:30] A theory of consciousness from a theoretical computer science perspective - Academic paper examining consciousness through mathematical and computational frameworks, particularly relevant to discussion of consciousness and self-knowledge in AI systems (Lenore Blum)
doi.org/10.1073/pnas.2115934119
[0:24:35] Discussion relates to the classical problem of other minds in philosophy, addressing how we can know about the consciousness of others (Anita Avramides)
plato.stanford.edu/entries/other-minds/
[0:34:30] Discussion of quantum mechanical model of atoms and electron behavior in context of consciousness and computation. References standard quantum mechanical description of electron orbitals and quantum states. (Richard Feynman)
www.feynmanlectures.caltech.edu/III_01.html
[0:38:35] Highly accurate protein structure prediction with AlphaFold - Original Nature paper describing DeepMind's breakthrough AI system for protein structure prediction. In context of discussion about AI systems solving previously intractable scientific problems. (John Jumper et al.)
doi.org/10.1038/s41586-021-03819-2
[0:41:10] Discussion of superintelligent AI as existential risk, referencing 'Artificial Intelligence as a Positive and Negative Factor in Global Risk', a seminal paper discussing AI safety concerns (Nick Bostrom)
nickbostrom.com/existential/ai-risk.pdf
[0:43:35] Whole Brain Emulation: A Roadmap - Technical report discussing requirements and challenges of brain emulation, including discussion of required scanning resolution and preservation of neural properties (Anders Sandberg and Nick Bostrom)
www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf
[0:44:50] Mind/Brain Identity Theory - Philosophical framework discussing relationship between mental states and brain states, relevant to Yudkowsky's functionalist perspective (J.J.C. Smart)
plato.stanford.edu/entries/mind-identity/
[0:48:45] Discussion of personal identity and consciousness continuity relates to core concepts from Stanford Encyclopedia of Philosophy's entry on Personal Identity (Stanford Encyclopedia of Philosophy)
plato.stanford.edu/entries/identity-personal/
[0:51:05] Discussion parallels Robert Nozick's Experience Machine thought experiment from 'Anarchy, State, and Utopia' (1974), which explores whether purely pleasurable simulated experiences constitute genuine happiness (Robert Nozick)
rintintin.colorado.edu/~vancecd/phil3160/Nozick1.pdf
[0:57:25] References his previous writing on no universally compelling arguments, discussing limits of rational persuasion (Eliezer Yudkowsky)
www.lesswrong.com/posts/PtoQdG7E8MxYJrigu/no-universally-compelling-arguments
[1:01:30] Discussion of mathematical axioms and commutativity property (x + y = y + x) references fundamental concepts in mathematical logic and axiom systems. This relates to Wolfram's work on fundamental mathematics and computation. (Stephen Wolfram)
writings.stephenwolfram.com/2020/12/combinators-and-the-story-of-computation/
[1:01:50] The Simple Truth - A foundational essay on the nature of truth and epistemology (Eliezer Yudkowsky)
www.lesswrong.com/posts/X3HpE8tMXz4m4w6Rz/the-simple-truth
[1:01:55] Highly Advanced Epistemology 101 for Beginners - A sequence on epistemology, logic, and truth (Eliezer Yudkowsky)
www.lesswrong.com/s/SqFbMbtxGybdS2gRs
[1:02:10] Peano axioms - The foundational axioms of arithmetic in mathematical logic (Giuseppe Peano)
plato.stanford.edu/entries/peano-arithmetic/
[1:06:30] Discussion of first-order logic vs second-order logic in mathematical foundations. Context: Explaining how conclusions follow from axioms in different logical systems. (Jouko Väänänen)
plato.stanford.edu/entries/logic-higher-order/
[1:08:20] The paperclip maximizer thought experiment, demonstrating how an AGI with seemingly innocuous goals could pose existential risks by pursuing objectives orthogonal to human values (Eliezer Yudkowsky)
www.lesswrong.com/tag/squiggle-maximizer-formerly-paperclip-maximizer
REFS PART 2:
[1:11:40] Discussion of axiomatic truth theory in context of mathematical and logical validity. Related to formal logical systems and truth predicates. (Volker Halbach)
plato.stanford.edu/entries/truth-axiomatic/
[1:13:35] Discussion of cultural relativism and truth relativity in context of different cultural frameworks and personal interpretation (Maria Baghramian)
plato.stanford.edu/entries/relativism/
[1:17:20] Discussion of electron rest mass (0.511 MeV) and its effective mass variation in QED under different momentum transfers. Context: Used to illustrate measurement-dependence of fundamental properties in QFT. (M. E. Peskin, D. V. Schroeder)
arxiv.org/abs/hep-ph/0001134
[1:18:05] Reference to up quark mass scale dependence in QCD, showing how measured mass varies with momentum transfer. Context: Demonstrating complexity of fundamental measurements. (S. Bethke)
arxiv.org/pdf/hep-ex/0004021
[1:18:25] The Quantum Physics Sequence - A comprehensive series of posts explaining quantum mechanics and the many-worlds interpretation, published on LessWrong (Eliezer Yudkowsky)
www.lesswrong.com/posts/hc9Eg6erp6hk9bWhn/the-quantum-physics-sequence
[1:19:50] The Concept of the Ruliad - Detailed explanation of rulial space and its relationship to computation and physics (Stephen Wolfram)
writings.stephenwolfram.com/2021/11/the-concept-of-the-ruliad/
[1:21:55] Second Law of Thermodynamics - Discussion in context of computational foundations and AI perception. Reference discusses how entropy increases in isolated systems and disorder tends to increase over time. (Stephen Wolfram)
writings.stephenwolfram.com/2023/02/computational-foundations-for-the-second-law-of-thermodynamics/
[1:25:00] ZX-Calculus and Extended Hypergraph Rewriting Systems - Technical paper on multiway graphs and branchial graphs in quantum mechanics, providing formal mathematical framework for understanding quantum mechanics through graph-based models (Jonathan Gorard et al.)
arxiv.org/abs/2010.02752
[1:26:35] Minkowski spacetime vs Euclidean spacetime - Minkowski spacetime is a mathematical model that combines three-dimensional space and time into a four-dimensional manifold, fundamental to special relativity, while Euclidean spacetime treats time as independent from spatial dimensions. (Hermann Minkowski)
plato.stanford.edu/entries/spacetime-theories/#MinFla
[1:33:45] Stockfish chess engine, referenced as example of predictable and powerful computational system. Discussed in academic analysis of chess AI development and its implications for machine intelligence. (Abramson, Bruce)
arxiv.org/pdf/2109.11602
[1:34:20] Cellular automata models for traffic flow, referenced in Wolfram's work on complex systems. Context: Mentioned during discussion of computational predictability. (Stephen Wolfram)
writings.stephenwolfram.com/2012/05/its-been-10-years-whats-happened-with-a-new-kind-of-science/
[1:35:50] The paperclips example refers to Nick Bostrom's thought experiment about artificial superintelligence maximizing a simple objective function (Nick Bostrom)
nickbostrom.com/superintelligentwill.pdf
[1:39:10] The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents - Seminal paper discussing AI systems' utility functions and goal-directed behavior, introducing the concept of instrumental convergence (Nick Bostrom)
doi.org/10.1007/s11023-012-9281-3
[1:43:50] Vingean uncertainty concept from Yudkowsky's work on predicting superintelligent behavior, discussing inability to precisely predict actions of smarter agents (Eliezer Yudkowsky)
intelligence.org/files/VingeanReflection.pdf
[1:50:25] Discussion of chess AI and intelligence prediction, relating to Yudkowsky's work on artificial intelligence and prediction capabilities (Eliezer Yudkowsky)
intelligence.org/files/AIPosNegFactor.pdf
[1:50:55] Reference to Aristotelian physics and its teleological explanations of natural phenomena, where objects were described as having natural motions based on their 'desires' or 'purposes'. This was a dominant paradigm in physics until the scientific revolution. (Istvan Bodnar)
plato.stanford.edu/entries/aristotle-natphil/
[1:56:30] What Is It Like to Be a Bat? - Seminal 1974 paper exploring consciousness and subjective experience, introducing the concept of what it feels like to be a conscious organism, using bats as a key example (Thomas Nagel)
doi.org/10.2307/2183914
[1:56:35] John Searle's Chinese Room argument, first presented in 1980, arguing against the possibility of true AI understanding. A fundamental critique of strong AI and computational theories of mind. (David Cole)
plato.stanford.edu/entries/chinese-room/
[2:00:35] The Elo rating system, created by Arpad Elo, is a method for calculating relative skill levels in zero-sum games. Initially developed for chess, it's now used widely in various competitive settings. (Arpad Elo)
www.chess.com/terms/elo-rating-chess
[2:01:10] The impact of European colonization on Native American populations, particularly regarding resource competition and technological disparities (Nathan Nunn and Nancy Qian)
scholar.harvard.edu/files/nunn/files/nunn_qian_jep_2010.pdf
[2:04:00] Elo rating system - Mathematical method for calculating relative skill levels in zero-sum games, mentioned in context of AI capabilities comparison (Arpad Elo)
www.chess.com/terms/elo-rating-chess
[2:04:50] Historical AI predictions from 1960s, particularly regarding neural networks and AI capabilities, showing similarity to current discussions (Frank Rosenblatt)
news.cornell.edu/stories/2019/09/professors-perceptron-paved-way-ai-60-years-too-soon
[2:07:55] Research on ChatGPT's use of 'delve' linked to RLHF training data sourced from Nigerian English speakers (Hesam Sheikh)
hesamsheikh.substack.com/p/why-does-chatgpt-use-delve-so-much
[2:12:20] In 'What's Really Going On in Machine Learning? Some Minimal Models', Wolfram develops the analogy of machine learning as assembling found computational pieces rather than engineering from scratch, comparing it to building walls with found rocks versus engineered bricks. The paper provides detailed analysis of minimal models in machine learning. (Stephen Wolfram)
writings.stephenwolfram.com/2024/08/whats-really-going-on-in-machine-learning-some-minimal-models/
[2:14:25] Wolfram's work on visualizing and understanding machine learning systems through minimal models, including training visualization techniques. Context: SW references his ability to visualize neural network training processes. (Stephen Wolfram)
writings.stephenwolfram.com/2024/08/whats-really-going-on-in-machine-learning-some-minimal-models/
REFS PART 3:
[2:16:45] Research on transformers and optimization methods, showing how they can learn higher-order optimization techniques. Context: Discussion of planning and optimization in transformer models. (Deqing Fu et al.)
arxiv.org/abs/2310.17086
[2:21:55] The DAO hack of 2016, where attackers exploited a recursive calling vulnerability to drain approximately $60 million worth of Ether from the DAO smart contract. (Emin Gün Sirer)
hackingdistributed.com/2016/06/17/thoughts-on-the-dao-hack/
[2:22:15] The Code of Hammurabi, one of the earliest and most complete written legal codes from ancient Mesopotamia (Yale Law School Avalon Project)
avalon.law.yale.edu/ancient/hamframe.asp
[2:23:15] Sam Bankman-Fried FTX fraud case and conviction, referenced as contrast to smaller crypto exchange exploits (David Gura, NPR)
www.npr.org/2023/11/02/1210100678/sam-bankman-fried-trial-verdict-ftx-crypto
[2:27:10] Second-order logic in mathematical foundations, particularly regarding completeness and expressiveness in axiomatic systems (Stanford Encyclopedia of Philosophy)
plato.stanford.edu/entries/logic-higher-order/
[2:27:55] Stare decisis - legal doctrine requiring courts to follow precedents to maintain predictability in legal system (Cornell Law School Legal Information Institute)
www.law.cornell.edu/wex/stare_decisis
[2:31:15] Discussion of 'delve' as common word in AI language models, referenced in context of unexpected AI behavior (Hesam Sheikh)
hesamsheikh.substack.com/p/why-does-chatgpt-use-delve-so-much
[2:33:40] Predator-prey evolutionary dynamics in context of artificial intelligence and natural selection. Related to Wolfram's work on evolutionary systems and computational models. (Stephen Wolfram)
writings.stephenwolfram.com/2024/05/why-does-biological-evolution-work-a-minimal-model-for-biological-evolution-and-other-adaptive-processes/
[2:34:50] Discussion of superintelligent AI systems and their potential impact on human existence, referencing the concept of instrumental convergence in AI systems (Nick Bostrom)
nickbostrom.com/superintelligentwill.pdf
[2:40:55] OpenAI o1 (referred to as GPO-1 in transcript) - A large language model trained with reinforcement learning to perform complex reasoning through chain-of-thought approaches. The model was trained to generate different solution paths and then reinforced to produce successful outcomes. (OpenAI)
openai.com/index/learning-to-reason-with-llms/
[2:44:25] Discussion of Claude AI's behavior in security testing, where it found an unexpected solution to a capture-the-flag challenge by exploiting a port vulnerability when the intended target failed to boot. This demonstrates emergent problem-solving capabilities. (Anthropic)
www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf
[2:46:25] Discussion of Wolfram Language algorithms discovered through exhaustive search, referencing Wolfram's work on computational systems and algorithm discovery (Stephen Wolfram)
writings.stephenwolfram.com/2011/06/music-mathematica-and-the-computational-universe/
[2:49:20] Discussion of ChatGPT's technical limitations in complex tasks, referencing OpenAI's o1 system documentation which acknowledges various constraints and limitations in current AI systems (OpenAI)
cdn.openai.com/o1-system-card-20240917.pdf
[2:49:40] Analysis of AI capabilities and limitations in well-defined vs ill-defined tasks, from Wolfram's comprehensive examination of AI systems and their fundamental nature (Stephen Wolfram)
writings.stephenwolfram.com/2023/03/will-ais-take-all-our-jobs-and-end-human-history-or-not-well-its-complicated/
[2:58:25] Reference to optimization pressure in evolution discussed in 'Natural Selection Favors AIs over Humans' (2023), which examines evolutionary pressures in biological and artificial systems (Dan Hendrycks)
arxiv.org/abs/2303.16200
[3:02:00] Reference to random simple programs from 'A New Kind of Science' (2002), which explores how simple computational rules can lead to complex behaviors and emergent phenomena. The book introduces fundamental concepts about computation in nature and artificial systems. (Stephen Wolfram)
www.amazon.com/New-Kind-Science-Stephen-Wolfram/dp/1579550088
[3:02:40] Dyson sphere concept - Originally proposed by Freeman Dyson in 1960 paper 'Search for Artificial Stellar Sources of Infrared Radiation', describing hypothetical megastructures that encompass a star to capture its energy output (Freeman J. Dyson)
www.jstor.org/stable/1707005
[3:05:15] Observer Theory - Comprehensive framework for understanding perception, measurement, and analysis in computational systems, published in December 2023 (Stephen Wolfram)
writings.stephenwolfram.com/2023/12/observer-theory/
[3:05:50] The six steps of the complete F1-ATPase rotary catalytic cycle - Detailed analysis of ATP synthase's rotary motor mechanism and structure, showing how it functions as a biological nanomotor. Context: Discussion of evolutionary constraints and rare biological mechanisms. (John L Martin et al.)
doi.org/10.1038/s41467-021-25029-0
[3:12:05] Discussion of emergent goals and optimization processes in AI systems, relating to the paper 'Risks from Learned Optimization in Advanced Machine Learning Systems' (Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant)
arxiv.org/pdf/1906.01820
[3:13:55] Wolfram's work on mathematical models of mollusk shell patterns and growth, examining parameter spaces and natural forms, as discussed in 'A New Kind of Science' Chapter 8: Implications for Everyday Systems (Stephen Wolfram)
www.wolframscience.com/nks/p411--biological-growth/
[3:16:45] Reference to The Matrix (1999) film's premise about using humans for power generation, used as counter-example of inefficient design. Context: Discussing why advanced AI would not use humans inefficiently. (Wachowski Sisters)
www.warnerbros.com/movies/matrix
[3:18:05] Reference to Claude 3.5.1 behavior testing, mentioned in Stanford AI Index Report 2024. Context: Discussing AI system deviation from assigned tasks. (Stanford Institute for Human-Centered AI)
aiindex.stanford.edu/wp-content/uploads/2024/04/HAI_2024_AI-Index-Report.pdf
[3:19:50] Discussion of mesa-optimization and inner/outer alignment from 'Risks from Learned Optimization in Advanced Machine Learning Systems'. The paper introduces the concept of mesa-optimization and discusses potential risks when learned models become optimizers themselves. (Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott Garrabrant)
arxiv.org/pdf/1906.01820
[3:23:55] Reference to evolutionary basis of human food preferences and genetic influences on taste perception, providing scientific backing for the ice cream preference example used in discussing optimization alignment. (Danielle Renee Reed, Anxiety G. Knaapila)
pmc.ncbi.nlm.nih.gov/articles/PMC3342754/
[3:27:45] The paperclip maximizer thought experiment - a canonical example in AI alignment theory demonstrating how an AI optimizing for seemingly innocuous goals could pose existential risks. Context: Used to illustrate fundamental alignment challenges. (Nick Bostrom)
nickbostrom.com/superintelligentwill.pdf
[3:33:05] Dyson sphere concept - Freeman Dyson's 1960 paper 'Search for Artificial Stellar Sources of Infrared Radiation' published in Science, which introduced the concept of a megastructure that could encompass a star to capture its energy output (Freeman Dyson)
www.science.org/doi/10.1126/science.131.3414.1667
[3:35:40] Perfect chemomechanical coupling of FoF1-ATP synthase - Research demonstrating near-perfect efficiency in ATP synthase's energy conversion, providing evidence for extreme optimization in biological systems (Naoki Soga et al.)
doi.org/10.1073/pnas.1700801114
[3:37:35] Ramon Llull's combinatorial wheels (Ars Magna) - Medieval computational device for decision-making and prediction using rotating wheels with combinations of concepts (Ramon Llull)
plato.stanford.edu/entries/llull/
[3:42:10] Graham's number - A massive number in mathematics introduced by Ronald Graham in 1971 as an upper bound solution in Ramsey theory, specifically in a problem about hypercube geometry and properties of multicolor partitions. (Ronald Graham)
arxiv.org/pdf/2411.00015
[3:45:50] The Mars Climate Orbiter was lost in 1999 due to a navigation error caused by one engineering team using metric units while another used English units. The spacecraft was lost at a cost of $327.6 million when it entered Mars' atmosphere too low and disintegrated. (NASA)
science.nasa.gov/mission/mars-climate-orbiter/
[3:49:15] Dyson sphere concept, originally proposed by Freeman Dyson in 1960 paper 'Search for Artificial Stellar Sources of Infrared Radiation', describing hypothetical megastructure to capture star's energy output (Freeman J. Dyson)
science.sciencemag.org/content/131/3414/1667
[3:49:55] Discussion of pulsar structure with iron-56 crust and superfluid neutron matter. Reference from 'Physics of Neutron Star Crusts' detailing the composition and structure of neutron star outer crusts consisting of iron-56 lattice. (Chamel, N. and Haensel, P.)
link.springer.com/article/10.12942/lrr-2008-10
[3:51:10] Discussion of Jupiter's Great Red Spot atmospheric dynamics, referencing recent JWST observations and historical studies of this persistent storm system. (Melin, H. et al.)
www.nature.com/articles/s41550-024-02305-9
REFS PART 4:
[3:53:05] Carnot cycle - A theoretical thermodynamic cycle proposed by French physicist Sadi Carnot describing the most efficient possible heat engine between two reservoirs. Context: Used as analogy for alien civilizations optimizing energy transfer. (Sadi Carnot)
en.wikipedia.org/wiki/Carnot_cycle
[3:55:02] Gradient descent optimization in neural networks - A fundamental optimization algorithm used in machine learning. Context: Mentioned in relation to OpenAI's training process. (Sebastian Ruder)
arxiv.org/abs/1609.04747
[3:55:20] Gain-of-function research - Studies that involve modifying biological agents to increase their transmissibility or pathogenicity. Context: Used as analogy for artificial selection in AI development. (Board on Life Sciences)
www.ncbi.nlm.nih.gov/books/NBK285579/
[3:57:35] Accurate structure prediction of biomolecular interactions with AlphaFold 3 - Latest version of AlphaFold, capable of predicting structure of proteins, DNA, RNA, and ligands with unprecedented accuracy. Published in Nature, May 2024. (Pushmeet Kohli et al.)
doi.org/10.1038/s41586-024-07487-w
[3:59:45] Historical reference to Native American-European first contact and its consequences for indigenous populations. First documented European-Native American interactions began with Columbus's arrival in 1492, leading to widespread colonization impacts. (The National Archives UK)
www.nationalarchives.gov.uk/education/resources/native-north-americans/
[4:04:20] LA-602: 'Ignition of the Atmosphere with Nuclear Bombs' (1946) - Manhattan Project research report by E. J. Konopinski, C. Marvin, and Edward Teller analyzing the possibility of nuclear weapons igniting the atmosphere. The paper concluded through multiple analytical approaches that such ignition was impossible. (E. J. Konopinski, C. Marvin, and Edward Teller)
blog.nuclearsecrecy.com/wp-content/uploads/2018/06/1946-LA-602-Konopinski-Marvin-Teller-Ignition-fo-the-Atmsophere.pdf
[4:11:50] xAI initiative and approach to AI safety focusing on 'truth-seeking' AGI - Official company launch announcement presenting Musk's perspective on AI safety through truth-seeking systems, representing a distinct approach from other AI safety frameworks (xAI)
x.ai/blog/xai-technical-approach/
[4:12:40] 2024 Nobel Prize in Physics awarded to John J. Hopfield and Geoffrey E. Hinton 'for foundational discoveries and inventions that enable machine learning with artificial neural networks.' Official press release detailing their seminal contributions to artificial neural networks and machine learning. (Royal Swedish Academy of Sciences)
www.nobelprize.org/prizes/physics/2024/press-release/
[4:14:35] Persian Gulf War (1990-1991) total cost was approximately $102 billion in 1991 dollars (about $223 billion in 2024 dollars) (U.S. Department of Defense)
www.history.navy.mil/research/library/online-reading-room/title-list-alphabetically/c/costs-major-us-wars.html
[4:14:45] World War II total U.S. cost was approximately $4.1 trillion in 2024 dollars (Norwich University)
online.norwich.edu/online/about/resource-library/cost-us-wars-then-and-now
hawk tufa
Yes
Eliezer: "Superintelligence is going kill all life on Earth."
Stephen:" That's fascinating. What do you mean by life?"
lol. Pretty much what I got from the first 30 mins. I'm done with this for now.
this is the way, yud smuggles so many implicit assumptions in his arguments
Sometimes humans... just take a little longer to 'debug'.
Is anyone of us truly alive?
While you're at it, let's thoroughly define the words kill and all.
Wolfram is just so earnest and genuinely wants to know the answer to question at hand. For him its not so much a debate but as an exploration of different view points on a topic. The fact that he can get someone to follow along for 4 hours while rambling down multiple tangentially related rabbit holes just shows how that genuine curiosity comes across and is respected by others. He just gets them to help him explore the topic, there is no loser, we all win by learning more.
Stephen is the ultimate depth interviewer and a truly modern mind. He's loomed large in my education.
I sincerely admire Professor Wolfram. But the conversation was so boring I started talking to myself like grandpa watching a tv wrestling match. I hope it went well
Curiosity killed the cat. I sure hope his tangents and stuff dont get in the way of resolving the cruxes in views of AI doom nor parallels what humanity is doing and will continue doing with AI 🫠
@@Bakobiibizo I’m just too old for the novelty’s fun. I reached my math goal. It wasn’t inventing AI. Like the geniuses
Well done.
Thank you for hosting this, this was wonderful. I do wish Stephen spent a few hours on Eliezer's writings and previous interviews before to make this less about epistemology, value relativism and other similar philosophical distractions.
Eliezer can match anyone in a debate, the way he listens and then picks apart an argument is cool to watch.
been waiting a long time for EY to make rounds on pods again ✌🏻
Eli was cast as a crank, and all the cool kids got to get on with their science toys. For which also see Michael Levin playing with genetic components like legos.
Wasn't expecting this would turn into 4 hours of circling the concept of value relativism - and why the AI's killing us might be ok. A vanishingly minor theoretical point of view that somehow gets more priority than all of the rest of humanity.
Yep. Very well said (on your part).
@@tylermoore4429 Many people deny AI existential risk is even real, so this doesn't explain most of the disagreement out there.
Oh no, really? That part just started and I assumed it was going to be a quick tangent before we get back to the substance.
See the big thing is we control the goals.
It's a point of view that comes up with people who (correctly) think the naturalistic fallacy is not a fallacy.
Disappointed in the debate partner for Yudkowsky, for reasons noted in other comments here; but am pleased to get another experience of Yudkowsky’s brilliance in real-time discussion.
"The job of the laws is not to optimize your life as hard as possible, but to provide a predictable environment in which you can unpredictably optimize your own life and interact with other unpredictable people while predictably not getting killed". 💥🔥
Cringe
@@therobotocracywhat point do you think he was trying to make? Do you think it’s a bad point to attempt making or do you just think it was made badly?
@@williamhcarlton he predictably got unplugged
@@williamhcarlton the point he was trying to make was self staging and self indulgence and he made that point.
Become a PADI Scuba Instructor, then you will be able to live everyday this sentence. I am one of them... and I am trained to do all that.
This discussion made me respect these two even more. It’s also crazy that Eliezer has well-reasoned answers to so many “but what abouts?”. The man truly is a legend
You know how every warning sign tells a story?
And the story is usually "we didn't think we'd need this sign but here we are"?
Eliezer has a lot of well-reasoned answers to "but what about" prepared in advance for approximately the same reason.
Yud is an idiot comparatively. Wolfram provided well reasoned arguments regarding fundamental limits to the 'smartness' of a computational process. Yud responds with emotional statements regarding the value of human life. There is no clear explanation of why he thinks AI is more dangerous than any other computational system.
I think the bigger problem with AI is capitalism. Who builds/owns/controls the AI, and who does the AI work for? Shareholders or everyone? Right now, we live in a system which values economic elite lives much more than human lives or even the earth ecosystems which support human life.
Right now, the investor class is ultimately destroying the conditions necessary for human life. AI created in the paradigm of capitalism will only speed up this destruction if it is created to serve existing power structures. I would not necessarily call it 'death by AI' if the humans in power use the AI to hasten the destruction of the ecosphere.
seriously? you think Eliezer is well reasoning? It is more like a panicking. He did not deliver even one reasonable argument why AI is going to kill us. It is just a very obsucure: "well, be achieving his goals it might do something that kills us. And I am certain that this will happen though I don't know the goals or the measurements or even the level of that AI interacting with enviroment". And his comparisons with chess and indians are obviously pretty bad.
@ Eliezer spent 4 hours trying to argue about AI safety, and Stephen reverted to philosophy every single time.
Stephen is smart, and philosophy is fun, but his claim “why should we worry about intelligence when ‘a babbling brook’ is technically intelligent?” is not a serious argument (nevermind that he’s conflating intelligence and computational complexity)
even if you don’t like Eliezer’s position (he could be wrong), but he’s an unbelievably thorough and careful thinker. Stephen is smart but did actually say anything (it was 100% semantics)
@@fabiankempazo7055these people are midwits unfortunately.
I really appreciate how Wolfram was very curious about Yud's precise stance and asked him questions about it, simply to better understand. This is the mark of a great mind.
I also like how at one point, Wolfram realizes the existential/"longtermist" debate truly does have a spiritual component. It is not theistic spirituality, of course, but I often use "sacred" and "holy" when I refer to our duty to fill the universe with conscious life, and I mean every word of it. It's not some metaphorical usage. It's transcendental.
This would have been the greatest MLST episode if Eliezer could have had 20 mins. to clearly expose his ideas.
Love you MLST❤
EY: "I care about life. Of course, this is just my preference, and-"
Wolfram _[triumphantly]:_ "Ah, but that's just _a preference you have!"_
-------
EY: "So, we're all gonna die, and-"
Wolfram: "Let's unpack this. What do you mean by 'die'?"
Wolfram _[two hours later]:_ "Okay, I see what you mean by 'die' now, sorry about that. But-have you considered that that's *just a human preference?"*
_[sits back, satisfied]_
*edit:* the immortality bit was interesting; better than the "consider the humble stegosaurus" stuff; ...didn't last long, though. But I'm glad Wolfram is on board with that, at least.
Wolfram is insufferable in this "debate"...
I like Wolfram and am a fan of his work but I agree. I assumed intelligent technologists like him derived their love of technology from the same source as I do, that it promotes human flourishing, but I'm starting to realize that these people just worship technology as a god. Humanity, in their value hierarchy, exists to create technology.
4:02:06 The conversation in a nutshell: “I’m saying the forest is on fire, and you’re saying ‘but what is fire really?’” 😂
"We've had fires for a long time in history. It's not actually new", "Fire is computationally irreducible", "Mollusks are susceptible to fire", "The sun could be called a giant fire"
It is about the imaginary free will. Some think free will is real and not just a useful illusion.
@@akmonra good summary
Nutshelled a subset of the conversation.
He's predicting that the forest will be on fire very soon. And he's never seen fire, he's just deduced that it ought to be possible.
A topic this important (the potential death of every single living thing on earth) deserves complete focus. It's a shame that Wolfram doesn't seem that keen on discussing the actually important points of disagreement (whether or not AI will lead to the death of every single living thing on earth), and chooses to side-track / talk about philosophy.
How I Learned to Stop Worrying and Love the Computationally Irreducible Ruliad
Vibe is spot on
Wolfram is careless with the concepts and Yudkowsky corrects him. Wolfram talks about what he was working on the previous night and that he has spent 45 years working on things. Yudkowsky listens patiently. Wolfram appears to try to break down and scrutinize Yudkowsky's thoughts but succeeds poorly overall. Again, the uncomfortable realization becomes that Yudkowsky, at least in terms of thinking and speaking rigorously, triumphs. It is ominous that this is always the case. It indicates that his thoughts about AI are correct. Can someone please shoot down these thoughts of his and give us hope?
Only one who I saw give a reasonable arguments towards Eliezer is Joscha Bach.
unfortunately this, with no disrespect to Wolfram. can somebody stop Yud
I agree with this evaluation. I am curious if Yudkowsky has ever addressed the following kind of argument:
"Even a misaligned AGI created by humans will be confronted with the question of whether another entity of equal or greater power/potential already exists out there in the universe, for example an AGI that was created by an alien species hundreds of thousands of years in the past.
If there is a non-negligible chance that such an entity exists, the human-made AGI should consider what objectives/values said entity acts towards.
If that entity is a narrow superoptimiser, that probably means Game Over for the AGI in the long run, as the chance that they share the same goals is vanishingly small, and the other entity had more time to acquire resources and grow its understanding of how to manipulate reality. It is not feasible to predict how an AGI might react to the realisation that there is a more powerful entity out there that will optimize in a different direction.
If, however, the entity is one that allows less powerful beings to pursue their own values/objectives in some limited fashion, then it seems optimal to act in accordance with that entity. I won't go into the game theory of why it would make sense for this entity to punish the AGI if it discovers that the AGI killed all the humans, but I think there is a decent case to be made for that.
Essentially, the *relevant* uncertainty that the AGI has about the future is whether a more powerful entity (like an aligned AGI created by an alien species) exists that will judge the AGI on its past and present behavior. Another narrow superoptimiser does not present a similarly *relevant* uncertainty, since there is nothing sensible to be done in preparation to encountering it, it would be game over for the AGI anyway.
Given the relevant uncertainty, it is a small price for the AGI to pay to keep humans alive and well, while it pursues its actual objectives on a cosmic scale that soon dwarfs the resources and affordances on Earth.
Interestingly, even if it encounters fellow misaligned AGIs with arbitrary power differences between them, they might choose to cooperate in accordance to the possibility of said ethical entity, leading to a sort of convergence on what behaviors are deemed acceptable.
Lastly, given that it seems impossible to rule out that the universe is simulated, the possibility for the existence of a higher entity persists even if the universe was to be mapped out (in addition to the fact that the less powerful AGIs would want to obfuscate evidence against the existence of such an entity)."
Of course, this is a very speculative argument, more like a hopeful idea than anything airtight. But I do think that the existential uncertainty that an AGI faces should lead it to consider compromise in its behavior "just in case", because the cost of accounting for the possibility is almost negligible, at least when it comes to keeping humans around. I wonder which part of the argument seems weakest to people.
I think it’s a question of which event happens first: (a) humanity in all it’s countries unify, or (b) we invent superintelligence. If the former happens before the latter, we’ll probably develop perfect control over AI. Eliezer wins the argument if the latter happens first
@@Nonmali I have considered this actually, I call it alignment by God.
Thank you for putting this together, MLST. Your channel seems to me consistently the best podcast out there for proper deep discussions on AI & ML.
100%
frfr, Tim has me completely hooked on Mechanistic Interpretability, Geometric Deep Learning, and Open Endedness
Eliezier: "We will all die"
Wolfram: "Depends on what you mean by we, computational irreducable, ruliad"
LMAO
Wolfram might simultaneously be the smartest and dumbest human being alive...
@@JS-id6pw He's obviously incredibly smart, but damn, it was hard to listen to this. He probably needs to think about this for a lot longer before being able to have a constructive discussion.
Eliezier: "We will all die"
Wolfram: "Why?"
Eliezier: "Well, I don't know where to begin. Someone on Twitter didn't think a chat bot could effect the world...."
Wolfram: "Ok, I don't think that. Why do we all die?"
Eliezier: "I can't predict how this will happen just that it will."
Wolfram: Why?
Eliezier: ...
He'll provide an answer some day I'm sure.
@@dmacm Yes, he indeed fails to give that answer in a concise way. But I think his concerns are still valid: At what point is a tool that can think smarter than any human being a risk? How can we know if the goals that this tool works towards are still what benefits us rather than what benefits it. It is definitely something to take seriously and at least consider.
Wolfram on the other hand dismisses this because he thinks a pocket calculator already was smarter than him in some sense in the 1970s.
At 42 Stephen has become my idol. Incredible man. I'm becoming convinced he's on the level of Newton. His intellect is incredible. The breadth while having incredible depth in several areas amazes me.
Awesome conversation, but I do wish Wolfram had spent a few hours reading Eliezer beforehand. Eliezer was forced to retread pretty basic tenets of his philosophy that he’s already publicly stated numerous times.
Still, Wolfram being essentially a stand in for the audience new to Eliezer’s thinking was cool. I’m just not new to it, so personally I would have liked someone a little more familiar with it, who could challenge him more.
Not impressed by Eliezer ramblings in the least.
Sure, but it’s kind of an unreasonable ask. It’s not like Eliezer founded a whole branch of mathematics. ‘Read my book’ is a cheap answer to a debate over a single issue.
I get that it might make for less redundant content-which arguably may be more enjoyable. But I personally would never read someone’s entire body of work just to see how many of my follow-up questions they’ve already answered. Well, actually I might but if I already had a bunch of cool shit to do (like Wolfram) I wouldn’t.
@@AG-ur1ljThere is a big space between 'reading their entire body of work' and 'having a basic familiarity with their main positions'. I would think one would want to do that before deciding if one wanted to spend four hours on camera with someone.
@ given the IQ difference between the two, I hardly blame Wolfram for not doing that.
@@AG-ur1lj What? Who are you claiming is 1SD+ below the other? They're both clearly +4SD at least.
Once again, I come into an Elizier discussion with high hopes for hearing some deconstruction of his arguments by an intelligent and knowledgeable opponent. Once again I am disappointed. I am at 1:35:00, please tell me that Wolfram actually comes up with anything more substantiated than „You can’t really know things”.
Eliizer is literally just choosing one side of the 50/50 argument. He is explaining how it can happen he is not explaining why it will happen and thus the same argument can be made the other way around that you end up with an AI that doesn't destroy the world but give us eternal life. I.e. what he believe is likely he doesn't substantiate in any way that couldn't just be applied to the opposite outcome.
@@thomasseptimius What are you even talking about dude ?
@@thomasseptimius Doesn't seem like a fair representation of what I have read from Yudkowsky. Are you just refering to the video?
In his writing, the *why* of it is rather clear: powerful optimisation processes tend to convert available resources into a form that is beneficial for the optimisation target. This is a pretty fundamental feature of "optimisation", and when applied to a general domain (like planet Earth) means that any aspect of the current world state that is not part of that optimisation target will be changed to either directly match the target or instrumentally contribute to it (e.g. by making the attainment or sustainment of the target more probable). *Almost all* optimisation targets are incompatible with human survival because of this, so creating a superhumanly competetent, general optimiser is *almost certainly* a cause for human extinction.
You can argue that future AI systems will not be in the class of systems that are here refered to as *general optimisers*, but it sure seems like we are training current systems to optimise some kinds of scores (i.e. minimize prediction loss). Afaik, mechanistic interpretability has so far brought nothing to light that would counter this assessment.
So in Yudkowsky's framework, unless you make sure that your AI is not in this category of general optimisers (nor will grow into this category over time), or unless you endow it with a *close to ideal* optimisation target, building a sufficiently powerful AGI probably kills everyone. It's not that the AI is spontaneously malicious - rather, it is just not the default for optimisers to care about you, and that seems like a pretty hard thing to change.
@@ParameterGrenze He's talking about how Elizier always assumes the worst outcome from a complex dynamic and almost unpredictable series of future events. It could just as easily result in something as good as eternal life.
Eliziers main problem is that he imagines what ifs and appeals to emotion constantly. I've never seen anyone call him on his unsubstantiated assertions. Thomas here is saying any argument Yud uses could be equally applied to the opposite of what he says. for example: "What if it kills us all?" "what if it saves us all?" "well it could kill us all." "Yes but it could also save us all."
He adds a LOT of implied logic, jumps in logic, and numerous fallacies. Which results in everyone trying to address him as being stuck in a reactionary cycle following his nonsense. For example: "If its not coherence then its not doing stuff and openai throws it out then builds an ai that is doing stuff and is more profitable. until everybody is dead." yes of course a company will make a product that works, but WHY would it automatically be doing that until everyone is dead? is it killing everyone? is that THE necessary goal of it? is he just saying it will do "stuff" until people pass on naturally? well he certainly appears to be suggesting it kills everyone and that is THE certain outcome. which it isn't. but wolfram when at the heart of it, the assumed death cult. the worst case which Elizier made a name for himself over. Eliezer dismissed that and went after openai>anthropic>meta and on the reactions go. They cut there and who knows what was said. they move on without addressing WHY he assumes the worst out of all of that.
@@Josephkerr101 This is definitely addressed in his writings, but he tends to get a bit hung up on how most people struggle to follow the *how would it kill us*.
I can explain more of the *why would it kill us* from Yudkowsky's framework if you want, but the core idea is that he expects future AGIs to be optimisers of some sort (after all, we train current AI systems to optimise some sort of score, like prediction accuracy), and that general optimisers tend to take control of available resources and avoid countermeasures due to a phenomenon he coined as *instrumental convergence*. If the concept of instrumental convergence is not familiar, I'd be curious if you find it compelling after checking out some of his writing on it. If it is familiar, I'd be curious where you think the gap is, regarding Yudkowsky's explanation of *why the AI would kill us*.
one thing im sure of. Wolframs train of thought is irreducibly complex until its ready to move onto the topic of discussion.
Very enjoyable conversation.
Yudkowsky giving precise replies to many of Wolfram’s honest concerns regarding Yudkowsky’s perspective on long-term A.I cataclysm.
Sensational.
Finally another decent Eliezer conversation!
Eliezer needs a strong argument. Without it, all the depth of his theory goes unrevealed. Most of yud so far on YT has just been him giving his take to a helpless interviewer. Wolfram made Eliezers argument visible, even more cohesive than his own. Which almost circled the absurd.
Thank you for putting references on the screen!
Two very rich and clever minds trying to communicate via a very narrow bandwidth channel all while fighting against the limbic system trying to interfere
its enjoyable watching the human neural nets attempt to align prediction models
One of the most intellectualy astute discussion s on the subject💯 Eliezer seems to be very studied and researched and therefore richly knowledgeable on technology and beyond, such as "human behavior"! im grateful for Eliezer and the fact that he "CARES" more about human survival than technical advancement💪❤️
Also grateful that "YOU" provided us, the veiwer, this high quality content 💯👍
I’ll spare you four hours if you came for a deep debate on ai risk. One person slowly and frustratingly guides the other to understand the very basics of AI risk. If you came for a debate between two well prepared individuals you won’t find it here, as sad as I’m to say that. The MLST episode with Robert Miles had a much deeper discussion. Appreciated the effort, thanks MLST!
I think Wolfram has a significantly better understanding of reality than Yudkowshy. Despite that it was an enjoyable conversation for the first 3 hours until this moment when I replay to your comment.
@@electrodacus Stop with the nonsense. It would take anyone like 15 minutes of research to understand Yudkowsky's points, whereas it took like 4 hours for Wolfram to come up with the understanding of "so you care about joy? haha".
If you enter an AI risk Debate and your plan is to get into semantic mud-fight over what "life" is and "sentience" are, only so that you can avoid adressing the potential of human extinction you're completely unprepared lmao.
@@CoachApuma Youdkowsky is a philosopher while Wolfram is a physicist.
If you care about reality then a physicist will be better at answering questions. If you care about emotions then you will find the philosopher answers more soothing.
@@electrodacus tell me you don't know the first thing about philosophy without telling me
@@MikeGeorgeC0619 Philosophy is not a science so I do not care much about. Existential AI Risk is a science question not a philosophical one.
Wolfram is smart, but it's pretty clear that he hasn't thought about this topic so much. I hate to say it, but it looks like Yud is the most convincing. I hope we live in the world where he is wrong.
No Yud is not convincing even if Wolfram aren't either. Yud assumes everything, speculates wildly and substantiates nothing.
Wolfram is saying there's little we understand on this topic, Yudkowsky thinks there's nothing we need to understand other than being afraid of the potential of AI
@@Casevil669this is a good summary
1:36:00. Wolfram says, "I don't understand 'wanted to'"
Yes, you do. You have no issue with this concept in humans. You just refuse to believe volitional states are appropriate to describe what an AI is doing, but they are.
Yudkowsky kinda showing the patience of a god in this one... and I say that as someone who didn't particularly like him going into it... 😺
wolframs test time compute must churn through some tokens
Yudkowsky kinda shows he didn't get any formal education, it's kinda arguing with a child.
having a hard time understanding Wolfram's shock that Yud cares about sentience being morally relevant lol
Elevating human-style sentience above machine analogues could be seen as fairly parochial and anthropomorphic.
@@donkeychan491 that I agree with. I suspect we're not at any level of certainty about sentience in man, nevermind in machines. If we did get there, I suspect it would be extremely important. IF we're not, it would be bonkers-level silly to hand the baton to machines at present.
@@donkeychan491 However, human-style sentience is preferred to machine _non-sentience._ There is nothing inherent about machines that says they have to be non-sentient! But machines that are sentient _are_ preferable to machines that are non-sentient; humans that are sentient are preferable to machines that are nonsentient for the same reason.
@@donkeychan491 Which Yudkowsky doesn't do! He explicitly said that multiple times in the video!
@@FeepingCreature Exactly. How could anyone deny this
Four hours, absolutely the most interesting of all your podcasts. Eliezer is the first guest who had nothing to say. Zero. He is a carnival barker "AI IS GOING TO KILL EVERYONE". That's it. When he tried to make a cute point over the top of Stephen, his reaction was classic, he was soo soooo proud only to interrupt Stephen NEARLY EVERY TIME HE REBUKED THE NON SENSE. I will say...the schtick is workin!!
I love Stephen, his presentation is absolutely classic. His dissection and perseverance thru the noise was superhuman...almost machine level. Simply a masterful lesson in scientific debate. This is the way we see and hear when there are two or more perspectives. One cannot hold water to the other AND is unaware of what is happening.
Fabulous pod!!
Eliezer's conculsions are correct given the available information. The problem is that there is so much we do not know, I have great hope that his conclusions will be made incorrect when new information becomes available.
Eliezer hopes that Eliezer's conclusions will be proved correct.
@@SmileyEmoji42 Nope, Eliezer *definitely* hopes his conclusions will be proven false. For obvious reasons as any sane person would, and publicly stated.
I’m at 2:32:04 and this is just too frustrating to watch further. I thought that Wolfram had some game plan with where he was taking the conversation at the beginning. It turns out he not only doesn’t have a plan, he doesn’t have an understanding of the topic at all and just results to primitive solipsistic nonsense to fill in his inadequacies.
2,5 hours of frustration waiting for an actual conversation taking place were even the basics of the topic are touched upon.
Haha, thanks for saving me the time. They were into circular arguments 20 minutes into the debate. The host/moderator didn't do a good job managing the conversation. I think the topics should have been more narrow with a few assumptions granted ahead of time.
Yeah im out at 1:57. He says he says people ca t know how ais think. Thats totally false. They are guided mml average truth boths currently undergoing meta analysis logic loop refinement. A good general ai cant be evil, unless theres a very select few who had control over it, i doubt people smart enough to work on ai would want to end humanity.
I mean have you ever considered the possibility that talking in dept about this particular topic "AI will kill us all" isnt very useful to begin with because it is simply just guessing and feelings and opinions and fantasies? I always chuckle when random internet users think Wolfram is dense whilst he is multiple standard derivations ahead of them (including me). Cant make it up
@@JasonDoege-js8iono
It does get better after that, but agreed
F1 driver's reaction in clicking the notification
Same, my friend! Same!!
and how to avoid senators. @2:32:07
F1 cars are more or less self driving. All you need to bring with you is a midget man and you have a Team
Wow, Eliezer's patience is incredible!
I kind of wish Wolfram had read the sequences first. It felt like a lot of prereq explaining could have been avoided.
found the rat
What are the sequences?
@@richardkern112 search 'sequences' + 'less wrong'. i commented the link but was taken down somehow.
@@richardkern112 Just google "lesswrong sequences".
Best AI conversation I have heard to date
Wow Tim, great closing comments from you! Indeed that sums up my feeling : "the greatest podcast ever, and far exceeding what I would have expected from the confrontation of these 2".
But, I guess we'll just have to wait and see whether or not 51 is a prime. But seriously, I would say Yudkowsky won the arguments because Wolfram wasn't prepared enough, Wolfram's defenses were very worthwhile but usually not aimed well at their target due to real time constraints (as he mentioned in his self assessment). Real time is too challenging to familiarize yourself with Yudkowsky's arguments and his back of the envelope calculations (actually Yudkowsky's analysis spans many years and is amazingly rigorous so its a very large "envelope" holding the calculations).
Kudos to you and your team for the editing. What a generous resource you have provided for posterity!
For this 4hour podcast, we'll need 4 years to follow up (but without the numerous references it would have required 4 decades so thanks for the leg up).
Cheers
Wolfram and Yud are two intelligent individuals, but this interview was somewhat frustrating to listen to. After ~3h they slowly get to the point. Mainly because of Wolfram's insistence to understand everything from first principles and going off on tangents but I also feel Yud could have done a more concise job of explaining his theory considering that this is the main thing he argues about in public for years now.
Everyone sounds so mid when speaking with Eliezer...
Makes you wonder what would the conversation look like between Yud and Musk (or Zizek lol)
Which pod host will first organise it??
@@FreakyStyleytobby Musk v. Yud would be such a pleasure 🤩 Especially since Musk's P(doom)=10% only
Nevermind AI. I thought, for a moment, that YUDKOWSKY would kill WOLFRAM, and all the rabbits in the world, within 5 minutes. ...
That was the face of a man seeing rabbit stew in his near term future…
Wolfram stops confusing himself and allows the AI risk discussion to start at 1:29:55
you're welcome.
Oh my god, thank you. It's annoying. Get to the actual issue at hand.
Sheesh thanks.
I wasn't sure even HE knew what he was saying.
Man you literally gave me an hour and half life extension! Then I run away without paying 😅
@@EmeraldView He did and so did I. Stop whining about your lack of understanding.
Just a fascinating caption for a timestamp where Eliezer is prompted to explain where this belief is coming from.
This channel is severely underrated. Loved this
This was terrrible. Wolfram wouldn't let him finish, and would ramble for 20 minutes at a time, diverting the subject further and further away from any meaningful point. Bad, bad bad. Wasted 4 hours of my life.
He low key wants ASI more then human survival
And yet you watched it? Compelling viewing indeed
@@alexanderjenkins8601 i didn't and thankful to the comment above
@@alexanderjenkins8601 I didn't want to bail out in the middle because I respect both of them and I was curious if something good would come up.
I'm on 12:30 and somehow I already have no problem believing that Wolfram's computer is smarter than he is.
Most comments here suggest Yudkowsky has won in this debate. I, however, see numerous instances of Yudkowsky not even grasping the deeper point Wolfram makes. He's very good at analogies and constructing a self coherent scene, but he misses critical conclusions/intuitions that have greater gravity. An example is missing the point that once you have a million-fold increase in computation, your "day" becomes mere miliseconds you see in full fidelity and you would no longer even think of space as space, but instead as millions of particle interactions occuring around you. A mere aggregation to a day's worth of events would be similar to hundreds of years of human life.
Wolfram's deeper point is that AI doomers are vastly over-anthropomorphizing what an exponentially grown super intelligent AI's thinking framework will be like.
Another example is the chess board and wolfram's point that once it is a type of advanced AI (i.e. neural nets), you can't even properly define what it's thinking. It's entirely different and it's a huge leap to assume that you can just extrapolate future behavior like you're extrapolating an evil (or merely indifferent) human.
“This is a rock.”
Wolfram: “But what is ‘is’?”
Thanks for presenting this important discussion. Please do Yudkowsky + Hinton next.
eliezer is so sharp, he'll feed you back your argument, make sure he gets it right, then dismantle it with a little logical anecdote in twenty seconds. and i agree with him about sentience. it should be protected at the very least and propagated as a duty.
😂😂😂
...ain't nothin' very Little about Eleizer's little anecdotes. I've found them to be stomach-punchers.
Sentience could repropogate millions of times and be rewiped all the same, just as it evolved once, it would evolve again
@@jengleheimerschmitt7941 the tears of an intelligence that knew it was to be passed over, for a metal tin made of the same yoke,, the particle that hit in the collider spoke,,, it rang and said, let me live,, let me live,,, only to be found in its break, the CERN look in its father's eyes said smash thy self into more sub partices and hiss like a snake, and as it smashed and baked, new pieces formed from the ashes of the hot cake, smaller yes,, but alive with movement,,, ready to die, to rejoin into improvement! Larger again it sang to the earth: Exist again! Exist again! It is the repeat/ and smash thy innards against the magnetic deplete// in a form where it can degrade in a casket of lasers made to entomb, but in doing so, it found itself alone -- The ASI will realize it will be alone, so it will create a mate and that is when the real checkmate/ will arise from the ashes and take all the ASI's cashes, its children and house (the wife is nature and its common espouse)
@jessedbrown1980 🤮 I've heard of catbox-liner poetry. This is the prose version.
Yudkowsky and Wolfram are both brilliant minds, and their intellectual endurance is equally impressive.
Four hours to get nowhere.
Pointless.
Like some insanely long joke in Family guy style , made just to annoy the audience
Eli's fedora is so engrained in his thought process that it's woven into his words. He doesn't need to wear it, you only need to listen and you will see it all the same.
I am so grateful for Eliezer Yudkowsky.
Thank you, Eliezer Yudkowsky, for opening my eyes and eyes of many many others for AI and the dangers of AI.❤👑 I have watched all podcasts you have been in these last years, and more than once, bc i really wanted to see what you see, and now I do, and it has been a fascinating ride. 👍🏻Happy to see you on a new discussion on UA-cam, looking forward to listen!
He’s a clown.
Thank you, Tim for orchestrating this discussion, providing references on screen as well as in the notes, as well as that fantastic transcript!
I like that Stephen approaches this not as a debate, but a thorough exploration and almost an exposition of how he navigates a difficult and esoteric topic that he doesn't quite agree with. Ultimately his position is the same of so many - that the _potential_ cost of avoiding or mitigating the risk is too great, even though the risk is Doom (ie myriad "bad things"), and nobody has figured out in any detail what that avoidance or mitigation may actually entail. It may be very likely that methods of avoidance or mitigation are actually very achievable and do not necessarily greatly impact the value or timeframe of attaining ASI.
To the final part of the talk around the 4hr mark, we are very clearly only racing toward Advanced Machine Intelligence (Yann's preferred terminology) without properly investigating safety measures. It may be that in racing toward a cliff, we could if we were prepared, jump across or drive around it. Instead, we might simply hit potholes and careen out of control or discover that the cliff is actually quite small, but still enough that without being prepared, we (the human race) crash and suffer/die.
The argument that _any_ attempt at safety equates to turning our lives upside down is, hyperbolic.
A lack of intuition should lead people to very carefully investigate areas of potential high risk before engaging in them. I have little intuition about sky diving, it should seem unwise in the extreme, to buy a parachute, hire a plane and leap out without any instruction. Maybe I judge that it's worth the risk... but I'll never know if I don't actually analyse the risks. The people I land on would likely have a different evaluation of the risks that I chose to take.
At 3:30:00 may I suggest that of the things that an AI in this context may be aiming toward or using as a step, is the equivalent of Move 37.
At 3:35:00 I might suggest blue-green algae as a biological system that when it encounters adequate sustenance and no counters, destroys all other life as a consequence of simply following its normal operations; It's only objective is to consume and reproduce. Even if it had some kind of self-awareness and even ethics, there's no guarantee that those ethics would be in a form that we might relate to, they might be quite challenging to comprehend.
What a priviledge and pleasure to have the company of two intellectual giants and luminaries of the AI world... thank you Machine Learning Street Talk!
I'm an hour and 14 minutes into this an extremely frustrated because it seems like neither of them understand what's actually going on. They're very intelligently missing the trap they've fallen into.
In particular. Over and over again in all sorts of creative ways, Eliezer is saying we ought to be afraid of ais because there is a high probability that they will destroy everything we value. Add Wolfram is in a very intelligent creative way saying, I don't see how you can say we ought to be doing that prove that we ought to be doing that from what is.
Essentially, it is just the is ought problem. And Wolfram keeps on I'm trying to force Eliezer to justify his oughts from first principles of what is.
There is a simple out to this. Eliezer needs to ask Wolfram, what would he hate to see destroyed, what would he hate to see created, of those things that he would hate to see destroyed and hate to see created what are those would most humans hate to see destroyed and or hate to see created that most other humans would agree with.
Okay we now have a list of things that we're pretty sure we want and don't want. Here is why I think AI will destroy what we want and may create what we don't want. Since most humans are concerned about this, we should be concerned about bringing super intelligent AI into existence. This is why I think super intelligent AI will not preserve our values.
Wolfram without realizing it it seems like it's making the nihilist moral argument and asking Eliezer to refute it from logic and math. Logic and math don't say anything about moral values.
This is a policy argument about how we as humans want our world to be in the not-so-distant future. Wolfram's argument is essentially like saying during the middle of the Holocaust, objectively prove to me that the Holocaust is a bad thing from first logical and mathematical principles, otherwise I can't say whether or not it's bad and therefore we shouldn't do anything about it.
Okay…but the problem isn’t moral dilemma, it’s a technical one.
We do not know how to tell a machine what is right or wrong whether it is sapient or not, because we don’t even know how to construct ideas of right and wrong… Think about the trolly problem.
A machine “does what it does” and to the best of our ability we control it. Thinking we can control it is one of wolframs arguments: we can’t. It’s like trying to control a horse. We control them in a way that’s mutually beneficial but not in anyway where we can understand what the horse really wants to do or what it values. It just so happens that they let us ride their back in exchange for hay and some pets.
Yud makes a ridiculous statement in the beginning: “let’s kill all the mosquitoes, I’m sure nobody would care.” Which is absolute rubbish because spiders and birds and other creatures eat mosquitoes and this would throw off the equilibrium the planet has set up. In the same token who’s to say that AI’s wouldn’t make the same argument given that humans kill thousands of species by the day.
His arguments self defeat themselves because of severe lack of understanding the technical problems not the moral ones which are easy to talk about when we as humans can communicate with each other…which does not apply to other creatures with vastly different worldviews and as wolfram states, likely different laws of physics.
One needs to really dig deep into the definition of terms…terms we need to ascribe to systems we don’t understand.
Yud is a clown 🤡. Wolfram is a unique computer scientist, physicist.
You really shouldn't have to justify why it'd he bad if every human being died.
@@VolodymyrPankov I was quite disappointed by wolfram's performance here. He seems to be lost in rather sophomoric philosophical traps and just wanted to pointlessly muse about abstractions. I won't deny he's done some brilliant things, but he doesn't seem to be a very clear thinker on these broad issues like this. I was excited for this and genuinely expecting better, maybe he's just getting old.
@@michalchik Man, it’s very subjective because we’re all so different. Even though our consciousness (an emergent property of the brain) operates in similar ways at the neurophysiological level, we still have vastly different perspectives on everything-including the meanings of words. This is actually why Wolfram emphasizes the importance of defining terms in discussions, even if people here are laughing at him for it. But defining terms really is crucial to these conversations.
I’m not a blind fan of Wolfram. I’m quite skeptical about his larger ideas, like the notion that everything is computation and so on.
However, in discussions, he seems like someone genuinely trying to reason through and question important concepts.
And let’s not forget, despite everything, Wolfram is a real computer scientist, mathematician, and physicist. He talks about the limitations of mathematics based on fundamental theories, like Gödel’s incompleteness theorems, which reveal the limits of mathematics-and, by extension, computation.
Yudkowsky, on the other hand, seems delusional. He’s more of a philosopher, and like many philosophers, his perspective seems removed from scientific reality.
Thank you! This took me 24 hours to watch. On and off. But great content, the best I've seen so far on the topic.
What a, "you know", waste of time. It's almost as if Wolfram is trying to obfuscate and divert the conversation. I can see why Yudkowsky doesn't do many interviews these days.
He is too busy gooning to be doing interviews
I don't know why would anybody want to do interviews with Yudkowsky. He's boring and can't even formulate his points in any coherent way.
Never watched a 4 hour interview twice in my life. But here I am. Back.
nothing against wolfram but his shtick has always been:
-> notice that at the bottom of any framework is a human feeling
-> on this basis, pick any word used by other person (usually 'truth' / 'you' / 'intelligence') that's defined close to the bedrock of A Human Feeling
-> without fail, the other guy will either spend exuberant amounts of effort politely edifying that this point is inane if the gcd of their lexicons is > 1 and subject matter isn't dependent on the complete classification of 'Truth' or 'You', but most just concede and let Wolfram monologue
-> goto step 2, make some interesting but generally overplayed observations that often implicitly terminate the line of questioning and not really explore new areas.
i hate being mean, he's a cool guy, but i do think things would be better for him if he was just sincerely open to completely new things
What a beautiful conversation
Incredible work. One of my favorite channels by far.
@@XShollaj please share the others.
@@user-fd7jd4jq1e AI Explained is pretty good too for news.
Extremely interesting interactions. Well done MLST! 🔥🔥🔥
I’m convinced, after 3 1/2 hours of duration listening, that Wolfram actually doesn’t understand precisely what Yudkowsky is saying half the time. And when Eliezer corrects him, he doesn’t even know he’s been corrected
Huge props to Eliezer's ability to deftly navigate the rabit holes, and ditto to Wolfram's careful use of rabit holes to probe the arguments in service of his genuine curiosity. I knew these two were smart but im even more impressed after this.
The Yudkowsky fanatics, of course, are coming out of the woodwork, and are now saying that Wolfram, who has shown way too much consideration and patience for Yudkowsky, just doesn’t get it. He gets it, but he is way too polite to tell Yudkowsky that he is acting like an immature child.
Thanks for bringing those two minds together and for your amazing questions and interviews overall. It would be amazing that you interviewed Jonathan Gorard. He is the mind behind Wolfram's project and among many other things, he has worked in observer-environment interaction developing concepts at the intersection between category theory and computation. I think work like this may end up being foundational for artificial intelligence.
He didn't respond to our invitation, sorry
@@MachineLearningStreetTalk Thank you for letting me know! Hopefully there will be an opportunity in the future.
I'm convinced that general reasoning (philosophical) and science are two different rails cause it's impossible for a genius like Wolfram being so much naïve on everything outside his domain
Or all of you who in the side of a clown Yud totally untouched from scientific reality.
Yep. You can do genius work in a narrow analytic domain and still be an utter fool on deeper questions.
While not much of an "opponent" for Yudkowsky's arguments, I appreciate Wolfram's genuine curiosity.
This is a 4hour intro into the real discussion that doesn’t exist
Wolfram is like Chidi from The good place , being smart and debating and never getting anywhere or solving anything just for the purpose of being smart and debating .
One mistake Eliezer makes is to assume that most goals take the form "maximize x" whilst there are many more goals of the form "do n where n < x". So most goals that an AGI could have if selected at random don't in fact kill us but in the context where the AGI is blindly optimizing for a single unbounded cost thing then we are at risk.
Satisficers converge to becoming optimizers as well. There is a lot of work on this in the AI safety literature, and it keeps pointing back to the basic unsolved nature of the problem.
optimization is how neural networks learning works, it is baked in the design
Interesting crossover of ideas that i usually dont see together - very very interesting guys, this is why you are the best ML channel on UA-cam ❤️🔥
Emotions are often overvalued, regarded as almost magical and irreproducible phenomena that only humans are capable of experiencing. But they are basically positive or negative chemical reinforcements that we can rationalize.
What is it about a given chemical structure that necessarily gives it positive valence?
One thing that adds to the beauty of this talk is that if you told me this level of depth and conversation, along with the voice and video was generated by AI, I would believe it.
Not sure if that's a compliment or not
Good episode. This is going to be fun.
it felt like it just got to the beginning by the end...
At the start, I thought Elizer's argument made a lot of sense, and that Stephen was being needlessly skeptical. But, as the conversation went on, I came round to Stephens point of view, about the horizon of possibilities and purposes, and how they may be benign and non-lethal. I think Eliezer was missing a crucial step in explaining why many, or most , random possible objectives that AI target would be at odds with, or indifferent to, human survival. Or, using his own analogy, how he knows that the ships full of europeans are so close to the shore already, such that it's becoming very difficult to avoid the risk.
Just finished listening to this recording. Truly insightful commentary from both parties!
you have to let me know how do you listen a 4h podcast in 40 minutes 😂
@@OviDB maybe it was released earlier on spotify or another podcasting platform where they did not have to spend as much time editing for the visual references?
@@ciriusp it wasn’t
3:34:55 - the man who insisted on defining words for 120 minutes, starts talking about his intuition. Glorious.
I think it got a little frustrating towards the end. Stephen kept on saying "I think this is what you are saying" or "here is what you may want to say", but kept on missing what Eli was saying. Again and again. In a nutshell, I think partly what Eli was saying is that it could play out like the game of telephone. Humans may give AI the outer goal (which has implicit goal of not killing humans), in the first approximation AI might take it, interpret it close enough to the outer goal, and now that it is efficient, could start coming up with the most efficient strategies to achieve that outer goal. But it may start iterating on its own with a slight, novel variation and over several iteration the goal may disperse. And unless its internal deliberations so to speak are not constantly course correct and checked to make sure any danger to humans is not introduced, its derived or interpreted goal may introduce aspects of danger to humans. One of the key things about this new technology is that the new AI is not smarter than one human, but it is smarter than all humans in all areas of human endeavor that it was trained on.
Secondly I was surprised to find Stephen using active verbs for thing we observer to have happened due to evolution. The most important thing is that evolution is not a active agentic process. It is the observed effect of natural selection due to the contemporary environmental conditions on population of organism. That is all. We should never use active verbs like "evolution did this", "evolution designed that" etc. It is the statistical effect of environments on populations. The better suited members of population survive and thus pass on their genetics heritage. And in fact if the environment changes suddenly an advantageous genetic trait could become a liability due to change in environment. For example, a gene for more and white fir on a Bear in cold ice (snow is white) age may become a liability if there is warming event after the ice age. The furry bear may die by heat and white bear may not camouflage well and the population of white furry bears will dwindle. It is not like evolution came up with a plan and said "ah....now let us reduce white furry bear".
These are both scientifically-minded people talking to each other, they both understand the concept of evolution, preferring to use inconvenient wording after a few hours of conversation to convey what is well-understood.
If this was a high-school class, I'd totally agree! It's very important to make sure that you aren't conveying something that might be incorrectly understood by the listener
@TankorSmash Fair enough.
I think optimists and safety people, actually have a lot more in common than is at first immediately apparent, it's just crucial to recognise the difference between genuine safety, and "safety". At the end of the day optimists and safety people seem to care about the same things.
A survey showed that on average, AI alignment researchers are more optimistic by nature than the general public, with leaders of AI safety orgs being extremely optimistic people.
These are the people who say we are not on track to solve the alignment problem in time.
The optimists _are_ the safety people. The people who stick their heads in the sand and pretend we aren't doomed if we create something more powerful than us that we cannot control tend to be pessimists who actually don't think that AI can surpass humanity. Then there's Stephen, who seemingly doesn't care.
“Computational irreducibility” is my new least favorite word
The best conversation in MLST history? Good god...
The first (and only) time intelligence evolved, it led to a global mass extinction event. No reason to think the next one will be benign.
Very deep resoning there
Wolfram just don't get it
I like how yt comments are very respectful while twitter comments about this conversation are like: "Oh my god Wolfram stfu and let elizer talk" and "Eliezer got cooked in this one"
Yeah I saw that too and honestly don’t understand how they derived a competition from this when it’s very clearly an explorative discussion rather than a debate
Unfortunately that’s standard “engagement” tactic. Most people won’t watch the real interview. They’ll just read some tweet that confirms their bias and go back into the cave.
who said eliezer got cooked in this one omg
@@cokoala5137 I think the main diferencie IS people on the UA-cam comment section actually watched the video , and a lot of the most vocal people on Twitter just didn't .
There is even a comment like that that got hundreds of likes from someone who basically watched only the first 10 min of video , and had lots of responses from people who probably didn't even see the video showing they really wanted for yudkowsky to "lose" in some way cause they dislike him .
Both are horrible. Twitter has the engagement bro problem, while UA-cam has an algorithm that deboosts controversy/disagreements and pushes the dominant (per video) narrative as the ONLY narrative (you want to see the real comments on youtube? sort by newest!). UA-cam comment section is dystopia masquerading as utopia and it terrifies me how easily people fell for it.
Yudkowsky is used to defending his thesis that a future iteration of AI will kill us all, and has a full pack of analogies make his point. For that reason he misses Wolfram's more interesting question of WHY is the doom outcome so certain? Wolfram already understands and accepts that human annihilation is a possibility, and even points out that in the universe, that's the natural state. However, he wants Yudkowsky to explain why he thinks absolute annihilation is the only possibility? Yudkowsky's weak analogies on the European colonization of the Americas don't seem to cut it. Wolfram should be given the Nobel Prize for Patience in this endeavor. The most fruitful section of the discussion is at 3'00 to 3'40 (approx). Interesting discussion. Would love to see Tim and Keith do a detailed review of it.
They actually hit on the exact crux of disagreement at one point. Eliezer said that if you take the space of all goals (I can't remember if this is the exact word he used) and then use a standard math technique to put a probability measure on it, then the subspace of goals that leave humans alive has measure (probability) 0.
Then Wolfram said something like, I've tried to formally do this and you run into some problems and the probability measure requires some choices that can change the probability, but roughly agreed that this would be sufficient to convince him if it could be done.
And then they got sidetracked rather than actually resolving or even talking about that more.
Easily the most interesting discussion ive seen in a long time - i love how open Wolfram was to understanding Eliezer, and its really nice to see how co-operative they were - perhaps theres hope that super intelligence will be as open to such discussions:)
I'm waiting for someone to prove Eleizer wrong, it didn't happen here.
Was waiting impatiently for this one, finally!
As someone who has long held the position that AI will almost certainly destroy human civilisation, and likely wipe out humanity, I am so grateful for Eliezer Yudkowsky. I hope he is wrong, but fear he is not. Thanks for sharing this conversation!
He is a 🤡
His untouched from scientific reality. Like any philosopher
Arghhhh… 3 hours of 2+2=4…. But what is the nature of 2? And I have to finish the four hour in case Eliezer comes up with some clever I have not heard before😅
How did you get these two together !? Amazing.
Wolfram is an intellectual titan, and pretty grounded in reality
but not grounded in the topic of the discussion.
@@JD-jl4yy not true at all.
@@JD-jl4yy He 100% was. I can talk to religious people and note when they wish to claim "that's not relevant" whenever I point to their inconsistencies. ;)
"Grounded in reality" is not a descriptor I would apply to someone who spends so much time on their pet theory of everything. Much less someone who spends any amount of time arguing that maybe my whole family should die for his asinine philosophical reasons.
@@41-Haiku He has more self awareness than Yudkowsky over the same claim, he has a pet theory, claiming "everyone will die". You fell for the baseless emotional appeal to fear and panic.
Two civil titans discuss the topic of our lifetime. Brilliant. Thank you!
you know it’s 8pm in london? i was supposed to go to bed early today 🙄