Code vs Data (Metaprogramming) - Computerphile
Вставка
- Опубліковано 25 сер 2024
- Why are code and data so separate? Robert Smith of Rigetti Quantum Computing explains how he uses Lisp code to generate Lisp data which he can use as Lisp code...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottsco...
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
Linear algebra has this same kind of duality between data and function. A matrix is just a collection of data in most cases but when you multiply a matrix by some other matrix or a vector the matrix acts as a transform or a "function/code" on the data that passes through it. A change of basis is a transform matrix that acts as a function while at the same time could be further transformed by rotations and scaling matrices and act again as just data.
Think of meta programming as inventing something new like multiplication or even pluss in math prior to those things being invented.
Or any other new concept.
For instance, there was a time when no one had invented zero.
Math at the time had to be done with numbers with a value since there was no zero.
In a programming language with metaprogramming you can invent such a new concept as zero, define it and use it instead of defining it every single time you need to use the concept.
Well, almost. A matrix is a *representation* of an operator relative to the given basis.
Love this comment
Please learn about geometry algebra
@@OverThinkKS
Linear algebra taught me to like math
Dude, please start a programmer's channel with a LOT more code. These videos are a tease.
Agreed. Show me some code.
If you want more code, check out "The Coding Train" channel ;-)
Its... Javascript... but there you go...
aah, that channel is so fun
Yes Daniel Shiffman is very cool. But hate JS are there any other channels which are not JS.
Yes and JS is very easy to understand. The problem arises when JS is used interchangeably with node and starts doing weird things.
@2:52: *Reversed reverse Polish Notation* or as sane people would call it... polish notation ;-)
@science no it's at 2:50
@Sourav Goswami - Thanks. Fixed it.
Or as people who prefer descriptive rather than arbitrary names for things, "prefix notation".
"So could you give us an example?"
"Oh sure, definitely. So lets say you wanted to draw an owl. First you just draw some circles. Right? Then you draw the rest of the owl, which I wont do here because its obvious."
as i understand it:
first, a function is a set of instructions.
second, we can create data during runtime.
third, we can interpret data as instructions.
fourth, we can run interpreted data as a function.
you created code at run time.
the program wrote parts of itself.
Andrew McAuliffe you can base code on input from the user if you desire so
The way he described it, metaprogramming is the same as creating a library, which doesn't sound right.
ElagabalusRex you can create libraries of syntactic abstractions. That’s part of the point. :)
Kris2510 great, written out example!
@5:20 - I've been there, too. Writing the loop in assembly and forgetting to INC the counter. Those days the only cure was rebooting your pc.
I love listening to this guy talk, I don't know why
This duality between data and code is much more common than most people think. One very simple example: most of us view a .c file as code. But to the compiler, it's data.
Exactly. The very existence of compilers and interpreters relies on that duality.
But can a C program evaluate, change, compile, and assemble it's own data at runtime?
I don't know, all this meta & functional programming seem so powerful because you can allegedly write compact code.
The problem I see is the same as with Perl: Can a third party actually decipher that code, or do we have to add 10 lines of comments for every line of code?
Well, you don't really have to decipher it. At least not any average Joe has to. This technique lets you develop your own DSL (domain specific language) which will be really intuitive to its users.
You can leave the writing of the actual interpreter to people that have the necessary skills to understand it.
@Arkhitekton Idk. So if I let users extend my program by letting them write plugins in a DSL is this already seen as meta-programming? What's the boundary - compilation vs. interpretation? Or has it more to do with Harvard vs. von Neumann architecture?
Can you give me a practical example?
Ever used CSS or JavaScript? You had the opportunity to use a DSL there. When you write "nav > ul li.top:hover" you are abstracted away from all of the details of how the interpretation of that line works. It is interpreted for you by some interpreter in the browser, and yet you know exactly what that line does.
This kind of support is quite hard to do in languages that do not support metaprogramming. It is not impossible in any way, but it does require you to spend more time on building up the scaffolding to make it work.
@@LordShigi yesss when I see ppl say "you can't do this in c/c++" I'm always like uhhhh yeah u can XD
Really nice broad pass over metaprogramming 👍
I want more from Robert Smith. This guy is good.
I remember back in the early 1980s, when programmable calculators starting blurring the lines between computers and calculators, I read a quote (which I'm sorry I don't recall the author of) which said that the difference between calculators and computers is that the latter can operate on their own programs as if they were data and vice-versa, whereas the former cannot.
I'd love to see someone debug a metaprogram that contains thousands of lines of business logic :)
During my job in a company i find a procedure that was modified costantly in 4 years ( a T-SQL store procedure) that has around 3000 rows with some part of code that write code that was executed...
It takes about 1 week to under stand it...and about 1 time over 2 month it need to be modified...The source of this madness is called SAP
A lot better than the Java mess.
"reverse RPN" is not Reverse Reverse PN, it is just PN. Polish notation is like + 1 2 or add 1 and 2, RPN is 1 2 +. Polish notation was specifically prefix notation, which was considered more natural than 1+2 because you can write +1 * 2 3 to mean 1+(2*3), and indeed, this is a common notation used in compiler intermediate codes. However, RPN is considered even more natural than this because 2 3* 1+ exactly expresses operands and the operators that use them in proper order. Thus get 2, get 3, multiply, then get 1, and then add to the result of 2*3. It also is stack machine friendly.
If there never were such a thing as polish notation, I'd agree that reverse RPN makes sense, but polish notation was a thing before RPN.
Oh, dear. The comment-formatting system has gotten tripped up by your example and parsed the asterisks for multiplication as instructions to make the text bold.
Thanks! Fixed it.
I really wish they'd just use a subset of HTML or pseudo-HTML for the formatting. These errors would be so much less likely.
Am I correct to say that Lisp as he describes it is better off for math problems while C is better off for dealing with memory... cause that makes scenes why you would rather use Lisp over C for something like Quantum problems as stated in last video
All abstract problems, while C is better for memory as you said
Fun fact: ".. prefix notation is similar to postfix notation such as RPN." Beginning with the HP 48 calculator instead of using RPN the manual refers to RPL which is not defined. A search on the web gives the unofficial acronym of "Reverse Polish LISP" which it is. If i enter "{1 2 3 * +}" and press ENTER that program is placed on the stack. Pressing EVAL then evaluates the expression leaving "7" on the stack. Notice how the parens dissapear in RPN/RPL. There is no PROGRAM/RUN mode switch on this machine; everything is eval mode and is placed on the stack!
To me that for-loop example seems very similar to defining a function that takes other functions as its arguments, though perhaps with a nicer syntax. Would love some more in-depth examples.
the for loop takes a variable name, the starting value, the ending value, the increment and the body of code to be executed with each iteration. Can you make a function that you give as input a body of code? Can you bend the syntax of the language you use in any way you see fit?
Is the standard C++ loop...
The right example in C++ for metaprogramming are Virtual classes / methods
Forth is another wonderful language that still stands the test of time. This vid mentions Postscript, which is very much a variant of Forth.
Duality: Tom Scott or John Green?
RE now I can’t unsee it 😂
Trick question, Griffin McElroy
Arguably better than both
Tom Green.
Or is is John Scott?
Would be interesting to see a clarification of how this topic compares to compilers in general -- because, of course, compilers are programs that write programs. I was waiting for the comparison and distinction to be made in this video, but it never happened.
I'm getting all warm and fuzzy when I see the _LISP_ syntax. Brings back good memories from when I used it with AutoCAD.
I will remain indebted for sparking my interest in Lisp.
I've yet to hear a convincing argument for LISP including this vid.
I did not really understand this, he seems to be skipping some important details in his explanation?
I mean, it's obvious what he's saying. I don't understand what you're finding difficult about it.
4:23 That’s what a compiler for a high-level language does. The original name for this concept, from the 1950s, was “automatic programming”. At that time, the high-level language source was not considered the “program”: that name was reserved for the actual machine code that the computer could interpret.
How different is that from ANY other interpreter? Or writing instructions directly to memory updating code on fly?
It's not. Most languages have some kind of 'eval' feature.
yeah i also don't understand what is so special about it.
It is a more abstracted way to modify code. You are modifying the code at an Abstract Syntax Tree level. (And with abstraction can come more power.)
TakanashiYuuji 0ⁿ
The special part is, that the program doesn't have to be recompiled by the programmer to introduce new features, formally this is called "at runtime". Simply speaking your program is conjuring itself while it's running, and doesn't come with all the necessary code (as defnined in the video) but instead it has the data necessary to create the code.
A popular counterexample would be C, which requires you to compile it for the given architecture before executing it. C cannot add (for example) a function that was passed from the keyboard as C-code, despite being valid C code, and it cannot be executed at runtime.
If you're looking for an "interpreted" language, you might find Java to be a suitable example (I know it's technically compiled, but it's also interpreted at the same time, so there's that). Although Java may modify the JVM runtime however it likes, changing the functionality of a method/function is prohibited, which severely limits its at-runtime capabilities (in practice it's possible but produces very patternized code and complex structures)
More Robert! :D
This is a rather confusing example of the usage of Metaprogramming
Unusual Activities you didn’t have for loops before, now you do. Is it confusing because everybody already has them? Maybe a better example would be: Imagine you wanted to specify logical constraints a la prolog. You can use metaprogramming to add such a facility to your program. But that means you’d have to explain what prolog is...
that's what you get when you assign that "confusing" guy to present this video..
I mean, I get it, actually I would to turn data into code for a project I'm currently working on but the language im working in (C++) won't let me :p
you could implement a scripting language then you could execute instruction that were created
Isn't RRPN just Polish Notation?
I want to look at the video teased at the end, "fighting obsolescence", but can't find it anywhere. Anyone got a link?
2:53 normally called Forward Polish Notation
Great video! I enjoyed the explanations and written examples. Thanks Mr. Robert Smith. 😊
How is the for loop example (and how it is an example of metaprogramming if a for loop didnt originally exist in the syntax) different from inline functions or #define
Could have introduced defmacro since that is often what is used at pre-compile time to turn the new syntax into something that the compiler will understand.
Where's the rest?
5:37 Assuming the increment of i lies at the bottom of that squiggle somewhere ... otherwise you’ve just written an endless loop.
i personally, since the moment I learned about variables that can store functions, see this distinction of "code vs. data" as wholly arbitrary.
code is data describing how to transform data.
it's all just data, the difference is in how you interpret it.
Every abstraction in programming has meaning in the correct context... otherwise we're just talking about 1's and 0's
Interesting to learn a bit about lisp
It's strange as a comparability theorist, I consider data and code to be the same thing. Though we explicitly use numbers for both.
wouldnt reversed reverse polish notation just be polish notation?
Yes.
It doesn't seem like Robert Smith understands what he is saying. There wasn't much of an explanation here.
Hmmm... can i think about this similar to Lambda-Expression in C# or Java?
Which, like... IS program code at compile time and BECOMES data at runtime?
Or, is that too far fetched? ^^
A lambda expression is a functional concept so yes, that's somewhat similar.
The comparison breaks down when you try to pick apart the lambda-expression - which you can't do in C# or Java. You can create a lambda, but you can't pull it apart and pick bits and pieces out of it.
With metaprogramming, you would be able to compose functions together, and then later on, pick them apart and, say, recompose the functions in a different way based on some conditions for optimization.
C# and Java have no way to do this.
Currying using lambdas is an example of how to achieve some of what the video talks about.
You can then get data that becomes code at runtime too...
What a joy it must be to debug a Lisp program.
You bet.
Just don't write any bugs. :p
Is it possible to get similar meta programming capabilities in languages other than lisp? IE are there other programming languages that can do similar meta programming?
Rust lang is another language with a somewhat powerful macro system. But it seems Homoiconicity is one of the qualities that make meta-programming easier to implement.
Nim, im not sure about "rust" but people have built whole DSL's with it in nim.
Are macros the same thing as Metaprogramming?
Haritha Wickremasinghe They are a form of meta-programming.
So... functional programming then?
Not quite.
It's a programming paradigm.
You *can* do this with a functional programming language (well some of them).
But this and functional programming isn't the same exact thing.
I had a guy demo lisp for me yesterday for the first time, now it is everywhere.
Von Neumann understood this perfectly well. The only ones who have forgotten tend to use languages like Java or C#. Oh, and PHP.
Don't be a hater. Use the best tool to solve problem at hand.
@@GreyDeathVaccine couldn't have said better
How is that different from other languages and interpreters?
I'm pretty sure you could do the same thing in C++ or Java or Haskell without any "duality".
Also I thought computer scientists had largely abandoned meta programming.
To iterate is imperative and to recuse is functional?
i feel bad for the computer scientists who worked on programs writing programs like this. because their way was the much more elegant, much more beautiful and pragmatic approach to programs that write programs. but instead we just went with this huge mess we call GPT-3 and language models. now nobody cares about this stuff even though it’s some of the coolest programming science out there.
Could you do a video on how to group logic gates to acomplish complex actions, such as multiplication, division, and converting binary to different two exponent bases
I think they already did, using domino bricks.
I would recommend you check the video game "Turing Complete" on steam, it goes there and beyond, allowing you to learn how to implement a full 8-bit Harvard Style computer.
I would just like to suggest to anyone that is interested in this, check out terra lang :) It's basically a scriptable compiler and deserves more love
At the end credit there's a snippet on the 'current state of quantum computing'. Where's that vid?
I can't find it.
*_...but tight code is used repeatedly-on scads of data; (Amer. scads, Brit. jillions)..._*
*_...on the other hand, the similarity shows in loop parameters, i++, list descriptors,..._*
*_...((in the control/addressing variables-but not, in the data itself/results per se))..._*
Hmmm still not getting it, I hope there are more of these.
It's just a method of abstracting useful functions as an evaluable data
Very well explained.
Is this the latest "Programmers are also human" video?
>Is there a simple example of that?
he literally gave an example
I'm not sure where he is going with this. So, you're not happy with the tools a language give you such as for loops. So, do you want to be able to make your code flow in whatever fashion you want? Use labels on C. Yes, it will be considered spaghetti code, because it is.
Also a c source file IS data, although stored less efficiently. So is Java code, and so is Java bytecode and assembly. A smart enough program could write programs in any of these languages. The difference is that some are readable, and some aren't. Judge for yourself which are.
It's not about not being happy about for loopes.
It's about being able to *implement* for loops on the fly in a language where it doesn't exist already.
Or any other feature for that matter.
If you can come up with a new consept you can implement it and use user input as a part of the implementation if desired.
You can change the code itself based on user input.
And sure, you can write a program that can write code for its own code.
However what about writing a program that can write the code used to run itself during the runtime?
Regarding the spagetti code argument.
Yes, partially.
But it doesn't have to be.
Not impressed at all with the last two videos on this. There's just no substantial concrete content, it's all "Lisp can do this" but doesn't show how
Unfortunately, the concept of metaprogramming is just like recursion/induction when trying to explain it to someone that hasn't tried it: you can't. "To understand recursion you must first understand recursion" is actually true in my experience. That means no matter how I try to explain it to someone that's never written a recursive function (or inductive proof), they won't get it, until suddenly their mental process steps back, sees the loop, and 'gets it'. The same thing for metaprogramming. You can see in the video that the speaker is having difficulty expressing the idea of writing code that writes code, because there's a similar (if not identical) nugget of self-reference at the center of the idea.
I think this channel is trying to cater to people that are not familiar with programming concepts, and that really ties the hands of the presenters, because they can't use the vocabulary necessary to express this idea, because it's all abstract nonsense until you have an actual problem to solve.
It’s very difficult to pack all of that in sub-10 minute videos with paper and pencil. Maybe Computerphile should use a computer in more videos? :)
5:37 this is what you had to write to iterate ten times, yes but in assembly you could:
times 10 instr
instr (some instruction you want to do 10
times)
I'm assuming that in Lispland eval isn't evil?
You don't use eval directly in Lispland, you define macros instead.
Love this man!
UA-cam just showed me a 24-minute ad--that’s over *three times* longer than the video I wanted to watch.
Never mind--I just let it play with the tab muted, while I did something else. That way you still get the ad revenue, right?
I really like this guy
nim-lang.org can do that and its procedural.
capoiosct If that is the case, then nim wouldn't just be procedural, it would be multi-paradigm, because then you could just write macros to add OOP, functional or any other paradigm functionality to the language. :-)
"write a programm that writes programms", many of us do that already. for example a few days ago i wrote a simple python script to dynamically to my needs created LaTeX source code. also,
automatically generating config files (config files may be considered as code sometimes) follows this scheme. more interesting: if the program and the generated program both are in the same language.
is all of this related to code? like Metaprogramming and/or Software defined Networking? Where can i learn these things? Are they being taught in Computer Science?
so how can we do this in ruby?
metaprogramming in ruby was a selling point but the only thing we hear today is rails. the language bent to the framework
how can we do metaprogramming in ruby?
Quite easy actually, due to the fact that everything is an object, you can construct many different things (classes, methods for specific objects, ...) with ease. Ruby's syntax also allows to create quite clean DSL's!
can you show me an example (external url)? im not a ruby dev so id like to see that working
people use metaprogramming in rails too?
Well, rails is already a full blown framework, I'm not sure if you ever really need to do metaprogramming there. However, a simple example would be the attr_accessor function, which creates getters and setters for a field of a class.
class Person
attr_accessor :name
end
Is a functions which basically gets converted to
class Person
def name
@name
end
def name=(value)
@name = value
end
end
This isn't exactly like the metaprogramming shown in the video, but you're still creating new "keywords" to make life easier. You can't do the same thing in Java for example (at least not when I used to write Java)
I think one of the best examples for metaprogramming are how ORMs (Object-Relational Mappers) create the Ruby classes for database tables. In Ruby you do not have to define methods to get and set the value for every single column. Most of the time, all you need to write down is your class name and point the ORM at the right table. The ORM will then for example use MySQL queries to get the column definitions of the table and define the methods for you, at run time. This defining of code on the fly and adding it to classes and objects is the metaprogramming. All this happens in memory. In contrast, Java for example would have the ORM literally write source code files for you, where it defines all those methods before compiling. You should then never touch those, because the next time you generate them, your changes are lost.
With Ruby then, you have the advantage that you don't have to change anything in case of a change to a part of a table, that your code does not touch.
Some keywords you can look up in ruby: "include", "extend" (both when called in a class definition and when called on an instance object), "send", "method_missing", "define_method" and of course the 'dreaded' "eval".
I want to see a non-contrived example
The trouble with metaprogramming is that all it does is give the user the same power as a programming language designer. What that means is if your programming language doesn’t have something, you can add it. The problem for explaining metaprogramming is the non-contrived examples have all either been implemented in other languages by language designers (so they seem silly nowadays), are really trivial (but nice) conveniences, or take a lot more time than they had here (like adding a new programming paradigm as a library).
Elijah Malaby this is a really great explanation!
quadricode being hyper sarcastic :)
Meta programming is just another layer of abstraction. You have assembly code to make processors do what you want, you have languages to make assembly code do what you want. This is a language to make a language to do what people want. A really powerful tool especially in academia and research. Simple examples will make not convey the power of the concept and always feel contrived. Non simple examples are non contrived but hard to explain
@@badsyntax173 The power in 'code as data' is thanks to macros. there's no real equivalent of a macro in languages like Java or Python. While Java's added a 'reflect' library that keeps a table of functions created in namespace, it still doesn't treat them equally 'first class citizens'.
There's no way for example to define '+' (plus) as '-' (minus)...or multiply or any function you want really. There's no constraint or workaround in having predictable code adjusting itself.
He forgot to increment i in the pseudocode
al Khwarismi And to have the initial check of the loop, which usually means an unconditional jump to its end, right after initializing i.
The pseudo code is definitely inaccurate!
hey phile can you explain exactly about symbolic data from the book structure and interpration of computer program
That "assembly" for the for-loop is not actually that correct. The evaluation of the repeat condition is actually made before you enter the loop in a standard for-loop. What he wrote down is more akin to a do-while loop, which runs trough the loop once unconditionally, and then repeats if the condition evaluates to true.
Actually, it's a bit of a hybrid, because he set the iteration-variable at the top, and is still using an iteration variable.
@Computerphile can we have a video about the Julia programming language?
Whats the difference between lisp and a langue where i can run the eval() from javascript?
A very powerful tool which I lost control over on day 2🤣🤣🤣now i kinda see where the elixir quote comes from...
code = evaluated data
reverse reverse polish notation = polish notation
Please try to get the interviewees to explain their subjects more concretely (i.e. with code samples)
data is to code what information is to knowledge
Aren't compilers in general an implementation of metaprogramming?
Eval is a common misspelling of evil
Shout out Julia
Please make a video about how code generation works. E.g. LLVM
Actually, everything is data (even machine code is made of "data").
Isn't it true most programming languages have an eval function where you could pass data (string, var, number, etc) to be evaluated and executed?
Only some scripting languages have this, and compiled languages generally can't. Eval also usually operates on program text, rather than directly on a generated AST.
"For loops exist in every language"...
ever used Haskell? No loops.
Skylar he probably meant "the concept of iteration/repetition"
Tail recursion or a fold/reduction give more or less the same thing as a for-loop.
Skylar, there is the forM function which is like a for loop for sequencing nomadic actions. Eg
forM [1..10] $ \i -> print i
Hmm, since code is written exactly the same as data, I wonder if I could write a program that reads a different program and then tells me if that program would eventually stop?
What do you think about using Marvin Minsky frames as the base for metaprogramming language?
I took his idea, only one element type: Frame object can hold scalar value, associative array (names slots points to other frames) and array holds pointers to other frames _in order_.
In results, I can construct networks of directed interconnected objects, looks very very close to the attribute grammar structure.
The key is some subset of data must have active nature: they must be executable, or more precise evaluatable in some execution context.
I played with this concept a lot, using some sort of FORTH script language to construct and manipulate such a structure type. Results look attractable. To represent say program-like structures I can inherit base Frame class to set of classes close to any generic script language -- functions, loops.
But what stops me going further this way: complexity rises too fast when I try to rewrite system in itself (metacircular implementation). Maybe I should not try to generate Python code too earlier, and I first should describe the system only at a very high level, I don't know.
Code is just data to a lower-level processor.
What is the lowest-level processor? Is it the laws of physics?
So Macros are also a type of MetaProgramming?? right!
Haskell doesn't have for-loops though :P
It uses recursion, it being functional and all.
Is this different than using eval in JavaScript?
Can't you also create assembler code in a C program and then execute it? Isn't that the same?
You can't take the generated code and make it into data again within the language. You'd have to disassemble the generated code, recreate the C from it (which is hard due to optimizations) and somehow make it recompilable.
The other way is for your C program to create a miniature VM to interpret byte code your program creates, but that, again, takes you out of the language.
I think it's meaningless to say that before high level languages For... did not exist. The only way that makes sense is to say that literally it didn't exist. Which is also meaningless really because conceptually it doesn't matter which language you use to implement the idea it is conceptually the same thing.
so its like the eval function in javascript and php which have been around for ages... and are generally considered as "dangerous", especially if you use it with user input.
Anytime you handle data from an unknown source it's dangerous... you have to predict what might come and handle it.
@@malusmundus-9605 and therein lies the hubris... unless you restrict the data, you can't possibly predict everything and handle it - especially with user input. A hacker can use these weak points to execute code without your knowledge and gain access to systems or data.
Beautiful
Isn't this metaprogramming sort of what High-Level programming languages are?
I remember programming in QBasic with subs and things like be for learning java and functions :p
Thanks