@@darkfllame0xC2 is also a return but should be followed by an number of bytes to be poped out of the stack For example: ret 4 is 0xc2, 0x04, 0x00 That line pops 4 bytes from stack after return
Another point to note is the further similarity between code and data in other situations. We often try to over-engineer our libraries / abstractions by taking in complex data types that offer the most flexibility as possible. As our data types being more and more general, they eventually, alongside the logic, become turing complete, and essentially you have written a mini-interpreted programming language. When you abstract too far, you are back to square one, a software developer trying to think of code to solve a problem.
And that is actually a powerful technique! While what you say is true, hopefully in doing that you've also constrained the domain of the program to the domain of the problem. So working in the problem now should be more natural and directly tackle the what needs to be done. Every program we write to solve something is modelling a very specific problem in a specific domain, so a key in writing a good solution is designing a domain specific language for it. In many programming languages, you can then design that DSL from within the host language to have an embedded DSL. One way of doing that in eg haskell, is by modelling the business logic with some dataype that then gets dispatched to a series of interpreters. The bad part is that the compiler sees datatypes as being opaque and won't do a bunch of optimizations that it would do if it were plain code.
Fun fact: I started developing a JIT compiler in C for fun about a week ago, my approach was even crazier but I wish I would've seen this video before I started, it would've saved me some time figuring things out ;)
I hope this video goes viral! It is the most concise and clear explanation I have ever seen on metaprogramming. Maybe you can do a video on zig's comptime next?
Zig comptime makes metaprogramming almost effortless, and is massively more readable and maintainable than code generation or AST analysis. The JSON serialisation / deserialization is an obvious example, but it ends up everywhere and really makes me turn to compile time generation as a casual fix for common issues while writing code - switching on different types for different behaviour, making an array have the length of some comptime derived number, its just nice.
The rare moment when I write comment to a video, but I wanted to note that your video presentation i well prepared, informative and easy to understand, along with nice voice! Thank you! +1 subsrciber :)
I'm not sure ASTs will be the best data structure to shove code into to then expose it for editing, but the power unlocked by allowing meta programs to do things is just to vast to not happen. Jonathon Blow's demos of JAI's abilities to metaprogram heavily modified how I look at programing language design.
@@chainingsolid Editors can have differet ways of representing AST. Like grapths or text etc. users would have no way of knowing the difference. But Version Control, Coding Convensions customization could be greatly improved.
@@rayneiit has but I think industry has shown that lisps are not favoured for large projects in part because of this capability. Would you agree or disagree?
Great job! I hope it sparks interest in such topics in curious minds. Excellent explanation which not hurts brain much and *best* part - practical examples.
@@E_m1ly_6302 Maybe I'm missing your point, but Lisp macros are orders of magnitude beyond C macros. Plus, the Lisp implementations I use provide eval and compiler use at runtime. It's not something I like using, but it's there.
@@E_m1ly_6302 In Lisp code is literally data. C macros are just text replacment, Lisp macros are tightly wound with the compiler/reader and can rewrite functions on the go. It's the strongest metaprogramming there is.
First, Thank you so much for introducing us to this world. Second, Thank you for the very nice explanation, and thank you for all the effort, And lastly, Thanks alot for doing something useful! That's awesome! Appreciate that, Thank you! Loay from Egypt!
D has typical templates, but also strong introspection capabilities, compile-time function execution and metaprogramming at the string level (via "string mixins" which effectively operate like a compile-time version of `eval`). this is kind of neat and means you can do things like making a parser-generator that is generated at compiletime instead of having a separate utility you need to do it. this also allows you to create domain-specific languages like vibe.d's diet templates for web programming. In vibe.d's case, it can also load the template from a file at compile-time and convert it into d code without any difficulty. D generally designs around making string-metaprogramming easy; for example the operator overloads in D all are called with a template parameter that contains the operation; so for example if you overload a+b, you end up with a.opBinary!"+"(b), where "+" is a template parameter (e.g. accessible at compile-time). this means that you can generate code for all binary operations with a single function. string metaprogramming is a little crude compared to how some languages let you access the AST, but it works great in D and is much easier to understand. the only problem with it imo is that sometimes I'd end up using it in situations where it'd be much better just to use a normal template, but it was just easier to make a string mixin (since templates can be finnicky at times)
You seem to always upload right when i’m returning to your videos :D just yesterday i watched some of your old videos after starting a Vulkan renderer in pure C
@@OctagonalSquare it's actually funny that I realized that as soon as I was done making the thumbnail. And it's certainly apt considering the amount of meta layers in the game. But if you'll believe it, it was a complete coincidence :P
@@voxelriftsI believe it. It does fit really well since the game basically makes logic edits to the code (probably in a more controlled way than what using concepts from this video would be) so I thought maybe it was intentional lol. Good video though!
9:35 won't work intuitively because of page execute-disable security features in most OSes. Many JITers work around this by manipulating the instructions then converting the page from RW to RX
Very clean explanations on metaprogramming and jit. Subsribed immediately. Don't pay attention to negative comments thwh don't know what they just watched
Hey brother your videos are amazing and inspiring especially the one in which you mentioned your journey in c.I am really longing to learn to render things on screen using c for a long time,is opengl a good way to start? And could u suggest any resources from which i can satrt learing modern opengl? Thank you
Hi, I want to comment on the Text used in the video. Text has a drop shadow effect that too a hard shadow which looks detached from the text. It gives the feeling that the text is blurry. Please fix this.
1:00 what do you mean it doesn't exist? we all use harvard-neumann hybrid architectures. data is literally on a separate bus and treated differently by the cpu cores
@@thezipcreator You can apply the paradigm to just about any language tho. Unless the language is specifically designed with another paradigm in mind. Zig is general purpose, and in its idiomatic form, it is structured and data-oriented
That's a great question. From my (admittedly limited) testing, it seems as though the stack isn't allocated within a page with the execute permission since the program just segfaults if I try
@@voxelrifts yeah, that's what I tried to say at the end. That in general I bristle at unnecessary heap allocations, but since you need a special allocation anyway, it makes sense to have that as the base for the explanation. Executing the stack sounds like something went very wrong with the callstack itself, the return address being overwritten by some pointer to a stack variable, and it's very reasonable to not allow that I'd think.
Honestly this video would have been so much better if you had included examples of metaprogramming on the AST level in languages that support it on language level like Elixir, Crystal, Rust (Rust technically has token-level metaprogramming) etc. This would make the concept much easier to understand and emphasise that there are languages that support it out of the box. Moreover, you assume that the viewer knows C quite well. I bet there are a lot of people interested in this topic who use other languages and do not understand C as well as you'd like them to. Adding a few trivial explanations for things like function pointers, malloc, free etc. could help them for sure.
@@wicked_sash metadesk is a particular library for metaprogramming. My goal wasn't to explain metadesk itself, but the merits and uses of metaprogramming as a whole :))
There are 2 kinds of VM: [1] guest ISA = host ISA. [2] guest ISA != host ISA. I believe that for the case [2], it is better to call it as Abstract State Machine instead of Virtual Machine.
"programming paradigms" - are political ideologies with 0 backing outside of "this feels better for me". The only program structure in the end is the CPU instructions. And the only one we, as engineers should care about "feeling better" - is the CPU. If something is better for the CPU, and therefore in the end for the user. I remind you that for example, python wastes 75X+ more energy than C. If you are an environmentalist, you really should think about that.
Too idealistic of a viewpoint for the real world. If a company needs a online storefront, you could get one up and running in a couple weeks using django and postgres or whatever, vs spending years doing it from scratch in C counting every CPU cycle for all possible code paths. Think about the time and money spent during development as well. All that time the company could've been making money from their python based app. And its not just about the money, why would you want to redo everything that has been done a thousand times before? Just take the abstractions that others have figured out for you and focus on business logic.
nope, code is for people. it's not that hard to write something working, it's about maintaining it. and decoupling - which is achieved using those paradigms - greatly helps with that you will spend way more energy recompiling and debugging your code with C should you choose it over python for something more demanding than embedded devices
I mean, it's about end results, and I may be wronge some things needs extreme speed, so we don't even use C, but even assembly, Now we can achieve in certain domains great and very fast results, using Python or JS for example Some times we see some operations needs more speed, so we write it in c and add it as extention to the python file. I'm wondering what domain you are in BTW But I get the point you are making anyway. Thank you + BTW, I'm really looking forward to try Cython, pasically a, some sort of compiler to python, So it's like as i've heard, a million times faster that the interpretter. it's like, a c code now?!
And a threaded code is few times more cache-efficient than a simple native code. You can never "just optimise", you always optimise for some problem-specific set of constraints. Anything else is a bad engineering.
Writing code that way will make it too platform specific as different CPUs have different ISAs even within the same CPU brand from the same manufacturer, So indeed there is a place for programming languages thst abstract away the hardware (while allowing to go deeper when needed)
Mixing data and program code is deadly for static analysis. Bugs, that could have been found by the compiler will now crash the program during runtime. Also, a very bad idea from a security point of view. Don't do this. The only use case I see is when the user has no influence on the data/program. For example your build pipeline. So, use metaprogramming for non-productive code.
It is yet another tool/technique. Fully relying on static analyzer also wrong. There is a QA, fuzzing, etc. Personally using analyzers too, but you just need to know what you are doing Also, it's a nice idea to convert during metaprogram from C to GLSL
Dude, you're a really knowledgeable guy and I like your explanations, but please set up your mic to suppress details of your mouth "mimics". Like I can hear you have a mouth full of saliva... and all the sounds related to that, that's not a very pleasant experience, especially while eating.
ret is 0xC3 not 0xC2
@@McSquizzy123 welp thats a certified dumb moment. Thanks for the correction
then what's 0xC2 ?
@@darkfllame0xC2 is also a return but should be followed by an number of bytes to be poped out of the stack
For example: ret 4 is 0xc2, 0x04, 0x00
That line pops 4 bytes from stack after return
Another point to note is the further similarity between code and data in other situations. We often try to over-engineer our libraries / abstractions by taking in complex data types that offer the most flexibility as possible. As our data types being more and more general, they eventually, alongside the logic, become turing complete, and essentially you have written a mini-interpreted programming language. When you abstract too far, you are back to square one, a software developer trying to think of code to solve a problem.
Which is why using power point feels just like coding all the time 😜
And that is actually a powerful technique! While what you say is true, hopefully in doing that you've also constrained the domain of the program to the domain of the problem. So working in the problem now should be more natural and directly tackle the what needs to be done.
Every program we write to solve something is modelling a very specific problem in a specific domain, so a key in writing a good solution is designing a domain specific language for it. In many programming languages, you can then design that DSL from within the host language to have an embedded DSL.
One way of doing that in eg haskell, is by modelling the business logic with some dataype that then gets dispatched to a series of interpreters. The bad part is that the compiler sees datatypes as being opaque and won't do a bunch of optimizations that it would do if it were plain code.
Fun fact: I started developing a JIT compiler in C for fun about a week ago, my approach was even crazier but I wish I would've seen this video before I started, it would've saved me some time figuring things out ;)
I hope this video goes viral! It is the most concise and clear explanation I have ever seen on metaprogramming. Maybe you can do a video on zig's comptime next?
Zig comptime makes metaprogramming almost effortless, and is massively more readable and maintainable than code generation or AST analysis. The JSON serialisation / deserialization is an obvious example, but it ends up everywhere and really makes me turn to compile time generation as a casual fix for common issues while writing code - switching on different types for different behaviour, making an array have the length of some comptime derived number, its just nice.
Wow, converting functions to GLSL shaders is a genius idea!
yeah, i wish i could access the AST of a zig function to convert zig to shader code ngl
The rare moment when I write comment to a video, but I wanted to note that your video presentation i well prepared, informative and easy to understand, along with nice voice!
Thank you! +1 subsrciber :)
Finally people are paying attention to this. The future of programming is direct AST editing.
Lisp programmers: -_-
I'm not sure ASTs will be the best data structure to shove code into to then expose it for editing, but the power unlocked by allowing meta programs to do things is just to vast to not happen. Jonathon Blow's demos of JAI's abilities to metaprogram heavily modified how I look at programing language design.
@@chainingsolid Lisp code is just a massive AST you can edit and it has worked fine.
@@chainingsolid Editors can have differet ways of representing AST. Like grapths or text etc. users would have no way of knowing the difference. But Version Control, Coding Convensions customization could be greatly improved.
@@rayneiit has but I think industry has shown that lisps are not favoured for large projects in part because of this capability. Would you agree or disagree?
Great job! I hope it sparks interest in such topics in curious minds.
Excellent explanation which not hurts brain much and *best* part - practical examples.
Could have been the perfect moment to talk about LuaJIT and DynASM.
Love the video nonetheless!
When you started explaining OS unique malloc() 's in part 4, I got very excited.
I’m confused but I’m gonna watch it again until I understand! 😂
so it's like compiling but with runtime errors
great
The LISP programmers be like: yeah, we do that
Lips has functions as data, but thats different then C macros by a long shot. Ans also not what he is showing in the video.
@@E_m1ly_6302 Maybe I'm missing your point, but Lisp macros are orders of magnitude beyond C macros. Plus, the Lisp implementations I use provide eval and compiler use at runtime. It's not something I like using, but it's there.
@@E_m1ly_6302 In Lisp code is literally data. C macros are just text replacment, Lisp macros are tightly wound with the compiler/reader and can rewrite functions on the go. It's the strongest metaprogramming there is.
Awesome thank you, what an amazing explanation of metaprogramming!
First, Thank you so much for introducing us to this world.
Second, Thank you for the very nice explanation, and thank you for all the effort,
And lastly, Thanks alot for doing something useful!
That's awesome!
Appreciate that, Thank you!
Loay from Egypt!
The part 4 is so cool !! thanks for the video.
Nice video, so basically code is just data in the right place at the right time
D has typical templates, but also strong introspection capabilities, compile-time function execution and metaprogramming at the string level (via "string mixins" which effectively operate like a compile-time version of `eval`). this is kind of neat and means you can do things like making a parser-generator that is generated at compiletime instead of having a separate utility you need to do it. this also allows you to create domain-specific languages like vibe.d's diet templates for web programming. In vibe.d's case, it can also load the template from a file at compile-time and convert it into d code without any difficulty.
D generally designs around making string-metaprogramming easy; for example the operator overloads in D all are called with a template parameter that contains the operation; so for example if you overload a+b, you end up with a.opBinary!"+"(b), where "+" is a template parameter (e.g. accessible at compile-time). this means that you can generate code for all binary operations with a single function.
string metaprogramming is a little crude compared to how some languages let you access the AST, but it works great in D and is much easier to understand. the only problem with it imo is that sometimes I'd end up using it in situations where it'd be much better just to use a normal template, but it was just easier to make a string mixin (since templates can be finnicky at times)
Great video!
Great video!
You seem to always upload right when i’m returning to your videos :D just yesterday i watched some of your old videos after starting a Vulkan renderer in pure C
I’m only 20 seconds in:
Is your thumbnail a Baba Is You reference? The colors and font are spot on along with the title being the same pattern
@@OctagonalSquare it's actually funny that I realized that as soon as I was done making the thumbnail. And it's certainly apt considering the amount of meta layers in the game. But if you'll believe it, it was a complete coincidence :P
@@voxelriftsI believe it. It does fit really well since the game basically makes logic edits to the code (probably in a more controlled way than what using concepts from this video would be) so I thought maybe it was intentional lol. Good video though!
"Metaprogramming", huh? Well, good to know that thing I do sometimes has a name!
9:35 won't work intuitively because of page execute-disable security features in most OSes. Many JITers work around this by manipulating the instructions then converting the page from RW to RX
GREENSPUN’S TENTH RULE
Great video! I was wondering, where do you learn all this stuff? I'm really new to this kind of low level of programming :0
Very clean explanations on metaprogramming and jit. Subsribed immediately. Don't pay attention to negative comments thwh don't know what they just watched
Just Brilliant, keep going bro.
one note tho, it will be nice to have some calm music :) just an opinion.
This is awesome! Can the same be achieved with stack memory? (13:35)
Hey brother your videos are amazing and inspiring especially the one in which you mentioned your journey in c.I am really longing to learn to render things on screen using c for a long time,is opengl a good way to start? And could u suggest any resources from which i can satrt learing modern opengl? Thank you
Hello, i’ve been trying to do a c project about chain reaction for months and it won’t work properly, can you help me with it?
This is a good video
great video! what editor do you use at 2:30?
I think he uses 4coder
Hi,
I want to comment on the Text used in the video.
Text has a drop shadow effect that too a hard shadow which looks detached from the text.
It gives the feeling that the text is blurry.
Please fix this.
1:00 what do you mean it doesn't exist? we all use harvard-neumann hybrid architectures. data is literally on a separate bus and treated differently by the cpu cores
I do Data Oriented Functional programming. Zig its great for that :)
zig seems pretty far from functional programming language
@@thezipcreator You can apply the paradigm to just about any language tho. Unless the language is specifically designed with another paradigm in mind. Zig is general purpose, and in its idiomatic form, it is structured and data-oriented
"because I cheated to the power of a script for this video" Love it hahahaha
Are the animations a library you made? I think tscoding did something similar
Animations were made using Motion Canvas: motioncanvas.io/
I use Julia btw
Wish you had a bajillion videos explaining everything programming....as an American its true you guys with accents do sound smart💀
I didn't understand why you'd allocate in the first place, why not just use the stack. But I guess you need special.
That's a great question. From my (admittedly limited) testing, it seems as though the stack isn't allocated within a page with the execute permission since the program just segfaults if I try
Do you mean always? If so, sure, lets store an object with 200k entries on the stack, lets see if your program even starts.
@@voxelrifts yeah, that's what I tried to say at the end. That in general I bristle at unnecessary heap allocations, but since you need a special allocation anyway, it makes sense to have that as the base for the explanation.
Executing the stack sounds like something went very wrong with the callstack itself, the return address being overwritten by some pointer to a stack variable, and it's very reasonable to not allow that I'd think.
@@E_m1ly_6302 we allocated a grand total of a hundred bytes here. And used like 5 or 6 of those. No need to be dramatic.
Honestly this video would have been so much better if you had included examples of metaprogramming on the AST level in languages that support it on language level like Elixir, Crystal, Rust (Rust technically has token-level metaprogramming) etc.
This would make the concept much easier to understand and emphasise that there are languages that support it out of the box.
Moreover, you assume that the viewer knows C quite well. I bet there are a lot of people interested in this topic who use other languages and do not understand C as well as you'd like them to. Adding a few trivial explanations for things like function pointers, malloc, free etc. could help them for sure.
Bro just explained ryan fleurys metadesk without explaining metadesk
@@wicked_sash metadesk is a particular library for metaprogramming. My goal wasn't to explain metadesk itself, but the merits and uses of metaprogramming as a whole :))
hi
There are 2 kinds of VM: [1] guest ISA = host ISA. [2] guest ISA != host ISA. I believe that for the case [2], it is better to call it as Abstract State Machine instead of Virtual Machine.
No. Data is code.
Code is data? Well, duh. The problems start when data is code (or at least is treated as code).
“Problems”? Yep, never written a line o lisp!
Code is an abstraction. Data is not.
"programming paradigms" - are political ideologies with 0 backing outside of "this feels better for me".
The only program structure in the end is the CPU instructions.
And the only one we, as engineers should care about "feeling better" - is the CPU.
If something is better for the CPU, and therefore in the end for the user.
I remind you that for example, python wastes 75X+ more energy than C.
If you are an environmentalist, you really should think about that.
Too idealistic of a viewpoint for the real world. If a company needs a online storefront, you could get one up and running in a couple weeks using django and postgres or whatever, vs spending years doing it from scratch in C counting every CPU cycle for all possible code paths. Think about the time and money spent during development as well. All that time the company could've been making money from their python based app.
And its not just about the money, why would you want to redo everything that has been done a thousand times before? Just take the abstractions that others have figured out for you and focus on business logic.
nope, code is for people. it's not that hard to write something working, it's about maintaining it. and decoupling - which is achieved using those paradigms - greatly helps with that
you will spend way more energy recompiling and debugging your code with C should you choose it over python for something more demanding than embedded devices
I mean, it's about end results, and I may be wronge
some things needs extreme speed, so we don't even use C, but even assembly,
Now we can achieve in certain domains great and very fast results, using Python or JS for example
Some times we see some operations needs more speed, so we write it in c and add it as extention to the python file.
I'm wondering what domain you are in BTW
But I get the point you are making anyway.
Thank you
+ BTW,
I'm really looking forward to try Cython,
pasically a, some sort of compiler to python,
So it's like as i've heard, a million times faster that the interpretter.
it's like, a c code now?!
And a threaded code is few times more cache-efficient than a simple native code. You can never "just optimise", you always optimise for some problem-specific set of constraints. Anything else is a bad engineering.
Writing code that way will make it too platform specific as different CPUs have different ISAs even within the same CPU brand from the same manufacturer, So indeed there is a place for programming languages thst abstract away the hardware (while allowing to go deeper when needed)
Mixing data and program code is deadly for static analysis. Bugs, that could have been found by the compiler will now crash the program during runtime. Also, a very bad idea from a security point of view. Don't do this. The only use case I see is when the user has no influence on the data/program. For example your build pipeline. So, use metaprogramming for non-productive code.
So... You're implying something like Lisp _can't_ do static analysis or what?
Do you have any sources for your claims?
mixing data and program code is deadly for **shit** static analysis. everything does semantic analysis nowadays. you're wrong.
Horrifying take
It is yet another tool/technique. Fully relying on static analyzer also wrong. There is a QA, fuzzing, etc. Personally using analyzers too, but you just need to know what you are doing
Also, it's a nice idea to convert during metaprogram from C to GLSL
who asked?
i did
Dude, you're a really knowledgeable guy and I like your explanations, but please set up your mic to suppress details of your mouth "mimics". Like I can hear you have a mouth full of saliva... and all the sounds related to that, that's not a very pleasant experience, especially while eating.
Yea, my misophonia literally prevents me from watching the video. Could anybody give me a tldr?