Accidently created JDSL | Prime Reacts
Вставка
- Опубліковано 3 лют 2024
- Recorded live on twitch, GET IN
/ theprimeagen
Become a backend engineer. Its my favorite site
boot.dev/?promo=PRIMEYT
This is also the best way to support me is to support yourself becoming a better backend engineer.
Article link: matsmoll.github.io/posts/how-...
By: Mats Eikeland Mollestad
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-Kinesis
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/deeznuts - Наука та технологія
![WC3 - [UD] Happy vs Infi [ORC] - LB Final - Warcraft 3 All-Star League - S1 - M4](http://i.ytimg.com/vi/g7DNzfoOhGY/mqdefault.jpg)
![WC3 - [UD] Happy vs Infi [ORC] - LB Final - Warcraft 3 All-Star League - S1 - M4](/img/tr.png)
Someone once said "any sufficiently complex program will contain a half-baked LISP interpreter."
Edit: Aha, it's called "Greenspun’s Tenth Rule"
Basically the douchiest way to say that sufficiently complex DSLs end up turing complete, as if lisp were the only turing complete language on the planet
@@isodoubIetLISP isn't the only Turing complete but it is simple and extremely flexible.
@@isodoubIetyeah C is just lisp with different brackets
The other reason for it being Lisp specifically is that Lisp programs literally _are_ their own abstract syntax tree-so if you ever parse anything, you're in a sense "converting it to a bad Lisp"
@@duncanw9901 C++ and Rust are just lisp, you heard it here first folks
If anyone is interested, one hot encoding is the process of taking a set of values (think enum) and convert them to a series of arrays with a single 1 and the rest 0s. You're essentially creating a bitset to define the set of possible values. This representation matches the output of a neural network classifier.
Good explanation. I like to think of the easy example like strings converted to one-hot vector embedding
Tom is a genius.
That ping tricked me >:[
Tom is a genius.
tom is a geneous
Me at 2:50: so we are writing parse trees by hand.
Prime at 3:00: we are writing an interpreter
Me: my CS degree is not for waste.
my take: just use Lisp at that point
"but muh parens"
@@spookyconnolly6072 JSON - but muh braces 😂
Would that require shipping a Lisp interpreter with the newly defined DSL? I think they were trying avoid exactly that interpreter lock-in, but one could argue that they now instead need to ship their in-house tech. Seems a bit overengineered to me either way.
@@leon.t8772 But Lisp is both small *and* deterministic, that's the point. Same would go for Lua, for example: it's not much of a challenge to bundle that with your application.
I came to say this. My experience with clojure is that code is not too far from data. Lisp (and descendants) make that very obvious. I once wrote a prototype of an automated rules engine in clojure where you'd describe logic as data files in edn format.
Man built JVM
JSON VM
This Mats is not Tom, but he seems quite smart
seems like he could really run a place
Mats is the codename of Tom
Yes one-hot encoding is a very standard thing in data science, not necessarily python.
Also if the success of your model hinges on slight numerical differences between different implementations of the same algorithm, your model is not very good.
Also nobody should ever say "python" and "type-safe" in the same sentence.
Give this man a cookie!
STROKE!
I lol’d out loud and lmao’d me ass off.
Glorious recovery.
I was there for the live stream, being a chat clown, but I just realized there is a briliant idea hiding in here. I have a real use for “git JDSL” in mind.
Do it!
Man, i feel bad for Keith, that will be tasked to maintain and work on Tom's atrocious and undocumented puppy when Tom leaves
all of this to still have his original issues around version differences, and machine/set up specific quirks... sounds like they need a container solution instead... But it does solve a significant amount of issues around data and lineage etc.
One day, all will be bespoke broken dynamic libs made in javascript with no comment support. Tom is magnanimous.
I built DDSL (double-diesel) at my company for our data transformations. The thing is awesome!
Having put LUA into 3 different things now, I also recommend LUA. It’s brilliant.
Also, if you use JLua, you can throw objects on the LUA stack without them being modified, so you can have them encapsulate your data and even functions, essentially turning LUA into an OO language. It’s evil and dark magic, but also very useful. (For example, I had a players object with sockets in them that LUA could not touch, but I could add a kick() function to any player object that uses closes the socket in Java, but it would only work or be available on the player objects. But whatever the LUA code did with the object, store it in an array, pass it along, return it, save it in other structures, it would keep its specialness and kick() still worked)
My take is that JSON is great for structured data that can be read by anything. It's very useful for shared lookup tables between disparate pieces of software in different languages that work together either remotely or on the same system. I recently used a JSON file to track modify times in a file sync tool and it went super smooth. Trying to use it as a language of its own seems ill-advised, but using it as a lingua franca between other languages is usually a solid tactic in my experience because both languages can work with their own ways of processing it.
It’s a decent way to safe structured data, until you need a datetime..
@MarcLucksch Unix timestamps, my man. They're just strings and almost everything has a converter.
@MarcLucksch Although I do feel your pain. Recently had a project that required me to do exactly that across a typescript front and a common JS node backend plugging into MSSQL. The time zone difference between the production server and my testing environment created no end of headaches.
@@andythedishwasher1117 @andythedishwasher1117 Yeah, there are workarounds, but saving it as a string is meh, and JS can't be trusted with numbers for timestamps. Also you always have to convert manually in JS, super annoying, even though JS has a Date type. And timezones are hell, as you said :)
Omg, i realized that i also made JDSL at work for processing excel files with an streamlit wrapper. It's also somehow pretty robust and worked really well.
I'm now convinced to write a Lexer for Lua. Writing a Lexer is something I've wanted to do for a while.
Rocking the arc
I did json coding in rust in my company, we wanted a service that could get events, hidrate, and send to multiple providers, some events go to all providers, some events only to some
what I ended up building was a macro in wich you write where your events go and what are the properties transformations in json like, and it builds all conversions, and the events themselfs are typed on a .proto file, and everything is checked in compile time by rust
Bro got so angry that Jake added code comments, but yeah, Tom is a genius.
I use objects like this for load balancing and view rendering configs for vanilla js SPA
"JDSL is an experience" Better words haven't been said
JSDL is inimitable. All hail Tom!
oh great another abstraction in ML. just what we needed.
4:10 - dog, pig, dog, pig. Loaf of bread.
man this gives me flash backs of logic being baked into xslt files via some local callback mechanism... *shudders*
Tom IS. A genius.
I guess that fits the requirements needed... let's convert that JDSL into XML, rename it to "DTSX", and pretend we used SSDT... and convert back legacy DTSXs into JDSL, so we can deploy on Camel and consider extending that data-oriented JDSL for enabling APIs with CamelK and Quarkus... great insight!
one-hot is a just a vector of zeros where you flip one (And only one) of the bits at a time. machine learning 001 bros
I think he's solving a problem that already has solutions behind it. Data processing is reasonably well accomplished via dbt , version editing spark or whatever. Model and data freshness can be maintained via tools like mlflow , kubeflow ect. I read his code and I'm just not seeing where to slot this in....
Data pipelines are also version controlled already on the code side and are run in controlled execution environments; your tech of choice will be available.
Freshness gets maintained via metadata tagging of your assets and rules.
I do like him using dataclass fields to define named operations.
the only thing worse than this is making a forth/stack machine embedded in json to be literal bytecodes for this
And when we needed the most, he returned....
Tom is a genius
Async python is genious.
Something Something, Haskel peerfunction ocaml.
Serious question, how do you call Java scripts (or compiled code) given by the user securely? let's say its purpose is to check things about data also given by the user
You don't
Well.. if you wanted to handle only very specific cases (ie. a very stripped down version of javascript where most functions don't exist anymore) then you could create a map of "allowed functions/operations" and then parse the string yourself to see if there's anything that's not allowed.. depending on what you're doing this could potentially be done (though you'd still need to be incredibly careful to not let any edge cases through).
Realistically, if you were doing anything like this you should also ensure that the code is only ever run on the client side by the client that created it - you definitely don't want that running on your own servers or on anyone else's computer. That way if the client does something stupid it doesn't affect anyone other than themselves.
If you mean just validating any javascript in general, that would be basically an impossible task and would probably be 100x more complicated than the entire rest of your program combined - it might even be mathematically impossible to do (I guess depending on what your definition of secure is..).
Noooooooooooo! jDSL is back!
Tom is a genius
I built an analysis pipeline that was entirely defined by (barely validated) YAML
The only thing that saved my skin was the incredibly tight scope
All organisms evolve to crabs and all programming languages evolve to json. Its inevitable just like the big freeze of universe is inevitable one day once whole universe has finished encoding it self into json.
Is this not how Redis databases are saved to disk? As in it records the full series of commands executed on the server and uses that to rebuild it in RAM?
This already exists. It’s called SQL.
I wonder what Vin Diesel would come up with
Usually this is done in Yaml for more pythonic review.
Not sure about this. I think I prefer Tom's solution. He's a genius after all.
JDSL is back! Hallelujah!
Symbols would help.
Wait till he sees Mojo...
Bro just invented Django
tom is a gnome
rust side-effect: s/in my life/in my lifetime/g
wait this this guy hears about graphs
Are we allowed to call JSON s expressions yet?
How long before somebody writes JSONlang?
So.... Why use SVN over Git? Real question.
Hehehe when I was much younger I created an action execution thing in xml. It was horrible.
So he recreated Unison.
JDSL: But does it have comment support though??
You can't have comments in JSON
Tom is a genius, we are the morons!
bro invented langgraph, wow
"torture-wise SVN"?
Developer discovers IR... But in json
I mean... It's easier to read than bytecode *shrug*
I would ask him to refund his salary if I employed him.
Now, I have a question: How do I add comments to JDSL?
skill issue
string literal?
OP ->'Otovo' | Prime -> 'Aviato'
Mats' a Genius!
Isn't this what we normal people call "a config file"? 😅
Should have Tom on, as a guest, and that other dude who did that stuff with Lua, Scandish name, I think, Jensen, somethin. I'd love to see a weekly with a panel, that would be increds.
haha the stroke yelling at 5:20: made me spit out my coca cola in laughter
Jom is a tenius.
TOM IS A GEEEENIUSSSS
Oh boy what is this 😂
This article provides too much documentation for this to be truly considered JSDL
Sype Tafe santrformations
aren't you that dr disrespectful guy? the moustache is about the same, its probably just you with glasses
No, it's Donut Operator
Turns out, Tom was right all along 😅
All hail Tom
Microsoft needs to watch this. Logic apps 😢
Prime, it’s time that you roll up your sleeves and create your own JDSL. You’ll never be as genius as Tom, but still it’d be pretty awesome.
Lua4DSL❤
Pkl moment
The man just reinvented serde and enums, in python.
Where enums exist in Python, natively xD
Just more sophisticated serialization/deserialization engine based on JSON
It's nothing like serde or enums
@@isodoubIetJSON is literally a wire format for Javascript objects.
He is serializing his pipelines, and then deserializing his pipelines. That is exactly what is happening.
@@duncanw9901Thank you for the definition but everyone already knows what json is. This is not just a serialization thing because it imposes additional structure. Python has json as a built-in, don't you think he would've used that if it was enough?
@@isodoubIet just because it is not using primitives to serialize pipeline objects, or is serializing pipeline objects at a more abstract level than you consider normal, does not mean it is not serializing them.
Serialization simply is the fundamental problem they are solving: "how do I get my object from this context to this other context automatically, with no corruption?"
tbh, i'm not sure why one would want to build data pipelines with json. it is probably better to use lua or python or some other language.
Tom é um gênio
Suineg a si Mot
09:17 very verbose assembly?
LISP
As a data engineer, this is awful. He is not a genius like Tom
Tom also thought JDSL was a great solution to the problem... The author is a unreliable narrator
Why not use the same programming language???
I can't help but think that article was written by AI
What an unfortunate last name.
This is shit. Im big dana engineer 10 year's. Support this shit is nightmare. Append something new for person who create is good. But in best case should be only framework or lib with basic generic transformarion related to domain. I wrote this kind stuff so many time, and understand this is shit.
JSON may feel bad until you've had to maintain 1k line YAML files. Even with the help of YAML aware code editor editing JSON is a pleasure compared to YAML. I often cheat and convert my YAMLs to JSON because JSON is a valid subset of YAML (therefore JSON is valid YAML syntax)
3:25 blerg! Why do peeps use JSON for this 💩. At least choose something easier to write and read like YAML.
10 seconds in: "programmed in JSON" ????? Is this amateur hour? PrimeTime thinks JSON is a programming language? I'm out.
Going to send this beatuty to my friend who is currently on his endeavor to build up his own Java-driven SQL-filled abomination of a JDSL, let's call it JSQDL (jei-sqwi-dl)
Tom is a genius.
Tom is a genius
Tom is a genius.