Migrating 3.7 Million Lines Of Code
Вставка
- Опубліковано 29 січ 2024
- Recorded live on twitch, GET IN
/ theprimeagen
Become a backend engineer. Its my favorite site
boot.dev/?promo=PRIMEYT
This is also the best way to support me is to support yourself becoming a better backend engineer.
Reviewed article: stripe.com/blog/migrating-to-...
By: Andrew Lunny | / alunny
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-Kinesis
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/deeznuts - Наука та технологія
The fact that 3.7 million lines of JavaScript exist in a project scares me
could probably replace it with a 100 line regex that only Steve upstairs understands
Most of it might be jsx not actual logic
They did mention it was a monorepo so could be a lot of different projects
And to think that it was used for processing a huge percentage of e-commerce transactions…
I once refactored a Next js project from 250,000 lines to about 15,000.
It was bad.
The other day I ordered two burritos from a local shop and the guy behind the counter stuffed fillings for both into same wrap. He tried to fold it but made a complete mess. It just overflowed everywhere. I asked what's up with that. He replied, "Don't ask me. I'm just an endofunctor following arrows." Lesson learned: don't go to Mona D's Burritos.
1:43 We still use Cobol at my current job (it's also my first soft dev job). When I have to work with it, I feel like an archeologist digging through fossils of the past
But do you like it or not? 🙂
I had a summer research position where we had to basically do a prototype for a system rewrite from a customized C server (not C++, just C), to a RESTful architecture for an industrial partner. At the time, they wanted us to use Node JS, which we had experience in due to learning web tech the school year before that. Basically we scoured through debug logs and legacy code to port the relevant functions into custom NodeJS modules. If only we knew about Golang or Rust at the time so that our successors wouldn't have to debug the untyped JS and learn the process flow... 😔
Edit: I really like JS, but not for backend. I want to either learn Go or Rust after I finish this semester so that I can actually write good backends. (I mean there's also C#, but I'd prefer not to deal with debugging that for web stuff, thanks to Unity)
You actually are an archeologist
@@GreyDeathVaccine I honestly like it! I know it sounds weird, but for me it's like a breath of fresh air lol
@@IndigoTeddy rust, go and C# are all good choices.
C# is not hard to debug.
21:34 "What's a monad? A monad is a burrito."
Thanks for leaving this in 😂
What is a burrito?
@@stevenhe3462 a burrito is a structure that combines food fragments (materials) and wraps their return values in a type
@@stevenhe3462I think it is a monad
gigachad.jpg
>jr dev struggling with issue
>gigachad sr netflix (btw) enigneer approaches
>oh you just need to use a monad for that
>what's a monad?
>what's a monad? A monad is a burrito
>refuses to elaborate
>leaves
@@Mikopet I love you for this.
One minor reason to order imports is to reduce merge problems.
e.g. if you add an import in one place and someone else adds the same import somewhere else in the same file you might end up with it there twice after a merge, but if you both add it in the same place it just disappears from the diff.
It should always be the work of a tool though, nobody should ever have to think about it.
Please don’t let your experience and expertise stay in the way of listening to Primegen gospel about his opinions on what is wrong or right based on his explicitly limited knowledge on the subject
3.7 millions lines of code happens when developers within a company don’t leverage each other tools and likely end up reinventing the wheel every project. smh
bro does the jest test memory thing every chance he gets
Re-ordering imports (or anything) listed alphabetically is super helpful when scanning code to ensure no duplicates or just to see if something you expect to be imported is or not
One of my favorite parts of using Go with nvim is the golines formatter which uses goimports/gofmt to automatically remove and organize imports for me. I like that it's organized in a consistent way, but above all that, I like that I don't have to think about organizing them at all.
For those curious, goimports is gofmt with the additional import stuff. Golines is goimports but it limits lines to a certain size
11:11 about function declaration with const and arrow
1) there is an option to use const declaration for function without arrow function syntax
2) const declaration benefits on part of type definition: instead of declaring a function with its argument types and its return types separately, you just write const Component: FunctionComponent = function(props) { ... } and all arguments with return type are inferred correctly and more important together
There are other benefits and differences too… But please, don’t let logic and facts distract you from worshipping a random guy on UA-cam bragging about his experience and talking BS about things he has very little knowledge about.
They just put :any everywhere
Better than having compiler errors 🤪
😂🫠
Typescript should be renamed to Anyscript
16:24 So that no one removes an import, adds it to the bottom and now git recognizes that as a change. It makes git gistory more consistent
it also makes it easier to visually search inputs to see if something is or isn't imported
And, there’s definitely no gain to ordering it arbitrarily
F# has this kind of file and function declaration order. It has the benefit of removing circle dependencies even inside of projects. Also it provides tools for mutually recursive types.
10:38 "this" binding in Javascript function and arrow function are difference, in arrow function it does not have it's own "this" so the "this" in arrow function inherit from enclosing lexical (surrounding) scope. I write Python btw, but I have to work with Javascript lately and I found this in MDN arrow function section.
About arrow function personally i use arrow function to indicate functions that live for a certain amount of time ex : arr.map(x => x+1 ) but also when a function return a function the inner one is a arrow function to clearly distinguish the two types of function
The main difference is how the this keyword works between the two. In a function declared with the function keyword, "this" gives you a reference to the object prototype for your function. In an arrow function "this" gives you a reference to the object that called the arrow function. Although this behavior may have changed...
If you've ever gotten a "keyword 'this' considered harmful, please do an immutable clone" this is why
@@ChungusTheLarge arrow functions also aren't hoisted vs if you use the function keyword. Might not matter for what you might be doing but it's something to be aware of.
@@bgill7475 isn't Javascript a joy to write?
ThePrimeagen, you just inspired me to rediscover being more hands-on technically again after 3 years of being an engineering manager.
I really like the explaination of monads in this video. Very educative, thank you
I always organize my imports, but not alphabetically.
First comes the imports from dependencies, like 'react', 'react-native', 'expo-whatever', etc
Second comes imports from global stuff and services created inside the repo itself.
Third comes imports from reusable components
And last comes imports from local components
And all these groups are split by a blank line, to visualize better.
I feel really happy with this system of my.
I have forgotten about flow crashing your machine was just a given and accepted industry standard
regarding ordering imports it helps in the following case:
you have a component with 10 relative imports, you moved it to another folder in the project, but your code editor didn't update import paths. now you need either go over imports and fix them manually or simply delete all of them and go over hightailed errored code one by one to auto-import stuff with you IDE. I personally prefer the second, because I hate remembering import paths, there is little autocomplete there, but auto-import via IDE works pretty fine. But it can cause order of imports to be changed based on the order of stuff you are importing. and here comes ESLint/Prettier with their import order plugins to get rid of this mess automatically before creating PR, so the PR is as clean as possible
I use a third method - I copy ALL the imports from the original file, including the ones that aren't part of the extracted code, then I save both files. This triggers an automatic "Organize Imports" action in VSC, which uses my ESLint rule to sort my imports AND delete any unnecessary ones.
I have import sorting by groups in my codebases, the idea is that it's easier to figure out what is going on in the file based on being able to read the imports more clearly, I can see ok, I am using these third party things, these components, these helpers etc. not insanely useful, but doesn't hurt (unless you don't have auto fix set up and you have to do it manually)
I work with a 30 year old application that we have tried to modernize for at least 8 years. That legacy system was written in smalltalk. The modernization is going to be at least another 3 years.
20:01 - lol, I hear that! Coming from mostly C / C# / go, I started a job where I think we’ve got ~12-13m lines of ObjC…? (That’s been churned out at breakneck speeds for the last ~20 years) That was fun. Plus we deploy on Linux so we gotta deal with differences between the Apple and GNUStep implementations….
It was a little rough to get used to the different syntax. But now I’m deep in, and mostly impressed with how well objc supports writing safely threaded code (granted, you need to use the correct patterns to do it safe, but ‘-performSelector:… onThread:…‘ to send data across threads is super easy).
16:23 reordering imports etc. can help reduce merge conflicts and PR noise, since added or remove + restored lines are consistently placed.
+1 to ordering imports: group by projects together, the closer the project, the higher priority in the list. Easily see what’s our code, what’s not. Keeping it alphabetically also reduces git conflicts.
One of the reasons for const functions is Typescript.
```
const Foo: React.FC = (props) => {
```
You can't do that with `function Foo`... you'd have to set the return type and parameter types directly.
That's no longer considered correct, but it's still around a lot.
At face value, I would agree that sorting imports doesn't make sense. However, after having dealt with many senseless merge conflicts that would have been completely avoided by sorting imports, I am now a proponent of sorting imports. There's easy to use prettier plugins that can automate this task. Reduces friction for larger teams :)
10:30 Functions should be like any other variable in the code. It makes no sense to treat them separately, especially given that in the runtime they *are* variables. Using const ... = (...)=>... forces you to treat them equally to a variable - e.g. no usage prior to declaring.
It also avoids the entire unnecessary complexity of the `this` keyword, as arrow functions don't have a `this` value.
21:09 I like your funny words, magic man
37 MILLION LINES OF CODE is bigger than the entirety of chromium.
1:33 most people write software as if they're going to be around for a year - resume driven development is just sensible for personal advancement (to new job that pays better)
perfect cut “a monad is a burri-“
“Having to define your functions in order is just annoying” - OCaml feels attacked
I've had import sorting in all repos for the past 4 years at work. It would be hard to explain it to a Vim user, but eslint autofix is pretty good - you write stuff without thinking of formatting and it all happens automagically, Imports are just one of the bits in this system, makes import blocks standard, easier to lookup when reading it.
As a neovim user, when I save a Python file, it gets auto formated, imports gets organized removing unused ones, and linting issues that are safe to autofix gets fixed.
I just need to ask, do you think all Vim users manually format their code? :D
@@Ubben1999well I haven’t seen them brag about it yet, so it’s clearly not a thing…
11:00 I think arrow functions capture the surrounding `this` in the enclosing context
edit: he literally mentions this later
Ya, i have to work 4 yr old code using classes and I'm like learning how to borrow a function in classes for some reason.
16:35 reordering imports' purpose is to help Git not create conflicts just because two people added their imports at the end of the list, even though one starts with b and the other with u. Keeping them ordered avoids such conflicts.
An expert system does exactly that. It shows the chain of reasoning.
Not to diminish Google's AI, but the idea and implementation is at least 50 years old.
Thank you so much for clarifying the difference between OCD and Assholes. Those of us who have OCD appreciate it. At least, I do.
Performance test the arrow function against the full function. Arrow functions lack bindings to "this" and "arguments" so they execute much faster. They are supposed to be much closer to true lambdas, but just like lamdas, you dont want to use them everywhere.
Come on now, don’t be coming here with your knowledge and performance hacks and be shitting on what the priest is saying…
The memory usage is certainly painful, I recently moved a ~34k LoC typescript project to project references so vscode wouldn't sudoku itself whenever you opened a ts file. Can't imagine 3.7 million; maybe they have 128gb of ram.
I can work with TS and PHP and wouldn't complain too much about Java or C#
but manual memory management and esolangs like Python are where I put the line
I only import modules in the following manner:
First paragraph - external crates
Second paragraph - internal crates and modules
Third paragraph - re-exports
i’m learning c++ and absolutely enjoying it and the challenges it brings.
the only thing i do not like so far is how pass by reference works. i mean, the calling code has no idea if the called function is accepting arguments by value or reference. that seems crazy.
btw, i have been coding in js for ~4 years
I just try all variants until error goes away xD
You seem alright, don’t check out c++ templates tomorrow
@@bignades1 i understand that templates are kind of tricky to implement. but i don’t understand why people hate it… ig it’s alright.
p.s. i have not worked on any large codebases in cpp, so im naive in this regard. is it primarily the additional compile time that yall hate?
I love c++ overall, I love syntax, functionality, standard, pre-processor, but managing dependencies, integrating libs and writing makefile/cmakelists is so bad, I hate that
@@oleksiistri8429 yes!! that’s a valid complaint.
I'm hyped for lmaolang to make it's way into Netflix!
10:53 arrow function is unscoped (eg. able to access 'this' of parent object) whereas non-arrow is scoped. There is a functional difference.
Sorting imports helps reduce merge conflicts. With tools like isort, it's a no brainier.
1. This guy's war-whoop opening makes Dead Poets Society look like a Quaker Friends meeting. For no apparent reason. 2. This isn't Software Engineering, it's a stunt. You're not curing cancer or surviving a bungee jump whose cord failed just as it started to stretch... you're just making other people money.
"In a single pull request" is WAYY outta pocket
"A Monad is a burrito" that is so much better than the self-je***ng "A Monad is just a Monoid in the Category of Endofunctors". Which no one with social skills is going to understand.
pretty sure most have been using it as a joke since monad's popularity exploded. chill out.
Somebody abridged a sub of the bunker scene from Downfall. Hitler is yelling at his generals:
"What the fuck is a monad?"
"Mein furher it is a monoid in the category of endofunctors"
"YOU JUST MADE THAT UP RIGHT NOW!"
ordering import can make every one code look the same, which can reduce conflict when merge other code. ordering css and package.json has the same benefit.
4:27 not having the right fonts installed for an editor plugin is such a mood.
I was imagining "the voice of prime™" as a podcast and I can say that he has a listenable voice when he's not trolling
Wow, not many companies would approve such a massive code migration project. Guess Stripe has upper management that does see software development as profit generator and not a cost center.
3.7 million lines of code, 3.5 million of which were deprecated and not used.
Import local modules
Import domain system libraries (gcp)
Import native libraries to language (dataclass)
Import external libraries to language (numpy)
Is how I organize my imports. That way for someone else they can look at my imports to see first what does the code touch immediately in the repo, then what apis im using from the system, then internal then external language things
how did you get the jest breadcrumbs with up arrow ? Is it a plugin or was it in temux ?
5:45 I work in a business of ~7000 employees, and most of IT has 0 care for internal tools. It's really sad.
Full ack any migration is a pain. Once had to migrate a large e-commerce database to a new software (~2 million product elements, 10 million users and their orders). First doing it the sophisticated ORM way it ended up stalling and taking ages to migrate the data. In the end a simple straight forward SQL script did the job. Harder to read but way faster and reliable. The ORM took 1 week or so runtime for the production DB, the script run within 10 minutes doing the job 😅
Why arrow functions written in that particular way? A reason I've seen, and practice, is if you are trying to give a more meaningful name to a set of instructions that should only be callable inside the function it is defined. In C# we have something called "local functions" that let us do this instead of having a private member function that is called in one place and then we have to hope people read comments saying not to call it anywhere else. It's not great but it allows you to narrow the scope as much as possible.
While I tend to agree with the argument against auto import ordering, I remember that cargo fmt also does this anyway.
Somewhere I worked there was a team of 10 people migrating python 2 -> 3 full time for 6 something months. As soon as it was finished at least half of them quit the company.
I'm glad the 2 months of work converting typed JS to typed JS went relatively well. However, there's something quixotic about going from one non-standard type system to another. If everyone moves to JSDoc will they have to do this again? if the "types in js" rfc is approved and a new type system is born will they have to do this again? Anyway I'm told Typescript will live forever so I'm sure this won't be a problem.
COBOL??? [planet of the apes scene. Pounding sand on the beach, in front of the Statue of Liberty] Damn you! Damn you....
Remember when we thoug that coffeescript was a good idea?
Everything was better than what we thought was a good idea in Vanilla js at the time
Average PR size when I’m the reviewer
I don’t want any developer spending time and effort on organizing their imports manually. Nor do I want IDE import organization clashes to be a problem.
Ordering imports avoids many useless merge conflicts when working within a large team.
Writing Java in 2024 = writing instant legacy code
They're migrating from JavaScript to JavaScript with types™. Woah!
Actually it's from Flow, so it's from JavaScript with types to JavaScript with types.
The difference is that Typescript takes linear (or maybe n log n) time to check, while Flow takes exponential time.
And then 10 years later typescript will become obsolete and they will be migrating 5 mil lines of code to a new fancy shiny thing
That's how programmers secure their jobs😂
It might be different now (haven't checked in years) but I default to arrow functions in JS because normal functions override "this" while arrow functions keep the same scope as the parent. I had so many random old jQuery libraries break because of it that I just never use function anymore. It just became normal that const ____ = () => {} is just how I write a function in JS and the other way is when I have to deal with writing stuff in ES5.
C# is server side of TS.
What's up with the square brackets in Objective-C though
The name... is monadgement
I have a converter from Typescript to C#, next step
Regarding the import order: It doesn't have to be alphabetically. At my last job we did it by "kind", so external packages first, then based on proximity to the file you're in.
It's really not an issue either way as it's all handled automatically by ESLint on file save so why care if the rule is there or not? 🤷♂
I think the reason do the arrow functions is because it automatically binds 'this' context. When you pass callbacks of named functions you have to add the .bind(this) and with arrow functions you don't. But I've seen a lot of people use the const arrow function pattern to define class methods. When you do that const/arrow method, every time you make an instance of the class the method becomes a new object so I think it actually takes up more memory? The function will no longer be on the prototype of the object if it isn't named. Idk people are wacky I don't see the benefit.
Yes, but I would never use it as class methods that would destroy classes xD
But otherwise I prefer them. You always know what "this" will do. That's the benefit at least for me.
Amd you don't need another keyword. Just const everywhere
I work for a company that transcodes COBOL into Java
10:36 smaller when return without "return" keyword
also with "function foo(){}", foo is not const ("foo=69;" is legal)
What is this blogging website?
I was curious so I ran an analysis on our application. It's 3.268.408 lines of C#.
worked at a company where they had no internal documentation, forget dev tools. That's not it, they also had internal libraries on which a large amount of codebase was dependent on, had their own version of docker but worse for deploying apps, their own modified gradle which had not been updated for years! and yeah, no documentation for anything at all.. no comments in code, and most of the experienced people had left long back!
I heking love LSP
The fact that stripe is using Javascript scares me
what else would the frontend be written in?
@@3_smh_3Stripe is a payment service, so I'm assuming they meant the backend is also in JS, which would be actually concerning since it's easier to mess up security in a "volatile" language which everyone learns without understanding how it works
@@3_smh_3 You don't need 3.7 million lines of code for frontend, i.e. it was not only used for frontend = scary
@@owlmostdead9492 I didn't imply it was only used for frontend.
Wait till you learn what the space shuttle code is written on… or IRS software… or … you know what, keep riding your high horse and don’t worry about how the world works
1:20 - Ackhtually, functional components were useless before hooks (which first appeared in 2019 in React 16). So that's 4-5 years.
I used to work on an ERP system running on an AS/400. It had Customer Order Management, Accounts Payable, Accounts Receivable, General ledger, Material Requirements Planning, Goods In, Shipping, Stores management, Customer Relationship management etc etc etc. It was 2.9 million lines of code. 3.7 million lines for a UI?
In what way would an LSP improve internal tooling? What is the use case? 6:20
What is Stripe doing with so much code? Surely their backend can’t written JS?
I mean...re: arrow vs normal functions, for starters their scopes are totally different. They are fundamentally different things with different uses.
01:40 libX11 has code from 1988 and still used
11:09 #1, You're not thinking enough about the effect of const, and #2, the effect of `this` reference capturing.
For #1, to get the same effect with function, you have to write `const foo = function() {}`, and now you will see that the arrow function syntax is shorter. You already know the benefit of const, const variables cannot be overwritten. If you omit the const, it's possible that your `function foo() {}` foo variable will be overwritten at a later point, either by you or third party script, and that may cause a silent error or difficult to find bug. Using const, you are immediately notified when a reassignment is attempted. And since arrow function syntax is shorter than the function syntax, it is used.
For #2, A person may still choose to use one or the other depending on if they want to capture the enclosing scope's `this` reference or not. Arrow functions capture it, function functions do not. Usually you want to capture, so arrow function seems to be a good default habit, especially since the syntax is shorter.
4:32 - O thats Y.
16:30 Justifications of reordering imports are clean diffs and consistency. I want code style to be globally the same, it's not what developers should care about. And you can setup fmt tool to run as git hook and never think about it again.
I don't agree with many things you say, but i totally agree with being against functions declared as const arrow functions lol
You got me in the first part, but then immediately dropped the ball…
I cant comprehend why you need that many lines for a project
Import restoring must be done by autoformatters. Not sure about JS world, but isort/black just enforce standard without your actual pain
In python
Or they could use jsdocs and have good documentation but this is better for the blog I guess
I thought for a sec that Prime's gonna switch to vim and generate 37M lines of something just to get the feel...
My guess is most (or ath least a lot) of that 3.7 mloc are generated by code generating tools. Generating interfaces and communication code to access 100s of legacy SOAP endpoints from 3rd parties would produce close to 1mloc. I remember that from back in the day when I generated code from WSDL into java. It wasn't pretty.
You say companies are still on React class components. Well boy, we are both on React classes and functional components and 3 different redux stores and a mix of typescript and javascript react components. Please help me 😢