As a former Amazon engineer, I empathize with you. Another annoying thing about using these internal tools is that the skills you gain are largely non-transferable; you are only able to apply the knowledge of using that tool to one company.
I think that's the whole point of these companies forcing "internal tools" on you... to lock you in as their (well-compensated, but immobile) labor slave.
@@clray123 I doubt it. They have no need -- the people there will have no issue learning other tools. I never felt immobile during my time there, even after wasting time fiddling with internal tools. And they don't have much trouble enticing engineers to come to work for them. These big companies just want to have full control over everything in their domain, and in theory it makes sense. In practice, it is great for tools that are maintained well, and terrible for all of the ones that aren't.
@@bbydykdyl7779 Clearly you have not spent much time working with those internal tools. Of course proprietary tools and platforms cause lock-in of users (and when it comes to developer tools, of developers). The whole business model of Microsoft revolves around it.
The code should be the documentation. Writing documentation is a waste of time, unless it's for the end user. Consistent and smart code structure is always the solution. All you need it PR be reviewed by senior employees, and code quality maintains. Sometimes redundancies appears and need to be refactored, but even so the code will be understandable and testable so you won't take too long getting to know it.
i work at IBM as a developer of operating systems and its exactly the same here, we literally use our own inhouse programming languages for development, scripting and testing and if we do use a well known public programming language, we actually have our own implementation of it and its compiler. We dont use Oracles java and jvm, we have our own implementation of java and jvm :) we have our own implementation of C as well. The hardware is our own, the operating system is our own, the languages we write it in are our own, known languages implementations are our own, EVERYTHING, literally the only thing that we haven't implemented ourselves internally at IBM is stuff like git and Jira, lmao.
Why do you guys do that . Could you explain the advantages and disadvantages of that ? Why not just use an open source tool like everyone else? Cheaper and ready to go right ? Or is it a security issue why you guys don’t use anything else other than your own? Hmm, quite curious.
@@weiSane we had our own version control system too, just recently we started migrating to git. The operating system is our own because the computers are our own, System Z.
@@weiSane Not an IBM dev, but I have worked for large companies. It's corporate inertia, high retraining cost, sunk cost fallacy, believing that because a tool was made for a specific purpose means it's the best at that, and not wanting to spend the time/money doing even the most basic customization or setup of 3rd party tools. Let me give a real example. It's common practice in some firms to have everyone using virtual machines that are reset to baseline every time they shut down, including monthly patching. At one point, I spent over a year using Visual Studio with 9Gigs of RAM, Microsoft's Perforce alternative VCS, and Visual Basic .NET Framework code living on a network share. Why? Because policy says everyone is treated the same. From the call center agents making terrible hourly pay to the salaried developers to management. This was post 2020 too... This is why large companies pay so well. Partly because they use such esoteric or old tools that finding devs who have every used them is hard. Partly because otherwise people would go to jobs that aren't stuck a decade or more in the past.
Hot take: A tool isn't complete unless its documentation is complete. The developer shouldn't have been promoted until their tool was completed. And at the very least the tool shouldn't have been implemented as a standard until it was documented.
And if the tool is still used (and I assume maybe even critical to some parts of the business) it should not be left unmaintained. Someone should have to come in to replace the original creator
No not for me. The creators of internal tools have come across a problem many people in the company have faced but they have decided to do something about it instead of pretending it doesn't exist. They create working code that solves the immediate problem. As soon as that code enters the internal domain, the creator should not be burdened with maintenance. The initial users need to contribute from the start. When they start using it and they find something was missing from documentation they need to make a PR to get it documented to help the next person. It would not have been that person's day job to create the tool, and they unlikely ever received formal work allocation to make or maintain it. They most likely did it unpaid. They should be promoted because they saw a problem and they took ownership. People that are expecting to just lift others work with no input themselves need to go back to reading the Little Red Hen. Usage is an interesting one. It might be prudent to groom the tool registry periodically. There will be tools with feature overlap or tools that have fallen out of use altogether. It might make sense to remove these from the registry as often a rewrite will be faster than a patch when the code is several years old and the original dev has moved on
There is no such thing as a complete tool in IT industry. Yesterday it seemed to be complete, today it's not: new user experience, new versions of the underlying libraries, etc. It's an institutional problem, which does not seem to be addressed by their management dickheads who only seem to be able to push everyone to using shitty tools, but not to ensure its ongoing maintenance.
We have this same issue at Amazon. Documentation sucks. I try very hard to avoid as much internal tooling in my design. Its very hard though, cause folks are so brainwashed with these tools / traditions, they can’t think outside them.
Having worked in small companies and startups and switching to a big tech company, saw a similar issue. Much of the time the big tech company wants "control" over a solution instead of using first party and community open source tools for a given problem. I was much more productive working in small companies and startups that let me choose the best tool for the job instead of relying on internal tools.
I worked at a FAANG and can relate. It is of course part of the job, but at a certain point it becomes completely unsustainable and you burn out. The entropy just increases non-stop. The biggest nightmare is when you need to CHANGE those external tools and there is no test suite. That's why it is so refreshing to work on personal projects.
I can't imagine working at a FAANG. I had a phone screening interview with Google after I finished my PhD that went very well and they wanted me to come to Mountain View for an interview and would check in with me every few months to see if I was interested... I never took them up on that offer: I ended up working for a non-profit science organization with excellent benefits that places the health of its employees above all else, and most of the software team there has been there for a minimum of 15 years. The work is challenging and incredibly fun and best of all, we get to know we're contributing to human knowledge of science. My manager is amazing, my coworkers are some of the greatest people I've ever had the pleasure of working with, and I love it. Oh, it has also afforded me both the luxury of living in Chile for eight years and in Hawaii for three years, working mostly in functional programming, which I hated at first but now love. I feel like I really hit the jackpot with this career and am extremely grateful for it. I think if I was stuck in a FAANG, I would burn out so quickly.
Not only that, but working on personal projects that have nothing to do with work subjects is awesome. I work on logic systems all day, but 3d/vision processing all night. Also at work the code doesn't physically touch the real world, but at night??? Arduino, Pi, AtMel, mmmmmm.... bring it on...
It's all about perspective. I'm working at a medium sized US company (not many internal tools), there is stress and burn out and stuff, and then the full-scale war against Ukraine has started - work has became the best way to distract from all that horror! Stress at work? - Now it's so low that it's pleasant to have some urgent issues everyone nervous about - I don't worry about it a bit! Stakes are so low!!! 😁 Oh, and talking (and even listening) to people who is not in constant stress is quite soothing - it's just great! Seriously. Now I take only short vacations (1 week), so I wouldn't have too much time thinking about what is going on and who said what about it. No fear of death, only fear to loose opportunities the life provides and to not be effective and die in vain if I would be drafted. Don't like increase of entropy - either find a way to deal with it or move on. Maybe it's time to move to another position. Do what YOU want!
As an Xoogler, I 100% agree with this. My first job was figuring out how to purge the old records in some program's database, and there was like three layers of abstraction between having the data in a protobuf and stuffing that protobuf into the database. I kept thinking "Is everything here really that complex?" It takes about a week to figure out how to compile and run your program, a couple weeks to figure out how to deploy it, and a couple months to learn all the bits you need to (say) write a Hello World program from scratch that can be accessed from the internet. And yeah, every library was google-specific. You didn't use the file system, you didn't use memory allocation, you didn't use the debugger, the compiler, etc etc etc.
I worked for a small startup that had an abstration hell. Its not really a big company problem, this is just human nature. We want to see patterns in everything and we create abstractions for that reason. I truly believe this has to do with how our brain works.
It's a matter of not admitting the problem in its entirety, which results from trying to get easy bucks. Usually managers push for using tools because these decrease the delivery costs as per their tiny minds. Would they drive a model of a car which does not have a single car service station in the country or has been designed to drive only on a few roads known to the constructors at the time of design? Probably they would walk away from the car showroom at the moment they hear about absence of a warranty period. But their brains suddenly start working differently in zero accountability conditions.
Not really. It's just a disease of junior programmers, spread by academicians with no practical experience promoting OOP and "clean code". In my small company it was an explicit design goal, set 15 years ago, to avoid advanced features of the programming language (using only a subset of syntax) and to also limit custom abstractions because most actual valuable functionality boils down to (1) executing some SQL queries and (2) manipulating some simple data structures. The idea was to make it easy for anyone to take over the codebase and to understand what is going on without specialized training (this also encompassed rejecting dependencies on external libraries and useless ideas such as object-relational mapping layers). This has worked out pretty well overall, the product is still there and still has simple structure. Although maybe sometimes you have to type a bit more to implement a new feature than if you had some glorious abstractions in place. Overall, it's a good tradeoff for productivity and makes developers feel in control (although bored).
@@darrennew8211 It's more popular to plug some half-fillting third party framework because it "saves effort". Then becomes one hell of "fun" to debug it and make it work for a specific task...
I doubt there is a place without abstractions, the only difference being whether they are known third party tools or something internal, developed due to lack of alternatives in the industry.
@@fr5229 i second that, it's one thing to use "abstractions" and yet another to use internal tooling that some dork used to get promoted, i've really seen that happening and i really wondered how those managers really let that happen, it makes no sense in the long term, and eventually they get abandoned.
The difference is whether you can or cannot get support. No tool can be built once and for all, it has to be developed according to newly discovered scenarios, get its bugs and oversights fixed, migrate to new versions of underlying someone else's implementations. So if it's just left in a vacuum ("oh we are Agile, which means no responsibility and not giving a shit about customers"), it becomes a net loss. Frankly, funky places like Google inevitably turn into a hell, there is no chance they could avoid it. That's the other side of the coin of buying developers with a false sense of freedom.
It depends entirely on how many people use the tool. The documentation for borg and blaze are complete and readable, because every single developer has to use the same version without change. The documentation for some half-ass internal ORM sucks because the developers wrote it for themselves, then someone else found it useful, and suddenly it's the go-to mechanism as enforced by the guy five levels up the management change of the guys who wrote it, because he wants credit for creating a tool that lots of people use. The power plays between management dictating which tool to use are half the problem. I once was told to evaluate the three possible databases we could use. I picked the first one, after several days of research, with a list of pros and cons. The boss's answer was "Well, would it be *impossible* to use the second one? Because boss's boss's boss wants that one." Like, why did you even tell me to research it?
You described the problem as abstraction, completely skipping over the actual problem: "Sometimes you are forced to use the tool" That's the actual problem. You can't get a sense of autonomy, mastery, and purpose working as one tiny cog within a big tech organization with tens of thousands of engineers. Terrible internal tooling and libraries and terrible legacy code that your work relies on is a problem at all organizations, even the small ones. And "abstraction" has really nothing at all to do with it, by the way; I've seen plenty of code bases where the problem is low-level procedural code that has been copy-pasted all over the place but is really concealing concepts that need to be abstracted. The difference I have found at small companies and start-ups is that I can have a bigger impact on the direction, I can say "this core code is no good, we need to rebuild it", or "this tool is no good, we're going to use something else" and I will likely be heard and respected.
Idk about the abstraction problem, but one thing these tools probably solve is inconsistency. Like you said, if everyone is forced to use the same tool, either the entire codebase breaks or nothing does. And that's what we want, deterministic behavior
So, you prefer having your entire infrastructure breaking rather than pieces of it? Just so you can say you have "deterministic behavior"? By itself your argument makes no sense. You don't won't your entire infrastructure to break, that's why we don't use mainframes anymore and we use "the cloud". Because the cloud is MANY different infrastructures working as if its one, but in reality it's just many data-centers behind the scenes. Many different infrastructures where if ONE breaks, you won't have a massive issue, probably not even a single data loss.
@@exapsy my friend, it is management. Good luck having to fix the same thing 100 different times at 100 different places. If you have one centralised dependency, you fix one and you're done.
I like the distinction between good abstraction and bad abstraction. If I wished I could just do it the way I wanted to rather than the way the abstraction enforces, the abstraction might be counterproductive.
One of the big differences is how leaky the abstraction is. C isn't leaky - you almost never need to know how the assembler works to use C. Borg and Blaze aren't leaky, because everyone uses them and releasing a new version is a big deal, and because everyone uses them the documentation is complete. It's when you get some half-assed ORM database layer where you have to look at the generated code and perhaps even clone and modify it to have it work for your solution that the abstraction becomes problematic.
I work as COBOL application programmer. Reading source code to understand how programs work is my daily routine (obsolete or no documentation). Static and dynamic analysis of assembly code is also something that happens from time to time. I'm not mad about this though, because I know it's something that makes my job secure. There are not many people that are capable of solving such problems. Imagine being that one guy responsible for one of internal Google tools. You are not likely to get fired.
The CEO and CTO of the startup I worked for came from AMD. He was obsessed with in-house engine, in-house source (even if it was just renamed from open source) and complete control, even though there was better open source available. Instead of focusing on our own technology, which wasn't open source. He wanted to do everything in-house, even invent the wheel. I thought it was just him, but from the comments here it seems to be a systemic problem in a large organisation, not just a few individuals.
They want to entrap their devs into proprietary garbage so 3 years later they're not employable anymore and can't use google knowledge to get hired elsewhere. It also serve as a barrier to entry for anyone trying to study or copy their garbage tech. It also increase the shares value when you have intellectual properties, even if they suck asses.
I experience this too. We have a library of generic frontend components that was developed by a team over several years. Now that team has scaled down significantly as no additional components are being made. My team discovered that none of the components supported right-to-left languages which we required, and adding the support was deemed too big of a task for the small maintenance team. So now we have to manually hack rtl support into all of the components we use, which has a high chance of breaking if we update the version of those components. It sucks.
If RTL was that important, surely someone could find the time to get the work done-like moving over to the other team to help out. It doesn't make sense to me to not concentrate effort where it's needed and sounds like a siloed company.
@@deallocThe challenge is resource allocation. Companies large enough to be siloed have crazy mobility to manage. There are too many moving pieces. When things get shifted too far, the business goes to hell pretty quickly. Consider that every single team is already running behind schedule. And every team is staffed for maximum productive utilization. So if a team loses a head, then they are short 40 hours of work every week at minimum. That’s work that gets picked up by already burned out people. When you rob Peter to pay Paul, it creates a domino effect. This week the tool maintenance team an extra head. Next month the team you poached the head from will be so far behind that THEY will need to poach a head. And the cycle continues. No. The strategic approach is to hold team managers accountable for allocating resources as per the scope of their delivery needs and to ensure they have adequate headcount at all times.
IMO, one of the main reasons that Google is struggling in the cloud market is that they have (or had?) a framework (its name begins with an H) that would have been instrumental for testing their cloud offerings, but they allowed it to fall out of maintenance in much the same way as you're describing. This is what happens when you have a "no promotion for maintenance" policy.
I liked the point that you seemed to make about abstraction, that abstraction is only good if you (almost) never have to worry about the level underneath it. If you constantly have to peek underneath to solve problems or do things that the abstraction isn't good at then you either have a horrible abstraction or maybe the abstraction isn't made for what you're trying to do.
I am one of those internal tool makers. Not to forget the love that users give is also a big contributing factor to the health of these projects. There is nothing like seeing one of your users champion your tool, not because they have to, but because they choose to. I am eternally grateful to such people. That, my friends, is what it means to be charitable in programming.
I’m very thankful my 45 year programming career was at a small company where we used C and C++. The first 20 years were spent developing multiuser operating systems. Then compilers, Internet service software, and machine learning software.
If you read about Motorola and talk to engineers, the core issue was the code base was so massive that everyone just keep adding to it and working around problems. So your day to day job becomes finding ways to work around old designs, old technologies and not designing new code using new tools. Stay at a company long enough and you spend a lot of time patching old code and reverse engineering old code to run on new hardware. Along came Apple with the iPhone (Moto said they were promising something that couldn't be done in the early stages) and they were blind sided. Their attempts to create an iPad/iPhone equivalent was comical and embarrassing. The same is true for Google now, the legacy infrastructure now gets in the way because you just can't sunset things that easy (it's tough). RSU's are viable because of growth and wall street allows a portion of that growth to be bleed off to support the employee's but when that growth stalls, it all fails and people that require RSU's to pay bills are stuck. It's going to get tough for engineers to live in CA without the RSU's. I don't see AI being able to solve the energy problem anytime soon since fusion is 35 years away:)
The Motorola Droid line were fantastic phones- and the last true smartphone to be made in USA. Not sure what you’re getting at. The quality and performance is superb. iPhone succeeded because of first mover advantage and sleek design. Not because of codebase.
lmao it is like that anywhere in large corporations. we have so many small repos that 1 person maintains and a ton that nobody knows anything about. but we have lots of externals
The assume the tool-"tick" a consideration from employers view perhaps a bit more than from an technical or engineering view. Abstraction layers can hide knowledge which they want to keep secret or limited to relatively few of their developers at least for some time. Abstraction makes developers work results less individual and allows to swap positions or replace staff more easily. The advantages for infrastructure issues had been already mentioned in the video and are technical reasons.
Google should spend more time hiring and firing tools. Treat each tool like a business entity. Does this tool constantly steal time from our engineers or give them time to be more productive.
Abstraction is the opposite of concretization, which means hiding the details. Abstraction is meant to make programming easier, but when you need to do complex stuff it just doesn't work until you gain more control by going lower.
@@Leonhart_93 But you do go lower. You have concrete abstraction generators. When you are working on the code to generate the abstraction, you worry about how it gets created. When you are utilizing the result of that work though, you just use the abstraction. Things like interfaces are a form of abstraction. We use abstraction everywhere. Abstraction separates the how, the why, and the what.
@@fieldmojo5304 I meant that by using high level languages there is no lower abstraction to go with, unless you switch to working with a lower level language, which is not always possible when working on existing projects.
Google does have an internal Stackoverflow and it's called YAQs. As for ad-hoc non supported tools, wise engineers avoid them. Google's problems is the gazillions of legacy systems that was developed 10-15 years ago and the current teams struggle to maintain them. Another Google problem is the bureaucracy that kills productivity, and the focus on social virtue signaling instead of innovation and excellence. Google became a sad place.
My god we have the same problem at my big tech company. Everyone has to create some abstraction for a common industry tool that’s riddled with bugs just to look good on their performance reports and say they created XYZ! Then it loses maintainers once they or the team moves on but we are stuck using it 🥴
Pro tip after programming for 45 years: simple, straightforward code is more reliable and much easier for you and others to maintain. Write clean code that is easy to read. Let the compiler optimize it.
This is called Impact Driven Development where devs focus on the promotion to the next level rather than real problems. Unfortunately this is how the game is played, you don’t need expertise you just need a strong promo doc.
honestly the solution for most of these problems is that google should have opensourced these tools rather than keeping it internal. promotion should only happen after you open source it
I feel kind issue applies to non internal repos as well, it basically comes down to the fact that most codes written weirdly by promotion seekers tend to be abstract and not easy to understand
In my 250 person company I am indespensible as I am one of the only programmers on staff, so I have literally built these kinds of bespoke tools and systems - but it is fascinating to me that Google has the same issue - I would have thought they wouldn't allow single points of failure like that.
I've been using your videos to prep for an upcoming interview. Thank you for making them. The video combined with the comment section have been very useful.
An other problem for me in a big tech company is that they have their own framework. Then the only thing u learn with them is their framework. U won’t learn new technologies in big tech company u will be stuck with their framework that nobody else use.
I can't see how that's a big problem. Frameworks can be learned quickly in most cases once you know what you're doing. It might take a couple of months, but when you come from Google and get a good onboarding at a new company that's hardly an issue.
@@benfrese3573 That's not true. You'll probably take a couple of months to learn most basic things and then couple years to become really good with it. Why would I hire ex-google dev that's going to perform on a level of a junior dev?
2:09 If that person worked at Google, they better have stellar clean code. But they probably don't, because leetcode questions are all about being a messy, tinkery, shit-eating grin programmer. It's all about being a mad tinkerer instead of a clean craftsman.
When I write a tool, or get requested to write a tool at my company, the initial question I always ask is "is it removing monkey labour". if it just aggregates a problem into some new issue, don't waste money and time making it. The tools should always be well documented, also tools in my oppinion should have code docs on top of that, just for the reason you said, the guy moved on and now it's someone else problem = it's an aggregated problem. Also I like when you don't create monster monolith tools, but smaller, and let the build system or what ever call them, scales better and also, that's what build systems normally do pretty well.
Another undortunate thing about working at Google is that there's little knowledge carry over to other jobs because a lot of tools are very Google-specific. At least Kubernetes is used across the board in industry. Borg is only used at Google and nowhere else.
That's funny, I always fight to choose the best tool for the job and it rarely happens that the best tool is the one you created. And, yup, for Google it probably can happen, but as you mentioned - only in widely used tools like those related to infrastructure or building, for smaller companies it virtually never happens.
At Amazon, not only are there internal tools specific to orgs, the people who created those tools have become SDE3s and have immense power at forcing you to use their internal tool. So much for customer centricity 😂
To be clear, there is assembly code (still "human readable") then there are the ones and zeros of machine code. Some compilers do indeed emit assembly code, which is then assembled by an assembler. I could be wrong, but I think the majority of them emit machine code in object files, which are then linked by a linker to produce an executable. When things go boom, or you didn't write (or have access to) the source of the code which went boom, then you have to deal with the machine code. I kind of agree, it's a slog to have to fix somebody else's code, but that's often part of what IT people are hired for.
Yup, that’s right. He (along with many modern programmers) doesn’t know that because he was hired to fulfill a diversity quota, not because he actually understands computer science.
Generally agreed, but regarding C, asm & abstraction: when you're optimising bottleneck pinchpoints, dipping into the compiled asm is very common. Inserting inline assembler, unrolling loops, avoiding pipeline stalls, and minimising load/store by maximising register usage is far from uncommon.
As someone coming more from hardware side, I've used assembler this way before. In retrospect, however, it's usually an indication of having chosen under-powered hardware for the problem at hand. If you're writing C for modern complex processors, you should probably only use assembler if you're in the process of developing a new variant of that processor, writing a compiler or simply feel curious about how stuff works under the hood. That last option is 100% valid, but it's all about learning instead of being productive at that very moment.
Those problem almost exists right now in almost tech company right now. The problem is that the quality of the tools really varies upon the devs knowledge with the language/problems in the first place. Sometimes, it really shows the "exp" & "growth ladder" problems in the company/tech team.
Ha as a hardware engineer, we also script somewhat. Sometimes we got a genius writing one flow, that every project use. They either left or moved or whatever and that hole is left unfilled. I certainly can't go in and start rewriting the code. So our company learned and made sure we have as little custom flow as possible, everything is industry standard tools, let those company handle code/model updates, we just use the tool. In the end, spending a little money and time now, avoid long long delays down the road.
This is the same also with non-internal tools that are not widely adapted. For example, a small open source project that is really useful, but then it takes forever to have small fixes etc. as there is only one or couple of people maintaining it. But then again every tool has to start somewhere. Sometimes there internal tool are released "into the wild" and everyone loves them.
I actually coded a serial communications protocol 'handler' in asm from a C version, as I had to put it on a Z80 card from a PC environment ... I left the C comments and code as comments in Z80 source, and added the appropriate Z80 assembly instructions to accomplish each C code statement.
I don't agree with the part about avoiding a tool and just creating it yourself. It is essentially the same thing the next person coming in will have to disect your code and figure stuff that you did. Instead I would say falling on open source tooling because then support and documentation is available. But I guess that's not possible there?
Imagine this at a company that programs primarily in a proprietary programming language. It was tough and took more time but eventually in the end it taught me you can find answers if you look hard enough through the source code.
Imagine when you figure out that you'd spend less time if you've just used YOUR tools, or other tools instead of fixing that shit because that guy didn't keep his code updated. IDK, let's say 30days vs 3hours, that would be gross.
Abstraction should be thought of as the same as a car part within a car engine. Just a black box you have a few controls in to do stuff. Purposely incredibly simple.
I think this problem is commonly faced by people working in large corporations. Someone would abstract the effort, but no documentation and no maintenance. And people who join later suffer and scratch their head understanding what is happening.
Have the exact same issue at this tiny startup I’m working at. The CTO is obsessed with building his own abstractions which for the most part do help us get stuff done quicker, but it’s so complex and over-engineered with zero documentation that as soon as something goes wrong (which it often does) he’s the only person who can understand and fix the bloated mess.
question: did Google have restrictions that prevent you from fixing the tool? I’m ok with having these tools. I have written many myself, some still in use at other companies I left years ago (I keep in touch with old teammates). If a tool or utility I wrote needs to be updated, etc… they just did it, why would they re-write something if all it needs is a new feature, or update? Might there be a little “no room left for me to write my own tools from scratch/ “not invented by me” issues going on here? If, on the other hand, some strange Google policies or process prevent you from contributing to the tool code, that’s unacceptable in my opinion. I can’t imagine why they would do this, I have never heard of this happening there.
We have are own ui library abstractions in out company. To use it, the security implementations we have to do is insane just so our application can download the css and js files for that library during runtime. I am part of a team (only member now as it was scaled down multiple times) that abstract over multiple saas vendors used by our company. Not only is it literal hell to maintain as a single dev as multiple people face issues all the time, but the thing is that I am not a direct POC for them. So if they face an issue, by the time it comes to me through various support teams, its already over or near the expected deadline. Since mine is not a product company, I have to justify my work in 30 minute timeframes. So sometimes I get loads of requirements for new implementations for the tool and work over time, while sometimes I dont get much immediately as it gets pushed to later sprints and have to hop on and take work on other applications to justify my time.
This is very common issue and i hate it, they have internal tools and dont even have proper documentation and if any issue comes up, you have to own regardless you want or not and fix it.
As a non-programmer but tech enthusiast and IT guy that has done tech support for years, I absolutely hated propietary software that did things that other non-propietary software did, and often better. Mainly because taking the time to learn to use their software and their quirks, was NOT transferable knowledge so it was a waste of time. I particularly hated my experience with the Sabre GDS while working there, mainly because it's very complex and can even be "fun" learning it, but then you can't really use it anywhere else except if you work in the travel industry. You also can't practice it on your own time at home if you don't work for them as there is no way to access it unless you're in a corporation. I have other examples like this, but I don't even remember the name of the software used in those companies (was also incredibly terrible).
An engineer’s job is to bridge the gap between the requirements and the desired output. How it can be done varies greatly from one eco system to another and that’s why engineers at big Tech pay big bucks, it’s not because they know XYZ technologies, it’s because they can be an expert in those technologies fairly quickly.
No its because their willing to slog through decades of soydev bullshit. Just mixing around labels and not really making it simpler, is not actually a "technology". Your not making anything faster lmfao. Almost everything software wise developed in the past few years, has been to code faster. Coding faster is not a technology believe it or not. So you get slower shit, thats not even faster, is 10x more complicated for no good reason, and your forced to use it cus some midwit worshipped overcomplicate.
Modern programming abstraction and WordPress (and the server infrastructure to support it) are all Planned Inefficiency to generate employment and the modern tech ecosystem. It's not all bad.
That's not how this works. If something really isn't working or isn't optimized, you should look for the bugs in the assembly level as well if you can reduce a single instruction from your code that will always help you in writing better faster code. Also, if you think you can maintain the code yourself, do it it may not pay you but it is doing good for the community that one should prefer
Bazel / open-sourced blaze is one of the worst build systems I’ve ever seen, and this was a solved problem. Google just wanted to keep their entire company’s source code in one place, and thought it’d be easier to create a build system from scratch to handle that than to bother trying to break things out. I’m like… what.
It's somewhat disturbing to see that a tech company as massive and complex as Google has the same kinds of problems as the mid level be bank I worked at.
SAP has this problem. In the 80's and 90's it was pretty muchg a case of the core application code that you were interested in was just under the skin and you could usually solve something in minutes by debugging. Then they got into OO concepts and started abstracting the basics into 'frameworks'; badly and just for the sake of it. Bang, now it takes half an hour of debugging until you find a code reference that looks like it might be using a concrete object (which then starts using abtraction in it's own component layer). TCO for customers wanting to enhance the product went skyhigh. It's probably great for them, internally, as frameworks do accelerate things, but as stated in the video, this needs a lot of knowledge and documentation. SAP are really bad at technical documentation. As a customer you are pretty much f-ed if an error happens in these frameworks now, your average SAP coder will flounder and experienced problem solvers like myself don't even want to go there.
0:42 Every compiler I've used at one point I've had to delve into the assembly to find a compiler/optimizer bug and figure out how to avoid triggering it.
Was just about to comment the same thing. Compiler optimizations can and will be the source of bugs which can only be properly diagnosed by examining assembly.
"i'd rather built a solution from scratch"... basically you want to be that guy that wants to get promoted and creates an internal tool to solve a problem
With 'abstraction hell' I associate too many layers of abstraction. 'tool hell'/ 'custom software hell' is what I'd rather call this. I'm guilty of doing this too. I cannot solve the problem. Either I try to automate parts of my job or have to do things manually. I can't change it. If someone else has to use my work I'm aware it might be a miss or hit thing and my work might end up in the bin. That's how life is sometimes. I still try to do my best, document and test my own stuff. But you need feedback from others to improve things. The feedback is often missing.
Not just big tech companies.. I work in a small one and they insist on reinventing all the wheels, which sucks because it means you learn how to use all their stuff which is useless in your next job
I think that something you haven't touched on (and maybe this is a clip from a longer stream/video) is that many times this abstraction is a result of tools that were started but were not finished, either because of the developer, or lack of funding, i.e. the level of abstraction was but one piece of a larger picture that was intended to be built, but organizationally could not get off the ground. we have many tools that are partially useful sitting around. curious -- did you feel like you were forced to use these tools with these levels of abstraction, or could you just solve your problem the way you chose (or remove the existing abstractions with standard issue/lower level solutions) also, sometimes a "local abstraction" (i.e. one you've built to solve an issue that gets on your own nerves over and again) can end up being taken on by a coworker, or team, and spread, becoming the defacto standard. in these cases, it can be annoying if you're on the other end of this in that as you take on the abstraction, only to realize that you actually have a better way to solve it with little effort at that lower level that makes the abstraction you originally leaned on a burden rather than a benefit. they say naming things and cache invalidation are the hardest issues in software dev, but i also posit choosing a level of abstraction given "software destination" or schedule is another toughie. The larger the company, the larger the errors in these decisions are multiplied. many times the core issue isn't that people are not just idiots or malicious (not saying you're saying that, but i get that feel from comments often), but that these things are actually genuinely sometimes complex issues that take some thinking, and cooperation, to avoid....heck someone could just be having a bad day and choose the wrong pattern, compounding their issues. which is also why review is useful.
I know that I know almost nothing about working in bigtech, but for me it could be a hell only in case of time limit. So if I debuged, fixed, wrote docs and tests without any pressure by deadline/manager/colleagues it would be ok for me to have like a small rest for few weeks from another work.
So why Google has all this internal tools to begin with? They run faster than their open source counter-parts? Other reasons? I prefer open and well documented technologies too.
I can't speak for Google. But at our company, we've had to create new tooling when what we needed didn't exist or was sufficiently lacking and it was faster to roll our own than modify existing open source. Then time passes and the open source tool gets better. But by that point, the effort to switch to the open source tool is too expensive.
Fxxk yeah!! I can't agree more, even I was a TVC there, I can still understand. It was a bittersweet story Luckily, I had opportunity to access those internal tools (no one but me in my team?! division?! corp?!) Unfortunately, I must study whole the related fxxkin internal tools before I started to complete my assigned task And because I was a TVC, that made thing even harder, how I was wishing it's just public tools with public names, so just stackoverflow should work it out
I personally love to create my own tools every time and have never used any 3rd party in any personal project (unfortunately I haven't used my own programming language). But honestly, I don't see any difference between debugging an external 3rd party or an internal 3rd party. Generally, debugging someone else's code sucks.
As a former Amazon engineer, I empathize with you. Another annoying thing about using these internal tools is that the skills you gain are largely non-transferable; you are only able to apply the knowledge of using that tool to one company.
I think that's the whole point of these companies forcing "internal tools" on you... to lock you in as their (well-compensated, but immobile) labor slave.
Yes, and it seems like one becomes more of a configuration engineer than a programmer.
@@clray123 I doubt it. They have no need -- the people there will have no issue learning other tools. I never felt immobile during my time there, even after wasting time fiddling with internal tools. And they don't have much trouble enticing engineers to come to work for them. These big companies just want to have full control over everything in their domain, and in theory it makes sense. In practice, it is great for tools that are maintained well, and terrible for all of the ones that aren't.
@@bbydykdyl7779 Clearly you have not spent much time working with those internal tools. Of course proprietary tools and platforms cause lock-in of users (and when it comes to developer tools, of developers). The whole business model of Microsoft revolves around it.
[...] largely non-transferable. Answer: that's exctly what it is meant to be.
No documentation? 😂
Funny this big ass company has the same problem as a small one.
documentation is costly
It's worse than a small company's problem. In a small company, the distance between the library's author and its users is orders of magnitude smaller.
you have proprietary internal tools in your company?
Smaller companies have the same egos and gatekeepers working there...
The code should be the documentation. Writing documentation is a waste of time, unless it's for the end user. Consistent and smart code structure is always the solution. All you need it PR be reviewed by senior employees, and code quality maintains. Sometimes redundancies appears and need to be refactored, but even so the code will be understandable and testable so you won't take too long getting to know it.
i work at IBM as a developer of operating systems and its exactly the same here, we literally use our own inhouse programming languages for development, scripting and testing and if we do use a well known public programming language, we actually have our own implementation of it and its compiler. We dont use Oracles java and jvm, we have our own implementation of java and jvm :) we have our own implementation of C as well. The hardware is our own, the operating system is our own, the languages we write it in are our own, known languages implementations are our own, EVERYTHING, literally the only thing that we haven't implemented ourselves internally at IBM is stuff like git and Jira, lmao.
Why do you guys do that . Could you explain the advantages and disadvantages of that ?
Why not just use an open source tool like everyone else? Cheaper and ready to go right ? Or is it a security issue why you guys don’t use anything else other than your own? Hmm, quite curious.
@@weiSane we had our own version control system too, just recently we started migrating to git. The operating system is our own because the computers are our own, System Z.
@@weiSane Not an IBM dev, but I have worked for large companies. It's corporate inertia, high retraining cost, sunk cost fallacy, believing that because a tool was made for a specific purpose means it's the best at that, and not wanting to spend the time/money doing even the most basic customization or setup of 3rd party tools.
Let me give a real example. It's common practice in some firms to have everyone using virtual machines that are reset to baseline every time they shut down, including monthly patching. At one point, I spent over a year using Visual Studio with 9Gigs of RAM, Microsoft's Perforce alternative VCS, and Visual Basic .NET Framework code living on a network share. Why? Because policy says everyone is treated the same. From the call center agents making terrible hourly pay to the salaried developers to management. This was post 2020 too...
This is why large companies pay so well. Partly because they use such esoteric or old tools that finding devs who have every used them is hard. Partly because otherwise people would go to jobs that aren't stuck a decade or more in the past.
Prevent employee experience from being fully transferrable, lock customers in to IBM systems.
Prevent employee experience from being fully transferrable, lock customers in to IBM systems.
This is exactly true (at other companies also). Promotion = make something new - not "save other people time in the long run." Incentive problem.
Reminds me of roman society wher you had to conquer some new land to get respect
The typical google conversation was "Well, I got my promotion." "For what project?" Like, what did you invent that got you promoted?
Hot take: A tool isn't complete unless its documentation is complete.
The developer shouldn't have been promoted until their tool was completed. And at the very least the tool shouldn't have been implemented as a standard until it was documented.
And if the tool is still used (and I assume maybe even critical to some parts of the business) it should not be left unmaintained. Someone should have to come in to replace the original creator
this feels like a lukewarm take at the warmest
No not for me. The creators of internal tools have come across a problem many people in the company have faced but they have decided to do something about it instead of pretending it doesn't exist. They create working code that solves the immediate problem. As soon as that code enters the internal domain, the creator should not be burdened with maintenance. The initial users need to contribute from the start. When they start using it and they find something was missing from documentation they need to make a PR to get it documented to help the next person. It would not have been that person's day job to create the tool, and they unlikely ever received formal work allocation to make or maintain it. They most likely did it unpaid. They should be promoted because they saw a problem and they took ownership. People that are expecting to just lift others work with no input themselves need to go back to reading the Little Red Hen. Usage is an interesting one. It might be prudent to groom the tool registry periodically. There will be tools with feature overlap or tools that have fallen out of use altogether. It might make sense to remove these from the registry as often a rewrite will be faster than a patch when the code is several years old and the original dev has moved on
There is no such thing as a complete tool in IT industry. Yesterday it seemed to be complete, today it's not: new user experience, new versions of the underlying libraries, etc. It's an institutional problem, which does not seem to be addressed by their management dickheads who only seem to be able to push everyone to using shitty tools, but not to ensure its ongoing maintenance.
its not a hot take. they literally teach this in software engineering 😂
We have this same issue at Amazon. Documentation sucks. I try very hard to avoid as much internal tooling in my design. Its very hard though, cause folks are so brainwashed with these tools / traditions, they can’t think outside them.
It's*
Having worked in small companies and startups and switching to a big tech company, saw a similar issue. Much of the time the big tech company wants "control" over a solution instead of using first party and community open source tools for a given problem. I was much more productive working in small companies and startups that let me choose the best tool for the job instead of relying on internal tools.
NIH syndrome
I worked at a FAANG and can relate. It is of course part of the job, but at a certain point it becomes completely unsustainable and you burn out. The entropy just increases non-stop. The biggest nightmare is when you need to CHANGE those external tools and there is no test suite.
That's why it is so refreshing to work on personal projects.
I can't imagine working at a FAANG. I had a phone screening interview with Google after I finished my PhD that went very well and they wanted me to come to Mountain View for an interview and would check in with me every few months to see if I was interested... I never took them up on that offer:
I ended up working for a non-profit science organization with excellent benefits that places the health of its employees above all else, and most of the software team there has been there for a minimum of 15 years. The work is challenging and incredibly fun and best of all, we get to know we're contributing to human knowledge of science. My manager is amazing, my coworkers are some of the greatest people I've ever had the pleasure of working with, and I love it. Oh, it has also afforded me both the luxury of living in Chile for eight years and in Hawaii for three years, working mostly in functional programming, which I hated at first but now love. I feel like I really hit the jackpot with this career and am extremely grateful for it.
I think if I was stuck in a FAANG, I would burn out so quickly.
The point about what makes personal projects so appealing is very insightful
Not only that, but working on personal projects that have nothing to do with work subjects is awesome. I work on logic systems all day, but 3d/vision processing all night. Also at work the code doesn't physically touch the real world, but at night??? Arduino, Pi, AtMel, mmmmmm.... bring it on...
It's all about perspective.
I'm working at a medium sized US company (not many internal tools), there is stress and burn out and stuff, and then the full-scale war against Ukraine has started - work has became the best way to distract from all that horror!
Stress at work? - Now it's so low that it's pleasant to have some urgent issues everyone nervous about - I don't worry about it a bit! Stakes are so low!!! 😁
Oh, and talking (and even listening) to people who is not in constant stress is quite soothing - it's just great! Seriously.
Now I take only short vacations (1 week), so I wouldn't have too much time thinking about what is going on and who said what about it.
No fear of death, only fear to loose opportunities the life provides and to not be effective and die in vain if I would be drafted.
Don't like increase of entropy - either find a way to deal with it or move on. Maybe it's time to move to another position.
Do what YOU want!
As an Xoogler, I 100% agree with this. My first job was figuring out how to purge the old records in some program's database, and there was like three layers of abstraction between having the data in a protobuf and stuffing that protobuf into the database. I kept thinking "Is everything here really that complex?"
It takes about a week to figure out how to compile and run your program, a couple weeks to figure out how to deploy it, and a couple months to learn all the bits you need to (say) write a Hello World program from scratch that can be accessed from the internet.
And yeah, every library was google-specific. You didn't use the file system, you didn't use memory allocation, you didn't use the debugger, the compiler, etc etc etc.
I worked for a small startup that had an abstration hell. Its not really a big company problem, this is just human nature. We want to see patterns in everything and we create abstractions for that reason. I truly believe this has to do with how our brain works.
It's a matter of not admitting the problem in its entirety, which results from trying to get easy bucks. Usually managers push for using tools because these decrease the delivery costs as per their tiny minds. Would they drive a model of a car which does not have a single car service station in the country or has been designed to drive only on a few roads known to the constructors at the time of design? Probably they would walk away from the car showroom at the moment they hear about absence of a warranty period. But their brains suddenly start working differently in zero accountability conditions.
Not really. It's just a disease of junior programmers, spread by academicians with no practical experience promoting OOP and "clean code".
In my small company it was an explicit design goal, set 15 years ago, to avoid advanced features of the programming language (using only a subset of syntax) and to also limit custom abstractions because most actual valuable functionality boils down to (1) executing some SQL queries and (2) manipulating some simple data structures. The idea was to make it easy for anyone to take over the codebase and to understand what is going on without specialized training (this also encompassed rejecting dependencies on external libraries and useless ideas such as object-relational mapping layers). This has worked out pretty well overall, the product is still there and still has simple structure. Although maybe sometimes you have to type a bit more to implement a new feature than if you had some glorious abstractions in place. Overall, it's a good tradeoff for productivity and makes developers feel in control (although bored).
Also, it's way more fun to work on a general framework than it is to work on an actual solution to a specific problem.
@@darrennew8211 It's more popular to plug some half-fillting third party framework because it "saves effort". Then becomes one hell of "fun" to debug it and make it work for a specific task...
It has NOTHING to do with how our brain works. Its just called being a soydev.
Your just overcomplicating shit that doesn't need to be complicated.
I doubt there is a place without abstractions, the only difference being whether they are known third party tools or something internal, developed due to lack of alternatives in the industry.
Yeah but.. have you seen how exaggerated it is at multinational corporations. It’s something else entirely
An internal abstraction is fine as long as it was the correct abstraction made after observing the need for one, vs an abstraction made preemptively
@@fr5229 i second that, it's one thing to use "abstractions" and yet another to use internal tooling that some dork used to get promoted, i've really seen that happening and i really wondered how those managers really let that happen, it makes no sense in the long term, and eventually they get abandoned.
There is no ‘correct’ abstraction only the correct abstraction for now. And Google only has two kinds of tools: experimental and drprecated
The difference is whether you can or cannot get support. No tool can be built once and for all, it has to be developed according to newly discovered scenarios, get its bugs and oversights fixed, migrate to new versions of underlying someone else's implementations. So if it's just left in a vacuum ("oh we are Agile, which means no responsibility and not giving a shit about customers"), it becomes a net loss. Frankly, funky places like Google inevitably turn into a hell, there is no chance they could avoid it. That's the other side of the coin of buying developers with a false sense of freedom.
google docs for their apis really suck, can't imagine internal ones
Couldn’t agree more
It depends entirely on how many people use the tool. The documentation for borg and blaze are complete and readable, because every single developer has to use the same version without change. The documentation for some half-ass internal ORM sucks because the developers wrote it for themselves, then someone else found it useful, and suddenly it's the go-to mechanism as enforced by the guy five levels up the management change of the guys who wrote it, because he wants credit for creating a tool that lots of people use.
The power plays between management dictating which tool to use are half the problem. I once was told to evaluate the three possible databases we could use. I picked the first one, after several days of research, with a list of pros and cons. The boss's answer was "Well, would it be *impossible* to use the second one? Because boss's boss's boss wants that one." Like, why did you even tell me to research it?
You described the problem as abstraction, completely skipping over the actual problem:
"Sometimes you are forced to use the tool"
That's the actual problem. You can't get a sense of autonomy, mastery, and purpose working as one tiny cog within a big tech organization with tens of thousands of engineers.
Terrible internal tooling and libraries and terrible legacy code that your work relies on is a problem at all organizations, even the small ones. And "abstraction" has really nothing at all to do with it, by the way; I've seen plenty of code bases where the problem is low-level procedural code that has been copy-pasted all over the place but is really concealing concepts that need to be abstracted.
The difference I have found at small companies and start-ups is that I can have a bigger impact on the direction, I can say "this core code is no good, we need to rebuild it", or "this tool is no good, we're going to use something else" and I will likely be heard and respected.
Idk about the abstraction problem, but one thing these tools probably solve is inconsistency. Like you said, if everyone is forced to use the same tool, either the entire codebase breaks or nothing does. And that's what we want, deterministic behavior
don't talk out of your ass
Meh, consistency is overrated. Over-indexing on consistency is a symptom of small minded, myopic thinking.
So, you prefer having your entire infrastructure breaking rather than pieces of it? Just so you can say you have "deterministic behavior"?
By itself your argument makes no sense. You don't won't your entire infrastructure to break, that's why we don't use mainframes anymore and we use "the cloud". Because the cloud is MANY different infrastructures working as if its one, but in reality it's just many data-centers behind the scenes. Many different infrastructures where if ONE breaks, you won't have a massive issue, probably not even a single data loss.
@@exapsy sure it's easier to break, but also much easier to fix in one go. it's the static vs dynamic linking dilemma
@@exapsy my friend, it is management. Good luck having to fix the same thing 100 different times at 100 different places. If you have one centralised dependency, you fix one and you're done.
I like the distinction between good abstraction and bad abstraction. If I wished I could just do it the way I wanted to rather than the way the abstraction enforces, the abstraction might be counterproductive.
One of the big differences is how leaky the abstraction is. C isn't leaky - you almost never need to know how the assembler works to use C. Borg and Blaze aren't leaky, because everyone uses them and releasing a new version is a big deal, and because everyone uses them the documentation is complete. It's when you get some half-assed ORM database layer where you have to look at the generated code and perhaps even clone and modify it to have it work for your solution that the abstraction becomes problematic.
I work as COBOL application programmer. Reading source code to understand how programs work is my daily routine (obsolete or no documentation). Static and dynamic analysis of assembly code is also something that happens from time to time. I'm not mad about this though, because I know it's something that makes my job secure. There are not many people that are capable of solving such problems. Imagine being that one guy responsible for one of internal Google tools. You are not likely to get fired.
Out of curiosity how old are you?
246 years @@zirkereuler5242
The CEO and CTO of the startup I worked for came from AMD.
He was obsessed with in-house engine, in-house source (even if it was just renamed from open source) and complete control, even though there was better open source available. Instead of focusing on our own technology, which wasn't open source.
He wanted to do everything in-house, even invent the wheel.
I thought it was just him, but from the comments here it seems to be a systemic problem in a large organisation, not just a few individuals.
They want to entrap their devs into proprietary garbage so 3 years later they're not employable anymore and can't use google knowledge to get hired elsewhere. It also serve as a barrier to entry for anyone trying to study or copy their garbage tech. It also increase the shares value when you have intellectual properties, even if they suck asses.
I experience this too. We have a library of generic frontend components that was developed by a team over several years. Now that team has scaled down significantly as no additional components are being made.
My team discovered that none of the components supported right-to-left languages which we required, and adding the support was deemed too big of a task for the small maintenance team.
So now we have to manually hack rtl support into all of the components we use, which has a high chance of breaking if we update the version of those components. It sucks.
If RTL was that important, surely someone could find the time to get the work done-like moving over to the other team to help out. It doesn't make sense to me to not concentrate effort where it's needed and sounds like a siloed company.
@@deallocThe challenge is resource allocation. Companies large enough to be siloed have crazy mobility to manage. There are too many moving pieces. When things get shifted too far, the business goes to hell pretty quickly.
Consider that every single team is already running behind schedule. And every team is staffed for maximum productive utilization.
So if a team loses a head, then they are short 40 hours of work every week at minimum. That’s work that gets picked up by already burned out people.
When you rob Peter to pay Paul, it creates a domino effect. This week the tool maintenance team an extra head. Next month the team you poached the head from will be so far behind that THEY will need to poach a head. And the cycle continues.
No. The strategic approach is to hold team managers accountable for allocating resources as per the scope of their delivery needs and to ensure they have adequate headcount at all times.
IMO, one of the main reasons that Google is struggling in the cloud market is that they have (or had?) a framework (its name begins with an H) that would have been instrumental for testing their cloud offerings, but they allowed it to fall out of maintenance in much the same way as you're describing. This is what happens when you have a "no promotion for maintenance" policy.
I liked the point that you seemed to make about abstraction, that abstraction is only good if you (almost) never have to worry about the level underneath it. If you constantly have to peek underneath to solve problems or do things that the abstraction isn't good at then you either have a horrible abstraction or maybe the abstraction isn't made for what you're trying to do.
It's called a "leaky abstraction" and is often worse than no abstraction at all.
I am one of those internal tool makers. Not to forget the love that users give is also a big contributing factor to the health of these projects. There is nothing like seeing one of your users champion your tool, not because they have to, but because they choose to. I am eternally grateful to such people. That, my friends, is what it means to be charitable in programming.
There’s also the internal culture of “if you can fix it, go ahead” (supported by code reviews).
Yuuuuup.
Diversity hire thinks he was hired on merit 🤦♂️
Creating a new internal project in order to get promoted, just so true and so common in big companies. The promotion process is the real cause of it
I’m very thankful my 45 year programming career was at a small company where we used C and C++. The first 20 years were spent developing multiuser operating systems. Then compilers, Internet service software, and machine learning software.
This is how it is in Oracle as well. I'd imagine most big tech companies as well.
If you read about Motorola and talk to engineers, the core issue was the code base was so massive that everyone just keep adding to it and working around problems. So your day to day job becomes finding ways to work around old designs, old technologies and not designing new code using new tools. Stay at a company long enough and you spend a lot of time patching old code and reverse engineering old code to run on new hardware. Along came Apple with the iPhone (Moto said they were promising something that couldn't be done in the early stages) and they were blind sided. Their attempts to create an iPad/iPhone equivalent was comical and embarrassing. The same is true for Google now, the legacy infrastructure now gets in the way because you just can't sunset things that easy (it's tough). RSU's are viable because of growth and wall street allows a portion of that growth to be bleed off to support the employee's but when that growth stalls, it all fails and people that require RSU's to pay bills are stuck. It's going to get tough for engineers to live in CA without the RSU's. I don't see AI being able to solve the energy problem anytime soon since fusion is 35 years away:)
The Motorola Droid line were fantastic phones- and the last true smartphone to be made in USA. Not sure what you’re getting at. The quality and performance is superb.
iPhone succeeded because of first mover advantage and sleek design. Not because of codebase.
lmao it is like that anywhere in large corporations. we have so many small repos that 1 person maintains and a ton that nobody knows anything about. but we have lots of externals
The assume the tool-"tick" a consideration from employers view perhaps a bit more than from an technical or engineering view. Abstraction layers can hide knowledge which they want to keep secret or limited to relatively few of their developers at least for some time. Abstraction makes developers work results less individual and allows to swap positions or replace staff more easily. The advantages for infrastructure issues had been already mentioned in the video and are technical reasons.
>Everything in Google relies on something called "Borg"
>Google is assimilating the world
Indeed.
Google should spend more time hiring and firing tools. Treat each tool like a business entity. Does this tool constantly steal time from our engineers or give them time to be more productive.
google isn't known for giving long-lasting support on their own products. they deprecate them all the time
This isn't what I think of when I hear abstraction in programming. I think of decoupling data sources from the consumption, etc. Abstraction is great.
Abstraction is the opposite of concretization, which means hiding the details. Abstraction is meant to make programming easier, but when you need to do complex stuff it just doesn't work until you gain more control by going lower.
@@Leonhart_93 But you do go lower. You have concrete abstraction generators. When you are working on the code to generate the abstraction, you worry about how it gets created. When you are utilizing the result of that work though, you just use the abstraction. Things like interfaces are a form of abstraction. We use abstraction everywhere. Abstraction separates the how, the why, and the what.
@@fieldmojo5304 I meant that by using high level languages there is no lower abstraction to go with, unless you switch to working with a lower level language, which is not always possible when working on existing projects.
Google does have an internal Stackoverflow and it's called YAQs. As for ad-hoc non supported tools, wise engineers avoid them. Google's problems is the gazillions of legacy systems that was developed 10-15 years ago and the current teams struggle to maintain them. Another Google problem is the bureaucracy that kills productivity, and the focus on social virtue signaling instead of innovation and excellence. Google became a sad place.
My god we have the same problem at my big tech company. Everyone has to create some abstraction for a common industry tool that’s riddled with bugs just to look good on their performance reports and say they created XYZ! Then it loses maintainers once they or the team moves on but we are stuck using it 🥴
Sucks to not be a senior developer. Just deal with it
@@one_step_sideways 🤣
Pro tip after programming for 45 years: simple, straightforward code is more reliable and much easier for you and others to maintain. Write clean code that is easy to read. Let the compiler optimize it.
This is called Impact Driven Development where devs focus on the promotion to the next level rather than real problems. Unfortunately this is how the game is played, you don’t need expertise you just need a strong promo doc.
idk about google backend, but why does the frontend suck so much
especially with youtube, everything is buggy and laggy
Because they have to run the spyware code in client. Saves them on infra costs
honestly the solution for most of these problems is that google should have opensourced these tools rather than keeping it internal. promotion should only happen after you open source it
I feel kind issue applies to non internal repos as well, it basically comes down to the fact that most codes written weirdly by promotion seekers tend to be abstract and not easy to understand
In my 250 person company I am indespensible as I am one of the only programmers on staff, so I have literally built these kinds of bespoke tools and systems - but it is fascinating to me that Google has the same issue - I would have thought they wouldn't allow single points of failure like that.
I've been using your videos to prep for an upcoming interview. Thank you for making them. The video combined with the comment section have been very useful.
We might need update soon, good luck
Seems like these tools are designed to keep you stuck at the same employer. Who wants to hire a programmer that only knows some obscure internal tool?
LOL. Borg. Resistance is Futile.
An other problem for me in a big tech company is that they have their own framework. Then the only thing u learn with them is their framework. U won’t learn new technologies in big tech company u will be stuck with their framework that nobody else use.
I can't see how that's a big problem. Frameworks can be learned quickly in most cases once you know what you're doing. It might take a couple of months, but when you come from Google and get a good onboarding at a new company that's hardly an issue.
@@benfrese3573 That's not true. You'll probably take a couple of months to learn most basic things and then couple years to become really good with it. Why would I hire ex-google dev that's going to perform on a level of a junior dev?
@@cloudboogie You won't, because you have obviously no idea what you are doing :)
2:09 If that person worked at Google, they better have stellar clean code.
But they probably don't, because leetcode questions are all about being a messy, tinkery, shit-eating grin programmer. It's all about being a mad tinkerer instead of a clean craftsman.
When I write a tool, or get requested to write a tool at my company, the initial question I always ask is "is it removing monkey labour". if it just aggregates a problem into some new issue, don't waste money and time making it. The tools should always be well documented, also tools in my oppinion should have code docs on top of that, just for the reason you said, the guy moved on and now it's someone else problem = it's an aggregated problem. Also I like when you don't create monster monolith tools, but smaller, and let the build system or what ever call them, scales better and also, that's what build systems normally do pretty well.
I need a T-shirt that says "TRIBAL KNOWLEDGE"
I have always heavily commented my code. A few years later when I have to work on it, I rely on the comments to remember how it works.
This has been an issue at pretty much every company I’ve worked at
Another undortunate thing about working at Google is that there's little knowledge carry over to other jobs because a lot of tools are very Google-specific. At least Kubernetes is used across the board in industry. Borg is only used at Google and nowhere else.
That's funny, I always fight to choose the best tool for the job and it rarely happens that the best tool is the one you created. And, yup, for Google it probably can happen, but as you mentioned - only in widely used tools like those related to infrastructure or building, for smaller companies it virtually never happens.
At Amazon, not only are there internal tools specific to orgs, the people who created those tools have become SDE3s and have immense power at forcing you to use their internal tool.
So much for customer centricity 😂
To be clear, there is assembly code (still "human readable") then there are the ones and zeros of machine code. Some compilers do indeed emit assembly code, which is then assembled by an assembler. I could be wrong, but I think the majority of them emit machine code in object files, which are then linked by a linker to produce an executable. When things go boom, or you didn't write (or have access to) the source of the code which went boom, then you have to deal with the machine code.
I kind of agree, it's a slog to have to fix somebody else's code, but that's often part of what IT people are hired for.
Yup, that’s right.
He (along with many modern programmers) doesn’t know that because he was hired to fulfill a diversity quota, not because he actually understands computer science.
Generally agreed, but regarding C, asm & abstraction: when you're optimising bottleneck pinchpoints, dipping into the compiled asm is very common.
Inserting inline assembler, unrolling loops, avoiding pipeline stalls, and minimising load/store by maximising register usage is far from uncommon.
As someone coming more from hardware side, I've used assembler this way before. In retrospect, however, it's usually an indication of having chosen under-powered hardware for the problem at hand. If you're writing C for modern complex processors, you should probably only use assembler if you're in the process of developing a new variant of that processor, writing a compiler or simply feel curious about how stuff works under the hood. That last option is 100% valid, but it's all about learning instead of being productive at that very moment.
Poorly done abstractions at work can be soooo painful
Those problem almost exists right now in almost tech company right now. The problem is that the quality of the tools really varies upon the devs knowledge with the language/problems in the first place. Sometimes, it really shows the "exp" & "growth ladder" problems in the company/tech team.
Ha as a hardware engineer, we also script somewhat. Sometimes we got a genius writing one flow, that every project use. They either left or moved or whatever and that hole is left unfilled. I certainly can't go in and start rewriting the code. So our company learned and made sure we have as little custom flow as possible, everything is industry standard tools, let those company handle code/model updates, we just use the tool. In the end, spending a little money and time now, avoid long long delays down the road.
This is the same also with non-internal tools that are not widely adapted. For example, a small open source project that is really useful, but then it takes forever to have small fixes etc. as there is only one or couple of people maintaining it. But then again every tool has to start somewhere. Sometimes there internal tool are released "into the wild" and everyone loves them.
Any issue you have in a new language has been solved in an older language, you just have to look up the JavaScript or Java or C solution.
Nice ... good to know
I actually coded a serial communications protocol 'handler' in asm from a C version, as I had to put it on a Z80 card from a PC environment ... I left the C comments and code as comments in Z80 source, and added the appropriate Z80 assembly instructions to accomplish each C code statement.
I don't agree with the part about avoiding a tool and just creating it yourself. It is essentially the same thing the next person coming in will have to disect your code and figure stuff that you did. Instead I would say falling on open source tooling because then support and documentation is available. But I guess that's not possible there?
I hate how every Google projects use Bazel and abseil.
I wouldn't call it an abstraction hell, more of a like too-many-internal-tools hell
Imagine this at a company that programs primarily in a proprietary programming language. It was tough and took more time but eventually in the end it taught me you can find answers if you look hard enough through the source code.
I didn’t even know that existed lol
This problem is present for any big company be it tech or non tech. We gotta deal with it
A nuanced take on UA-cam in 2024? Not just: "Abstractions bad! *gargle*" You have earned my subscription.
Imagine when you figure out that you'd spend less time if you've just used YOUR tools, or other tools instead of fixing that shit because that guy didn't keep his code updated. IDK, let's say 30days vs 3hours, that would be gross.
Abstraction should be thought of as the same as a car part within a car engine.
Just a black box you have a few controls in to do stuff. Purposely incredibly simple.
Haha it's kind of a relief that this even happens at Google 😂
I think this problem is commonly faced by people working in large corporations. Someone would abstract the effort, but no documentation and no maintenance. And people who join later suffer and scratch their head understanding what is happening.
Have the exact same issue at this tiny startup I’m working at. The CTO is obsessed with building his own abstractions which for the most part do help us get stuff done quicker, but it’s so complex and over-engineered with zero documentation that as soon as something goes wrong (which it often does) he’s the only person who can understand and fix the bloated mess.
question: did Google have restrictions that prevent you from fixing the tool?
I’m ok with having these tools. I have written many myself, some still in use at other companies I left years ago (I keep in touch with old teammates). If a tool or utility I wrote needs to be updated, etc… they just did it, why would they re-write something if all it needs is a new feature, or update?
Might there be a little “no room left for me to write my own tools from scratch/ “not invented by me” issues going on here?
If, on the other hand, some strange Google policies or process prevent you from contributing to the tool code, that’s unacceptable in my opinion. I can’t imagine why they would do this, I have never heard of this happening there.
We have are own ui library abstractions in out company. To use it, the security implementations we have to do is insane just so our application can download the css and js files for that library during runtime.
I am part of a team (only member now as it was scaled down multiple times) that abstract over multiple saas vendors used by our company. Not only is it literal hell to maintain as a single dev as multiple people face issues all the time, but the thing is that I am not a direct POC for them. So if they face an issue, by the time it comes to me through various support teams, its already over or near the expected deadline.
Since mine is not a product company, I have to justify my work in 30 minute timeframes. So sometimes I get loads of requirements for new implementations for the tool and work over time, while sometimes I dont get much immediately as it gets pushed to later sprints and have to hop on and take work on other applications to justify my time.
This is very common issue and i hate it, they have internal tools and dont even have proper documentation and if any issue comes up, you have to own regardless you want or not and fix it.
Lots of companies have proprietary internal tools. It's interesting to hear that even the top guys are not exempt.
As a non-programmer but tech enthusiast and IT guy that has done tech support for years, I absolutely hated propietary software that did things that other non-propietary software did, and often better. Mainly because taking the time to learn to use their software and their quirks, was NOT transferable knowledge so it was a waste of time.
I particularly hated my experience with the Sabre GDS while working there, mainly because it's very complex and can even be "fun" learning it, but then you can't really use it anywhere else except if you work in the travel industry. You also can't practice it on your own time at home if you don't work for them as there is no way to access it unless you're in a corporation. I have other examples like this, but I don't even remember the name of the software used in those companies (was also incredibly terrible).
abstraction hell clickbait title. Its moreso tooling hell
As a solo developer who doesn't comment code, it's a nightmare going back to old code to try and understand what I even created and trying to fix it.
Most new hires are opinionated about which abstractions should/shouldn't be. It's almost as if you're new here.
Basically making internal stuff and being slow with docs makes you irreplaceable and kinda automatically lead of something.
An engineer’s job is to bridge the gap between the requirements and the desired output. How it can be done varies greatly from one eco system to another and that’s why engineers at big Tech pay big bucks, it’s not because they know XYZ technologies, it’s because they can be an expert in those technologies fairly quickly.
No its because their willing to slog through decades of soydev bullshit.
Just mixing around labels and not really making it simpler, is not actually a "technology". Your not making anything faster lmfao.
Almost everything software wise developed in the past few years, has been to code faster. Coding faster is not a technology believe it or not.
So you get slower shit, thats not even faster, is 10x more complicated for no good reason, and your forced to use it cus some midwit worshipped overcomplicate.
Modern programming abstraction and WordPress (and the server infrastructure to support it) are all Planned Inefficiency to generate employment and the modern tech ecosystem. It's not all bad.
every large company company works this way. as long as they have docs and they have support teams, you should be fine
This issue is also a "thing" at T-Mobile with dozends of internal propriertary tools
That's not how this works. If something really isn't working or isn't optimized, you should look for the bugs in the assembly level as well if you can reduce a single instruction from your code that will always help you in writing better faster code.
Also, if you think you can maintain the code yourself, do it it may not pay you but it is doing good for the community that one should prefer
Bazel / open-sourced blaze is one of the worst build systems I’ve ever seen, and this was a solved problem. Google just wanted to keep their entire company’s source code in one place, and thought it’d be easier to create a build system from scratch to handle that than to bother trying to break things out. I’m like… what.
imagine if facebook kept react internal 😬
It's somewhat disturbing to see that a tech company as massive and complex as Google has the same kinds of problems as the mid level be bank I worked at.
SAP has this problem. In the 80's and 90's it was pretty muchg a case of the core application code that you were interested in was just under the skin and you could usually solve something in minutes by debugging. Then they got into OO concepts and started abstracting the basics into 'frameworks'; badly and just for the sake of it. Bang, now it takes half an hour of debugging until you find a code reference that looks like it might be using a concrete object (which then starts using abtraction in it's own component layer). TCO for customers wanting to enhance the product went skyhigh.
It's probably great for them, internally, as frameworks do accelerate things, but as stated in the video, this needs a lot of knowledge and documentation. SAP are really bad at technical documentation. As a customer you are pretty much f-ed if an error happens in these frameworks now, your average SAP coder will flounder and experienced problem solvers like myself don't even want to go there.
0:42 Every compiler I've used at one point I've had to delve into the assembly to find a compiler/optimizer bug and figure out how to avoid triggering it.
Was just about to comment the same thing. Compiler optimizations can and will be the source of bugs which can only be properly diagnosed by examining assembly.
"i'd rather built a solution from scratch"... basically you want to be that guy that wants to get promoted and creates an internal tool to solve a problem
I can relate to that is issue of "internal tools" making one's life hard rather than solve problems.
With 'abstraction hell' I associate too many layers of abstraction. 'tool hell'/ 'custom software hell' is what I'd rather call this. I'm guilty of doing this too. I cannot solve the problem. Either I try to automate parts of my job or have to do things manually. I can't change it. If someone else has to use my work I'm aware it might be a miss or hit thing and my work might end up in the bin. That's how life is sometimes.
I still try to do my best, document and test my own stuff. But you need feedback from others to improve things. The feedback is often missing.
What I hated about google: They didn't let me leetcode all day during my shift.
Not just big tech companies.. I work in a small one and they insist on reinventing all the wheels, which sucks because it means you learn how to use all their stuff which is useless in your next job
I think that something you haven't touched on (and maybe this is a clip from a longer stream/video) is that many times this abstraction is a result of tools that were started but were not finished, either because of the developer, or lack of funding, i.e. the level of abstraction was but one piece of a larger picture that was intended to be built, but organizationally could not get off the ground. we have many tools that are partially useful sitting around.
curious -- did you feel like you were forced to use these tools with these levels of abstraction, or could you just solve your problem the way you chose (or remove the existing abstractions with standard issue/lower level solutions)
also, sometimes a "local abstraction" (i.e. one you've built to solve an issue that gets on your own nerves over and again) can end up being taken on by a coworker, or team, and spread, becoming the defacto standard. in these cases, it can be annoying if you're on the other end of this in that as you take on the abstraction, only to realize that you actually have a better way to solve it with little effort at that lower level that makes the abstraction you originally leaned on a burden rather than a benefit.
they say naming things and cache invalidation are the hardest issues in software dev, but i also posit choosing a level of abstraction given "software destination" or schedule is another toughie. The larger the company, the larger the errors in these decisions are multiplied.
many times the core issue isn't that people are not just idiots or malicious (not saying you're saying that, but i get that feel from comments often), but that these things are actually genuinely sometimes complex issues that take some thinking, and cooperation, to avoid....heck someone could just be having a bad day and choose the wrong pattern, compounding their issues. which is also why review is useful.
Hello from Germany. I have to tell you, that working at a big car company with to letters will be the same.
I know that I know almost nothing about working in bigtech, but for me it could be a hell only in case of time limit. So if I debuged, fixed, wrote docs and tests without any pressure by deadline/manager/colleagues it would be ok for me to have like a small rest for few weeks from another work.
Don't talk to be about abstraction hell. I've lost about 5 years to it
So why Google has all this internal tools to begin with? They run faster than their open source counter-parts? Other reasons?
I prefer open and well documented technologies too.
I can't speak for Google. But at our company, we've had to create new tooling when what we needed didn't exist or was sufficiently lacking and it was faster to roll our own than modify existing open source. Then time passes and the open source tool gets better. But by that point, the effort to switch to the open source tool is too expensive.
Fxxk yeah!! I can't agree more, even I was a TVC there, I can still understand. It was a bittersweet story
Luckily, I had opportunity to access those internal tools (no one but me in my team?! division?! corp?!)
Unfortunately, I must study whole the related fxxkin internal tools before I started to complete my assigned task
And because I was a TVC, that made thing even harder, how I was wishing it's just public tools with public names, so just stackoverflow should work it out
I personally love to create my own tools every time and have never used any 3rd party in any personal project (unfortunately I haven't used my own programming language). But honestly, I don't see any difference between debugging an external 3rd party or an internal 3rd party. Generally, debugging someone else's code sucks.
Yeah can completely relate. I love using one tool for everything. not one tool per thing.