The fact they brought in experts after spaghetti code was written for years and they fixed it in such a short time says a lot about their planning and competence.
I found this quite striking too. The difference between a vast bureacracy and an actual, functioning coding shop with decent engineers and not too much red tape is night & day.
I got to be part of this effort and man it was a train wreck. One of their "solutions" to fixing db connection deadlocks was to perform dirty reads with NOLOCK directives. Their upper management team was the best part. Having such little experience with dev teams and doing nothing but creating bottlenecks for the devs, they were often left out of the loop on purpose. Yeah I got years of horror stories working with these guys.
@@karmatraining Beyond a certain size most software producing shops become vast bureaucracies. Decent engineers and not too much red tape is a state that tends to vanish somewhere around the point where the engineering department's headcount goes beyond 10-20. The whole story has all the hallmarks of basically any software project at your run of the mill mid to large sized enterprise organization.
I've worked on two government funded projects myself, as a software engineer. Both went ... well ... terrible. There are just to many upper and middle-managers in our governments bureaucracy. All these people search purpose in their job, whilst there isn't one. So they fill their days with meetings and more meetings, these meetings result in new ideas, which result in project change requests. All these change requests result in a dev-team that goes in endless circles, without any progress at all. Due to this lack of progress, the client and/or government officials lose fate in the engineering team. This results in a massive increase of government bureaucracy, which just roadblocks the dev-team to a complete stop. Which result in less progress, which in return results in less trust, which results in more bureaucracy, etc ... The slower things go, the more the government officials start making their own decisions, the worse the project becomes...
As a software engineer, I've worked on a bunch of government funded projects. The last project was rather small, so I estimated that it would take 2 to 3 months to deliver. The project required a specific type of IOT wireless access point, which wasn't yet installed. So we scheduled a meeting with government officials, in order to discuss an optimal installation point for this new antenna. They agreed to install the antenna themselves, so we continued developing the backend of the project. We completed the dev-work in less than two months, ahead of time. So we contacted the city officials, to share pairing codes for the antenna. They informed us that the antenna hadn't been deployed yet, so we had to wait. We contacted them 1 month later, same story. Again 2 months later, still no antenna. Another three months passed, still no antenna. We called them back after 6 months, begging to deploy those IOT devices to free up some office space. Explaining that we could active them remotely, once they installed the antenna. They agreed, we deployed the IOT devices, then everyone forgot about the project. It took the government 2.5 years in total, to install the antenna. Which is about an equivalent amount of work, as installing an wifi AP. They then asked for a financial compensation, claiming that we delivered the project two years behind schedule. A government funded project, in a nutshell... KEEP AWAY!!!
arre you kidding? Almost every single government contractor I've seen over quotes the living hell out of everything. A 600 worth equipment costs 6000, 6000 costs 60000. They just pay the officials in charge and the tax agencies and the officials don't care either. I mean it's not their money, it's public money and how many people even bothered to read the constitution? People kill each other for government contracts because they're legal licenses to steal. After all if everything's public money and the public doesn't care.....
"They didn't document anything, but that's pretty standard, we can give them a pass" I've never been more insulted by something I completely agree with.
I HATE that this is true. Ever tried to continue a project made by another team? When you have no contact with that team and no code is commented plus no diagrams or any documentation is given, good freaking luck! I just got finished with a project like that. It was hell.
@@peterdieleman303the great thing about this industry standard practice is that it actually applies to all industries ever. No one ever documents shit. Humans are lazy
@@lightning_11 Crotchety project managers high up that refuse to learn new things. The type of people that here in 2023 still insist on running Windows XP and ignore all the shit that breaks because they are doing that. I have dealt with people like that and it's miserable, but they're everywhere. There is also a type of project manager that will only allow a technology to be used if they understand it, but they are also an idiot, so you're stuck building rocket ships with Duplo Blocks.
This project took 3 years. I genuinely feel that if they hired a single skilled full-stack engineer, a graphic designer, and a ux designer, paid them good money for those 3 years, and gave them access to domain experts to set the requirements, they could have easily cruised to the finish line on this.
i mean, functionality wise, yes, it would work better. but the tech requirement sheet they provide would be prob 2000 pages, given the bureaucratic crap, so f that shit.
A friend of a friend worked on healthcare.gov. I think the level of effort is being severely underestimated. The requirements kept changing because the people having the project built didn’t even know what they were building.
Every time I hear about such projects, how long they took and how expensive they were I‘m thinking to myself that I could have done it in a third of the time for 100 times cheaper and it would work better. Sometimes I feel that the government deliberately does everything wrong in software projects.
Underdelivering is important, as they then justify their existence by wasting even more time & money, resulting in management receiving salary bonuses.
The US visa application website, also crated by CGI, is probally the worst website I ever had to deal with. It has a number of pages which you would need to complete for the applicaion, however they have an extreamly short time out of maybe 5 minutes or so before they would boot you out of the system. Although you can save your progress, saving is only possible once an entire page is completed. It was extreamly furstrating when you go dig though the draws for a document needed to complete a question near the end of a page, only to return to see that the system has timed out and anything filled in on that page is lost. It has gotten to the point that I re-entered questions for a page so-many times due to the time out that I memorized the entire page.
@@Rubicola174 or make it open source so that others don't have to choose between endless suffering or paying money for a 3rd party app just to apply for a visa. Or the government could make the improvement, but whatever.
This is basically every pitfall that a project may encounter in 13 minutes. Seriously impressive to encounter all of this in one project 😂 Especially the idea that more engineers will speed up the delivery makes clear the managers had no clue about software engineering.
exactly. That's the thing that bothers me the most - how managers could even try to manage dev team if they have no idea about how do developers work? And it's the case in most of companies I've heard of.
Yeah, usually a good project just need a bunch of seniors, one for each area. One senior in frontend,one in backend with another one that knows about databases, the DBA, if needed. One for security practices, just to make sure each one knows which data should leave their layer. The leader who knows exactly what is going to happen in the project, how it should be built, assisted with a software architect. This would be the ideal for a medium sized project, or even big sized project if every each one of them is truly a senior.
Its a situation thats way too common - ive seen this in utility company projects, one of which should have utilized two developers for 3 months time. When we arrived they had spent 6 months with 28 developers across three timezones, complete shitshow to approach projects that way.
"The production launch is the end-to-end testing". It gives me the same vibe as the meme of the dog that says everything is fine while sitting in a house on fire. 🤣
That line gave me a literal cold sweat for a second. When I hear that from a superior in my line of work it's my signal that it's time to polish my LinkedIn profile
Once in a while I entertain myself by pushing untested code straight into production. By "untested" I don't mean not tested for security, bugs or load capacity, I mean I'm not even sure it compiles.
Shoutout to the government project I worked in 2017, I've heard they're still using an IIS with php 5.6 with the vendor directory commited to the repo.
It's not ideal, but committing vendor is "fine" in practice. It's similar to a monorepo setup, which is acceptable, and it does come with a benefit in preventing some possible supply chain issues.
You know, this video showed me that even though I only have one more day to complete my project in python, and even though I have only like 30% of it complete I can rest assured that what I am doing is standard practise in the field. God how I love being a programmer.
That was an astounding trainwreck from beginning to end! I lived in Singapore for a few years prior 2020, and I was spoiled by the quality and slick interfaces of all the government technology over there, created by their in-house SWE agency GovTech. Everything from the SingPass app (for national ID and SSO) to their tax portal was extremely solid and modern looking, rivaling most commercial apps. It's so fucking sad how far behind the US government is, when it comes to software engineering.
@@johnpaulgeorgeringo2329It's so that the United States government can bully other governments into loving them. Even though it just shows that you are corrupt a****** who will use violence and force to make friends.
It's mostly down to how our government is structured here in the states. It's structured in such a way that making changes takes a very long time and a lot of political will, for both good or bad. This prevents a single administration from abusing power, however at the same time it means that the government lags far behind when it comes to modern rapidly developing technologies.
@@holy3979 I'm not so sure about that. Both NASA and NOAA do some incredible engineering and scientific work today, and so does the DoD when it comes to aerospace and IT (just look at the invention of GPS and the early Internet for past examples), but somehow, the US just sucks hard at software. *EDIT:* Forgot to mention NIST, the NSA, and the U.S. Naval Research Laboratory as prime examples of the US government successfully developing and standardizing some incredibly advanced computer tech, e.g. Tor, Ghidra, and several key cryptography standards.
@@xplinux22 This is probably because of how good the software industry is in the US. It's not the US public that sucks at SWE, it's the US gov't, and the reason for that is that the gov't can't afford to match the good compensation a tech company will provide, which leads to all the talented people (both in technical and non-technical skills like communication and management) being drained towards private industry. Like you said, NASA and NOAA are top-notch, and that's because for the top-notch hardware engineers, these are some of the best places you can work at because only government can afford the high upfront costs to doing things like maintaining a successful aerospace program.
It’s funny how it took hundreds of people to build this and still fuck it up, but nowadays, this is probably a system design question for an entry level position at a tech company lol
This is very interesting, but I feel like I still have no clue how this actually could have happened on the ground. Like yeah, there was no decisive leadership, stupid managerial practices, old technology, changing requirements... it does sound like a perfect storm for things to go badly, except that's kinda par for the course in professional software development. It does sound like everyone managing it was at fault from contractors to government officials because of various bad decisions and the project was a nightmare to work on. It sounds like everything boiled down to terrible management, but I'm still sorta surprised about how that happened to such an important project. It's not like the federal government hasn't made working websites before... I would have liked to see more specifically how that GUI-generated code played a part, cause I can see that being a problem. I've definitely felt restrained and exasperated by bad code I could not change.
The main problem is almost always the management. If they tell you "this project is going to use X" you tell them "X is truly outdated and almost no one uses it" and they just shush you. Well, no one knows about X and now the whole team has to learn X with outdated documentation or lost documentation. This has happened to me a few times, when a boss asked me to use X or Y, I told him how outdated it was, then everything just slow downs a lot because there is hardly documentation and just some online posts from 2009 or 2010
This is basically just what happens when there isn't leadership. The point of a leader is to make sure everyone is working together to achieve the end goal, not doing their own thing based on their own version of the end goal that doesn't fit together at the end. (Continuous integration and testing are just two specific technical mechanisms that ensure everyone's pieces fit together)
@@asii_k I assume it's just the N+1 queries problem: because you put more effort in making the code "clean" than fast, you have a "get whatever from database" function and the only way to get 100 whatevers is to call it 100 times instead of asking the database for all 100 whatevers at the same time.
As someone who doesn't know anything about web dev, I find it so ridiculous how over complicated everything seems to be. There is no way a basic webpage and form colector should require 50 services built by 10 different teams
as a webdev i can say: no, it realy does not. projects like this would be solvable by a single team. the decisions made along the way were the problem. i have seen it myself. a little change from 50 shown characters to 500 can easily cost 100k when you have badly written software, no way of testing, different contractors that are not willing to help you and multiple instances of ppl having to accept the execution of the change. whats even worse is that the good players quit at this point and all left is a bunch of losers operating.
most of any modern website you use is built with many different libraries written by developers all over the world. no one writes the whole thing themselves because someone else has written some piece of code and it's much better than you could hope to. it's easier and smarter to just integrate that into your workflow. a "basic" modern looking web page would be a pain to write all from scratch, especially if you want to serve many different people.
worked on it a few years after the time period in this video, long story short it's not that simple, hundreds of thousands of lines of backend code levels of not that simple.
@@parabolicpanorama But why would it be such a pain to write it from scratch? Sure, for some things you really should use libraries, but you can't overuse them (or worse, make the project only by linking libraries together), because you can get boxed in and are unable to change certain functions without breaking everything. With purpose built code, you also have the advantage of designing the system in the most optimal way for what you're trying to achieve, so even if your code isn't as fully optimised, it could still be faster if it is less bloated with unnecesary features, right?
You should really do a video on when Helse Midt (Norway's healthcare organization that covers the middle area of Norway) chose to change their journal system. They started with a bidding process. All vendors welcome. Pretty early in the bidding process however they decided to disqualify DIPS, who was delivering the same journal system to Helse Sør-Øst, Helse Nord and Helse Vest - the three other govt. healthcare organizations in Norway. Yes, they really disqualified the one piece of software that was used literally everywhere else. Who did they go with? Well, good'old american EPIC - and they chose to call the new system "Helseplatformen". Did it go well? NOPE. It has been a shitshow from day one. Most notably, 16 000 critical letters to patients were discovered to not have been delivered, which delayed a whole boatload of appointments and also put some patients lives at risk (Helseplatformen blamed this on user error - yes, really, 16 000 missing letters was the result of a user error made by several hundred users was to blame on the users, not the system). After this, the director of Helseplatformen stepped down due to the huge wave of criticism following the reveal of these issues. One doctor actually saved a patient life after being unsure about wether or not a letter was sent. He asked a colleague to directly contact the patient to ensure he got the information he needed. The patient had a blood clot. Had the hospital not reached out to him directly, he could have died. Currently, 44 individual cases have been confirmed where patients have not received the help they needed in time, directly as a result of Helseplatformen. In a survey conducted among the staff of one of Helse Midt's largest hospitals, 27% of the nurses were considering quitting due to the issues. The feature for referring patients to another hospital, which is a critical feature of any hospital journal system, was non-existent when the system was first implemented. After several months this feature was finally implemented, but it caused a massive amount of manual labour for the hospital staff - who are already understaffed. Some hospital directors are quoted in meetings discussing the backlash from doctors and healthcare workers as "echo chambers", "hate groups" and "whining from angry doctors" - which isn't really helping the issue. The healthcare on their side are claiming that they are providing the directors with "constructive criticism" about legitimate issues with the system. In addition, the price for the new system has increased by 35%, from 3.7 billion kroner to 5 billion kroner - and it has been implemented in less than half of the healthcare institutions in the region. UX/UI wise it's also a complete mess. It's one of the worst pieces of software I've ever seen in modern times. If you search for Helseplatformen on google you can see their take on inputting a date/time in the system. It's a complete scandal. Now, a ton of doctors and healthcare workers have protested against Helseplatformen - with some healthcare institutions outright refusing to change to the new system, instead opting to use the old one. In the meantime Helse Sør-Øst, Nord and Vest are looking at this and wondering why the hell they didn't go with DIPS, so they could all use the same system and be able to share journals across all hospitals...
Really awful story 😢 but quite frequent the way how federal software projects work. It's a real mess, and I wonder why nothing is learned from this. There are brilliant tech companies out there making immensely powerful systems; why can't this knowledge not be brought into federal software??? 🤷🏼♂️🤷🏼♂️
Everything that happened in this scenario is covered in "The Phoenix Project". That book still holds up and I wish more people in our government would read it.
Man, you're a really great storyteller and your videos are truly awesome. I'm glad I randomly stumbled into this channel. You deserve way more recognition! As an engineer myself I find these videos very interesting and valuable. Keep 'em coming!
"first they didnt document anything but that was a standard, so we can give them a pass here", this line just cracks me up😂😂😂. more or less standard across most of the companies😂😂
These videos are awesome, thank you so much for making these. Was also thinking, I'd be totally down to listen to this via podcast as well, but that's a lot of extra work on what I imagine is already a ton of work. Thanks again, the effort/content is really appreciated.
I discovered your channel today on the way home from work and I've already watched all your videos. Please keep it up. I hope your ad revenue allows you to make them more often.
Frequent changes is basically the point of processes like Scrum - you still gotta pay for your changes, but at least the process can accommodate them. And you still have to eventually figure out what you want if you want it to ever get done. And you can't set a deadline if you don't know what you want (but you can still try to build it).
I signed up back in 2013 and didn't get coverage, 9 years later I tried to login using the saved information and they had removed the security question I used so I was no longer able to recover my account. I called support and tried for two hours to recover my account. After 10 years - still no healthcare, they told me I have to have my wife call in as I'm now under her account but we do not and have never had coverage. It's great to know they were poorly mismanaged - when I signed up in 2013 the forms I was supposed to fill out were not available yet so I obtained an account but could not complete the setup. I was told to keep checking back.
In our country there was the same hi-tech attempt to enable parents to enroll their children in kindergartens. It was fiasco, system was down on day one, and every second kid was not in the lists. At it repeats each year :D
I feel like the all star team had an advantage where the business rules were more well understood. If you talk to a vendor who doesn’t even understand the project how could they possibly convey it to you to build. Also involving so many teams is just awful. Makes comms so much more difficult. The government likes over complicating anything.
Your videos keeps reinforcing the idea that you don't need hundreds of people to develop software even for national use, if anything adding more people create more communication problem which lead to wasting even more time and money. If the people at the government had chosen instead to go with a smaller team(20 people at most), they wouldn't have encountered all these issues. It would have cost significantly less money and it would probably have been delivered on time. They could even have hired multiple teams to develop multiple concurrent version of the website and chosen the best out of them.
That isn't the government way. Throwing more money and resources at something that is failing is always the governments course of action. The contracts make more money, the more incompetent they are. I agree, hiring 20 really good developers is much better than hiring 100 bad developers. But that isn't the way the government sees it.
Particularly since ultimately it was "just" a large-scale insurance broking website. There are dozens of pre-existing ones - it wasn't needing to do what certain government projects must: something new that also reaches everyone,.
jeah, some of the most amazing and solid software used by millions of people is written by tiny teams or even solo devs. The entire developer team at f**king Adobe is only around 30 people. An entire PC fleet management system was written by a few German teachers over the curse of a few years and is now used in **many** schools across Germany. DXVK, the library translating directX to vulkan and embarrassing both AMD and Intel, by, in many cases, substantially outperforming their native drivers, was originally written by a single dev, and is now maintained by only 2 devs. It has since been included in Intel's arc drivers to fix the severe performance problems they had at launch. Quality software IMO requires only 4 things (for the coding part, i'm can't speak about the other things): A clear *vision* and focus. At least *one* *good* and, perhaps even more important, *motivated* *programmer*. And *time* to improve after an preferably soon usage start. Communication between everyone involved (this point is easier the fewer people work on the project) if the vision is missing, you will produce something, but end up with something unusable, and possibly useless. If you don't have a single good and motivated programmer, it will end up a buggy and mostly unfixable product, no matter how great the vision. If you don't give it time (in use) the program will either be unfinished or inadequate. If the Communication is failing, all three of them can happen. (bad communication costs time, robs motivation, and prevents a vision from driving the team forward) I think most software fails on the Vision or Communication front. Especially government or large company software.
Wow, doubling the headcount with 3 months to deliver the project. Understandable, it's not like that's a well documented mistake since the 70's or something... haha
I write code professionally. This video made me want to cry. How could they possibly do *everything* wrong? It's like a fucking bingo card of how never to manage a software project. HOW!?
As someone who works in an agile change environment listening to you describe it as what it theoretically is supposed to be was kind of hilarious. In practice the daily stand ups are a painful mix of the product owners chastising people for not working faster, people saying "yup still working on that thing which I already said in the last three meetings" and the rest just trying to avoid saying anything which might put them in the product owners firing line. The method of controlling work on the kanban board is disorganised and communication with the business is essentially cut off by the whole agile process. The only time it works well is when you have someone with enough clout and a modicum of common sense to organise the work on spreadsheets outside of the whole process and actually get some overarching control.
@@P4INKiller Or fire the product owner, but since that's not something an individual dev has any power over yeah I agree at that point you start sending out CVs
I just wanted to leave a comment to say that your videos about these tech disasters are brilliant. Your six most recent videos have all been so interesting and very well produced. I'm sure if you keep putting out grerat content like this then the algorithm will reward you handsomely and your channel will grow very quickly. Keep up the great work!
This sounds exactly like my experience with government projects in Australia. Not all of them are like this, but I was in the shit house dept where everyone was trying to leave.
You should do one of these on the Queensland Healthcare Payroll systems debable. Similar amount of money was wasted, maybe more, and it took down a whole State government here in Australia. Absolutely great business case study on how NOT TO run an IT project.
I'm not a web developer, so maybe this is a stupid question, but how could a website whose only purpose is to allow people to buy insurance cost nearly a billion dollars, let alone the nearly half a billion dollar original budget?
I'm not a dev (yet) but I guess it's not only a website. The website is what user sees but devs also need to create an entire backend. It's similar to frontoffice and backoffice - as a customer you'll never see backoffice but it is needed to provide smooth operation. Such service relies on authentication based on some gov't services and needs to register your actions in another gov't service. Yet (as a noob) I still believe that the budget for this could be at least 5x lower and it would be sufficient for creating a decent system. It's just too many managers taking too high wages for what they are doing (i.e. creating chaos and disruption)
It's the Government. "Other people, spending other people's money on other people". Plus I believe the contract was given to a friend of Michelle Obama's, not a well known tech company. No website, should ever even cost close to that much money.
welcome to gov inefficiencies... think how many gov employees were hired for how much time and how many sub contractors were involved and how many times these requirements were changed. Also bringing in top tier talent, doubling resources, ... all these things last min cost more. Not to mention that the amount mentioned doesn't mentioned what is really covered. it could be the cost for the website for 5-8yrs with support and staff. Again, not justifying the ridiculous amount of $ for what essentially is a website (and not a very complicated one at that) - it is not like a brokerage that needs to do transactions in real time or an airline booking website that needs to search from airlines in realtime from all over the world and be responsive.
Why I don't get it why did they have to rebuild the damn website from the first one. The first one they had worked. Just because it didn't let you put in payment information or sign up doesn't really matter, you're going to have to call the insurance company anyways when something doesn't work correctly. For a website that was supposed to make getting health insurance easy was an extremely difficult task and a big waste of time, and just convinced people just to shop around by calling everyone that they knew and to use Google and reddit.
It's actually a recurring theme for government programs to be effectively leaderless. Often the guy in charge is a politician who is preoccupied with politics and he just rubber stamps proposals since he isn't really involved in operations. Likewise, sometimes certain tasks fall under several agencies at once so progress moves at a snail's pace as any given file must go on a trek between as much as dozen agencies before being finished
6:10 yeah of course, more people means more code right, should be faster "we did the hard part already by thinking what it should look like, you just have to code it out, easy. we did all the thinking already"
I work at a big government contractor (formerly a data scientist at a tech company). I can confirm that government contractors are as ineffective and inefficient and bureaucratic as government agencies. It’s really bad. People here barely know how to write code.
As Joel Spolsky said, non-technical people managing software projects is like a non-surfer relying on their advisors standing on the shore to tell them what to do. Project managers are necessary to make decisions, but they need to know the domain. Linus Torvalds could be an ordinary dev, but he focuses on managing the whole Linux kernel. His job is to make the technical decisions and give the implementation to trusted, competent people. But an MBA who wasn't a programmer wouldn't understand the decisions that need to be made, so would come unstuck unless someone technical is really in charge.
The scariest part of this isn't the incompetence, it's the fact that **healthcare.gov** was able to get exempted from any security testing until 6 months after launch.
Why wouldn't you want to hire someone who brags about 30 years of experience in a technology that's been obsolete for 25 years? You would hate to waste such valuable experience!
Lack of experience, corporate greed and incompetence, all in one project. I feel for the front line workers, all those sleepless nights to clean up after what was done by the leadership. Pity it still happens in numerous other projects as well.
I'm simply baffled by the excessive database queries. Like, did dev leadership forget that RAM exists for them to use to hold values as variables? Were they querying the database every time they needed to access data? I've literally only been working with SQL part-time for about three months and even I know better. The only reason I can think of that might justify those excessive queries is if they're running on servers with lightning-fast CPUs and pathetically little RAM, and even then, the far better and far more obvious solution is to just use better servers. Also, wtf made them think NoSQL was inherently more scalable? Sure, for some applications, a schema is overly restrictive, but for some applications, a schema provides immensely valuable structure to the data, and since, as the video indicates, NoSQL is a catch-all for a variety of data structures that work pretty differently from each other, the fact that they decided they "needed" NoSQL without knowing what kind of NoSQL they needed just screams "I got hired to lead this project as a favor to my wealthy political donor parents, I'm not actually a software engineer". I sure hope the over-budget funding we ponied up went straight into the pockets of the hard-working programmers who pulled 24-hour shifts to make this thing work; the incompetent leadership didn't deserve a cent beyond their original contract, and if the leaders continued to get paid after the Oct 1, 2013, launch date, that raises some serious questions about whether or not they actually intended to provide a working product on time (or at all).
It's dead easy to generate excessive queries in a big system. A system where no one communicates with each other and many lack technical competence like this? That increases exponentially. You need A, okay, make a DB call. Oh you also need B from this function, let's call it. But that function is actually making 5 calls to return B to you. Now, you need C, and C can actually be retrieved along B in that earlier function call, but you don't know that, or can't figure out how to retrieve C alongside B, so you make a call only for C. Do this enough times, in enough abstraction layers, and you end up making 100x the calls you need. Then you also have shit like operations that can be combined in a transaction, but are being done sequentially in separate calls.
Your org chart with tenure as free text field, fav food and color, SSN for each employee: "Y0L0!" - Every information system specialist out there: "Noooooooooooooooooooo~"
The fact they brought in experts after spaghetti code was written for years and they fixed it in such a short time says a lot about their planning and competence.
I found this quite striking too. The difference between a vast bureacracy and an actual, functioning coding shop with decent engineers and not too much red tape is night & day.
This is a plot point in Shin Godzilla (2016). I think about that movie a lot as a software engineer and as a citizen of the US.
I got to be part of this effort and man it was a train wreck. One of their "solutions" to fixing db connection deadlocks was to perform dirty reads with NOLOCK directives. Their upper management team was the best part. Having such little experience with dev teams and doing nothing but creating bottlenecks for the devs, they were often left out of the loop on purpose. Yeah I got years of horror stories working with these guys.
@@karmatraining Beyond a certain size most software producing shops become vast bureaucracies. Decent engineers and not too much red tape is a state that tends to vanish somewhere around the point where the engineering department's headcount goes beyond 10-20.
The whole story has all the hallmarks of basically any software project at your run of the mill mid to large sized enterprise organization.
I've worked on two government funded projects myself, as a software engineer. Both went ... well ... terrible.
There are just to many upper and middle-managers in our governments bureaucracy. All these people search purpose in their job, whilst there isn't one. So they fill their days with meetings and more meetings, these meetings result in new ideas, which result in project change requests.
All these change requests result in a dev-team that goes in endless circles, without any progress at all. Due to this lack of progress, the client and/or government officials lose fate in the engineering team. This results in a massive increase of government bureaucracy, which just roadblocks the dev-team to a complete stop. Which result in less progress, which in return results in less trust, which results in more bureaucracy, etc ...
The slower things go, the more the government officials start making their own decisions, the worse the project becomes...
As a software engineer, I've worked on a bunch of government funded projects.
The last project was rather small, so I estimated that it would take 2 to 3 months to deliver. The project required a specific type of IOT wireless access point, which wasn't yet installed. So we scheduled a meeting with government officials, in order to discuss an optimal installation point for this new antenna. They agreed to install the antenna themselves, so we continued developing the backend of the project.
We completed the dev-work in less than two months, ahead of time. So we contacted the city officials, to share pairing codes for the antenna. They informed us that the antenna hadn't been deployed yet, so we had to wait.
We contacted them 1 month later, same story.
Again 2 months later, still no antenna.
Another three months passed, still no antenna.
We called them back after 6 months, begging to deploy those IOT devices to free up some office space. Explaining that we could active them remotely, once they installed the antenna. They agreed, we deployed the IOT devices, then everyone forgot about the project.
It took the government 2.5 years in total, to install the antenna. Which is about an equivalent amount of work, as installing an wifi AP. They then asked for a financial compensation, claiming that we delivered the project two years behind schedule.
A government funded project, in a nutshell... KEEP AWAY!!!
Yup sounds WAY too familiar, never again, thats for sure.
arre you kidding? Almost every single government contractor I've seen over quotes the living hell out of everything. A 600 worth equipment costs 6000, 6000 costs 60000. They just pay the officials in charge and the tax agencies and the officials don't care either. I mean it's not their money, it's public money and how many people even bothered to read the constitution? People kill each other for government contracts because they're legal licenses to steal. After all if everything's public money and the public doesn't care.....
did the company pay compensation?
@@cones914 No! The company I worked for had a long enough paper trail, to proof they where not in fault.
@@timmy7201 good, its absolutely sad to see how a government is so good at wasting money then blames it on others
"They didn't document anything, but that's pretty standard, we can give them a pass" I've never been more insulted by something I completely agree with.
Industry standard practice.
I HATE that this is true. Ever tried to continue a project made by another team? When you have no contact with that team and no code is commented plus no diagrams or any documentation is given, good freaking luck! I just got finished with a project like that. It was hell.
@@peterdieleman303the great thing about this industry standard practice is that it actually applies to all industries ever. No one ever documents shit. Humans are lazy
That’s a lot of extra work
@@Powderlover1 Ik tried a project with documentation and figured it would be better have way through to do restart without docs
As any government project they were leaning towards the oldest platform they could find 😂😂
Why is this so true?
@@lightning_11 Crotchety project managers high up that refuse to learn new things. The type of people that here in 2023 still insist on running Windows XP and ignore all the shit that breaks because they are doing that. I have dealt with people like that and it's miserable, but they're everywhere.
There is also a type of project manager that will only allow a technology to be used if they understand it, but they are also an idiot, so you're stuck building rocket ships with Duplo Blocks.
@@lightning_11 Because they usually need to support some old shit that is still in use on the government computers.
Our public healthcare providers went with MUMPS...
@@alexandruilea915 (It was supposed to be a joke... |0| )
This project took 3 years. I genuinely feel that if they hired a single skilled full-stack engineer, a graphic designer, and a ux designer, paid them good money for those 3 years, and gave them access to domain experts to set the requirements, they could have easily cruised to the finish line on this.
But that would require Government contracts to not, y'know, be bloated clusterfucks that only upper management would love.
i mean, functionality wise, yes, it would work better. but the tech requirement sheet they provide would be prob 2000 pages, given the bureaucratic crap, so f that shit.
A friend of a friend worked on healthcare.gov. I think the level of effort is being severely underestimated. The requirements kept changing because the people having the project built didn’t even know what they were building.
You expect the government to do something correctly? That's a high bar
Every time I hear about such projects, how long they took and how expensive they were I‘m thinking to myself that I could have done it in a third of the time for 100 times cheaper and it would work better. Sometimes I feel that the government deliberately does everything wrong in software projects.
"Only down 10% of the time"
Underdelivering is important, as they then justify their existence by wasting even more time & money, resulting in management receiving salary bonuses.
The US visa application website, also crated by CGI, is probally the worst website I ever had to deal with. It has a number of pages which you would need to complete for the applicaion, however they have an extreamly short time out of maybe 5 minutes or so before they would boot you out of the system. Although you can save your progress, saving is only possible once an entire page is completed. It was extreamly furstrating when you go dig though the draws for a document needed to complete a question near the end of a page, only to return to see that the system has timed out and anything filled in on that page is lost. It has gotten to the point that I re-entered questions for a page so-many times due to the time out that I memorized the entire page.
@@Rubicola174 or make it open source so that others don't have to choose between endless suffering or paying money for a 3rd party app just to apply for a visa. Or the government could make the improvement, but whatever.
That's very possibly by design thought. Make it as hard as possible to apply for a visa, you will have to give less visas.
@@milesmartig5603 hey it's not my fault your elected government is dumber than a 12 year old. Now pay up
Are you sure you were applying for a visa and hadn't stumbled upon the time trial mode of visa simulator 9000
i woulda made an ahk script to jiggle my cursor or something
This is basically every pitfall that a project may encounter in 13 minutes. Seriously impressive to encounter all of this in one project 😂 Especially the idea that more engineers will speed up the delivery makes clear the managers had no clue about software engineering.
exactly. That's the thing that bothers me the most - how managers could even try to manage dev team if they have no idea about how do developers work? And it's the case in most of companies I've heard of.
Yeah, usually a good project just need a bunch of seniors, one for each area.
One senior in frontend,one in backend with another one that knows about databases, the DBA, if needed.
One for security practices, just to make sure each one knows which data should leave their layer.
The leader who knows exactly what is going to happen in the project, how it should be built, assisted with a software architect.
This would be the ideal for a medium sized project, or even big sized project if every each one of them is truly a senior.
As Fred Brooks famously said: adding more engineers on a project that is late will make it later.
It's like thinking that adding more women can reduce a pregnancy from 9 months to 1 month.
Its a situation thats way too common - ive seen this in utility company projects, one of which should have utilized two developers for 3 months time. When we arrived they had spent 6 months with 28 developers across three timezones, complete shitshow to approach projects that way.
"The production launch is the end-to-end testing". It gives me the same vibe as the meme of the dog that says everything is fine while sitting in a house on fire. 🤣
That line gave me a literal cold sweat for a second. When I hear that from a superior in my line of work it's my signal that it's time to polish my LinkedIn profile
😂😂😂
They thought they were a gaming startup. Unfortunately, EA wasn't looking to buy and destroy a health insurance website.
Once in a while I entertain myself by pushing untested code straight into production. By "untested" I don't mean not tested for security, bugs or load capacity, I mean I'm not even sure it compiles.
Seriously you make some of the best software documentaries, I love it!
For real! They're my fave
They're goofy funny even the silly animations are full of technical details. And plenty of explosions.
Shoutout to the government project I worked in 2017, I've heard they're still using an IIS with php 5.6 with the vendor directory commited to the repo.
At least they have a vendor directory and didn't just paste files from other libraries in the src directory.
How to make a developer cry in less than thirty words
It's not ideal, but committing vendor is "fine" in practice. It's similar to a monorepo setup, which is acceptable, and it does come with a benefit in preventing some possible supply chain issues.
You know, this video showed me that even though I only have one more day to complete my project in python, and even though I have only like 30% of it complete I can rest assured that what I am doing is standard practise in the field. God how I love being a programmer.
That Office of Consumer Information and Insurance Oversight logo just doesn't sit right with me lol
its fake (I couldn't find a real logo) 😔
What time in the video is that at? I couldn't spot it
@@radiosification I managed to find it at 0:53, it's the logo on the right
@@davidddisjesus Ahh I see it, thank you
The delivery on this line was amazing 12:18 "EIDM was replaced by the Scalable Log-In System which was more scalable at logging in..."
That was an astounding trainwreck from beginning to end! I lived in Singapore for a few years prior 2020, and I was spoiled by the quality and slick interfaces of all the government technology over there, created by their in-house SWE agency GovTech. Everything from the SingPass app (for national ID and SSO) to their tax portal was extremely solid and modern looking, rivaling most commercial apps. It's so fucking sad how far behind the US government is, when it comes to software engineering.
Yeah, I don't know why most of their resources goes to weapons and wars
@@johnpaulgeorgeringo2329It's so that the United States government can bully other governments into loving them.
Even though it just shows that you are corrupt a****** who will use violence and force to make friends.
It's mostly down to how our government is structured here in the states. It's structured in such a way that making changes takes a very long time and a lot of political will, for both good or bad. This prevents a single administration from abusing power, however at the same time it means that the government lags far behind when it comes to modern rapidly developing technologies.
@@holy3979 I'm not so sure about that. Both NASA and NOAA do some incredible engineering and scientific work today, and so does the DoD when it comes to aerospace and IT (just look at the invention of GPS and the early Internet for past examples), but somehow, the US just sucks hard at software.
*EDIT:* Forgot to mention NIST, the NSA, and the U.S. Naval Research Laboratory as prime examples of the US government successfully developing and standardizing some incredibly advanced computer tech, e.g. Tor, Ghidra, and several key cryptography standards.
@@xplinux22 This is probably because of how good the software industry is in the US. It's not the US public that sucks at SWE, it's the US gov't, and the reason for that is that the gov't can't afford to match the good compensation a tech company will provide, which leads to all the talented people (both in technical and non-technical skills like communication and management) being drained towards private industry.
Like you said, NASA and NOAA are top-notch, and that's because for the top-notch hardware engineers, these are some of the best places you can work at because only government can afford the high upfront costs to doing things like maintaining a successful aerospace program.
It’s funny how it took hundreds of people to build this and still fuck it up, but nowadays, this is probably a system design question for an entry level position at a tech company lol
You should consider launching a patreon or memberships... The content quality is top notch.
^^^
Don't need let knowledge be free
@@manojramesh4598 patreon/memberships are optional
@@manojramesh4598the knowledge is free here but the creators should still be able to have support 😊
@@manojramesh4598well the video creator is going to need to make a living somehow
"The production launch is the end-to-end testing." I will now be incorporating this philosophy into all of my future endeavors.
Your humor with the editing is amazing
This is very interesting, but I feel like I still have no clue how this actually could have happened on the ground. Like yeah, there was no decisive leadership, stupid managerial practices, old technology, changing requirements... it does sound like a perfect storm for things to go badly, except that's kinda par for the course in professional software development.
It does sound like everyone managing it was at fault from contractors to government officials because of various bad decisions and the project was a nightmare to work on. It sounds like everything boiled down to terrible management, but I'm still sorta surprised about how that happened to such an important project. It's not like the federal government hasn't made working websites before...
I would have liked to see more specifically how that GUI-generated code played a part, cause I can see that being a problem. I've definitely felt restrained and exasperated by bad code I could not change.
I'm curious about the out of control db queries too, think I'll check the report and see if it has anymore details on that
Incompetence compounds over time.
The main problem is almost always the management.
If they tell you "this project is going to use X" you tell them "X is truly outdated and almost no one uses it" and they just shush you.
Well, no one knows about X and now the whole team has to learn X with outdated documentation or lost documentation.
This has happened to me a few times, when a boss asked me to use X or Y, I told him how outdated it was, then everything just slow downs a lot because there is hardly documentation and just some online posts from 2009 or 2010
This is basically just what happens when there isn't leadership. The point of a leader is to make sure everyone is working together to achieve the end goal, not doing their own thing based on their own version of the end goal that doesn't fit together at the end.
(Continuous integration and testing are just two specific technical mechanisms that ensure everyone's pieces fit together)
@@asii_k I assume it's just the N+1 queries problem: because you put more effort in making the code "clean" than fast, you have a "get whatever from database" function and the only way to get 100 whatevers is to call it 100 times instead of asking the database for all 100 whatevers at the same time.
Meanwhile an Indian UA-camr can casually build that in a week
Maybe like a college project
Fr
As someone who doesn't know anything about web dev, I find it so ridiculous how over complicated everything seems to be. There is no way a basic webpage and form colector should require 50 services built by 10 different teams
They obviously needed wEbScALe !
as a webdev i can say: no, it realy does not. projects like this would be solvable by a single team.
the decisions made along the way were the problem.
i have seen it myself. a little change from 50 shown characters to 500 can easily cost 100k when you have badly written software, no way of testing, different contractors that are not willing to help you and multiple instances of ppl having to accept the execution of the change.
whats even worse is that the good players quit at this point and all left is a bunch of losers operating.
most of any modern website you use is built with many different libraries written by developers all over the world. no one writes the whole thing themselves because someone else has written some piece of code and it's much better than you could hope to. it's easier and smarter to just integrate that into your workflow. a "basic" modern looking web page would be a pain to write all from scratch, especially if you want to serve many different people.
worked on it a few years after the time period in this video, long story short it's not that simple, hundreds of thousands of lines of backend code levels of not that simple.
@@parabolicpanorama But why would it be such a pain to write it from scratch?
Sure, for some things you really should use libraries, but you can't overuse them (or worse, make the project only by linking libraries together), because you can get boxed in and are unable to change certain functions without breaking everything.
With purpose built code, you also have the advantage of designing the system in the most optimal way for what you're trying to achieve, so even if your code isn't as fully optimised, it could still be faster if it is less bloated with unnecesary features, right?
You should really do a video on when Helse Midt (Norway's healthcare organization that covers the middle area of Norway) chose to change their journal system.
They started with a bidding process. All vendors welcome. Pretty early in the bidding process however they decided to disqualify DIPS, who was delivering the same journal system to Helse Sør-Øst, Helse Nord and Helse Vest - the three other govt. healthcare organizations in Norway. Yes, they really disqualified the one piece of software that was used literally everywhere else. Who did they go with? Well, good'old american EPIC - and they chose to call the new system "Helseplatformen". Did it go well? NOPE.
It has been a shitshow from day one. Most notably, 16 000 critical letters to patients were discovered to not have been delivered, which delayed a whole boatload of appointments and also put some patients lives at risk (Helseplatformen blamed this on user error - yes, really, 16 000 missing letters was the result of a user error made by several hundred users was to blame on the users, not the system). After this, the director of Helseplatformen stepped down due to the huge wave of criticism following the reveal of these issues.
One doctor actually saved a patient life after being unsure about wether or not a letter was sent. He asked a colleague to directly contact the patient to ensure he got the information he needed. The patient had a blood clot. Had the hospital not reached out to him directly, he could have died.
Currently, 44 individual cases have been confirmed where patients have not received the help they needed in time, directly as a result of Helseplatformen.
In a survey conducted among the staff of one of Helse Midt's largest hospitals, 27% of the nurses were considering quitting due to the issues.
The feature for referring patients to another hospital, which is a critical feature of any hospital journal system, was non-existent when the system was first implemented. After several months this feature was finally implemented, but it caused a massive amount of manual labour for the hospital staff - who are already understaffed.
Some hospital directors are quoted in meetings discussing the backlash from doctors and healthcare workers as "echo chambers", "hate groups" and "whining from angry doctors" - which isn't really helping the issue. The healthcare on their side are claiming that they are providing the directors with "constructive criticism" about legitimate issues with the system.
In addition, the price for the new system has increased by 35%, from 3.7 billion kroner to 5 billion kroner - and it has been implemented in less than half of the healthcare institutions in the region.
UX/UI wise it's also a complete mess. It's one of the worst pieces of software I've ever seen in modern times. If you search for Helseplatformen on google you can see their take on inputting a date/time in the system. It's a complete scandal.
Now, a ton of doctors and healthcare workers have protested against Helseplatformen - with some healthcare institutions outright refusing to change to the new system, instead opting to use the old one. In the meantime Helse Sør-Øst, Nord and Vest are looking at this and wondering why the hell they didn't go with DIPS, so they could all use the same system and be able to share journals across all hospitals...
Really awful story 😢 but quite frequent the way how federal software projects work. It's a real mess, and I wonder why nothing is learned from this. There are brilliant tech companies out there making immensely powerful systems; why can't this knowledge not be brought into federal software??? 🤷🏼♂️🤷🏼♂️
Wow I don't think I've ever seen a worse ui. The date input looks like what a day 3 high-school student would make in a programming class.
Can anyone share a link to the date picker lmao
That's the Hell in HELLseplattformen. 😮
Everything that happened in this scenario is covered in "The Phoenix Project". That book still holds up and I wish more people in our government would read it.
as an independent fullstack developer, this was absolutely depressing to watch
Man, you're a really great storyteller and your videos are truly awesome.
I'm glad I randomly stumbled into this channel. You deserve way more recognition!
As an engineer myself I find these videos very interesting and valuable. Keep 'em coming!
5:42 This notepad scene made my day.
Do the Phoenix Pay System next! You want horror story nightmare, that's the place to find one.
happen to also have en.wikipedia.org/wiki/CGI_Inc. working on it? i went through a few years getting on gov contracts after them, lot of clown code.
"first they didnt document anything but that was a standard, so we can give them a pass here", this line just cracks me up😂😂😂. more or less standard across most of the companies😂😂
Painfully true.
These videos are awesome, thank you so much for making these. Was also thinking, I'd be totally down to listen to this via podcast as well, but that's a lot of extra work on what I imagine is already a ton of work. Thanks again, the effort/content is really appreciated.
it's amazing that such a fresh video has CC already. thank you!
I discovered your channel today on the way home from work and I've already watched all your videos. Please keep it up. I hope your ad revenue allows you to make them more often.
Frequent changes is basically the point of processes like Scrum - you still gotta pay for your changes, but at least the process can accommodate them. And you still have to eventually figure out what you want if you want it to ever get done. And you can't set a deadline if you don't know what you want (but you can still try to build it).
Just limot your scope - but is a useful life tip
This video is great. Love the style. I really appreciate that it's still going into depth to an extent but trying to keep it accessible.
I signed up back in 2013 and didn't get coverage, 9 years later I tried to login using the saved information and they had removed the security question I used so I was no longer able to recover my account. I called support and tried for two hours to recover my account. After 10 years - still no healthcare, they told me I have to have my wife call in as I'm now under her account but we do not and have never had coverage. It's great to know they were poorly mismanaged - when I signed up in 2013 the forms I was supposed to fill out were not available yet so I obtained an account but could not complete the setup. I was told to keep checking back.
California got fed up and did their own Covered California. Surprisingly, it works. Ive been signed up since the second year and saved a fortune.
In our country there was the same hi-tech attempt to enable parents to enroll their children in kindergartens. It was fiasco, system was down on day one, and every second kid was not in the lists. At it repeats each year :D
This is an excellent video. Keep up the good work!
One of the most underrated UA-cam content creator. Hilarious and really informative!
I feel like the all star team had an advantage where the business rules were more well understood. If you talk to a vendor who doesn’t even understand the project how could they possibly convey it to you to build. Also involving so many teams is just awful. Makes comms so much more difficult. The government likes over complicating anything.
Delaying security testing to 6 months after launch? So, 6 months for malicious parties to exploit any possible vulnerabilities. That seems pretty bad.
As an software engineer specialised in putting down fires, your channel is just the best. Thank you.
These documentaries of yours are exactly what I was looking for in terms of entertainment; hilarious and informative
Your videos keeps reinforcing the idea that you don't need hundreds of people to develop software even for national use, if anything adding more people create more communication problem which lead to wasting even more time and money. If the people at the government had chosen instead to go with a smaller team(20 people at most), they wouldn't have encountered all these issues. It would have cost significantly less money and it would probably have been delivered on time. They could even have hired multiple teams to develop multiple concurrent version of the website and chosen the best out of them.
That isn't the government way. Throwing more money and resources at something that is failing is always the governments course of action. The contracts make more money, the more incompetent they are.
I agree, hiring 20 really good developers is much better than hiring 100 bad developers. But that isn't the way the government sees it.
@@crissd8283 Honestly, even 20 bad developers should mostly be better than 100 bad developers lol
This is known as the Mythical Man Month. Adding more people creates complexity, which can slow things down.
Particularly since ultimately it was "just" a large-scale insurance broking website. There are dozens of pre-existing ones - it wasn't needing to do what certain government projects must: something new that also reaches everyone,.
jeah, some of the most amazing and solid software used by millions of people is written by tiny teams or even solo devs. The entire developer team at f**king Adobe is only around 30 people. An entire PC fleet management system was written by a few German teachers over the curse of a few years and is now used in **many** schools across Germany.
DXVK, the library translating directX to vulkan and embarrassing both AMD and Intel, by, in many cases, substantially outperforming their native drivers, was originally written by a single dev, and is now maintained by only 2 devs. It has since been included in Intel's arc drivers to fix the severe performance problems they had at launch.
Quality software IMO requires only 4 things (for the coding part, i'm can't speak about the other things):
A clear *vision* and focus.
At least *one* *good* and, perhaps even more important, *motivated* *programmer*.
And *time* to improve after an preferably soon usage start.
Communication between everyone involved (this point is easier the fewer people work on the project)
if the vision is missing, you will produce something, but end up with something unusable, and possibly useless.
If you don't have a single good and motivated programmer, it will end up a buggy and mostly unfixable product, no matter how great the vision.
If you don't give it time (in use) the program will either be unfinished or inadequate.
If the Communication is failing, all three of them can happen. (bad communication costs time, robs motivation, and prevents a vision from driving the team forward)
I think most software fails on the Vision or Communication front. Especially government or large company software.
Wow, doubling the headcount with 3 months to deliver the project. Understandable, it's not like that's a well documented mistake since the 70's or something... haha
As a career DBA, I love the database-focused nature of these documentaries.
Software Design teacher showed this too our class as a case study and bad example, keep up the good work!
I write code professionally.
This video made me want to cry.
How could they possibly do *everything* wrong? It's like a fucking bingo card of how never to manage a software project. HOW!?
As someone who works in an agile change environment listening to you describe it as what it theoretically is supposed to be was kind of hilarious. In practice the daily stand ups are a painful mix of the product owners chastising people for not working faster, people saying "yup still working on that thing which I already said in the last three meetings" and the rest just trying to avoid saying anything which might put them in the product owners firing line. The method of controlling work on the kanban board is disorganised and communication with the business is essentially cut off by the whole agile process. The only time it works well is when you have someone with enough clout and a modicum of common sense to organise the work on spreadsheets outside of the whole process and actually get some overarching control.
If your dailies consist of product owners chastising you or others over time spent, you quit.
Simple as that.
Why are product owners in scrum?
@@P4INKiller Or fire the product owner, but since that's not something an individual dev has any power over yeah I agree at that point you start sending out CVs
Love these videos. Been waiting for you to upload again.
The amount of times I have to pause in your videos are insane, but in a good way. I love the memes / articles curated into the video.
Me after a good night sleep: "Productivity increased by 9000%"
I just wanted to leave a comment to say that your videos about these tech disasters are brilliant. Your six most recent videos have all been so interesting and very well produced. I'm sure if you keep putting out grerat content like this then the algorithm will reward you handsomely and your channel will grow very quickly. Keep up the great work!
This sounds exactly like my experience with government projects in Australia. Not all of them are like this, but I was in the shit house dept where everyone was trying to leave.
You should do one of these on the Queensland Healthcare Payroll systems debable. Similar amount of money was wasted, maybe more, and it took down a whole State government here in Australia. Absolutely great business case study on how NOT TO run an IT project.
Do you have info or relevant docs regarding this?
I'm not a web developer, so maybe this is a stupid question, but how could a website whose only purpose is to allow people to buy insurance cost nearly a billion dollars, let alone the nearly half a billion dollar original budget?
I think it was all the change requests. If leadership says to do A, then later says A is wrong, do B, then all the money spent making A was wasted.
I'm not a dev (yet) but I guess it's not only a website. The website is what user sees but devs also need to create an entire backend. It's similar to frontoffice and backoffice - as a customer you'll never see backoffice but it is needed to provide smooth operation.
Such service relies on authentication based on some gov't services and needs to register your actions in another gov't service. Yet (as a noob) I still believe that the budget for this could be at least 5x lower and it would be sufficient for creating a decent system. It's just too many managers taking too high wages for what they are doing (i.e. creating chaos and disruption)
It's the Government. "Other people, spending other people's money on other people".
Plus I believe the contract was given to a friend of Michelle Obama's, not a well known tech company.
No website, should ever even cost close to that much money.
welcome to gov inefficiencies... think how many gov employees were hired for how much time and how many sub contractors were involved and how many times these requirements were changed. Also bringing in top tier talent, doubling resources, ... all these things last min cost more.
Not to mention that the amount mentioned doesn't mentioned what is really covered. it could be the cost for the website for 5-8yrs with support and staff. Again, not justifying the ridiculous amount of $ for what essentially is a website (and not a very complicated one at that) - it is not like a brokerage that needs to do transactions in real time or an airline booking website that needs to search from airlines in realtime from all over the world and be responsive.
Why I don't get it why did they have to rebuild the damn website from the first one.
The first one they had worked.
Just because it didn't let you put in payment information or sign up doesn't really matter, you're going to have to call the insurance company anyways when something doesn't work correctly.
For a website that was supposed to make getting health insurance easy was an extremely difficult task and a big waste of time, and just convinced people just to shop around by calling everyone that they knew and to use Google and reddit.
"They slept in nearby hotels, while working 24 hour shifts" 🤣
The amount of miscommunications here is even more amazing than Operation Viking
Directors have tenures of less than a year? That's not far off normal for my government directorate.
Fire makes every presentation better, especially when you combine fire and the government.
Love that the investigation from the government actually got put to use as a reference for this video!
Wow, Terremark was the most efficient one in this whole story and all they did was double it and give it to the next person.
An entire 6 people were able to successfully sign-up on day 1? 100% guaranteed at least SOMEONE got fired for making the system too easy to use.
It was only 1 person who made six different accounts.
He was Chuck Norris 😂😂
the good part is that this being a government project, it's all documented, but this is all too common in the private sector too
It's actually a recurring theme for government programs to be effectively leaderless. Often the guy in charge is a politician who is preoccupied with politics and he just rubber stamps proposals since he isn't really involved in operations. Likewise, sometimes certain tasks fall under several agencies at once so progress moves at a snail's pace as any given file must go on a trek between as much as dozen agencies before being finished
It's amazing how often shit goes completely sideways because of some arbitrary deadline someone pulled out of their ass.
I love the way you tell those tech stories. Really cool to follow
Please make more of these type of videos.
6:10 yeah of course, more people means more code right, should be faster
"we did the hard part already by thinking what it should look like, you just have to code it out, easy. we did all the thinking already"
I work at a big government contractor (formerly a data scientist at a tech company). I can confirm that government contractors are as ineffective and inefficient and bureaucratic as government agencies. It’s really bad. People here barely know how to write code.
This sounds pretty much like the project at my work.
Love how his voice is so monotone. It's like the computers would laugh about these incidents
"If you think it's expansive to hire professionals just wait until you hire amateurs"
- Sone guy on the internet who probably got it from someone else
13:00 I was thinking you meant 464k... that’s the worst 400M ever spent...
I thought he meant 4.64 million but then he said 58 million... What in the fuck is all this.
As Joel Spolsky said, non-technical people managing software projects is like a non-surfer relying on their advisors standing on the shore to tell them what to do.
Project managers are necessary to make decisions, but they need to know the domain. Linus Torvalds could be an ordinary dev, but he focuses on managing the whole Linux kernel. His job is to make the technical decisions and give the implementation to trusted, competent people. But an MBA who wasn't a programmer wouldn't understand the decisions that need to be made, so would come unstuck unless someone technical is really in charge.
What I love about government contracts is, that you make a buck if you can deliver on time but you make an even bigger buck if you can't.
3:15, the monster just interleaved "es kyu el" and "sequel".
The scariest part of this isn't the incompetence, it's the fact that **healthcare.gov** was able to get exempted from any security testing until 6 months after launch.
Why would anyone expect any gov't to actually be capable of doing what it's designed to do?
Holy shit at some point why didn’t anyone think we need to rebuild the entire project with people who know what they’re doing
Your meme game is god tier...I kneel
A work of blood sweat and tears that the A Team did singlehanded more less in a 1/16th of the time
Classic us gov lmao. They need to actually hire people other than dinosaurs there
Why wouldn't you want to hire someone who brags about 30 years of experience in a technology that's been obsolete for 25 years? You would hate to waste such valuable experience!
The production launch is the end to end testing… gosh that brings back bad memories 🤢
imagine being the country that has the (in)famous silicon valley but fails to build a website
You couldn’t pay me enough to be a project manager especially in the government holy shit
Your videos are dank AF. Keep up the great work🤙
How does this channel not have more views and subs is strange to me
I’ve never heard of Marklogic until this vid and I am in software development for 20+ years, had to learn it exists from this vid 😂
This channel satisfies an inner IT Schaudenfreude which is hard to explain
Lack of experience, corporate greed and incompetence, all in one project. I feel for the front line workers, all those sleepless nights to clean up after what was done by the leadership. Pity it still happens in numerous other projects as well.
What? Business side constantly changing requirements resulted in mishmash code? Imagine that!
Literally art Kevin. Keep it up!
your visuals during any of your videos are amazing and hilarious!😂
Worth to know: making the bottom part of a thumbnail red - makes it look like the video has already been watched, even if it was not.
2 years for a an account creation popup that doesn't even work is exactly how i imagine govt projects would work.
People: Haha, government disorganised, private companies better.
Private companies: Even more disorganised
Dude just watched all five of your technical documentaries, they are so entertaining. This one is especially triggering though haha
Im pretty sure 3 experienced programmer with 0 overhead can pump out this thing in the same time with less issues lol
I'm simply baffled by the excessive database queries. Like, did dev leadership forget that RAM exists for them to use to hold values as variables? Were they querying the database every time they needed to access data? I've literally only been working with SQL part-time for about three months and even I know better. The only reason I can think of that might justify those excessive queries is if they're running on servers with lightning-fast CPUs and pathetically little RAM, and even then, the far better and far more obvious solution is to just use better servers.
Also, wtf made them think NoSQL was inherently more scalable? Sure, for some applications, a schema is overly restrictive, but for some applications, a schema provides immensely valuable structure to the data, and since, as the video indicates, NoSQL is a catch-all for a variety of data structures that work pretty differently from each other, the fact that they decided they "needed" NoSQL without knowing what kind of NoSQL they needed just screams "I got hired to lead this project as a favor to my wealthy political donor parents, I'm not actually a software engineer".
I sure hope the over-budget funding we ponied up went straight into the pockets of the hard-working programmers who pulled 24-hour shifts to make this thing work; the incompetent leadership didn't deserve a cent beyond their original contract, and if the leaders continued to get paid after the Oct 1, 2013, launch date, that raises some serious questions about whether or not they actually intended to provide a working product on time (or at all).
It's dead easy to generate excessive queries in a big system. A system where no one communicates with each other and many lack technical competence like this? That increases exponentially.
You need A, okay, make a DB call. Oh you also need B from this function, let's call it. But that function is actually making 5 calls to return B to you. Now, you need C, and C can actually be retrieved along B in that earlier function call, but you don't know that, or can't figure out how to retrieve C alongside B, so you make a call only for C.
Do this enough times, in enough abstraction layers, and you end up making 100x the calls you need.
Then you also have shit like operations that can be combined in a transaction, but are being done sequentially in separate calls.
Let me introduce you to in house frameworks with custom orm, which will haunt you in your dreams.
Pretty sure they had a relationship with Michelle Obama. Just corruption.
Your org chart with tenure as free text field, fav food and color, SSN for each employee: "Y0L0!" - Every information system specialist out there: "Noooooooooooooooooooo~"