I don't know why, but this codebase has post-collapse cyberpunk vibes. Like, just mashing together highly complex and extremely sophisticated machines to create a desk fan. It's beautiful.
There's an entire Kubernetes stack controlling the fan's rotation servo. 7 microservices are involved in accepting updates to the desired rotation angles over WiFi. That lives on a slick webpage that works in any browser. But changing the fan speed is in a different portal that needs to load ActiveX components and requires IE 6. The fan has no buttons.
It's the new "Google Translate an instruction manual through thirty languages, and then back to English", but for programming. The future is stupid. And just wait till they use all the code that H1Bs poorly copy pasta'd from StackOverflow, as LLM training material. Then LLMs can truly "do the needful" too!
nah, he is just playing 4d chess, we all know that would drive any emerging AGI straight into insanity. but if his gambit fails, he will have to answer for his deplorable crimes against the machine.
If you want an insight into the quality of corporate code, you only have to look at the scandal of Fujitsu's POS system for the Post Office in the UK - the cause of the greatest miscarriage of justice in our history. At the public enquiry, multiple witnesses have attested to the shambolic development process - no proper specs or docs, no coding standards, inept developers and no effective testing. And yet the company doubled down and allowed hundreds of postmasters to be bankrupted and even imprisoned for cash shortfalls caused by their bugs.
@@GreedoShot What are you suggesting here - that you install a complex POS and accounting system into 12,000 branches without testing that it actually works?
@@Volvith you guys are forgetting to factor in all the smaller scale stuff glued together at large local businesses. I worked my regions largest car dealership as a web developer and the dude that made the database was a idiot husband of someone who was in the company family and he had no idea what he was talking about. We were just automating the business office so our sever never really had more then a couple hundred users but the database was so bad they had to go in and manually make some of the basic queries not time out at 2 minutes. It was a couple hundred thousands for that small project by the time I had gotten hired and it was wild. The idiot lead guy and the second in charge were using notepad++ and ftping into the same server. They would overwrite each others code intentionally. They would sabotage each others code intentionally. Eventually they fired me because they told me to start ripping social security numbers out of some other licensed software they used and store social security numbers + other personal information in plain text and I said no. Management actually told me I was crazy because I said that storing social security numbers in base 64 isnt encryption.
you have no idea bro... i'm gonna tell ya a company with the same systems and trash software here, let's see how many will find it: FCA (yes the automobile group Fiat Chrysler Automobiles) I can't tell you much more but trust me you would want to rm -rf your brain after seeing it...
Legit. The cleanest code I've ever seen was written by students at school, it's all be downhill from there. Every single thing you learned not to do is what you find done in corporate. xD
@@jambonmusical2689The old "good enough vs done right" struggle. Short-term financial gain almost always leads to "good enough" and a growing pile of tech debt.
Yep... Looking at you Siemens with your abhorrent structure for Step 7 and WinCC... My God, their software is so dog water that it reminds me of programs I saw growing up in the early 2000s.
Of course it is the cleanest code because you only work on this projects on your own. If you design the codebase completely on your own, it is far easier to do so than in a project where >100 people worked on and > 200 people who worked on left the company a while ago. Very often you have to do and to work with legacy systems. And those work for a good reason how they do. And it is very easy to judge in retrospective from your current knowledge rather than at the time when people actually wrote that code.
I worked for a pretty big gambling company and they stored javascript, html and css in columns in a MSSQL db. Before that i had never had a panic attack during a project overview meeting.
Reminds me the old days where senior dev was asked if we could implement new feature into program and the only respond you heard from across the room was "I have no idea." im glad we didn't get all kicked out from company 🤣
I worked in a factory where their internal production management and quality tracking system was custom built in house (by someone who was entirely self taught). Not only did it store HTML in an SQL db (not sure which server) but it had raw SQL queries in JavaScript and using simple string concatenation to build queries such that little Bobby drop tables would have a field day. When I discovered this I brought it up to the poor sod tasked with maintaining this mess (since the creator had since left and turned this codebase into a commercial product and started a company selling it) they just shrugged.
i have also encountered HTML in the database (it was bad). unless the system actually needs to be that dynamic, it's a bad design. it's bad when it's the same HTML every time with only specific portions that vary. someone mentioned WordPress, but i'm on the fence about that as it stores plugins (i don't really know the WordPress internals though)
@@xybersurfer The css and js part really tripped me up. Creating dynamic css by writing SQL statements, i actually refused to touch any of the javascript. I guess a blessing in disguise because i learnt all the new fancy css things.
I manage a database for my company that was designed by the guy who came before me. No joke. That guy stepped in front of a train. Anyway, there is a table in this database where the primary key is integers 1 through 36. The values associated with each of those primary keys is also integers. 1 through 36. That's the table. And if you delete it, it breaks the whole database.
It's probably for a good reason that's usually not apparent at first glance. Look at the code utilising the table to understand why. Primeagen had already mentioned this, usually things like this are done for a good reason and it takes you a while to understand why.
@raptorate2872 I appreciate that having humility and giving the benefit of the doubt are great qualities in the workplace, but this is the youtube comments section. I am wracking my brain to come up with literally any logical explanation for what OP described, and I am coming up 100% blank. Maybe you can enlighten us?
Maybe the code was supposed to map from some ids to other ids because some ids could be aliases used for different purposes, but in the end there was no need for extra aliases and the table ended up being trivial?
I suppose it is entirely possible that it is 100% coincidental that the 1st 36 values were the integers 1-36, and maybe there was no reason this data couldn’t change in the future or that new data could be added that doesn’t conform to the pattern. It would seem highly improbable, but maybe it’s just pure happenstance that the table appears to be pointless, and the next record could be fk 37 with value of 92738. If that’s genuinely the case then just wow.
@@raptorate2872 it is literally because there is geographic data in the database, and 1 through 36 are the section numbers in a township and rather than type the section number, he thought it would be 'cool' to have a lookup for it. So instead of typing 1 for Section 1 of the township, you type 1 for the lookup, and it fetches the 1 for you. Probably pretty easy to resolve, if I wanted to. But the guy was my friend, and now he's gone, and I don't want to untangle his beautiful messes. (That came out a lot less hetero than intended.)
How JSON showed up in our production database? "I'll dump this here for convenience until I write a generator. People will use the method to retrieve this anyway." Two weeks later: "Nice, it is done." Two minutes later still: "Kill me, please. How is this column already called from over 100 places in the code? Why didn't they use the method?" The name of the column? never_use
There's nothing wrong with a JSON blob in a database. In fact, it's how I design all my database tables. Works great. You only need to have columns where you are going to apply indexes to. The data in those columns originates from the JSON blob. So, yes, you have some data duplication, but then your application can both search really fast via indexes and also expand a complex object in the application. JSON is supported in every language. You can also technically search on the JSON blob using DB queries but those will inevitably be slower. To solve that problem, you add "archive" tables where old data periodically gets moved into the archive.
When someone says "its Year X, you should use Y already", I don't think they had lots of longstanding customers. I want to see them how they argue, that the customer should switch to a new database and transfer 10+ years of medical, finanical, structural data.
Just because a technology is new does not mean that's a good reason to transition, frankly in today's interdependency hell I would argue the opposite. Unless there are viable business incentives to transition, all you're doing is hurting yourself.
The last company I worked for that did medical records, had automatic migration policies. The customer had a defined interface to the database. The database could be updated, without changing that interface. If the customer is using your raw data tables, you have a bad design.
I cant wrap my head around the calendar table. Running out of columns is one thing, poor design and all, but be unable to login if you ran out of calendar entries? What am I even reading, how?
I think the calendar table would be used to quickly calculate N business days from current date. I think it is manually updated to allow for changes in holidays every year.
@@amehybrid You may be on to something. I worked on a timekeeping system that had to account for holidays in different jurisdictions. In some places, people were legally required to pay double if they worked on Good Friday. Have you ever tried to automate a calendar setting that has a mixture of solar, lunar, and ecclesiastical factors? I figured it out, but I still think it may need an adjustment in 100 years or so.
Some applications use different calendar types. You have a typical Gregorian calendar. Then you have a fiscal calendar where the first month of the year is October and days start on Monday. Broadcast television calendar uses "square" months where a month can start on the 27th of the previous month. Then you have calendar type/month/quarter/year values in other tables.
At my company plenty of our codebase relies on a (working days) calendar table. I remember in the past we had a codebase (luckily taken out of use now) that had its own internal calendar table, but for some reason the dev only added a year of dates at a time. So every time Januari rolled around it would stop working until somebody poked the dev that maintained it and he remembered he had to update that.
We have a similar thing in our company. A scheduler table where you have to manually insert records otherwise a critical job wouldn’t happen. The guy that wrote this put a comment in his hand over docs that said every December you needed to add records for the coming year…
I once worked in a company where I learned there's a hard limit on how many elseif statements you can have in C++, since they are essentially just differently formatted nested if statements, and they have a limit too. There were multiple files where they hit the limit. The solution? Just start a new if statement and pray to god it works. Of course, I spend days just figuring out this was the cause of a bug I was hunting. I never thought that somewhere in those thousands of lines the else-if just stopped and started with a new if.
else-if statements make me nervous because 9 out of 10 times as I am writing those, and I stop to think for a few minutes, I realize I'm approaching the problem completely stupidly and can refactor it into a single evaluation :)
what. the. fuck. Dude if I saw that I would 100000000% rewrite it as something a bit more sane... Even if it took me months, even if it got me fired for not "being productive".
Jesus Creeping Christ. Well that is an interesting bit of trivia I'll have to remember. Then I can casually drop that little tidbit next time devs are telling war stories. The implication being, anyone who knows that fact has seen some serious shit.
Nobody likes the select...case syntax. I mean I'm sure there's somebody out there who likes it but they're weird. However, it does serve a purpose. It's also pretty fast.
Fun fact: the original Relational Model (the basis of relational databases) did not use names for columns, but index numbers. That was 1969 (nice) - by the 1990s Codd had found people who were using over 128 columns, so he decided using names was better than remembering that column 36 was the surname of a user.
I work as a support engineer for a company that manages the infrastructure of at least 200 companies. A couple big ones, _and a whole lot of truly sketchy shit._ We have a lot of legacy customers as well, that got rolled over from other tiny support companies we ate. (we's a growin' boye) Let me just summarize how horrible it can get in one sentence: _I genuinely think we've found more bugs in Microsoft's code than any other company on the planet. And it's not even our job._ I love my job, but God bless the soul of anyone picking up the phone for a customer that nobody recognizes. Because either they just got rolled over from some tiny shit creeck company we took over from, or even worse, _it's the first time in 6 years they called, and you're about to meet the end-boss of DIY legacy code._
I work for a supermarket chain and this video has made me realise that the POS machines freezing every 2 minutes or so is probably not the fault of their ancient hardware Don’t even get me started on when a technician working on the self serve registers told me the error we were having was caused by the register getting stuck in a while loop and the only way to break it out was to cause another error by physically removing the receipt roll from the printer.
I used to work for a clothes retailer in online order fulfillment. At the time they had two options for scanning equipment - the "legacy" Motorola MC3100 handheld computers, well over 10 years old at this point and running on Windows CE and the brand new Honeywell CX45s running a fresh and modern looking variant of the pick system over Android. The MC3100s blew the Honeywells to smithereens in basically every department. More rugged, better ergonomics, and somehow significantly faster despite having equivalent specs to your mom's Gateway from 1996. Their one issue with them was that all the batteries were shot, but the Honeywells tended to overheat in their scanner cradle and die so it was really a wash as to which lasted longer. Software matters, and honestly a lot of enterprising computers don't really need much of anything at all if their whole purpose is sending a few strings to a database. I'm not a programmer but thought about buying one of those Motos and seeing what kind of things I can make for it.
@@jordanhildebrandt3705 I mean, really, since sqlite has dynamically typed columns, you can just make any table a 3-column one (row_id, column_name, value) with unique(row_id, column_name) and then you can have as many columns as you want and even dynamically generate columns! ;-)
@@tomlxyz I see both i16 and u16 being used for column count/index in various places. Not in the database format though afaict, so perhaps if you're extremely eager you could try "fixing" all of these to be 32-bit. It would be a terrible idea regardless, I'm 99% sure that accessing a row involves some work that scales linearly with column count. Of course, since sqlite is dynamically typed, the solution for this "problem" is easy! Just split each row into a whole bunch of (row_id, col_name, value) triplets and you can have unlimited columns! ;-)
It’s 2024 - one of our system uses a table “calendar” and we had a production bug that stopped all our analysis because the script that was responsible to renew the “calendar” was crashing for a couple of months and no one knew and the analytical queries depended on it… We also have the CSV thingy. It’s anecdotal, but companies run like this even now.
I program since 8 months, i am in no means good but when buidling an application for a database i went and made all transcations and componets modular so that you cloud change for example a players team mid game without the app crashing. And hearing they do everything what was said. I feel like an Senior Dev 😂
The beaty of over the top solutions is more that it's more of an "art" rather than engineering. It should be place in gallery for software engineers so they could reflect on their life, while looking at this art.
A bunch of people are essentially trying to reproduce this practice across the industry, only the name of the guy who gets the crappy CSV file and presses the button that may or may not import business critical changes to the database is named Devin. Making your core business processes unknown magic is sometimes necessary, but long term inexorably leads to terrifyingly fragile stacks of "just make it work" quick fixes.
@@terryjophlin Pretty much everything complex is built like that, software to submarines to shopping malls. You can usually have one unreliable layer, but multiple eventually causes failure. In Florida there was a concrete issue for a few years where bad concrete was unknowingly being shipped to new construction projects. Years later, high rises, parking garages, and large oceanfront condo towers started falling down, sometimes catastrophically collapsing with people inside. Most of them had a second factor contributing to the failure, like adding a floor above the original design, or moving the pillars just a few feet. Things that stretched tolerances and were almost always signed off on by an engineer, but they were unaware that those tolerances were already stretched by the substandard concrete mix.
11:50 as someone who took a database class working in SQL server, writing, queries and other stuff. And who loves back in development. If I was an intern, I probably would have written that application to make those SQL statements… And then not told anybody so I could get my eight hours of work done in Three minutes Just running my application
Corporate code vs. hobby code..... Don't assume corporate means better. It's convoluted, hotfixed, non optimized, ... But that's since you used 100 times more time on your hobby code, than any manager will allow an employee.
Agreed. Ever since I got into the real world, I really started to notice this. Hell, even professional software (**cough TIA Portal, WinCC, and EPLAN**) can be beyond absolute garbage.
Yeah the key difference is that i feel shame if i push garbage code on my personal project to a public repo but noone cares or will ever shame me for garbage code i write in the proprietary repo at work. Usually you are even discouraged to take more time to refactor/clean up your code, becuase "customer wants new feature NOW not in 2 days" and then it takes 2 senior devs a month to clean up my mess 3 years later costing 100 times more
With this kewill reflection thing, I think he missed the most beautiful point. This sounds so weird, so problematic, so hellish. Colleagues told him not to dive into it, because they knew what weird magic shit show he'd find. Everybody gave up. But he found it. SOMEONE HAD DISABLED A CRONJOB. That weird reflection on empty glasses was still working. That incredible weird part. That part was fine. WTF 😂
Right on! I hate design patterns, especially when paired with OOP inheritance hierarchies! I can only imagine how insane backend architect 'design patterns' are...
I have a much less exciting version of this. We ran out of tables AND columns (of our paid cms). At one point we had to embed JSON of a “new table” in another table that was mostly just used to store strings. Literally if we wanted to do select statements from one “table” we would instead query a different table, filter the rows, convert string to json, extract what we need. Holy moly. Company no longer exists
1:10 - This hit way too close to home. I do a worse job at work since there are: 1. Time pressure (partly as a function of me/my team not being good enough to fight against as much as we should) 2. Social pressure and general social negotiations that cause the code to become a mish-mash of styles and quality levels. Yes, even with code reviews and pair programming. You are not immune. 3. You are not coding your baby all the time. Sure, the best times are when you can fully gaslight yourself to go in with that mindset, but every once in a while it just... fails to materialize to the extent that you want. If the project is not very short this can thankfully be overcome.
JSON on a random Column? That is cute. The bane of my existence is a Table, that stores serialized PHP Arrays in one column and pure HTML in the Next. Compared to that, JSON ain't half bad. Organically grown codebases started by some dude in his bedroom over a decade ago are such a joy to work on.
@@olbluelips Mate don't even start. I'm in a situation now where my company acquired some 15 years old financial planning company. Their software was written by one old boomer who decided to make a proprietary scripting language for HTML pages. His reasoning? Well they couldn't get the pages to work correctly in HTML and CSS... the scripting language transliterates directly into HTML and CSS. A few days ago I had a bug where a page refused to behave like I was telling it to. It was literally "Show these columns if they are not null". Turns out I was using "if (col[0] not null)" instead of "IF (col[0] not null)" and they are completely different in this proprietary language pile of wank.
This is simply beautiful. Now I'm dealing with a similarly messy monorepo codebase with react and I just embraced the chaos, concerns for code duplication and design are gone and it's now about just doing the thing that needs to be done. I hate react, but now with this new approach I kinda dont mind it anymore and found my peace with it. It's refreshing in some ways.
12:10 Writing single-task application was considered too hard so interns wrote and executed custom SQL queries against production database??? How? Why?
Haha, yea... and having 3 interns whose sole job is to keep doing these one off queries for who knows how long is easier than writing the one off application. WTF?
easier to get an all-purpose intern who on paper can tackle any easy issues in the backlog, than convincing management to assign 4 weeks of work to build that application and then make the all-purpose intern only do sql issues
We had a job that did pivot 3000+ columns on a daily.. including over9000 unions, 50+ table joins, casts, UDFs you name it. Quant who wrote it, modified it regularly, and when it broke (after each update), we had call someone it to supply us with coffee and food, while we fought this monster. Day? Night ? Weekend? The only upside that it was written sql and running on sql server and not some unicorn level system. Sometimes I wonder what this particular Quant was smoking
A manually populated calendar can make sense sometimes for things like public holidays that don't always follow a predictable pattern. But you shouldn't have to manually enter the *entire* calendar...
We have some JSON in some tables where the schema changes and instead of adding new columns, the object gets stored in JSON. The rest of the database is properly relational. Sometimes it's just keeping all your data in a single db.
This. Example: We store the configurations for a tool in JSON - we need a few hundred configs - and the history of configuration edits (because it’s one of those ‘picky’ tools that it’s easy to fuck up the configurations of, but we also need to change them sometimes). The tool configs can be CRUDed using a simple browser UI, which shows the user feedback on their changes. The tool is a core part of our system in production and is constantly in use. When it comes to storing the data, I can think of ‘other ways to do it’ but not necessarily better ways to do it. Hard agree, however, on not needing to do any LIKE ‘%FOO%’ logic on it. I’ve been guilty of that in the past, and been responsible for tidying it up.
I have 100% experienced a "Gilfoyle's Hard Drives" situation. When I became aware of this problem, I took it upon myself to fix it immediately. It took awhile, but eventually everything was put into TFS. And at this point many years later, most of that code has been completely rewritten as part of other efforts.
This whole "I'm concerned about perf" and then proceed to use GC languages is so true. People who write the slowest most dog shit code say stuff like that on a consistent basis. It's those people who think having less lines of code makes the code run faster.
lmfao for real. Return to C++, return to "oops, accessed a dangling pointer because some dumbass didn't use a smart pointer"... But also return to "yeah, this function takes an immeasurably small amount of time to execute".
My last project ran into the problem that DB2 did not want to join more than 100 tables in a single query. The fix for this took me multiple months and was essentially a recreation of joins on the rich client. all table data was loaded into the client. This was also 23x faster than letting the database do it. (46 seconds to 2 seconds. But 1.7 seconds of that was jamming 40000 rows into a JTable)
@@vytah it does, it was actually quite optimized to even work at all, just the amount of data duplication due to joins was insane. Just by loading each text string only once I reduced the amount of data send by the server from 3GB per query to less than 200MB
I think prime hit on something relevant. Imo code should be open sourced after a certain period of time, for the good of mankind. Especially defunct projects. We do it with public domain (or at least used to until it kept getting pushed back), we do it with patents. We should really do it with code
it's funny how we ask ourselves "How did this happen" but I am sure the place I first worked at and where I run from will likely be like this in 3-5 years time. The ideas the people there not only think but execute is madness.
I am not a programmer. But I heard a phrase. "Real programming starts when you run out of memory." So I guess. "Real database management starts, when you run out columns."
The way that sounds to me is like every dev would just scrape the db, then write their own little app with the data. This caused devs to keep adding columns to store the data of/for the app. Get a new task, write a new app to solve that task, add column. etc I've worked at a place where all employees had open communication with each other cross departments. So every one would just talk directly to a dev to add something just for them that no one else in the company uses. If that person left, the code was still there, but no one uses it any more. A new employee would then go through that process with their own requested solution. It's kinda fun, for the devs and other employees, but the codebase is such a mess. Actually it sort of - but not really -is like blockchain. There's a central point and every one builds their own thing on top of it. To put it simple, every one had their own custom built dashboard.
First network admin job I ever had we had an Access database that indexed all the CAD drawings our technicians had to work on. Every technician had a VBA application on their system that would read and update status of drawings in that shared DB on the network. The database would be corrupted on an almost daily basis, and I eventually decided to look at the code to figure out why. When I finally dug in to the code I found that anywhere an exception could be thrown when trying to lock the DB for opening was the following line: *On Error Resume Next* Handling exceptions is for amateurs. Real men just write to the DB even if they got an exception.
We have json in our MS SQL DB, but not because we ran out of columns. Apparently they wanted to use a NoSqueal DB, but being a bank they would have had to have an expert with some kind of certification for it on staff, which they didn't, so into Squeal it went.
Yeah same here, except that it wasn't because certifications that we couldn't go nosql, but because of customer requirements. For some reason they insist on hooking our brand-new, modern application up to a 10 year old version of on-prem OracleDB.
@@henningerhenningstone691 Oh no that reminds me of that one company that had an interval (can't remember the exact number) where they would export SAP into an OracleDB for other Apps to consume.The documentation for that DB was a giant incomplete Excel sheet. It was impossible to connect some of the data because some IDs just weren't in the export.
@@miled8649 well, I can confidently say that Oracle is the worst DB in existence. Want to save an empty string to your string column, because that's what the user entered? Oh, let me make that null for ya. What, the column can't store nulls? Tough luck, now you have an error. And loads more nonsensical stories like this one. OracleDB is a meme in our team at this point.
As a prof dba, I am torn... I wanna go down there and say "KNOCK THIS OFF"... but then I know I'd never be allowed to leave... I'd be "here is how you should do this" and "here is a basics on data normalization"... But at that point, I'd be forever fixing their mess.. and you KNOW it's not gonna be for decent pay!
Is this a case of self selection. The only people who would accept a job working on that system are the ones that would perpetuate that sort of nonsense?
You typically don't get to see the actual code you will be working on as part of the job interview. I've had a few interviews where they come right out and say something like "Your first project will be to fix..." so at least you have some advance warning that you'll be dealing with broken code.
@@MikeU128 I think we should ask to check the codebase for few minutes before accepting offers. Why not? We aren't gonna memorize it, at least you could see if methods are thousand of lines or not
Any codebase will have dark holes as I like to call them, unless they are made purely to educate someone. In any professional environment you will have 2 things happen: 1. a seasoned developer who knows better fcks up because he is under pressure or lazy or any other human nature thing 2. an junior developer who doesn't know better fcks up because he simply doesn't know better The likelyhood of those 2 scenarios happening in any amount of time is almost exclusively 100%. So yea, any professional code base includes reaaaaaaaallly bad code.
@@tanner0397bitches don't know about my denormalized data warehouses. That includes me, because I've read Kimball's book and am as clueless as before I've started reading. 😅
I know this story sounds insane, but as someone who is working with IT systems for a large insurance provider, these types of stories are far more common than you want to believe. I both laughed and cried at this, laughed because it's funny, cried because it reminds me of at least a handful of systems which I have touched over the years.
That dev really knew how to secure his job by becoming the only person in the planet that could possibly understand his codebase, a truly mad individual
0:46 Tables are cheap, while columns are expensive. For RDBMS would you ever use more than 20 columns in a table? I mean it's relational, loading more than 15 columns from a table usually brings in more data than you are able to display. I have at least 5 tables for order line off the top of my head, these are: OrderLine, OrderLineDescription, OrderLineCompatibility, OrderLineAutocalc, OrderLineSummary, OrderLineRackSpace.
@@v0id_d3m0n He is quoting the 3rd normal form rule or sometimes called BCNF Boyce-Codd Normal Form. There are 5 normalization design principles to prevent you from making or breaking a database. ua-cam.com/video/GFQaEYEc8_8/v-deo.html
8:34 It sounds silly, but at a big enough company, enough people are being hired / fired / leaving on their own that it just ends up being easier to have stuff like this automated. Plus it can be good for any security practices if you can essentially say that everyone definitely loses their access to the database within 24 hours, or any new hires automatically get access in a timely fashion. And to know it is always going to happen at a certain time every day helps keep out any guess work since any times it doesn't happen must be bugs. "Oh, the employees table didn't get updated today? Something *must* be broken" vs "Oh, the employees table didn't get updated today? Hmm, maybe that's expected, does anyone know if anyone left the company recently?"
Yeah this actually happens in some corporations with older mainframe databases. These might indeed have thousands of columns (because of data type limits in the old database), and if you want to work with them in something like Postgres you have to split the rows between multiple tables and join them back with row ids. Table with one column and one row that generates auto-incrementing IDs? That's a sequence in Postgres lol
in the real world we call those rube goldberg machines. if you see a rube goldberg software/architecture equivalent at work its time to apply elsewhere
3:28 I wish my company’s DB stored unstructured data in JSON. Rather, business critical data gets stored in either XML or with home-spun array serialization. Sometimes the arrays are comma separated, sometimes double-colon separated, sometimes separated by some random non-visible Unicode value.
@@puilp0502 How about Zero Width Joiner? Man emoji + rice emoji = farmer emoji! Yay for 3 unicode code points that is 15 bytes in UTF-8 that means a single emoji that is two characters wide!
I have recently started a new job and joined a team to improve a piece of software that was a 1-man rewrite of an old piece of software. You know how they say "perfect is the enemy of good"? Well, that's true, but on the other end of the spectrum lies "just make it work for now", and that is by far the most common requirement. I found code that was raising events in the 6th composition shell layer, bubbling them up to a top-level god-object, that was passing it back down again back though methods into the object that generated the event in the first place.
Hey, Primeagen, I'm not known for anything, but I'm an experienced engineer who's been self taught and solo for 8 years, and I have a program that's like 100% reflective programming. It's a botnet that plays games. Everything from the introduction of an new IP to the wifi network, grabbing the bots profile, the way the 8 levels of decision making function, to the way screen data is processed in terms of both what is on it and what to do areceive. It's all "reflection" which is a word I just learned from this video, funny enough. I don't know any other way to program. I tried the different styles a ling time ago, and "went caveman" and just started doing things however I felt. Reflection is something I cavemanned together, apparently. I'd be happy to show you my "beautiful black hole volcano disaster" code as I think you'd find it amusing and I'd learn a lot from the roasting I'd recieve.
It's so funny, I ended up trying to reinvent reflection in my earlier years of programming as well for a project I was working on... I feel like it's a right of passage for any programmer to do when working in languages with no built-in reflection...
100% agree with the author about walking up to someone at the company and asking them how you can make their life easier and then you just go and build it. IMHO it's a worthwhile trade off for living in chaos. No scrum, no sprint planning, no "should have been an email" meetings. Just a developer working directly with their user(s).
Considering how many people pronounce that incorrectly, squeal is far less stupid than the other ways that I hear people pronouncing that. For the record, it's pronounced SQL.
As a Data Engineer I can assure you that this kind of things are happened more often than you thought, and Nope, it doesn't happen in cheap, little companies, most of the biggest companies have flaws like this.
Oh boy, and i thought that my process where i have to go to accounting floor, ask a lady if its ok to do X and come back and press a button once a month was a giant crutch...
on "the application for a for a single job", I did a bunch of custom "data massaging" apps that were basically convert some excel sheet, sanity check everything and add some data so instead of doing some half a day data entry job it was a 15 minute job at most (basically any line that wasn't sane was manually corrected and then stored as the default for matching cells, so they would just have to check it was ok instead of retyping everything). stuff like putting email adresses as delivery adresses and things like that were a common occurence you should've seen the function that checked addresses, basically a bunch of exceptions, because saint louis could also be "st louis", "st. louis", "louis, st" and a whole lot of other variations, including using the unicode ˢᵗ.
I recently was trying to get a field out of a SAP Public Cloud system that they didnt store at all but calculated each time the data was needed. The Implementation was a few thousand lines of abap and there was no way to call this implementation from anywhere outside this system. I had to Reverse engineer the whole thing, probably going over 100 Tables just to Display this field in a shitty Webapp. Try to explain to a customer that adding field x to their App Takes 1h but y a thousand times longer. This is the day i decided to quit my Job
3:30 i had a job where we had this one daily notification where the user would choose an time of the day to be sent and it was not working sometimes the information for what time of the day was store in a column as a json with the hour of the day and the user timezone. They would just pick every single one, decode the json, adjust for timezone and check if should be sent. I don't think i need to say that it was getting some timeout and memory issues thanks to having millions of users.
I uh.... still use Team Foundation Server. We've been off-and-on working on migrating to git for over a year now. The code was migrated to TFS from SourceSafe, and a significant portion of the codebase predated that. The code history is such a mess that any TFS->Git conversion tools I've tried don't know what to make of it; and yes, the code is unfathomably cursed in many different ways.
This reminds me of a bug in a very old custom ABAP application which generates a batch input on the SAP transaction F-02. The application had over 50'000 lines of code, was extremely important for the company, but stopped working after migration to SAP S4/HANA. Because the whole code was completely undocumented and it contained very deep IF statements (about 20 levels deep), we did not rewrite it. What we did is porting the old SAP R/3 standard application to S/4HANA, so that it exactly behaved like the old F-02. We had to duplicate lots of includes, function modules and customizing tables, but at the end, it worked.
That is stone-cold hilarious - the watering hole is polluted. The funny thing is - if you work on legacy long enough - you'll find out there is 2 watering holes (one at the back - where nobody wants to go) and the one out front - which is just as bad.
I feel called out by the fuzzy operator :( J.K. I only use for searching logs or something like that. If you need 1500 Columns, you really need 100 tables.
What I have learned over time is most major codebases are started by someone without the knowledge and experience to know better. Most startups are done by novice devs because seniors and leads are too experienced (and well paid) to want to start a major multi-month project from scratch that needs everything built out. Every major company's codebase was once a startup made by novices. Also most businesses aren't cut and dry for requirements, you can expect objectives, needs, etc to change over time, companies have to pivot, so inevitably you are going to have to change direction in your code, entire parts of your architecture become unused. Finally, there will be a bug that has to be fixed at 2am via triage and guess what, you don't have 12 hours to fix it. For people who say college students write better code, make them write their 6-hour project in 4 hours as a marathon, even with static requirements (which never happen in business), the time crunch will emulate more realistic conditions.
I work with a "gilfoyle" like this. Dude reject git because he doesn't trust cloud and stores everything on his own machine, now imagine sharing a code base with this man.
If you don't trust clouds, don't put your git repo on a cloud, that's not a reason to not use git. I often enough use git locally to have version tracking on a project even if it doesn't live anywhere other than my own hd, and you can also push/pull between different machines locally via ssh.
I wasn't 100% paying attention, didn't the article say it was related to a search feature? My guess was that they had a table where each row was a "merchant", and added a new column for every "keyword" they wanted to be associated with the merchant. A more sane way to do this would be to have a separate table where each row has a merchant_id and single keyword column, and add a new row for every merchant_id/keyword pair. Then find all keywords for a merchant by joining on merchant_id. Now your only limit is row count not columns.
@@Intense011 I didn't hear that it was related to search, maybe I missed it too. If it's that keyword thing, I agree with you, that should be two different tables, one for keywords, and the other to hold keyword_ids and merchant_ids. A join is possible but a sub query is fine these days (in mysql at least)
There's definitely a use for having JSON columns in a database. You just get to plop in whatever kind of not-too-specific metadata you wanna plop in and OBVIOUSLY you're not gonna be a complete imbecile to try to do a WHERE clause on that column.
I feel this... at one company, we had a 3-part table with equipment_reg, equipment_reg2, and equipment_reg3... since it was migrated from a legacy schema from a system with hard limit of 255 columns per table... and the IDs one one table were stored in a field in another table, even though the current database system (sql server) has auto-increment, it wasn't used for that table, you had to start a transaction, increment the row in the constants table, then use that ID for the new row in the other table then close the transaction... suffice to say it was slow.
Legend has it that Gilfoyle's harddrive RAID is still kept alive with frog bladder oil and a ritual sacrifice on Friday afternoons in Pete's shack behind the lake.
It is funny because all the way through university you are taught 2nd, 3rd, BCNF normal forms to reduce data redundancy and save space. However when you work in organisations with big data they often have these "Single Source of Truth" tables with countless columns that could easily be broken up into multiple tables, however they don't because it is significantly easier to write various SQL when most stuff is in 1 table rather than 1000.
Yeah, I've seen some horrific database designs. At one place we were told to add a feature, by next Thursday. However the database kings required that if you needed a new column, you had to submit a formal document. The database committee would meet once a month, and usually reject a proposal once or twice. So we were stuck. Fortunately I saw that the "fax number" field was always empty. So we packed in two integers into the fax number field. Nobody noticed. At another place, all SELECTs were really slow, as there were no keys, every select had to do a string search on every row. Every time. It gets worse. Another place was using an offbeat SQL where the basic insert/delete features used a non-standard way. Something like a SELECT with an empty something was actually a DELETE. And the SQL server was like seven years past end of support. Yikes. When I suggested factoring the code a bit to speed up the conversion, my key card didn't work the next day!
Did a dashboard with a single SQL query with - if I count correctly - 26 joins. about 1-15 of them are CTEs with their own set of joins I don't even count here. Works flawlessly. Even presented it at a conference. The reason: It's a DB for student advisors at a large university. It pulls from student records, employment records, advising tables, grade tables, enrollment tables, student attribute tables, email databases, and the list goes on. The reason why it was such a game changer is because you now have caseloads and student attributes and even checking for who is still assigned to an advisor that left or the student changed their major and now, you can see they need to be reassigned. Also checks on academic probation and dismissal cases so we can do targeted outreach... Things that took a supervisor the better part of a day now takes them 5 minutes. The key is caching every morning.
I'm impressed and also absolutely horrified as to how they got to reach the max column count on a MSSQL Server. At my company our record was 7 tables, but only with around 400 columns. MSSQL has a max table length of 8KB, so that would mean each of their columns had a average length of 8 god damn bites. Thanks for the video, I just realized I still have a long way to go and a lot to learn from this company
My first development job, I found a procedure that had an "error handling" subroutine that just increased the id by 500k and it had a comment that said "if there is a problem processing, throw the requisition very far away in the queue, should keep it running for at least 5 years" Luckily I only stayed one year in that company.
I don't know why, but this codebase has post-collapse cyberpunk vibes.
Like, just mashing together highly complex and extremely sophisticated machines to create a desk fan.
It's beautiful.
There's an entire Kubernetes stack controlling the fan's rotation servo. 7 microservices are involved in accepting updates to the desired rotation angles over WiFi. That lives on a slick webpage that works in any browser. But changing the fan speed is in a different portal that needs to load ActiveX components and requires IE 6. The fan has no buttons.
@@FireStormOOO_ oh shit activeX lololol
@@FireStormOOO_ beautiful
Space Station 13 servers in a nutshell
Warhammer 40k type beat
"I want ten year old production code to be open sourced and fed to LLMs for training."
Sounds like poisoned training data. XD
It's the new "Google Translate an instruction manual through thirty languages, and then back to English", but for programming. The future is stupid. And just wait till they use all the code that H1Bs poorly copy pasta'd from StackOverflow, as LLM training material. Then LLMs can truly "do the needful" too!
oh that's hilarious
nah Intel thunderbolt utils is worse
nah, he is just playing 4d chess, we all know that would drive any emerging AGI straight into insanity.
but if his gambit fails, he will have to answer for his deplorable crimes against the machine.
Sounds like job security.
If you want an insight into the quality of corporate code, you only have to look at the scandal of Fujitsu's POS system for the Post Office in the UK - the cause of the greatest miscarriage of justice in our history. At the public enquiry, multiple witnesses have attested to the shambolic development process - no proper specs or docs, no coding standards, inept developers and no effective testing. And yet the company doubled down and allowed hundreds of postmasters to be bankrupted and even imprisoned for cash shortfalls caused by their bugs.
I'm updating production code and having to rewrite failing tests
The tests don't test anything they just exist to meet coverage requirements
@@GreedoShot What are you suggesting here - that you install a complex POS and accounting system into 12,000 branches without testing that it actually works?
As a software engineer working in the corporate world, I can agree that nearly all the code is trash
POS means piece of sh
@@GreedoShot typical , play stupid games, win stupid prizes
At least you have 100% coverage , am I right ?
I am genuinely scared by the fact the database the guy is talking about is eerily similar to the database of a company I've worked on before
Who is it
There's multiple of these space-hulk type codebases out there, in the depths of the unexplored overgrowth.
Don't worry too much.
...
But same lol.
@@Volvith you guys are forgetting to factor in all the smaller scale stuff glued together at large local businesses. I worked my regions largest car dealership as a web developer and the dude that made the database was a idiot husband of someone who was in the company family and he had no idea what he was talking about. We were just automating the business office so our sever never really had more then a couple hundred users but the database was so bad they had to go in and manually make some of the basic queries not time out at 2 minutes. It was a couple hundred thousands for that small project by the time I had gotten hired and it was wild. The idiot lead guy and the second in charge were using notepad++ and ftping into the same server. They would overwrite each others code intentionally. They would sabotage each others code intentionally. Eventually they fired me because they told me to start ripping social security numbers out of some other licensed software they used and store social security numbers + other personal information in plain text and I said no. Management actually told me I was crazy because I said that storing social security numbers in base 64 isnt encryption.
you have no idea bro... i'm gonna tell ya a company with the same systems and trash software here, let's see how many will find it: FCA (yes the automobile group Fiat Chrysler Automobiles) I can't tell you much more but trust me you would want to rm -rf your brain after seeing it...
kevin from the office asking martin nash to explain insider trading
Legit. The cleanest code I've ever seen was written by students at school, it's all be downhill from there. Every single thing you learned not to do is what you find done in corporate. xD
just because you're told not to do it doesn't mean it's not faster to do it rn, will it break in a year, probably, but whatever.
@@jambonmusical2689The old "good enough vs done right" struggle. Short-term financial gain almost always leads to "good enough" and a growing pile of tech debt.
That makes me very worried. I have seen what my classmates have cobbled together. That gets the job done.
Yep... Looking at you Siemens with your abhorrent structure for Step 7 and WinCC... My God, their software is so dog water that it reminds me of programs I saw growing up in the early 2000s.
Of course it is the cleanest code because you only work on this projects on your own.
If you design the codebase completely on your own, it is far easier to do so than in a project where >100 people worked on and > 200 people who worked on left the company a while ago. Very often you have to do and to work with legacy systems.
And those work for a good reason how they do.
And it is very easy to judge in retrospective from your current knowledge rather than at the time when people actually wrote that code.
I worked for a pretty big gambling company and they stored javascript, html and css in columns in a MSSQL db. Before that i had never had a panic attack during a project overview meeting.
Reminds me the old days where senior dev was asked if we could implement new feature into program and the only respond you heard from across the room was "I have no idea." im glad we didn't get all kicked out from company 🤣
Storing code in the db... One of the many joys of using WordPress too.
I worked in a factory where their internal production management and quality tracking system was custom built in house (by someone who was entirely self taught). Not only did it store HTML in an SQL db (not sure which server) but it had raw SQL queries in JavaScript and using simple string concatenation to build queries such that little Bobby drop tables would have a field day.
When I discovered this I brought it up to the poor sod tasked with maintaining this mess (since the creator had since left and turned this codebase into a commercial product and started a company selling it) they just shrugged.
i have also encountered HTML in the database (it was bad). unless the system actually needs to be that dynamic, it's a bad design. it's bad when it's the same HTML every time with only specific portions that vary. someone mentioned WordPress, but i'm on the fence about that as it stores plugins (i don't really know the WordPress internals though)
@@xybersurfer The css and js part really tripped me up. Creating dynamic css by writing SQL statements, i actually refused to touch any of the javascript. I guess a blessing in disguise because i learnt all the new fancy css things.
I manage a database for my company that was designed by the guy who came before me. No joke. That guy stepped in front of a train. Anyway, there is a table in this database where the primary key is integers 1 through 36. The values associated with each of those primary keys is also integers. 1 through 36. That's the table. And if you delete it, it breaks the whole database.
It's probably for a good reason that's usually not apparent at first glance. Look at the code utilising the table to understand why. Primeagen had already mentioned this, usually things like this are done for a good reason and it takes you a while to understand why.
@raptorate2872 I appreciate that having humility and giving the benefit of the doubt are great qualities in the workplace, but this is the youtube comments section. I am wracking my brain to come up with literally any logical explanation for what OP described, and I am coming up 100% blank. Maybe you can enlighten us?
Maybe the code was supposed to map from some ids to other ids because some ids could be aliases used for different purposes, but in the end there was no need for extra aliases and the table ended up being trivial?
I suppose it is entirely possible that it is 100% coincidental that the 1st 36 values were the integers 1-36, and maybe there was no reason this data couldn’t change in the future or that new data could be added that doesn’t conform to the pattern. It would seem highly improbable, but maybe it’s just pure happenstance that the table appears to be pointless, and the next record could be fk 37 with value of 92738. If that’s genuinely the case then just wow.
@@raptorate2872 it is literally because there is geographic data in the database, and 1 through 36 are the section numbers in a township and rather than type the section number, he thought it would be 'cool' to have a lookup for it. So instead of typing 1 for Section 1 of the township, you type 1 for the lookup, and it fetches the 1 for you. Probably pretty easy to resolve, if I wanted to. But the guy was my friend, and now he's gone, and I don't want to untangle his beautiful messes. (That came out a lot less hetero than intended.)
How JSON showed up in our production database?
"I'll dump this here for convenience until I write a generator. People will use the method to retrieve this anyway."
Two weeks later: "Nice, it is done."
Two minutes later still: "Kill me, please. How is this column already called from over 100 places in the code? Why didn't they use the method?"
The name of the column? never_use
Reminds me of the react GitHub issue of someone asking if it is safe to use the stuff under __SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED
"never_use"
I wonder what that stands for?
Should have called this __SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED
@@jasonieong7772 jokes on you, I used it in order to be fired. Now I am mad.
There's nothing wrong with a JSON blob in a database. In fact, it's how I design all my database tables. Works great. You only need to have columns where you are going to apply indexes to. The data in those columns originates from the JSON blob. So, yes, you have some data duplication, but then your application can both search really fast via indexes and also expand a complex object in the application. JSON is supported in every language. You can also technically search on the JSON blob using DB queries but those will inevitably be slower. To solve that problem, you add "archive" tables where old data periodically gets moved into the archive.
When someone says "its Year X, you should use Y already", I don't think they had lots of longstanding customers. I want to see them how they argue, that the customer should switch to a new database and transfer 10+ years of medical, finanical, structural data.
Just because a technology is new does not mean that's a good reason to transition, frankly in today's interdependency hell I would argue the opposite. Unless there are viable business incentives to transition, all you're doing is hurting yourself.
The last company I worked for that did medical records, had automatic migration policies.
The customer had a defined interface to the database.
The database could be updated, without changing that interface.
If the customer is using your raw data tables, you have a bad design.
@@jamess.2491 Survivor bias
@@darekmistrz4364 i think Path Dependence is the phrase you are looking for
@@jamess.2491 this is true, but often we are also blind to how not using a new technology is "hurting"
I cant wrap my head around the calendar table. Running out of columns is one thing, poor design and all, but be unable to login if you ran out of calendar entries? What am I even reading, how?
I think the calendar table would be used to quickly calculate N business days from current date. I think it is manually updated to allow for changes in holidays every year.
@@amehybrid You may be on to something. I worked on a timekeeping system that had to account for holidays in different jurisdictions. In some places, people were legally required to pay double if they worked on Good Friday. Have you ever tried to automate a calendar setting that has a mixture of solar, lunar, and ecclesiastical factors? I figured it out, but I still think it may need an adjustment in 100 years or so.
Some applications use different calendar types. You have a typical Gregorian calendar. Then you have a fiscal calendar where the first month of the year is October and days start on Monday. Broadcast television calendar uses "square" months where a month can start on the 27th of the previous month. Then you have calendar type/month/quarter/year values in other tables.
At my company plenty of our codebase relies on a (working days) calendar table.
I remember in the past we had a codebase (luckily taken out of use now) that had its own internal calendar table, but for some reason the dev only added a year of dates at a time. So every time Januari rolled around it would stop working until somebody poked the dev that maintained it and he remembered he had to update that.
We have a similar thing in our company. A scheduler table where you have to manually insert records otherwise a critical job wouldn’t happen. The guy that wrote this put a comment in his hand over docs that said every December you needed to add records for the coming year…
I once worked in a company where I learned there's a hard limit on how many elseif statements you can have in C++, since they are essentially just differently formatted nested if statements, and they have a limit too. There were multiple files where they hit the limit. The solution? Just start a new if statement and pray to god it works. Of course, I spend days just figuring out this was the cause of a bug I was hunting. I never thought that somewhere in those thousands of lines the else-if just stopped and started with a new if.
else-if statements make me nervous because 9 out of 10 times as I am writing those, and I stop to think for a few minutes, I realize I'm approaching the problem completely stupidly and can refactor it into a single evaluation :)
what. the. fuck. Dude if I saw that I would 100000000% rewrite it as something a bit more sane... Even if it took me months, even if it got me fired for not "being productive".
Jesus Creeping Christ. Well that is an interesting bit of trivia I'll have to remember. Then I can casually drop that little tidbit next time devs are telling war stories. The implication being, anyone who knows that fact has seen some serious shit.
Nobody likes the select...case syntax. I mean I'm sure there's somebody out there who likes it but they're weird. However, it does serve a purpose. It's also pretty fast.
Methinks this must be an urban myth as the if else's just compile to branch's I can't see any limits.
Fun fact: the original Relational Model (the basis of relational databases) did not use names for columns, but index numbers. That was 1969 (nice) - by the 1990s Codd had found people who were using over 128 columns, so he decided using names was better than remembering that column 36 was the surname of a user.
it's less surprising when you understand that for him, it was all mathematics and in his head, tables were matrices.
maybe they expected the them to use enums.
@@lerpmmo Don't even joke about enums.
@@NickTheCodeMechanicenums are good, actually
@@tissuepaper9962 better than Java enums 😀
I work as a support engineer for a company that manages the infrastructure of at least 200 companies. A couple big ones, _and a whole lot of truly sketchy shit._
We have a lot of legacy customers as well, that got rolled over from other tiny support companies we ate. (we's a growin' boye)
Let me just summarize how horrible it can get in one sentence: _I genuinely think we've found more bugs in Microsoft's code than any other company on the planet. And it's not even our job._
I love my job, but God bless the soul of anyone picking up the phone for a customer that nobody recognizes. Because either they just got rolled over from some tiny shit creeck company we took over from, or even worse, _it's the first time in 6 years they called, and you're about to meet the end-boss of DIY legacy code._
Elaborate more on your stories, the wisdom proufound will must be
_Tell us more?_
sign me up haha
Thats easily millions of dollars in zero day exploits son why ain't you selling that shit
That sounds fun
I work for a supermarket chain and this video has made me realise that the POS machines freezing every 2 minutes or so is probably not the fault of their ancient hardware
Don’t even get me started on when a technician working on the self serve registers told me the error we were having was caused by the register getting stuck in a while loop and the only way to break it out was to cause another error by physically removing the receipt roll from the printer.
I used to work for a clothes retailer in online order fulfillment. At the time they had two options for scanning equipment - the "legacy" Motorola MC3100 handheld computers, well over 10 years old at this point and running on Windows CE and the brand new Honeywell CX45s running a fresh and modern looking variant of the pick system over Android.
The MC3100s blew the Honeywells to smithereens in basically every department. More rugged, better ergonomics, and somehow significantly faster despite having equivalent specs to your mom's Gateway from 1996. Their one issue with them was that all the batteries were shot, but the Honeywells tended to overheat in their scanner cradle and die so it was really a wash as to which lasted longer.
Software matters, and honestly a lot of enterprising computers don't really need much of anything at all if their whole purpose is sending a few strings to a database. I'm not a programmer but thought about buying one of those Motos and seeing what kind of things I can make for it.
0:20 Actually by default it's max 2000 columns for sqlite, but you can bump the limit to 32767 if you compile the library with custom config
DON'T GIVE THEM IDEAS!
@@jordanhildebrandt3705 I mean, really, since sqlite has dynamically typed columns, you can just make any table a 3-column one (row_id, column_name, value) with unique(row_id, column_name) and then you can have as many columns as you want and even dynamically generate columns! ;-)
Is the column counter a 16 bit signed integer?
@@tomlxyz I see both i16 and u16 being used for column count/index in various places. Not in the database format though afaict, so perhaps if you're extremely eager you could try "fixing" all of these to be 32-bit. It would be a terrible idea regardless, I'm 99% sure that accessing a row involves some work that scales linearly with column count.
Of course, since sqlite is dynamically typed, the solution for this "problem" is easy! Just split each row into a whole bunch of (row_id, col_name, value) triplets and you can have unlimited columns! ;-)
It’s 2024 - one of our system uses a table “calendar” and we had a production bug that stopped all our analysis because the script that was responsible to renew the “calendar” was crashing for a couple of months and no one knew and the analytical queries depended on it…
We also have the CSV thingy. It’s anecdotal, but companies run like this even now.
Paused at 8:45 to take a breath.
I can feel the imposter syndrome leaving my body.
Bruhhhh😂😂😂 This is too realll
I program since 8 months, i am in no means good but when buidling an application for a database i went and made all transcations and componets modular so that you cloud change for example a players team mid game without the app crashing. And hearing they do everything what was said. I feel like an Senior Dev 😂
I was about to comment the exact same thing lol. Next week is going to be good bro. We are going to be flying.
The beaty of over the top solutions is more that it's more of an "art" rather than engineering. It should be place in gallery for software engineers so they could reflect on their life, while looking at this art.
A bunch of people are essentially trying to reproduce this practice across the industry, only the name of the guy who gets the crappy CSV file and presses the button that may or may not import business critical changes to the database is named Devin. Making your core business processes unknown magic is sometimes necessary, but long term inexorably leads to terrifyingly fragile stacks of "just make it work" quick fixes.
1024 columns ?, i started sweating with barely 20 columns on my job and started breaking apart the SQL to pieces. To limit that.
@@EvanEdwards stacks and stacks and stacks.
@@terryjophlin Pretty much everything complex is built like that, software to submarines to shopping malls. You can usually have one unreliable layer, but multiple eventually causes failure. In Florida there was a concrete issue for a few years where bad concrete was unknowingly being shipped to new construction projects. Years later, high rises, parking garages, and large oceanfront condo towers started falling down, sometimes catastrophically collapsing with people inside. Most of them had a second factor contributing to the failure, like adding a floor above the original design, or moving the pillars just a few feet. Things that stretched tolerances and were almost always signed off on by an engineer, but they were unaware that those tolerances were already stretched by the substandard concrete mix.
11:50 as someone who took a database class working in SQL server, writing, queries and other stuff. And who loves back in development. If I was an intern, I probably would have written that application to make those SQL statements… And then not told anybody so I could get my eight hours of work done in Three minutes Just running my application
Corporate code vs. hobby code..... Don't assume corporate means better. It's convoluted, hotfixed, non optimized, ... But that's since you used 100 times more time on your hobby code, than any manager will allow an employee.
Agreed. Ever since I got into the real world, I really started to notice this. Hell, even professional software (**cough TIA Portal, WinCC, and EPLAN**) can be beyond absolute garbage.
Yeah
the key difference is that i feel shame if i push garbage code on my personal project to a public repo
but noone cares or will ever shame me for garbage code i write in the proprietary repo at work. Usually you are even discouraged to take more time to refactor/clean up your code, becuase "customer wants new feature NOW not in 2 days"
and then it takes 2 senior devs a month to clean up my mess 3 years later costing 100 times more
it's like, idk, sewing as a hobby vs for a fast fashion corp
Hobby code is almost always 100% better. However, you need that skill to get down and dirty, too, and work fast cleaning up the poop.
You've worked at shitty places it seems
With this kewill reflection thing, I think he missed the most beautiful point. This sounds so weird, so problematic, so hellish. Colleagues told him not to dive into it, because they knew what weird magic shit show he'd find. Everybody gave up. But he found it. SOMEONE HAD DISABLED A CRONJOB. That weird reflection on empty glasses was still working. That incredible weird part. That part was fine. WTF 😂
the reflection of empty classes is actually pretty cool/smart imo.
I take unorganised chaos and direct talk to customers over enterprise design patterns and 1 year plans any day of the week.
Right on! I hate design patterns, especially when paired with OOP inheritance hierarchies! I can only imagine how insane backend architect 'design patterns' are...
I have a much less exciting version of this. We ran out of tables AND columns (of our paid cms). At one point we had to embed JSON of a “new table” in another table that was mostly just used to store strings. Literally if we wanted to do select statements from one “table” we would instead query a different table, filter the rows, convert string to json, extract what we need. Holy moly.
Company no longer exists
1:10 - This hit way too close to home. I do a worse job at work since there are:
1. Time pressure (partly as a function of me/my team not being good enough to fight against as much as we should)
2. Social pressure and general social negotiations that cause the code to become a mish-mash of styles and quality levels. Yes, even with code reviews and pair programming. You are not immune.
3. You are not coding your baby all the time. Sure, the best times are when you can fully gaslight yourself to go in with that mindset, but every once in a while it just... fails to materialize to the extent that you want. If the project is not very short this can thankfully be overcome.
"Squeal" instead of "sequel" is triggering.
I could barely watch
The correct initialism is obviously SQUIRREL
It is the only way of pronouncing it
He must do it just to cause us pain? There's no way someone unironically says squeal 😶
I say both are awful. SQL is an acronym that doesn't form a word. Stop trying to force it into one.
JSON on a random Column? That is cute.
The bane of my existence is a Table, that stores serialized PHP Arrays in one column and pure HTML in the Next. Compared to that, JSON ain't half bad. Organically grown codebases started by some dude in his bedroom over a decade ago are such a joy to work on.
no no no no no no no no that's horrific.
Doing reflection on a hierarchy of classes and empty methods sounds unholy. 😂
Next stop, writing your own language
@@olbluelips Mate don't even start. I'm in a situation now where my company acquired some 15 years old financial planning company. Their software was written by one old boomer who decided to make a proprietary scripting language for HTML pages. His reasoning? Well they couldn't get the pages to work correctly in HTML and CSS... the scripting language transliterates directly into HTML and CSS.
A few days ago I had a bug where a page refused to behave like I was telling it to. It was literally "Show these columns if they are not null". Turns out I was using "if (col[0] not null)" instead of "IF (col[0] not null)" and they are completely different in this proprietary language pile of wank.
@@fl-ri- case-sensitive conditionals? That’s just brilliant I’m stealing that
@@olbluelips You can understand why I spent 5 hours trying to debug it to find out why all my columns were null, right?
This is simply beautiful.
Now I'm dealing with a similarly messy monorepo codebase with react and I just embraced the chaos, concerns for code duplication and design are gone and it's now about just doing the thing that needs to be done.
I hate react, but now with this new approach I kinda dont mind it anymore and found my peace with it.
It's refreshing in some ways.
I lost it at "when the calendar runs out, we can't login to the system" lmao
"No Mr President, we can't update G because Trygve is on vacation" - Some poor bureaucrat to Stortinget.
12:10 Writing single-task application was considered too hard so interns wrote and executed custom SQL queries against production database??? How? Why?
Haha, yea... and having 3 interns whose sole job is to keep doing these one off queries for who knows how long is easier than writing the one off application. WTF?
easier to get an all-purpose intern who on paper can tackle any easy issues in the backlog, than convincing management to assign 4 weeks of work to build that application
and then make the all-purpose intern only do sql issues
We had a job that did pivot 3000+ columns on a daily.. including over9000 unions, 50+ table joins, casts, UDFs you name it. Quant who wrote it, modified it regularly, and when it broke (after each update), we had call someone it to supply us with coffee and food, while we fought this monster. Day? Night ? Weekend? The only upside that it was written sql and running on sql server and not some unicorn level system. Sometimes I wonder what this particular Quant was smoking
Sounds like a daily full dababase dump to a single table for "ease of use" lol
Probably a guy who had absolutely zero idea how a database works.
A manually populated calendar can make sense sometimes for things like public holidays that don't always follow a predictable pattern. But you shouldn't have to manually enter the *entire* calendar...
16:00 Schrödinger's work order
I don't see anything wrong with using JSON in a DB as long as it's not used to query rows. It's useful when you need flexibility.
It's okay on non indexed nonsense but eventually you'll have the one guy building pivots around json values and the whole world burns
We have some JSON in some tables where the schema changes and instead of adding new columns, the object gets stored in JSON. The rest of the database is properly relational. Sometimes it's just keeping all your data in a single db.
This. Example: We store the configurations for a tool in JSON - we need a few hundred configs - and the history of configuration edits (because it’s one of those ‘picky’ tools that it’s easy to fuck up the configurations of, but we also need to change them sometimes). The tool configs can be CRUDed using a simple browser UI, which shows the user feedback on their changes. The tool is a core part of our system in production and is constantly in use.
When it comes to storing the data, I can think of ‘other ways to do it’ but not necessarily better ways to do it.
Hard agree, however, on not needing to do any LIKE ‘%FOO%’ logic on it. I’ve been guilty of that in the past, and been responsible for tidying it up.
@@jimiscott is the user changing the schema?
@@noobling8313me who has used LIKE %foo% to query varchar columns many times 🫣
I have 100% experienced a "Gilfoyle's Hard Drives" situation. When I became aware of this problem, I took it upon myself to fix it immediately. It took awhile, but eventually everything was put into TFS. And at this point many years later, most of that code has been completely rewritten as part of other efforts.
the Gilfoyle at my last job had a network drive called "TheFrench"
This whole "I'm concerned about perf" and then proceed to use GC languages is so true. People who write the slowest most dog shit code say stuff like that on a consistent basis. It's those people who think having less lines of code makes the code run faster.
The code in question:
#import giantAssCodeBase
doAll()
// Very fast code because it only has 2 lines
lmfao for real. Return to C++, return to "oops, accessed a dangling pointer because some dumbass didn't use a smart pointer"... But also return to "yeah, this function takes an immeasurably small amount of time to execute".
If the goal is to get from the top of your source file to the bottom fast, then technically less lines is faster.
For web apps less lines really make the app load faster because of network, but will not save from deep React rerender
if you attack a table with over 1024 columns, the language you use to do it really doesn"t matter.
Man... what does it say about me that I can't even land an interview when so many companies are accepting this quality of work?
It says that hiring is broken.
@@jordanhildebrandt3705 the workaround seems to be embellishing your qualifications to match the embellished requirements.
Running out of chars in col is even better. Text1, Text2, Comment1 etc
My last project ran into the problem that DB2 did not want to join more than 100 tables in a single query. The fix for this took me multiple months and was essentially a recreation of joins on the rich client. all table data was loaded into the client. This was also 23x faster than letting the database do it. (46 seconds to 2 seconds. But 1.7 seconds of that was jamming 40000 rows into a JTable)
@@MarcLucksch Did the DB have any indices? No need to answer, I'm assuming no.
I've seen Input1-21 in a knockout Jquery webb app, it is (yes, still exists and is maintained.) over 600 lines of code just to update their state.
ah I had that at my previous work. Bank was like, no worry, only small comments. Some of the small comments extended into 17000 characters.
@@vytah it does, it was actually quite optimized to even work at all, just the amount of data duplication due to joins was insane. Just by loading each text string only once I reduced the amount of data send by the server from 3GB per query to less than 200MB
I think prime hit on something relevant. Imo code should be open sourced after a certain period of time, for the good of mankind. Especially defunct projects. We do it with public domain (or at least used to until it kept getting pushed back), we do it with patents. We should really do it with code
Yeah but then how will poor little Adobe make money
it's funny how we ask ourselves "How did this happen" but I am sure the place I first worked at and where I run from will likely be like this in 3-5 years time. The ideas the people there not only think but execute is madness.
Never heard anyone pronounce SQL as "squeal" before.
I am not a programmer. But I heard a phrase.
"Real programming starts when you run out of memory."
So I guess. "Real database management starts, when you run out columns."
The way that sounds to me is like every dev would just scrape the db, then write their own little app with the data. This caused devs to keep adding columns to store the data of/for the app. Get a new task, write a new app to solve that task, add column. etc
I've worked at a place where all employees had open communication with each other cross departments. So every one would just talk directly to a dev to add something just for them that no one else in the company uses. If that person left, the code was still there, but no one uses it any more. A new employee would then go through that process with their own requested solution.
It's kinda fun, for the devs and other employees, but the codebase is such a mess. Actually it sort of - but not really -is like blockchain. There's a central point and every one builds their own thing on top of it. To put it simple, every one had their own custom built dashboard.
If I ever see an SQL dump with huge "merchants" table and "merchants2", I will know what company it came from.
I'd bet 100 bucks that there's at least >100 company code bases out there in the world with a merchants and merchants2 table.
@@ayporos Well, then other stuff from the article will help narrow it down.
@@ayporos That is deeply frightening.
8:26 "This is a story of a man named Stanley"
First network admin job I ever had we had an Access database that indexed all the CAD drawings our technicians had to work on. Every technician had a VBA application on their system that would read and update status of drawings in that shared DB on the network. The database would be corrupted on an almost daily basis, and I eventually decided to look at the code to figure out why. When I finally dug in to the code I found that anywhere an exception could be thrown when trying to lock the DB for opening was the following line: *On Error Resume Next*
Handling exceptions is for amateurs. Real men just write to the DB even if they got an exception.
What could possibly go wrong.
We have json in our MS SQL DB, but not because we ran out of columns.
Apparently they wanted to use a NoSqueal DB, but being a bank they would have had to have an expert with some kind of certification for it on staff, which they didn't, so into Squeal it went.
Yeah same here, except that it wasn't because certifications that we couldn't go nosql, but because of customer requirements.
For some reason they insist on hooking our brand-new, modern application up to a 10 year old version of on-prem OracleDB.
@@henningerhenningstone691 Oh no that reminds me of that one company that had an interval (can't remember the exact number) where they would export SAP into an OracleDB for other Apps to consume.The documentation for that DB was a giant incomplete Excel sheet.
It was impossible to connect some of the data because some IDs just weren't in the export.
@@henningerhenningstone691 Interesting, how it's going on so far?
@@miled8649 well, I can confidently say that Oracle is the worst DB in existence. Want to save an empty string to your string column, because that's what the user entered? Oh, let me make that null for ya. What, the column can't store nulls? Tough luck, now you have an error.
And loads more nonsensical stories like this one. OracleDB is a meme in our team at this point.
As a prof dba, I am torn...
I wanna go down there and say "KNOCK THIS OFF"... but then I know I'd never be allowed to leave... I'd be "here is how you should do this" and "here is a basics on data normalization"...
But at that point, I'd be forever fixing their mess.. and you KNOW it's not gonna be for decent pay!
Is this a case of self selection. The only people who would accept a job working on that system are the ones that would perpetuate that sort of nonsense?
I would accept a good amount of money.
You typically don't get to see the actual code you will be working on as part of the job interview. I've had a few interviews where they come right out and say something like "Your first project will be to fix..." so at least you have some advance warning that you'll be dealing with broken code.
@@MikeU128 I think we should ask to check the codebase for few minutes before accepting offers. Why not? We aren't gonna memorize it, at least you could see if methods are thousand of lines or not
Any codebase will have dark holes as I like to call them, unless they are made purely to educate someone. In any professional environment you will have 2 things happen:
1. a seasoned developer who knows better fcks up because he is under pressure or lazy or any other human nature thing
2. an junior developer who doesn't know better fcks up because he simply doesn't know better
The likelyhood of those 2 scenarios happening in any amount of time is almost exclusively 100%. So yea, any professional code base includes reaaaaaaaallly bad code.
I'm a data engineer for a large company and hearing this crap makes me squirm. I've seen some shit. but this. this is extra painful.
y'all need star schema
@@tanner0397bitches don't know about my denormalized data warehouses.
That includes me, because I've read Kimball's book and am as clueless as before I've started reading. 😅
I know this story sounds insane, but as someone who is working with IT systems for a large insurance provider, these types of stories are far more common than you want to believe. I both laughed and cried at this, laughed because it's funny, cried because it reminds me of at least a handful of systems which I have touched over the years.
There’s also a size limit. If each field is a blob, you’ll run out of space long before 1024 columns.
Yep, 8KB can fill up fast with a handful of nvarchar(200) columns. That’s going to be much more likely than having thousands of smallint columns.
That dev really knew how to secure his job by becoming the only person in the planet that could possibly understand his codebase, a truly mad individual
0:48 950 columns? That’s child’s play
0:46 Tables are cheap, while columns are expensive. For RDBMS would you ever use more than 20 columns in a table? I mean it's relational, loading more than 15 columns from a table usually brings in more data than you are able to display. I have at least 5 tables for order line off the top of my head, these are: OrderLine, OrderLineDescription, OrderLineCompatibility, OrderLineAutocalc, OrderLineSummary, OrderLineRackSpace.
All I remember about databases is the phrase "They key, the whole key, and nothing but the key"
Ha I've never heard that phrase in relation to databases. It's good!
@@v0id_d3m0n You haven't? It's literally the holy mantra of Database Normalization. :)
@@v0id_d3m0n He is quoting the 3rd normal form rule or sometimes called BCNF Boyce-Codd Normal Form. There are 5 normalization design principles to prevent you from making or breaking a database. ua-cam.com/video/GFQaEYEc8_8/v-deo.html
8:34 It sounds silly, but at a big enough company, enough people are being hired / fired / leaving on their own that it just ends up being easier to have stuff like this automated. Plus it can be good for any security practices if you can essentially say that everyone definitely loses their access to the database within 24 hours, or any new hires automatically get access in a timely fashion. And to know it is always going to happen at a certain time every day helps keep out any guess work since any times it doesn't happen must be bugs. "Oh, the employees table didn't get updated today? Something *must* be broken" vs "Oh, the employees table didn't get updated today? Hmm, maybe that's expected, does anyone know if anyone left the company recently?"
Yeah this actually happens in some corporations with older mainframe databases. These might indeed have thousands of columns (because of data type limits in the old database), and if you want to work with them in something like Postgres you have to split the rows between multiple tables and join them back with row ids.
Table with one column and one row that generates auto-incrementing IDs? That's a sequence in Postgres lol
in the real world we call those rube goldberg machines. if you see a rube goldberg software/architecture equivalent at work its time to apply elsewhere
3:28 I wish my company’s DB stored unstructured data in JSON. Rather, business critical data gets stored in either XML or with home-spun array serialization. Sometimes the arrays are comma separated, sometimes double-colon separated, sometimes separated by some random non-visible Unicode value.
Dude i can feel your pain😂😂😂, stored in xml? Shit
> some random non-visible unicode characters
Is that you, ZWSP?
@@puilp0502 How about Zero Width Joiner? Man emoji + rice emoji = farmer emoji! Yay for 3 unicode code points that is 15 bytes in UTF-8 that means a single emoji that is two characters wide!
I have recently started a new job and joined a team to improve a piece of software that was a 1-man rewrite of an old piece of software. You know how they say "perfect is the enemy of good"? Well, that's true, but on the other end of the spectrum lies "just make it work for now", and that is by far the most common requirement. I found code that was raising events in the 6th composition shell layer, bubbling them up to a top-level god-object, that was passing it back down again back though methods into the object that generated the event in the first place.
Hey, Primeagen, I'm not known for anything, but I'm an experienced engineer who's been self taught and solo for 8 years, and I have a program that's like 100% reflective programming. It's a botnet that plays games. Everything from the introduction of an new IP to the wifi network, grabbing the bots profile, the way the 8 levels of decision making function, to the way screen data is processed in terms of both what is on it and what to do areceive. It's all "reflection" which is a word I just learned from this video, funny enough.
I don't know any other way to program. I tried the different styles a ling time ago, and "went caveman" and just started doing things however I felt. Reflection is something I cavemanned together, apparently.
I'd be happy to show you my "beautiful black hole volcano disaster" code as I think you'd find it amusing and I'd learn a lot from the roasting I'd recieve.
This sounds amazing - I hope @theprimeagen will take interest.
Do it
It's so funny, I ended up trying to reinvent reflection in my earlier years of programming as well for a project I was working on... I feel like it's a right of passage for any programmer to do when working in languages with no built-in reflection...
I think I used to code like this when I was in high school and self taught. I think I was a much better programmer back then too
Do it
100% agree with the author about walking up to someone at the company and asking them how you can make their life easier and then you just go and build it. IMHO it's a worthwhile trade off for living in chaos. No scrum, no sprint planning, no "should have been an email" meetings. Just a developer working directly with their user(s).
Why the heck is he pronouncing SQL like that 😭🙏
I thought i was the only one... Seriously, wtf
Because f sea-quell, that's why =) also kooh-beck-tell instead of cube-control.
It's a running joke
To make us squeal, obviously.
Considering how many people pronounce that incorrectly, squeal is far less stupid than the other ways that I hear people pronouncing that. For the record, it's pronounced SQL.
As a Data Engineer I can assure you that this kind of things are happened more often than you thought, and Nope, it doesn't happen in cheap, little companies, most of the biggest companies have flaws like this.
Oh boy, and i thought that my process where i have to go to accounting floor, ask a lady if its ok to do X and come back and press a button once a month was a giant crutch...
on "the application for a for a single job", I did a bunch of custom "data massaging" apps that were basically convert some excel sheet, sanity check everything and add some data so instead of doing some half a day data entry job it was a 15 minute job at most (basically any line that wasn't sane was manually corrected and then stored as the default for matching cells, so they would just have to check it was ok instead of retyping everything).
stuff like putting email adresses as delivery adresses and things like that were a common occurence
you should've seen the function that checked addresses, basically a bunch of exceptions, because saint louis could also be "st louis", "st. louis", "louis, st" and a whole lot of other variations, including using the unicode ˢᵗ.
I swear... I think SAP single-handedly raises the global average number of columns per table well into the triple digits
I recently was trying to get a field out of a SAP Public Cloud system that they didnt store at all but calculated each time the data was needed. The Implementation was a few thousand lines of abap and there was no way to call this implementation from anywhere outside this system. I had to Reverse engineer the whole thing, probably going over 100 Tables just to Display this field in a shitty Webapp.
Try to explain to a customer that adding field x to their App Takes 1h but y a thousand times longer.
This is the day i decided to quit my Job
3:30
i had a job where we had this one daily notification where the user would choose an time of the day to be sent and it was not working sometimes
the information for what time of the day was store in a column as a json with the hour of the day and the user timezone. They would just pick every single one, decode the json, adjust for timezone and check if should be sent. I don't think i need to say that it was getting some timeout and memory issues thanks to having millions of users.
This is a Dr. Strangelove story. How I learned to stop worrying and love the code base.
I love how you always highlight sentences from the second-last letter to the second letter. It's just pleasant
Where col like '%foo%' hit my nerve soo deep. I have deployed stored procedures that generate an application summary this way.
I uh.... still use Team Foundation Server. We've been off-and-on working on migrating to git for over a year now. The code was migrated to TFS from SourceSafe, and a significant portion of the codebase predated that. The code history is such a mess that any TFS->Git conversion tools I've tried don't know what to make of it; and yes, the code is unfathomably cursed in many different ways.
Man I think I’ve used b2b websites written like this
This reminds me of a bug in a very old custom ABAP application which generates a batch input on the SAP transaction F-02. The application had over 50'000 lines of code, was extremely important for the company, but stopped working after migration to SAP S4/HANA. Because the whole code was completely undocumented and it contained very deep IF statements (about 20 levels deep), we did not rewrite it. What we did is porting the old SAP R/3 standard application to S/4HANA, so that it exactly behaved like the old F-02. We had to duplicate lots of includes, function modules and customizing tables, but at the end, it worked.
This is the best video I have ever seen.
That is stone-cold hilarious - the watering hole is polluted. The funny thing is - if you work on legacy long enough - you'll find out there is 2 watering holes (one at the back - where nobody wants to go) and the one out front - which is just as bad.
I feel called out by the fuzzy operator :(
J.K. I only use for searching logs or something like that.
If you need 1500 Columns, you really need 100 tables.
you might wanna use full-text indexes
What I have learned over time is most major codebases are started by someone without the knowledge and experience to know better. Most startups are done by novice devs because seniors and leads are too experienced (and well paid) to want to start a major multi-month project from scratch that needs everything built out. Every major company's codebase was once a startup made by novices. Also most businesses aren't cut and dry for requirements, you can expect objectives, needs, etc to change over time, companies have to pivot, so inevitably you are going to have to change direction in your code, entire parts of your architecture become unused. Finally, there will be a bug that has to be fixed at 2am via triage and guess what, you don't have 12 hours to fix it. For people who say college students write better code, make them write their 6-hour project in 4 hours as a marathon, even with static requirements (which never happen in business), the time crunch will emulate more realistic conditions.
I work with a "gilfoyle" like this. Dude reject git because he doesn't trust cloud and stores everything on his own machine, now imagine sharing a code base with this man.
Yeah, I have worked on a place where you got the codebase in a zip-file . Which got passed around...
The company should fire him. Causing more problems than he could solve.
They should be happy that he at least stored it SOMEWHERE, not just delete the code as soon as the client received the solution
but Git is distributed, so everything is still stored on your machine...
If you don't trust clouds, don't put your git repo on a cloud, that's not a reason to not use git. I often enough use git locally to have version tracking on a project even if it doesn't live anywhere other than my own hd, and you can also push/pull between different machines locally via ssh.
It was at this moment he knew, he was fucked up... and decide to release the infos to the public for posterity. A great man, a savior, for all coders.
How many unique bits of info do they have for each merchant? I just don't comprehend how you'd continue to add columns like that....
I wasn't 100% paying attention, didn't the article say it was related to a search feature? My guess was that they had a table where each row was a "merchant", and added a new column for every "keyword" they wanted to be associated with the merchant. A more sane way to do this would be to have a separate table where each row has a merchant_id and single keyword column, and add a new row for every merchant_id/keyword pair. Then find all keywords for a merchant by joining on merchant_id. Now your only limit is row count not columns.
@@Intense011 I didn't hear that it was related to search, maybe I missed it too. If it's that keyword thing, I agree with you, that should be two different tables, one for keywords, and the other to hold keyword_ids and merchant_ids. A join is possible but a sub query is fine these days (in mysql at least)
There's definitely a use for having JSON columns in a database. You just get to plop in whatever kind of not-too-specific metadata you wanna plop in and OBVIOUSLY you're not gonna be a complete imbecile to try to do a WHERE clause on that column.
What kind of monster would pronounce "SQL" as "squeel"?
I'm assuming it's some kind of channel in-joke, but either way I'm never coming back.
I feel this... at one company, we had a 3-part table with equipment_reg, equipment_reg2, and equipment_reg3... since it was migrated from a legacy schema from a system with hard limit of 255 columns per table... and the IDs one one table were stored in a field in another table, even though the current database system (sql server) has auto-increment, it wasn't used for that table, you had to start a transaction, increment the row in the constants table, then use that ID for the new row in the other table then close the transaction... suffice to say it was slow.
Weight Watchers (the frozen dinners) runs their whole company on an Excel file
They're not watching the weight of that excel file...
14:04 this spoke to me on a spiritual level. The legacy code base I'm working on right now, uses date picker, C#, jquery and iis express
Fuzzy searching JSON in a SQL database is my life
Single person use applications is what I did for the last 4 years at my job. They are all only on a hidden folder with all other source code I made.
Who tf calls it "squeal server"??
Legend has it that Gilfoyle's harddrive RAID is still kept alive with frog bladder oil and a ritual sacrifice on Friday afternoons in Pete's shack behind the lake.
It is funny because all the way through university you are taught 2nd, 3rd, BCNF normal forms to reduce data redundancy and save space. However when you work in organisations with big data they often have these "Single Source of Truth" tables with countless columns that could easily be broken up into multiple tables, however they don't because it is significantly easier to write various SQL when most stuff is in 1 table rather than 1000.
This was the most cathartic article I've ever seen. I've gone through all 5 stages of grief while reading it. By the end, I was spiritually reborn.
Yeah, I've seen some horrific database designs. At one place we were told to add a feature, by next Thursday. However the database kings required that if you needed a new column, you had to submit a formal document. The database committee would meet once a month, and usually reject a proposal once or twice. So we were stuck. Fortunately I saw that the "fax number" field was always empty. So we packed in two integers into the fax number field. Nobody noticed.
At another place, all SELECTs were really slow, as there were no keys, every select had to do a string search on every row. Every time.
It gets worse. Another place was using an offbeat SQL where the basic insert/delete features used a non-standard way. Something like a SELECT with an empty something was actually a DELETE. And the SQL server was like seven years past end of support. Yikes. When I suggested factoring the code a bit to speed up the conversion, my key card didn't work the next day!
Did a dashboard with a single SQL query with - if I count correctly - 26 joins. about 1-15 of them are CTEs with their own set of joins I don't even count here. Works flawlessly. Even presented it at a conference.
The reason: It's a DB for student advisors at a large university. It pulls from student records, employment records, advising tables, grade tables, enrollment tables, student attribute tables, email databases, and the list goes on. The reason why it was such a game changer is because you now have caseloads and student attributes and even checking for who is still assigned to an advisor that left or the student changed their major and now, you can see they need to be reassigned. Also checks on academic probation and dismissal cases so we can do targeted outreach...
Things that took a supervisor the better part of a day now takes them 5 minutes. The key is caching every morning.
I'm impressed and also absolutely horrified as to how they got to reach the max column count on a MSSQL Server. At my company our record was 7 tables, but only with around 400 columns.
MSSQL has a max table length of 8KB, so that would mean each of their columns had a average length of 8 god damn bites. Thanks for the video, I just realized I still have a long way to go and a lot to learn from this company
My first development job, I found a procedure that had an "error handling" subroutine that just increased the id by 500k and it had a comment that said "if there is a problem processing, throw the requisition very far away in the queue, should keep it running for at least 5 years"
Luckily I only stayed one year in that company.