Stress Testing Games
Вставка
- Опубліковано 16 січ 2025
- I talk about how to stress test games to make sure they work with a wide variety of different machines, beyond just the minimum specification.
Videos I reference:
Playtesting The Outer Worlds: • Playtesting The Outer ...
The Outer Worlds: Eight Playthroughs: • The Outer Worlds: Eigh...
Min Specs And Ratings: • Min Specs And Ratings
I worked for a mobile game dev during the flip-phone/JavaME times (before smartphones), they had literally a wall of phones, probably 200, just to test the games on. The phones had such a range of capabilities that several separate builds with different content had to be made.
That must’ve been astoundingly frustrating
I assume now they’re all done on emulators
@@ashuggtube The development itself was mainly run on emulators. But they where just an approximation, so the builds had to be tested on the phones at the end. Nokia being one of the worst contenders, as they had a bad Java implementation but where so popular that they required porting and could not be left out. Sony Ericsson made great phone in comparison.
The games often had several graphics sets or even different content, depending on what the phones could support.
It all changed when the iPhone came out.
@@vast634 look at me, assuming that the hardware actually worked as well as the dev kits / emulators 😂
Could they deploy/patch to multiple phones at once automatically at least?
@@PeppoMusic There was a build system that compiled the code and archives (jar) to a number of phone configurations. Back then there where many different app stores from many different phone suppliers. But for testing, the tester had to use each phone and connect it to a local server and download the build.
There where some benchmark apps that cold test the general capabilities of the phone hardware before finalizing development, so the final build was reasonable for the phone model.
Everything that you make is just so insightful on how hard it is to be a developer of games. I would’ve never guessed in 1 million years that sound would have caused as much issue as it did. Simply amazing.
As for the monitors, I can only imagine how bad it’s going to get with the VR included in some of these projects that are also to be played on screens.
I never understand what was the problem reading sound card port address, IRQ and DMA setting from environment variable.
What I mean, every sound card what I saw back in DOS days and also installed myself, just set environment variables at boot time. And in my understanding, DOS had API call to read environment variables.
Still, nearly every game had some setup program to set those configurations.
@@gruntaxeman3740 No, the environment variables had to be set by the user. But not every user did this. Some systems just missed the information set blaster.... in the autoexec bat and then there were other soundcards that did it differently. Some even required a driver to load first. Usually the PCI based ones, that came later.
Most setup programs were easier to use by the user than configuring the autoexec bat. And most game manufacturers licensed such tools as a 3rd party application and together with a sound driver library, thus it could be shipped with the game.
@@OpenGL4ever
When I did clean install DOS 6.22 + Win 3.11 + Soundblaster 16, Soundblaster setup program added environment variables to DOS, and drivers to Windows. And that was ISA card.
Sure I can misconfigure any system, remove sound device / driver module away but I found it odd that application program have sound card setups because back then there was also couple of games that just worked without any setup program and read environment variables.
However, I haven't install other sound cards than Creative cards. I know that there was clone cards, there was Gravis Ultrasound, Adlib or computer was connected to Roland, and DOS didn't have any kind of way to choosing audio device if there was multiple.
I understand that sound API licensing well, to have someone else worry. So if it has setup program, that is in game then.
But I didn't understand WTF happened in early 90s that suddenly almost everyone was using Watcom compiler in games. Because what I tested, it wasn't very good, every game made with that make Windows 3.11 crash, and I found that DJGPP distribution was essentially price of some shareware CD-ROM collection (almost free) and did job better. It was also used in Warcraft, Quake and some other titles and I run some games under Windows 3.11 using that.
In my memories, Borland was famous with their cheap Pascal compiler in 80s that was largely used so I wonder wasn't there any path to build DPMI software with their tools.
The first time I ran into the term "stress test" was when I was asked to help stress test World of Warcraft before it launched (yeah, long time ago). At one point it was so bad that every time we looted a mob, it would get stuck trying to grab the loot that was dropped from the server so our characters would be stuck in the pick-up animation pose. You'd see a bunch of us sliding around the map on bended knee.
Trump wouldn't like that kek
That's super interesting. Wow, at least classic wow, was known for its low latency.
Knee gliding is the way to travel
Omg! When you talked about read errors in the days of floppies and CD rom drives, I thought, "Hmm, do I remember that?" I flashed back to being a little boy and playing games like Stonekeep in that era.. Immediately after, you mentioned shipping Stonekeep! Haha. I fondly remember sitting around hearing our computer's CD rom drive go "Whhrt-whhrrt, beep veep.." I remember my friends' families back then when we were learning about computers, largely through the games that were out at the time. "Yeah it's probably doing that because the CD rom drive's too slow. It's better to have a fast one." "I just got a 16x CD rom drive!" "He just got a 32x CD rom drive!" I wonder how much we were sitting there idly due to errors! Had no idea.
It's still painstakingly wild west on Android for games that ship on mobile. Modern, higher-end devices from reputable companies are reliable, but as soon as you start having people with $50 mobile tablets they bought off Amazon/Wish or devices popular in China/India/Brazil it's a free-for-all. We've had _numerous_ manufacturers tell us that Vulkan/OpenGL is the most stable on their device & then we switch to the opposite and see a 50% reduction in driver crashes. Or sometimes they add additional bloatware into the OS which results in loading system libraries requiring drastically different amounts of address space (issue on 32-bit devices) from one device to another. And sometimes these devices can be excruciatingly difficult to acquire, like the Vivo Y15S which was consistently showing up in our top unstable devices but wasn't available for purchase in the US. It's not mobile-exclusive though, like the recent Windows 11 crash issue in Ubisoft games. It's a gut-punch to deal with these too because you put out a solid game & then you get all of these crash reports from people with terribly buggy hardware through no fault of your own.
Using latest and fanciest graphics API features, game targetted to too new devices, or too new C++ version can easily cause trouble. Web developer mentality where they have compatibility tables for everything is good thing.
But that available system memory thing is really nothing new. Back in DOS days, some game developers did really dumb thing making game require 600k conventional memory. They didn't though that people may have some TSR software running. I could accept some 576k but they just didn't though how world works.
Based on Android documentation, 32-bit ABI devices the memory available to the kernel and userspace should have 416Mb free. So if game is designed to run
I love it when I see a game review on the play store and it's like they game is quite laggy and my phone gets hot and it's like an S7+ like yeah it's gonna be laggy and yeah your phones supposed to get hot.
@@mryellow6918
Mobile phone chips really can't run full speed long period of time because they heat and then they start throttling.
I don't know how it is done but there should be somewhere switch to developer to turn off all those boost clocks and run phone on 100% load continuously where it doesn't overheat, and use that ensure that game can run constant speed.
@@gruntaxeman3740 why that poor management there's a reason boost clocks happen, the heats always gonna be there if your locking it down to a low frequency but hammering the cores its gonna be worse performance for the same heat. the reason cores boost and park is because there's areas where they arn't needed so they save power and heat for another core that's being used or to boost later for longer as the heat sink has more headroom.
@@mryellow6918
I know but worse performance is acceptable to do stress testing to ensure, that software works all time properly.
What I mean that game developer, should have "work backwards" mindset. That is that you fix game to run example 30 fps, then you set boost clocks away. And now you got 33ms time to do everything you want for every frame. Make it happen. It really should run deterministic way.
Modern game development disease is that game developers don't set reasonable constraints and are out of clue some basics. I can guarantee that 90% of developers don't really know how computer works, and many developers don't even know difference between stack and heap.
Where is the line between "we need to get this to work on this specific type of machine" and "this specific combo is so rare we shouldn't spend time making it work"?
Thank you Tim, you made this video at a perfect time for me! I am looking into deterministic simulation testing, which is basically writing a simulator that would play the game with certain modifiers on and in case something wrong happens you have a seed and can easily replicate it.
The things you've mentioned would make for great modifiers in this case.
The remark about peripherals being plugged in and out reminded of how in World of Warcraft there's a specific check for players plugging their headphones in and out during a cutscene in order to correctly handle that, cause before that the game didn't recover the audio at all even after the cutscene ended and players had to restart the game.
Here's a list of other things I thought about testing:
- varying frame rates
- sending multiple inputs at once, from varying sources (like mouse kb gamepad)
- losing focus (alt+tab or it got stolen)
- changing resolution (especially relevant for games with adaptive resolution rather than fixed resolution)
- machine going to sleep mid-gameplay (consoles and Steam Deck)
- when there's less RAM available than there needs to be reserved (combined with what you mentioned, fast/slow and little/lots of it)
I found that cheap CD/DVDROM burner drives often only lasted about a year if you were using them regularly. This was one of the reasons I used to use NOCD cracks on most of the games I bought. Otherwise it was often a toss up whether a game I owned and had completely installed would actually run because the disc drive struggled to correctly detect the presence of the game disc.
So many drives killed by DRM and I rarely burned anything.
My Blue Ray player doesn't read most of the Blue rays I have...
Related: having the game check the hardware at startup and pick a custom setting that is a stable optimum, so players dont need to jump through the graphics settings when starting the game. Many games just have a default that is lower than it could be on good machines or too high on slower ones.
Shoutout to you timothy. After watching the notorious david jaffe meltdown regarding grifters ruining dragon age veilguard this was refreshing, enjoyable and informative to watch.
The "labor of love" and the joy derived from the "breaking" of things.
Thanks for sharing with us! always good to hear other testers thoughts.
:)
Thank you so much for these videos Tim. It truly is so fascinating to get a look behind the scenes of video game development. The problems that could come up and solutions to them.
Wow... Listening to all those anecdotes makes me thankful that now hardware is much more standardized and reliable on PC
I remember hearing about the presentation done as part of a demonstration for a publisher. That was for the game Astroids. The developer could only get the game to boot up and run on one computer in particular.
I remember hearing that the original Fallout1/2 determined random encounters by how fast the CPU cycled or something (not a computer guy), so on modern machines, you would basically get a random encounter every pixel.
I have a memory of playing (around 2008) and this happening, and being massively overleveled by the time I got to the Den.
Happy Thanksgiving all!
Yes I love a good fast UI! Atlus jrpgs have a stylish speedy UI it's my favorite.
Love your videos in general, but this thumbs up is for the sleeping dog.
She is the MVP
Great stuff, thanks Tim!
Perfect thumbnail
When Alone in The Dark New Nightmare released, the screen/camera change via CD was enfuriating , since it was a slower game than the one from 1992 😂
Speaking about PC speaker sounds, Scream Tracker for DOS could do some solid 4 channel digital tracks :)
Responsiveness to mouse clicks is important. I remember back in the day the windows version of Panzer General was sluggish and it gave me hand pain to play, i think i was pressing the mouse buttons harder because of the show response time.
One thing you didn't test for, i had an apple 2 with a hardware failure in a hard drive, when asked to write data it would not write anything but also did not return any errors. I discovered this in Wasteland when everything in a temple in Las Vegas kept respawning. It took me a while to figure out the problem but by then i had farmed it for XP and a large amount of ammo for laser weapons which was hard to come by at that point in the game.
Load times... Multiplayer Satisfactory has an issue with this, it fails if it takes over 30 seconds to load which isn't hard at all for a remote player on a large save file. There are config files on the server that can fix this but they aren't clear and you have to do some web searching to find the fix.
Hey Tim you are amazing.
I have suggestion for channel to add a new content type which is talking abith games in trend for more eyeballs and your corr content is goodmine i hope you soace then out a bit and bring more audience to thede topics by making in trend contrnt. Thank you for your great advices ❤
so thats what that snoring noise is hah
We love her 😂
@@ashuggtube honestly it kinda scares me
I was actually hoping for a Timothy Cain Special Thanksgiving Spectacular
I couldn't get the dance moves down. Sorry.
@@CainOnGames For Christmas, there's enough time to put up a picture of your dog wearing a Santa costume. I think that would suffice. :P
@@CainOnGames we will need a UA-cam short of you dancing with the dog!
pretty much what I did for my game servers, just make some headless "user" that spams the hell out of whatever actions it can do that the server will have to read and react to lmao
2:45 you were referring to the infamous pentium floating point division bug right?
I've for a long while thought that there must be a service out there that owns one of every possible piece of hardware going back at least a few generations, and for some fee will stress test and benchmark your software for you. If it isn't already a thing, I should invest in it, because even large companies don't want to deal with managing that much random hardware and would pay for anyone else to do it for them.
I can't imagine the software update nightmare that would be.
I remember about 20 years ago John Carmack going on a huge rant because he had three computers that he thought were identical but gave vastly different results, two played their development version correctly but the third had the camera spinning wildly making the game pure chaos. It turned out to be some library file that windows quietly updated on just that one computer.
@@roberteltze4850 Oh, absolutely. That's why a group dedicated to understanding all the intricacies of the hardware and drivers would do a much better job than each individual development studio trying to do it on their own.
I don't think testing hardware alone matters because there is also software involved, so test matrix requires several driver versions, OS releases etc. Few generations also don't matter because number of hardware generations, and hardware capability generations are increasing, because computers are more long lived than before. There is combinatorial explosion of different configurations.
I believe that analytical way can cover more, because there are certain things that can be calculated.
1. Memory usage
2. Storage space usage
3. Bandwidth usage (between GPU-VRAM, CPU-RAM, client-server, hard disk-memory etc)
4. Latency in many places.
5. Draw calls
6. How many iterations there is in loops.
And those hardware specs are usually easy to found so many decisions can be made before there is target hardware to test.
Optimizing things to fit in GPU caches is micro-optimization and usually unnecessary, except finding some tile/instancing size where no cache misses happen that are really repeated a lot. Caches are kind of made to speed up calculation, while bandwidth has been almost every case through gaming history, major limiting factor.
I don't see performance as issue if developer intentionally doesn't push things to limits or over the limits when something weird can happen. Or just do something very dumb, like drawing pixel size polygons like Cities: Skylines 2. But hardware+software real capability is different thing because there is buggy drivers, or some feature is not yet implemented so they need some compatibility tables.
ive played a couple modern indie games that still have the issue where changing audio output while the game was running and then they would just break their audio output
Happy Turkey day!!
This certainly sounds expensive. Having access to all these test systems, designing for each of them, deciding wether or not to support them and the manhours of testing each machine. And you need people who actually understand these machines.
This is why when you read the EULA of open source projects they excempt themselves of any responsibility for damages caused by their software.
Here's a question: how do you get bad hardware? I've heard this idea of testing against bad hardware before, but it was never explained to me how to get my hands on said bad hardware. Than again, not my department is also a thing. Just curious.
Over time, we collected bad hardware naturally. Someone had defective RAM...we kept it. Someone had a malfunctioning CPU...into the pile. Some brands had inherent defects. At one point at Interplay, I could regularly produce read errors on three different brands of CD-ROM drives.
@@CainOnGames The buggy CPU you mentioned was maybe (probably?) a result of the Pentium FDIV bug (discovered in 1994).
Also, an interesting way to potentially force RAM errors is to run them at a higher temperature than they're rated for (e.g. by removing heat spreaders, blowing hot air on them with an air gun (obviously that's potentially dangerous though, do at your own risk!). There was an interesting talk about bitsquatting a few years ago which had data demonstrating that games consoles had a higher rate of bit flips than desktops because they tend to run at higher internal temperatures and the vents are more prone to being clogged by dust/debris.
I am now running Arcanum on a VM on my M2 Mac - I bet this was not a part of the stress testing scope :)
However, the game runs fine. Surprisingly, the worst lags and hiccups are related to the main menu screen, so I curse every time I hit the Esc button accidentally 😂
Gaming consoles should be great for minimizing the amount of hardware variations a game developer would need to account for and the customer should be able to just buy a console and be sure that any game for that console will run. There are still console games that don't run very well and consoles sometimes get hardware changes depending on year or upgraded versions but it should still be generally much less variation to account for than with general use personal computers. I have thought that this should make gaming consoles ideal for competitive gaming since every player would have basically the same hardware making for a more even playing field but in reality it seems like serious competitive gaming is more common on desktop computers rather than gaming consoles. One of the reasons I don't feel too interested in competitive gaming is because of how much hardware can potentially give you an advantage and basically make the whole scene a pay to win sport.
What are your thoughts on GFI (Goals, Feedback, Interpretation) as a lens for narrative design. I'm only a student so from what I've seen so far it looks nice, but I wanted to know from an industry veteran if it's something one could actively use or not.
Out of curiosity, is there a specific tech jump or development you've seen during your career that really took you by surprise or changed game development dramatically, or has it always been a mostly gradual and predictable advancement?
ua-cam.com/video/rr61gn2k46o/v-deo.html
@CainOnGames Thanks!
How many of these problems do you think should be brought up early and incorporated into the technical practices of the delivery team?
7:50 Mitsumi Double-Speed CD-ROM drive, it was so bad that I had to refund it.
How do you feel about Digital Foundry and like who goes in depth on performance and technical excellence.
i would love hearing Tim's opinion on bespoke channels
Happy Thanksgiving, Tim. Minor question, is there a way to predict to a reasonable degree the performance cost of a game?
I usually make sure to test a game on the minimum spec machine. So now the question is how to select that spec. A good way is to look at the results of Steam hardware surveys, to see what hardware is owned by the aggregate millions of Steam users. You can pick a cutoff for older hardware that loses a % of users that you are comfortable with.
@@CainOnGames Thanks 👍
I think you answered the what part of this but not really the how, perhaps. As a solo gamedev how do you test for these? An obvious way is beta testers, but is there a better way to "simulate" some of these tests? Maybe a service to test your game on various hardware?
Having never been a solo dev, I am not sure. Maybe early access?
Some of them can be simulated just writing normal test code but those hardware/platform variations are mess. Operating systems, GPU manufacturer, different driver versions, ARM/x86 CPU, consoles and their generation... It is just not something that solo dev really have resources. So in my opinion for solo dev, it is best to mitigate this as much as possible abstracting platform away.
Best way to abstract hardware away, and most limited one, is write game in browser but affects to game distribution so and limitations may be too much for some games. Not likely for solo dev projects. For ready solutions there are game engines. I've evaluated myself huge amount of game engines. Like Unreal Engine, Unity, Godot, IdTech, Armory and some others. It feels like Godot has cleanest abstraction so far. That go actually very far, there isn't even way to change resolution so changing screen mode doesn't mess things or window doesn't disappear and go to other monitor so easily. Game code running bytecode top of runtime and there are prebuild runtimes for different platforms, so no need to compile native code. Graphics APIs are also thought, that there was OpenGL ES 3.0 and OpenGL ES 2.0 on 3.x series, and 4.x series has Vulkan and OpenGL ES 3.0. OpenGL ES APIs are subset and those very same subsets are found from browser so it is likely that it is not so easy to hit some driver bug on those. I assume that they follow same practice on Vulkan and limits features to same found from webGPU.
Same ideas found from Godot can be used in Unity and Unreal Engine, like don't use native code and use blueprints instead.
And one way to mitigate is to make decision on target and just calculate stuff. Data that is read from VRAM should fit on bandwidth, find what is capability of different target hardware like shader level, texture sizes, cache sizes etc. some optimization decisions can be made before single line of code is written, like ensure that every triangle is at least 10px size, or make shaders that uses at most four textures, limiting textures to 2k and so on. This kind of forward thinking can save from trouble.
@@gruntaxeman3740 Thanks for the detailed reply! I'm not sure whatever I'd make would run in a browser, and I'm not too keen on using Godot, their PR is terrible. 🙂
I've started learning Unreal Engine a while ago, and I can honestly say, that I don't know anything about it :D
My question was more along the lines of possibilities of let's say test the game on a few hundred variations. I tried to google something like a company that would take my game, and run it on various PCs in their warehouse to help determine minimum system config and such. I haven't found any. That lead me to believe that most people do this my means of testing and optimization before release - like alpha and beta testing.
I do have more than 1 hardware to test the stuff on, but I'd perhaps willing to pay for some professionals to specifically run it on various hardware just to see if it runs and how well with what settings.. Strange that there's no market for that.. I guess it's the de-volution of the Beta tester role.. In the old days that was a proper paid job, and that evolved into most companies expected that to be done for free by enthusiastic gamers..
If you have the funds, it can be worth to outsource porting and testing your game to different platforms. As an alternative to early access, which isn't an option for console and phone releases. However, gotta be realistic as a solo dev as well and properly scale to your capacity, this includes going for safe options, well trodden roads and minimizing what you actually have to test for.
@@PeppoMusic
I believe that console ports are likely better to do this way, perhaps iOS/macOS too but it is fairly easy to get access to used machines, friends machines and so on so it is fairly easy to set up testing for Windows 10 / 11, Debian 11 / 12 and ChromeOS. Android devices are found from almost every pocket. And if the game isn't especially demanding, buying Raspberry Pi 5 gives nicely low bandwidth / tile rendering found from mobile devices.
It is important that game is made for certain hardware capability that is all targets because porting & testing is likely straightforward but modifying, re-engineering and optimizing it to work totally different hardware is likely hard.
Okay Tim but why does Spacers Choice not work with Global Illumination on AMD cards and the original version doesn't work on AMD cards without hitching? Testing is fine and all but implementing the necessary fixes seem to be beyond Obsidian's ability.
Any comment on current obsidian drama? Would be cool to get a glimpse into that world to see if it’s just a one off loon or if the inmates are running the asylum.
No idea what Obsidian drama. Everything they are releasing seems absolutely on point.
Tim won't speculate on stuff like this btw. Or even say anything if he knows anything.
I was so looking for comment like this. While the situation is unfortunate, I don't think Tim has anything to do with it - the bad situation that arose is something that Matt Hansen has to deal with on his own/with help of his colleagues, but certainly it is not something that would be talked about in a public way such as this/not yet.
Personally I think that Matt had good intentions when writting what he wrote, but he chose the worst possible words to write it in. Elon Musk is definitely not a good samaritan that came to redeem us all, he personally creates his sort of cult of personality so that everyone loves him and it is clear that he absolutely dislike LGBTQ+ people and politics related to it.
I am only sad that it is very likely Avowed will suffer bad sales thanks to current drama situation.
@@lukaskubinec9608I didn’t think he had anything to do with it, I was just looking for insight into how things worked while there. I agree. I wasn’t expecting much of a response I was just shooting in the dark. I am appreciative for the reply. Thank you.
@@Mark-wq7wd It's okay, I just feel like Tim wouldn't want to get into how the working conditions and relations among workers are there at Obsidian. What Matt wrote crossed the line by a lot and I hope he will answer to some sort of justice, despite that I think his original ideas were good in its meaning, he wrote something in a way that is absolutely unacceptable for a person of his position at Obsidian.
Still I do not think Tim should be bothered by this drama, he is very easy-going person and doesn't deserve any BS thrown in his way - that being said, from what I seen most people are nice to him since they know Tim is a wise and nice person that knows which words to choose to speak with others (even if he doesn't like something...I recall in one of his videos he spoke about LGBTQ+ people being demonized back in 90s - to be fair that is not something very nice and I sort of understand him and people that are being treated poorly only for being different - despite that I think the social politics should be integrated into games in much better way than it was in Veilguard - Larian studio with BG3 integrated social politics nicely).
Some resolutions are such nonsense tbh, even more than twice the horizontal length than the vertical. Why do people need a noodle screen?
Man, I hate slow menus, especially when they're not tied to the game's framerate.
Lately, it's common for games to run at a high framerate and when you enter the menu, it drops to 30 or 60fps. The input response is slow, the animations are rough, everything is blurry.
If it's a game where you need to enter the menu frequently, it kills the mood. The probability of stop playing is high.
a good video idea:
can you talk about what caused fallout to have issues before release, and got you less pay
He already has a video about this: ua-cam.com/video/6i8Bx4zYIKQ/v-deo.html and also this is important ua-cam.com/video/UGfaCXEu0tE/v-deo.html
@DxAxxxTyriel thank you👍
Happy Thanksgiving
Happy Thursday
Happy Turkey Day
Which ever one you pick.
It's just Thursday for me.
Probably get a Turkey sub from the gas station.
Just Thursday for me. We don't have Thanksgiving. Plus I don't like Turkey so that's fine by me.
Was that dodgy CPU Cyrix Tim? 🤣
I think it was the first Pentium with the floating point division bug.
Hi Tim, I wonder if you already spoke about "early access" of games? I think it is very nice way to receive feedback from players and shape the game to their liking - at least partially. It worked absolute wonders with BG3.
Also loving that cute Abby snores 🥹🥹🥹😄😂
P.S.: I already see that you did speak about it slightly in "How the internet changed game dev". I will come up later with a valid question 😂