Automatically Testing Hazel - My Game Engine
Вставка
- Опубліковано 27 тра 2024
- To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/TheCherno. The first 200 of you will get 20% off Brilliant’s annual premium subscription!
Patreon ► / thecherno
Instagram ► / thecherno
Twitter ► / thecherno
Discord ► / discord
Hazel ► hazelengine.com
🕹️ Play our latest game FREE (made in Hazel!) ► studiocherno.itch.io/saving-c...
🌏 Need web hosting? ► hostinger.com/cherno
📚 CHAPTERS
0:00 - Hello
2:17 - What happened to Tim
3:05 - A better QA test strategy
8:48 - Automated testing
17:52 - How should I make dev logs
💰 Links to stuff I use:
⌨ Keyboard ► geni.us/T2J7
🐭 Mouse ► geni.us/BuY7
💻 Monitors ► geni.us/wZFSwSK
This video is sponsored by Brilliant.
Thanks for watching! Please let me know what you think of these dev logs, I want to make better videos for you guys ❤
Also don't forget you can try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/TheCherno. The first 200 of you will get 20% off Brilliant’s annual premium subscription!
I actually really like these videos, they actually show how the process is, just just those average dev logs where only the good stuff is showed. This shows the actually struggles and I really like it
Call the QA PC "Testing & Integration Machine", or TIM for short.
Brilliant. Just brilliant.
But it might quit
this should be the name.
poor TIM, sry I mean Tim
lmao
You should go down the GH actions self-hosted runner route. It's not as bad as it seems
Yeah, imo this probably makes it easier to maintain + run tests and there's a big plugin ecosystem for things that would otherwise require a lot of work to implement.
Could also use Jenkins - my team uses it for a similar-ish usecase (not graphics-related but something that requires a lot of compute where we have our own machines). The instance we use is managed by a central team so I'm not sure how hard setting up the server is, but configuring build and regression pipelines, and the actual "runners" is pretty straightforward.
Yeah I was gonna just suggest to install a self hosted runner. It’s not hard to install.
Came here to say the same thing.
This!
Oh yeah, I never use this before, interesting, this might be the long term solution that he should look into.
I feel bad for Tim
why
He was basically forced to quit. The entire progress of the engine was on his shoulders, while cherno got all the praise.
how so?@@topg3067
@@topg3067it's always like that unfortunately. elon musk, mark zuckerberg, steeve jobs and so much more
@topg3067 just curious as to how you know this to be true? On assumption I would have thought he just found something more lucrative or just a better fit for him. If you know otherwise, do tell... this may be more interesting than I thought.
RIP Tim
I love this video so much as an SDET (who got a boost in his C++ skills in no small part due to this channel). Automated testing is a really interesting problem space to solve and a lot of people have a mistaken idea of what automated testing is.
github actions with self hosted runners are actually really easy to setup and use. it's basically just 1 command to setup the runner.
probably easier on linux but yes
As someone who works with automated testing, I am actually interested in hearing all your thoughts, not just when all of it is ready
I really miss the Ray-Tracing series.
yeah he has a way with starting series' and abandoning them
same
@@maxi-g yeah unfortunately his videos are all over the place. I would guess that he didn't proceed with the ray tracing series just because it was getting less and less engagement from viewers.
Really enjoy these types of videos, it really shows the real world problems and the journey itself. Also liked the part about the lens at the beginning. Such DevOps problems are always interesting for me. Keep going!
I enjoy these dev logs a lot. Please keep making
14:39 I had the same problem some weeks ago, I used a tunneling service, as it would not need port forwarding and had an SSL certificate included. It helped me set up a signaling server for webRTC, not the same thing, but I figured it can work in your case too. Anyway, great video!
Haha great video man, love the random approach to filming
When working with offline computing that is based on some online computation, always prefer a poll solution or a pull based mechanism to let your machine know about anything online.
Alternatively, you can open your system to a network(where you set up your ci tool) and let some ci tool remotely access or ssh into it, run some scripts and done.
The explanation for what's to come in the next vid is perfectly fine. Better to get it out now and have a clear understanding of what you need to do later, great vid.
The QA Pc cameo is mindblowing!🤯
Enjoying the hazel content for sure! For me, I would want to see more of the thought process and how you get to the end result. Even thought you went with X, I might not know that there is Y, Z to consider before making that X decision if that makes sense.
These dev logs are awesome!
Actually would love to see how you go through the process of intergration, your thoughts, see what worked and what didn't. How you deal with it, etc. It's a great learning opportunity I think.
Hey Cherno - I noticed during this video from 18:00 onwards that your camera was constantly making micro adjustments to the focus, so maybe that has something to do with why your lens crapped itself? I don't know enough about photography to say for sure, but I know enough about machinery that cycling constantly when it's not supposed to or designed to can lead to premature wear and failure of parts. I think this is the reason why there is still a market for camcorders as opposed to dslr's for video production purposes (not specifically the lens thing, but more that the autofocus feature being far better on them than most dslr's because they can be tuned better for a fixed lens). Don't know if I'm wrong here, but just my assumption from what little I do know. Hope you can get it sorted in any case. :)
stoked for the journey! If bring us along with you, then we can celebrate successes with you!
the test automation sounds like an interesting project :)
Great video.
I actually think making videos of your process like this, is fantastic.
But I also think a well made edited video of the final result ( like after a month of work ), is also very nice.
So, post both if you can.
In my engine I use constexpr and consteval for testing my shades and rendering. During compile time it will render what the result screen should look like into a buffer, then do a render loop and compare the actual back buffer with that. My engine has almost 100% unit test coverage as of now
It's more like you talking about what's going on mixed with a bit of you talking about talking about what's going on. Seems to be working. Keep it up 😊
To automate looking at images, you can have the models spell things out and then use off the shelf OCR.
I enjoy watching the whole process while you build or develop things. I've setup CI/CD using self-hosted bot from Azure DevOps, it's pretty easy and I like their docs. There is also Jira, which seems to be another good option.
0:55 bro I really liked that wide view,
You can try to automatically compare screenshots to check whether it renders correctly.
@@samoatesgames No one says you need to match per every pixel. Just comparing the overall difference between pixels for image chunks can tell you if something is different color or missing.
Yes this is the plan, maybe for next release
@@samoatesgames I have never unit testing GUI and renderer, but usually with unit tests we only test output functions and their coverage (where passing inside the if-then-else) given a known context and inputs. With virtual methods, the good tool is gmock (googletest): ypu change thier behavior during the test. For modifying the context of a class, I use the trick inside the MyClassTest.cpp file #define private public #define protected public #include "MyClass.hpp" #undef private #undef protected You can now access to private member variables/methods of your desired class, modify them, call the method with inputs and assert expected outputs (and modified member variables). For testing shaders I guess you have to emulate them in the CPU and a window is just a big matrix of pixels. After why not simply using Github actions. This works perfectly (this was not the case with Travis, Appveyor, Jenkins is just a crap) for pure unit tests. The big deal is that is running inside a docker: so no exported display is a pain: you cannot call your OpenGL functions. So yes I guess you have to run your benchmark tests on a personal computer. For public Github projects CoveryScan is free.
Snapshot based visual testing? 😂
@@samoatesgames ye bro just make AGI for that.
i really enjoy these videos
I find hearing about your thought process at least as useful as hearing about the result
Awesome video
Count of word "Tim" used here made me understand how much you miss him
listening to your journey and train of thought, the problems and solutions you come up with, and the result of what worked and what didnt is really interesting and helpful. Eg: we now know setting up git webhook for a local build is too much effort, instead of just hearing about how you set up a polling system and wondering why you didnt use actions.
This style can help us with similar problems we might face ourselves, and also the comment section might be able to give you feedback on good solutions
Awesome Vlog
As a other developer. I appreciate these types of dev logs. The journey is just as important as the result. These are the types of videos that show new devs the actual content of what a software developer will go through.
I have been a developer for 5 years and I have to say, I wish I had videos like these going into it.
This is awesome
I'm setting up some github CI stuff for my current employer, they already have it for some projects and they use their own github workflow runners. They said that setting up local runners is easy and I'm planning to test this on my personal computer for game jams. While we do use unity which makes this a lot easier for a bunch of reasons, I ran into a lot of cross platform issues and I also had to rethink how I solve issues with a build. For example, a script was missing from a build and I could just add it to a scene I know is in the build, but that of course doesn't scale and it's easy for somebody to come in a few months later and delete the instance because it's "not doing anything in the game". I ended up making my own script dependency collector which runs with the build system.
Run Hazel with some -debug flag, that allows you step each frame.
Have the C# app use send keys or something to trigger next frame. Screenshot each frame.
Compare the two images using a diff tool (Image magik?). If the diff is above some threshold, raise it for a human to inspect.
That way you don't have to check everything, just the flagged differences if there's a potential regression.
I like the idea, but I would imagine that would introduce a lot of overhead, which would mean having to run each test twice - one for raw performance and another for frame inspection. IDK - might be suitable and more thorough depending on what he's going for.
Btw in production show cases, we would like to see the thing built. On dev logs, we would love to hear your thought process and what you might consider etc.
I am not a c developer and I don't build game engines, I don't know why I watch your channel. I think I like people who do hard things. Good video. enjoyed. all the best to the guy that left.
I do build serious and slightly complex system and "testing" is something i'd like to implement, maybe thats why I clicked on this one. Will tune in for followup with what you did.
There are so many CI software options out there, you don't have to reinvent the wheel. GHA would be my first choice because it's right in GitHub and you can make self-hosted runners.
My suggestion would be to have one computer (or VM) to build the engine at a commit and then send it off to other computers (or VMs with specific GPU passthrough - you could do it all on one machine that way) to test on different GPUs. There would be some overhead from this specific setup of course, both from virtualisation and putting GPUs potentially on PCIe sockets that go through the mobo instead of directly to CPU. It's not the only way to do it for sure.
There are ways to handle dynamic IPs and ssl for nginx and it isn't that hard. But in this case having a simple script sending requests to git from the client side is probably the best and the most flexible solution for this case scenario. Well done.
i had the same problem with GIT actions. I setup a RasberyPI to be exposed on the Internet working basicly as a Proxy. Git Actions Calls my Exposed Linux server which calls my lokal test Maschines. The linux handels all the internet problems like changing the External IP and SSL and Windows can just be Windows with a little service for RPC calls from linux
I personally would love to hear what you come up with. This I think holds a ton of value, at least for me.
I am doing the same type of things.
More of this content, plz. Integration tests are key!
I realize you may or may not see this comment, but specifically talking about the requirements for robust testing at around 12:00 into the video - would it be possible and practical for you guys if you added the individual computers yall use to an on-premise k8s cluster (going with the thought that these are dev machines and not daily driving PCs) and schedule an automated testing scheme outside the working hours that they're being used to automate the testing? I feel like, while complex to set up, the usage of node taints and scheduler configs along with wrapping a test suite in manifest files to deploy on each node (the individual workstations that you currently have and any that you may add later down the line with specific hardware to test on) might be a way to achieve this in-house. You could even set it up in a way that would pull a specific branch, test that branch's code, and report back to some sort of endpoint for statistic collection.
The only downside really is the relatively massive learning curve to setting it up and configuring the cluster and nodes the way you would need them to behave (in this case replicating the pods for tests without load balancing the resources to ensure each node is accurately using the test suite on its own hardware), but the upside has a few points to be made, such as the ability to not only test the framework, editor, and engine modules, but also be able to test reference doc deployments, app deployments made from the engine, and health checks on the pods that run your manifests (while being able to collect logs for each particular machine in case of errors or crashes)
Edit: Just reached the point at 16:00 minutes in and honestly, the script idea sounds so much less complicated, especially if you end up roping the community into being CI endpoints with an opt-in script run -> either way, best of luck in all the headaches traversing through this stuff! I enjoy watching these dev logs along with the entertaining videos you produce because of the sole fact that there's so much insight to glean from them!!
I've also had my Sony 24 1.4 GM die on me, I managed to resurrect it by thoroughly cleaning and brushing the electronic contacts of both my lens and my a7iv
For testing if the images look right:
Can't there be a public website in which everyone can look and vote for each image if it looks fine?
I have no interests in such a matter like Games and the like, but I like your dialect, man. So, yeah, I'm enjoying watching your videos, keep going.
As a viewer I'm down to seeing uncomplete stuff, if that means more videos.
As a developer I'm enjoying to see possible struggles when you do this kind of development.
Either way, keep it up man :D
I'm not sure if you plan to run automated test on any PR that is opened on your repository but if that's the case, be really thoughtful about the fact that then anyone with bad intentions could execute some code that, I guess, run on your local network. Someone could replace the main function with one that installs a backdoor, let it run then close the PR. Running untrusted is not an easy subject (spinning up a VM in your case, for performance testing isn't the best choice but that might be sufficient with hyperv)
You should always disable the workflows from forked repo PRs.
i don't know if Hazel supports DAP right now, but if you implement it and some kind of headless mode, you can probably automate a lot of things just using a script that hooks up to it
I think the older you will improve on the V language, or on JAI or Odin.
Good Stuff. The Discovery of workflow automation was a huge step for men, in the internet history and in general software development.
Just wondering, why not have a git(lab) CI/CD pipeline to do most of that stuff for you?
Why not use TeamCity? You can configure that testing PC to be the "TeamCity Agent". Create a project, add a VCS root that points to develop branch, then create a build configuration that pulls on every commit & run custom tests.
l'd use bash for the git script and something like cron to run periodically and post the report. As you might have many commit or pr at one.
For the game engine crashes you need to have some way to log it at least. For performance you could compare the old numbers of the report.
The editor maybe an automatized script that goes and try general functionally. but it's a hard kind test to do auto.
An idea would be to do a hook (or action) to plug it to ansible instance that would have a range of ips in your network and then with the module provided by ansible it does the cloning the builld and the launching on each and every machine you have.
I don't know if this the right solution for this case or maybe there are issues i don't see.
Basically a whole infrastructures
Tim was a funny dude
1:58 buff cherno 😂
more devlogs
I'd love just a weekly "Hey this is what I've been up to".
You should not have to use any person to do checks if a screenshot from a Hazel scene is correct or not. You should test if the scene is rendered correctly by comparing it to a previous screenshot (one you already know is correct) automatically.
Waiting for Hazel official release
Enjoy the journey
journal the enjoyment
Joy over joy
I was screaming at the screen for half the video asking "why doesn't he just poll it", glad you landed on it
I feel your pain though, getting tunnel visioned into something overly complex happens
I'm only a hobbist game dev, but professional software engineer for 20+ years, your final approach is pretty much how I'd do it too
Are you looking for QA people to replace Tim? Ones that could build out the CI, testing and stuff for you? :)
I don't see how you need nginx for that hook thingy. You can just make a rest server in any language you want, no certificate required.
But the polling should suffice too.
Really depends on the amount of control you want, it seems.
if you want more exciting videos, maybe you or someone else could make some demo game's or scenes to show off Hazel
Good video. I don't think we need to see the code when you are explaining your thoughts.
So indeed Hazel did make Tim a better person... by leaving
toxic but ok
Testing? If compiles, ship it!!!! 🤣
can you use ia to make a game engine
What about Github action or webhook to send an email, then a AHK script to detect the email and parse it to configure and trigger the builds? Unrelated, are you looking for an intern? 👀
Wait, so has Hazel dropped yet?
ImGui has test engine and you can run it headless I think.
Noo, Cherno has became crazy (I dont mean it) :U, he is naming his computer
Not a devops engineer or anything, but wouldnt telebit or ngrok solve your issue to receive events from github?
hi
19:40 can you prove that's the real world and not just a high fidelity scene rendered in hazel? don't know if I believe you
PyAutoGui for editor actions. CI is definitively possible. Setup something like Jenkins on your extra machine and let it run on the background with webhook like you mentionned. You can define many agents that will each run on their own machine (and so with their own graphic cards). You can run script through them. Also maybe having an actual QA dev engineer (that deals with automation, scripts, etc. and not manual testing) would be really beneficial for your project. Good luck!
Please God no, no Jenkins
@@LetsDark Well he can use something else if he wants. At least Jenkins is free and all. What do you use?
@@davidb4020 GitLab runners or whatever integrates the best with my git environment. One time I even had to use azure dev ops and there you can setup test suits as well. I currently still have one Jenkins instance and it is driving me insane. So many small bugs, slow as hell… But I can not get rid of it just now.
bugs cant avoid making bugs, multiply style, the github rep can do the compiling in any case, split the burden, you dont have to run tests on the github stuff, or you could, also. k.i.s.s. as always :) take a breath, if something seems breath taking, let God give it to you, perfectly, so you can enjoy whatever you are doing. you could make all the sub-parts so that you can test them individually, ie, keep the gpu code away from the math code, or in other words, make the math testable separately from the gpu instructions.
interesting, document something not exist at all, had me for this week
Mann, Do you accept testers by chance? I would love to help test, as i can't afford paying to even help report bugs etc..
Automation is hard, just concentrate on capacity and keep the time you need for maintenance of the system as low as possible.
Апхахпахп, Тим. Видимо выгорел)
ImGui test engine is a thing.
Could I possibly get a job over at hazel?
nice camera too bad you didn't show a single step of the process
Edit: If you really have some skills Contribute in Godot or similar project. We will thankful to you in future !
what is hazel Engine ? Answer: I don't know but i think its something in the skies that we will see in our dreams. I'm following you since 2020 maybe but I don't actually Understand you have built ?
At least one person here cares about lenses and cameras! (Me)
"Hey Tim, can you do EVERYTHING?"
... "Tim?"
...
...
...
"...Tim?"
great video. but srsly show us some gameplay
test 1060 please
nooo, Tim come on! sales??? =/
Does australians like americans walk with shoes inside?
Sounds like you could use some work on making a good place to work, maybe? 😂😂 Also I doubt you’re going to be able to automate testing your game engine easily after it’s been built!!
Sir, Github Pages. That's all I'm going to say.
his computer is female (she is) and my PC is male (He is).
see if ngrok could help you with your situation.