C++ Weekly - Ep 413 - (2x Faster!) What are Unity Builds (And How They Help)
Вставка
- Опубліковано 28 січ 2024
- ☟☟ Awesome T-Shirts! Sponsors! Books! ☟☟
Upcoming Workshop: Understanding Object Lifetime, C++ On Sea, July 2, 2024
► cpponsea.uk/2024/sessions/und...
Upcoming Workshop: C++ Best Practices, NDC TechTown, Sept 9-10, 2024
► ndctechtown.com/workshops/c-b...
This episode is sponsored by think-cell. think-cell.com/cppweekly
Episode Notes: github.com/lefticus/cpp_weekl...
T-SHIRTS AVAILABLE!
► The best C++ T-Shirts anywhere! my-store-d16a2f.creator-sprin...
WANT MORE JASON?
► My Training Classes: emptycrate.com/training.html
► Follow me on twitter: / lefticus
SUPPORT THE CHANNEL
► Patreon: / lefticus
► Github Sponsors: github.com/sponsors/lefticus
► Paypal Donation: www.paypal.com/donate/?hosted...
GET INVOLVED
► Video Idea List: github.com/lefticus/cpp_weekl...
JASON'S BOOKS
► C++23 Best Practices
Leanpub Ebook: leanpub.com/cpp23_best_practi...
► C++ Best Practices
Amazon Paperback: amzn.to/3wpAU3Z
Leanpub Ebook: leanpub.com/cppbestpractices
JASON'S PUZZLE BOOKS
► Object Lifetime Puzzlers Book 1
Amazon Paperback: amzn.to/3g6Ervj
Leanpub Ebook: leanpub.com/objectlifetimepuz...
► Object Lifetime Puzzlers Book 2
Amazon Paperback: amzn.to/3whdUDU
Leanpub Ebook: leanpub.com/objectlifetimepuz...
► Object Lifetime Puzzlers Book 3
Leanpub Ebook: leanpub.com/objectlifetimepuz...
► Copy and Reference Puzzlers Book 1
Amazon Paperback: amzn.to/3g7ZVb9
Leanpub Ebook: leanpub.com/copyandreferencep...
► Copy and Reference Puzzlers Book 2
Amazon Paperback: amzn.to/3X1LOIx
Leanpub Ebook: leanpub.com/copyandreferencep...
► Copy and Reference Puzzlers Book 3
Leanpub Ebook: leanpub.com/copyandreferencep...
► OpCode Puzzlers Book 1
Amazon Paperback: amzn.to/3KCNJg6
Leanpub Ebook: leanpub.com/opcodepuzzlers_book1
RECOMMENDED BOOKS
► Bjarne Stroustrup's A Tour of C++ (now with C++20/23!): amzn.to/3X4Wypr
AWESOME PROJECTS
► The C++ Starter Project - Gets you started with Best Practices Quickly - github.com/cpp-best-practices...
► C++ Best Practices Forkable Coding Standards - github.com/cpp-best-practices...
O'Reilly VIDEOS
► Inheritance and Polymorphism in C++ - www.oreilly.com/library/view/...
► Learning C++ Best Practices - www.oreilly.com/library/view/... - Наука та технологія
I worked on a project that we got from a different studio, it was setup to use unity builds, it drastically reduced build times, without unity builds it would take 45+ minutes to do a complete rebuild, but with a unity build only took 15-20 minutes to compile. HOWEVER do not take this as an endorsement of unity builds, compile times for comparable projects in house that didn't use unity builds were only taking 7-10 minutes, so we decided to clean up all the includes on the project instead (they were horrendous) and using forward declarations where possible we got the non-unity build down to about 6 minutes doing that while the unity build was actually slower and took about 7-8 minutes (we were using Xeon CPUs with MANY cores but mediocre single thread performance which I'm guessing isn't as suitable for unity builds).
So yeah, unity builds can certainly help, but in my experience you are better off just writing your code better so your includes are kept to a minimum.
I'll pin this comment, it's a great story from someone with the experience on the matter.
Yup, we use it in the games industry all the time. Pain in the butt because it means your static function names have to be much longer to avoid name conflicts.
Yep we use unity builds, and sometimes it feels like we're not writing C++, but are instead writing in a semi conformant C++ subset
Unity builds do decrease compile time a lot. However, be wary of the following new pitfalls:
- Static functions or symbols in unnamed namespaces might now clash if they have the same name.
- A using namespace in a cpp might break something in another cpp (almost better not to use them at this point).
- A macro defined in one cpp (e.g. NOMINMAX, etc) might affect other cpps (you might want to undef your defines at the end of files).
- pragmas in one file might affect other files (e.g. disabling warnings), use push/pop.
In projects I've worked on, we call these "unified" builds -- when arriving here, I first thought you were going to talk about the Unity game engine. :-)
In my experience, the main downside is that over time, `#include` lines go wrong:
When writing new code that relies on some header, you forget to add the corresponding `#include`, but the project still compiles because another file earlier in the same group already includes it.
Later, the real problem arises, when you start adding/removing/renaming files, the unity groupings shift, and now the new code above may be higher in its group, the build will fail because of the now-missing `#include`, and it can be a pain to understand what's going on -- a long time may have passed, and/or it was someone else's code and you have no familiarity with it.
But overall, I think the benefits (faster builds for everyone, sometimes size & perf improvements) still outweigh this issue (occasional "undeclared identifier" failures affecting one person at a time).
for this reason I have two types of builds - one with unity set to on and one without - IMO best option is to work with unity build daily but check non-unity build before submit to CVS after being done with code changes
Worth noting that unity builds as configured by default (in so called batch mode) are not guaranteed to be reproducible, especially if you have conditionally added source files depending on platform and internal project toolset. The generated unity source files that include project's source files don't have to be identical between different machines so it is entirely possible that the project will compile fine on your machine but your coworker's configuration will consistently fail. To ensure reproducibility, especially with CI pipeline setup, it might be necessary to set UNITY_BUILD_MODE to GROUP and manually define unity groups for source files. This can help speed up compilation even further because you can group up related source files together. You may also need to use this if you have platform-dependent code that must not be arbitrarily embedded into any unity file, such as when you have a source file that includes Windows.h; you don't want to pollute other sources with macros.
This actually sounds like a feature to me. If you are dependent on the order of unification, you should fix the .cpp files instead!
Identical names with internal linkage in multiple compilation units, will be a problem for unity build
Thus it is important to build both unity and non-unity in CI to validate the code
thanks for this. had never come across unity builds before today. Still counting on modules to be the go to tech for build time improvement though😅
Unreal 5 uses unity builds by default all the time. It sometimes causes issues but most of the time it's actually very reliable. The smaller binary size is interesting I wonder if this could help with performance in certain cases
UE also has very nice feature that when working with unity builds you can disable them for currently modified files by using global option - no need to go to any modules explicitly. So most of project uses unity builds and only part I'm working on not - very useful.
The raw file size (eg. ls -l) of a binary does not necessarily reflect the size of the executable code. The raw file size will include things like debug info, relocation info, etc. Instead, you might want to take a look a the 'size' command, which gives the size of text, data and bss segments, as well as the total space used by the executable. For example, if I compile a simple "Hello world" program on my system, a debug version has file size of 20K, but a non-debug version comes in at 8.5K. But size shows that both use 2.6k for text, data and bss segments. Further using "strip" to remove symbols from both files, both files are reduced to the same 6.2K size, which would be expected, since the only difference between the two builds is the -g flag for debug.
Using unity build forever here, nearly always work in most cases,
quite rare it is counterproductive
Wow, this is a brilliant idea! Do you think it can also have a potential improvement with C++ modules? Or are they compiling them already in a smart-enough way that this wouldn't affect much?
This could cause compile errors when using anonymous namespaces too, couldn't it?
Thank you for video.
We've been doing something similar in Dlang. This is really great to see this come to C++.
does this not defeat part of the purpose of unnamed namespaces? and can't it in unlucky cases make build times worse? Ex. you don't use LTO and you change one source file, and now every source file in the unity chunk needs to be recompiled as part of the chunk recompilation. those two concerns are why I haven't experimented with it. (add to that that my projects are small enough that having a "normal" build system is good enough, and I already opt to use LTO to not leave that optimization "to chance").
I'd have to see an example to see if it defaulted an anonymous namespace use case.
LTO is a great option as well, and I'm certainly not saying that everyone should use unity builds. But I do think it's a Best Practice to test your code against Unity and LTO and see how well it goes!
Those are great, but CI is red due to missing headers 😢
Poor man's LTO
How to get this much knowledge on a programming language?. I am doing competitive programming from a year but I don't have deep knowledge of cpp. Any Resources or suggestions please?
I spend lots and lots of time experimenting and learning.
i would assume those merged files do not get compiled in a muiltithreaded fashion. well, that depends on the compiler, i suppose. and it will negate any benefits of ccache.
i wonder in what case would we benefit from this exactly. i would assume big gains in lto ?
I don't think it would negate the benefits of ccache - as that works on hashes of the preprocessed input files. That should still work in this case.
Doesn't this mean that #define and #include are leaking between the cpp files, which means that you can, or will, end up in situations where the cpp files no longer compiles on their own?
Oh yes, we converted a project using unity builds to not use them and the number of errors from that exact situation was staggering. Even if you keep using unity builds it can still be an issue if you rename a file and the order they are included in gets altered, before the conversion we renamed one file and got nearly 8000 new errors to fix because of that. We also had issues with scope getting messed up where you can call different functions of the same name from different scopes accidentally if you use the using keyword to avoid having to type out the fully qualified function, usually it is caught as an ambiguous call, but if the functions have slightly different definitions and you are expecting some implicit casts it is surprisingly easy for it to go unnoticed when the parameters exactly match a completely different function.
that's why you need to compile your project with and without unity builds.
Costs 15 cents per build 😆
The primary reason for unity builds is runtime performance, but a linker (mold) is not the type of architecture that is likely to benefit as it is likely to be a naturally modular dataflow. The type of architecture that will benefit from a unity build is code that might be modularized for human readability (and by extension maintenance) purposes, but from a dataflow perspective is a more monolithic architecture. That is where the global optimization will likely be a win as interfaces can be elided. Also, a good LTO implementation can do nearly as well as a chunk=N (where N is all the TUs) unity build so if LTO is on across all the tests, there will likely be little difference and if N < number of TUs, LTO will likely be better. The benefit of unity builds is that (for portable code) you might be building with a wide variety of toolchains, including some that don't have a good LTO implementation so in that case unity may provide better optimization coverage across all targets.