Repository navigation
Cap the libxc build parallelism at 4 - #337
Conversation
The libxc 7.0.0 build for x86_64/amd/zen4 in EESSI/software-layer#1772 was killed twice with the same OOM in cc1 at half the cores. A divider of 3 gives 5 jobs on a 16-core build node instead of 8.
|
Testing this in EESSI/software-layer#1772 (comment) |
|
Still went OOM - or so it seems. We can try to reduce parallelism even further. But by now I'm not convinced that it isn't the memory consumption of one of the threads that is a bottleneck... The good news I guess is that the build time wasn't all that different. Maybe we should just really crank it, go for a reduction of 8 (i.e. 2 cores), see if that works. It'll take the least amount of human time to just try this. |
HaoZeke
changed the title
casparvl
changed the title
Eleven libxc 7.0.0 translation units peaked at 2 GiB or more with GCC 15.2.0 at -O3. The hook compiles those files at -O1 for libxc 7.0.0 and newer when the easyblock is CMakeMake. Those builds use the normal job count. Older libxc and ConfigureMake libxc keep the 2-core cap. The A64FX limit of 6 cores is unchanged.
libxc 7.1.2 cc1 peaked at 2.153 GiB on lda_xc_ksdt.c with -O3. The same translation unit peaked at 2.530 GiB with -O1, so -O1 is the wrong flag. The eleven-file list is unchanged.
|
The -O1 hook is about memory at high -j, not correctness. On 8 cores, make -j 8 of libxc 7.0.0 peaked at 13.259 GiB and oom_kill stayed 0. |
|
Honestly, I wouldn't change the optimizations just because of a memory issue at compile time - this is a one time issue, we can wait longer (even if it is annoying). If that in the end gets us a better optimized binary, that's worth it. Also: a reduction in parallelism is a standard procedure, and thus easy to maintain - this solution would require more maintenance. The annoying part is figuring out the exact limit for Anyway, with 2 it works. I'd like to double check if it also works at 4, since build times have now increased quite substantially (even though we saw little difference between -j 8 and -j 5). Can you revert the -O1 change and set parallelism to max 4 cores? I'll give that a try in EESSI/software-layer#1906 then, see if that still works. If it does, I'll build all the other targets as well. |
Completely agree with Caspar here, and, ultimately, we could even spin up some machines with more memory like we've done for the hipBLAS builds (though that's not possible for e.g. A64FX, RISC-V, Grace). |
casparvl
changed the title
bedroge
left a comment
bedroge
left a comment
There was a problem hiding this comment.
The build in EESSI/software-layer#1906 (comment) succeeded, so lgtm
|
bot: build repo:eessi.io-2023.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx |
|
New job on instance
|
|
New job on instance
|
|
New job on instance
|
|
Staging PR merged |
Companion to EESSI/software-layer#1772.
The libxc 7.0.0 build for x86_64/amd/zen4 there was killed twice by the out-of-memory killer in cc1.
This caps the libxc build at 8 parallel jobs on every target, instead of dividing the core count by 3, so a large build node no longer starts more cc1 processes than its memory can hold.
The A64FX limit stays at 6.
AI disclosure: an AI coding assistant helped write this change.
I reviewed it and checked the builds and tests myself.