For parallel compilation in makepkg, I came across these links of using MAKEFLAGS https://arcolinux.com/how-to-speed-up-your-computer-during-building/ and https://unix.stackexchange.com/questions/268221/use-multi-threaded-make-by-default
and I was wondering why is there a recommendation of +1 to number of cores for MAKEFLAGS in those articles, whereas the archwiki just has nproc which is cores as it is.
How could the +1 help?
For many modern systems with multiple cores the compilation time for a ceras can be reduced by performing a “parallel
make
” by either setting an environment variable namedMAKEFLAGS
equal to the number of logical cores (or threads) on your host system or by telling themake
program how many logical cores are available.Since glaucus is automated, the environment variable
MAKEFLAGS
is set in/home/glaucus/scripts/flags
equal to-j1
which is the sane default.To enable parallel building, you need to first know the number of threads on your host system. Simply run nproc and it’ll return that number (let’s say that
nproc
returned a value of8
). Now you can setMAKEFLAGS
to one of the following values:
MAKEFLAGS=-j$(nproc)
which in our case isMAKEFLAGS=-j8
:
This sets the number ofMAKEFLAGS
equal to the number of the current threads on your host system. Some say that this will enable parallel building while giving some room for other applications to run as there will be an extra overhead thread that ensures your system doesn’t lag or freeze.
MAKEFLAGS=-j$(nproc + 1)
which in our case isMAKEFLAGS=-j9
:
This sets the number ofMAKEFLAGS
equal to the number of the current threads on your host system plus one. This is similar to the first option, but it ensures that the overhead thread is also used in the build process. This might build faster than option one, but will cause the system to lag a lot especially if you’re using other applications such as a web browser.
MAKEFLAGS=-j$(nproc * 1.5)
which in our case isMAKEFLAGS=-j12
:
Some say that in order for parallel building to work properly you need to set to set yourMAKEFLAGS
equal to the number of current threads times1.5
.
I assume on Arch it means same
Thanks, still not sure what “overhead thread” is here
Looks like it’s literally “little more threads than CPU can chew”
Here’s some reading as well, that topic seems to be opinionated
https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/
Somehow i feel @jonathon would have more knowledge possibly
Essentially, it’s relatively rare to have all threads running at 100%, so adding an extra one (or more) can speed up the process overall (assuming you have proper multiprocessing and not just old crappy hyper-threading).
For example, here’s a snapshot of compiling a kernel:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15477 jonathon 7 137320 115500 24340 R 59.0 0.7 0:01.81 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15618 jonathon 7 134480 111584 24404 R 42.7 0.7 0:01.31 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15632 jonathon 7 130008 107512 24112 R 42.7 0.7 0:01.31 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15649 jonathon 7 111300 92460 24360 R 39.1 0.6 0:01.20 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15681 jonathon 1 123812 95224 18616 R 29.3 0.6 0:00.90 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15704 jonathon 7 100432 73144 17184 R 22.8 0.4 0:00.70 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15710 jonathon 1 98320 70516 17168 R 21.8 0.4 0:00.67 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15782 jonathon 7 92692 65700 17184 R 19.2 0.4 0:00.59 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15794 jonathon 7 90880 63872 19088 R 17.9 0.4 0:00.55 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15778 jonathon 1 90476 64460 19504 R 17.6 0.4 0:00.54 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15834 jonathon 7 77136 50248 17184 R 12.1 0.3 0:00.37 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15846 jonathon 1 74936 47404 17184 R 10.4 0.3 0:00.32 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15856 jonathon 7 70092 43584 17176 R 8.5 0.3 0:00.26 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15874 jonathon 1 59520 33876 17268 R 5.2 0.2 0:00.16 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15884 jonathon 5 56992 31268 17260 R 3.9 0.2 0:00.12 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
15894 jonathon 1 54828 29196 17204 R 3.3 0.2 0:00.10 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.