Doubt on MAKEFLAGS in makepkg

For parallel compilation in makepkg, I came across these links of using MAKEFLAGS https://arcolinux.com/how-to-speed-up-your-computer-during-building/ and https://unix.stackexchange.com/questions/268221/use-multi-threaded-make-by-default
and I was wondering why is there a recommendation of +1 to number of cores for MAKEFLAGS in those articles, whereas the archwiki just has nproc which is cores as it is.
How could the +1 help?

For many modern systems with multiple cores the compilation time for a ceras can be reduced by performing a "parallel make" by either setting an environment variable named MAKEFLAGS equal to the number of logical cores (or threads) on your host system or by telling the make program how many logical cores are available.

Since glaucus is automated, the environment variable MAKEFLAGS is set in /home/glaucus/scripts/flags equal to -j1 which is the sane default.

To enable parallel building, you need to first know the number of threads on your host system. Simply run nproc and it’ll return that number (let’s say that nproc returned a value of 8). Now you can set MAKEFLAGS to one of the following values:

  1. MAKEFLAGS=-j$(nproc) which in our case is MAKEFLAGS=-j8:
    This sets the number of MAKEFLAGS equal to the number of the current threads on your host system. Some say that this will enable parallel building while giving some room for other applications to run as there will be an extra overhead thread that ensures your system doesn’t lag or freeze.

  2. MAKEFLAGS=-j$(nproc + 1) which in our case is MAKEFLAGS=-j9:
    This sets the number of MAKEFLAGS equal to the number of the current threads on your host system plus one. This is similar to the first option, but it ensures that the overhead thread is also used in the build process. This might build faster than option one, but will cause the system to lag a lot especially if you’re using other applications such as a web browser.

  3. MAKEFLAGS=-j$(nproc * 1.5) which in our case is MAKEFLAGS=-j12:
    Some say that in order for parallel building to work properly you need to set to set your MAKEFLAGS equal to the number of current threads times 1.5.

(c) https://wiki.glaucuslinux.org/constructing-the-toolchain-and-the-chroot-environment/final-preparations/about-ccus-and-ceus/

I assume on Arch it means same :upside_down_face:

1 Like

Thanks, still not sure what “overhead thread” is here

Looks like it’s literally “little more threads than CPU can chew”

Here’s some reading as well, that topic seems to be opinionated :slight_smile:
https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/

Somehow i feel @jonathon would have more knowledge possibly :upside_down_face:

1 Like

Essentially, it’s relatively rare to have all threads running at 100%, so adding an extra one (or more) can speed up the process overall (assuming you have proper multiprocessing and not just old crappy hyper-threading).

For example, here’s a snapshot of compiling a kernel:

  PID USER        PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  
15477 jonathon     7      137320 115500  24340 R  59.0  0.7   0:01.81 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15618 jonathon     7      134480 111584  24404 R  42.7  0.7   0:01.31 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15632 jonathon     7      130008 107512  24112 R  42.7  0.7   0:01.31 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15649 jonathon     7      111300  92460  24360 R  39.1  0.6   0:01.20 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15681 jonathon     1      123812  95224  18616 R  29.3  0.6   0:00.90 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15704 jonathon     7      100432  73144  17184 R  22.8  0.4   0:00.70 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15710 jonathon     1       98320  70516  17168 R  21.8  0.4   0:00.67 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15782 jonathon     7       92692  65700  17184 R  19.2  0.4   0:00.59 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15794 jonathon     7       90880  63872  19088 R  17.9  0.4   0:00.55 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15778 jonathon     1       90476  64460  19504 R  17.6  0.4   0:00.54 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15834 jonathon     7       77136  50248  17184 R  12.1  0.3   0:00.37 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15846 jonathon     1       74936  47404  17184 R  10.4  0.3   0:00.32 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15856 jonathon     7       70092  43584  17176 R   8.5  0.3   0:00.26 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15874 jonathon     1       59520  33876  17268 R   5.2  0.2   0:00.16 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15884 jonathon     5       56992  31268  17260 R   3.9  0.2   0:00.12 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+ 
15894 jonathon     1       54828  29196  17204 R   3.3  0.2   0:00.10 /usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/cc1 -quiet -nostdinc -I ./arch/x86/include -I ./arch/x86/include/ge+
2 Likes