I am missing something with distcc

I am trying to get distcc set up to compile a kernel with Multiple Odroid N2’s

I am starting with two Odroid N2’s. Both can compile the kernel as stand alones.

I have set up static IP addresses for the Client distcc as 192.168.50.141 and the Volunteer or Server distcc as 192.168.50.142 and I am starting with the plain method as opposed to the pump method.

I currently have firewalld disabled on both computers so it is not a firewall problem.

On the client distcc computer, I get multiple outputs of the following:

distcc[3717] (dcc_readx) ERROR: unexpected eof on fd5
distcc[3717] (dcc_r_token_int) ERROR: read failed while waiting for token "DONE"
distcc[3717] (dcc_r_result_header) ERROR: server provided no answer.
  Is the server configured to allow access from your IP address? Is the server performing
 authentication and your client isn't? Does the server have the compiler installed?
 Is the server configured to access the compiler?

On the volunteer or Server computer, I get the following in /tmp/distcc.log

distccd[508] (dcc_check_compiler_whitelist) CRITICAL! aarch64-unknown-linux-gnu-gcc not in /usr/lib/distcc or /usr/lib/distcc whitelist.
distccd[508] (dcc_job_summary) client: 192.168.0.141:49810 OTHER exit:0 sig:0 core:0 ret:0 time:39ms 
distccd[507] (dcc_check_compiler_whitelist) CRITICAL! aarch64-unknown-linux-gnu-gcc not in /usr/lib/distcc or /usr/lib/distcc whitelist.
distccd[507] (dcc_job_summary) client: 192.168.0.141:49796 OTHER exit:0 sig:0 core:0 ret:0 time:46ms 

I read the Archlinux wiki manual and wiki entry and come up with this cheat sheet.
I followed this cheat sheet to set up the computers, but I am obviously missing something.

DISTCC instructions

INSTALLATION                                                  client  vol1  vol2
1 Set static IP Adress for all involved devices        192.168.0.141  .142  .143
2 Install distcc package on all involved devices                ___   ___   ___
3 Set firewalld to allow distcc trafic on port 3632/tcp         ___   ___   ___

VOLUNTEERS
1 To config file /etc/conf.d/distccd add:
  DISTCC_ARGS="--allow-private --log-file /tmp/distcc.log"            ___   ___
2 systemctl enable distccd - on every volunteer                       ___   ___
3 edit /etc/ssh/sshd_config
   change "#Port 22" to "Port XXXX" [XXXX = desired ssh port number]
   change "#PermitRootLogin prohibit-password" to "PermitRootLogin no"
   change "#PasswordAuthentication yes" to "PasswordAuthentication yes"
   change "#PermitEmptyPasswords no" to "PermitEmptyPasswords no"         ___   ___
4 systemctl enable sshd.service                                       ___   ___
5 configure firewalld
    firewall-cmd --reload
    firewall-cmd --permanent --zone=public --service=ssh --remove-port=22/tcp
    firewall-cmd --permanent --zone=public --service=ssh --add-port=XXXX/tcp
    firewall-cmd --permanent --zone=public --remove-service=dhcpv6-client
    firewall-cmd --permanent --zone=public --add-source=192.168.0.0/24
    firewall-cmd --permanent --zone=public --remove-forward
    firewall-cmd --reload                                             ___   ___
6 reboot                                                              ___   ___

CLIENT - For use with makepkg

Edit /etc/makepkg.conf in the following sections
1. The BUILDENV array will need to have distcc unbanged
   i.e. list it without exclamation point. See example          ___
2  Uncomment the DISTCC_HOSTS line and add the host name or
   IP addresses of the volunteers. Optionally, follow this
   with a forward slash and the max number of threads they
   are to use. The subsequent nodes should be separated by a
   white space. This list should be ordered from most
   powerful to least powerful (processing power). See example   ___
3. Adjust the MAKEFLAGS variable to correspond roughly twice
   the number max threads per server.
   In the example below, this is 2x(5+5+5)=30.                  ___
4  Change PKGDEST
   PKGDEST=/home/$USER/Packages  [or whereever you want]        ___
5  Change PKGEXT
   PKGEXT='.pkg.tar.zst'                                        ___

EXAMPLE /etc/makepkg.conf for three Odroid N2 devices

Plain mode example:
BUILDENV=(distcc fakeroot color !ccache check !sign)
MAKEFLAGS="-j30"
DISTCC_HOSTS="localhost/5 192.168.0.142/5 192.168.0.143/5"

Pump mode example:
BUILDENV=(distcc fakeroot color !ccache check !sign)
MAKEFLAGS="-j45"
DISTCC_HOSTS="localhost/5 192.168.0.142,cpp,lzo 192.168.0.143,ccp,lzo"

Pudge

1 Like

Hi @Pudge, how about permissions and ownership (I don’t know what they should be but it’s worth checking :slight_smile: ) of the files /etc/conf.d/distccd and /etc/ssh/sshd_config

can you show their content as well?

I hope you’re doing good!

Hi Fernando. I am not too bad, for an old guy. :man_white_haired:
I hope you and your family are all doing well.

I do not know why I overlooked the bane of mankind. Permissions and ownership !
Permissions and ownership are on many levels of life besides Linux.

We are off to get Breakfast and we have some errands to run. Then I will wrestle with distcc some more. Thanks for the suggestion and I’m glad to see you around the forum.

Pudge

Yeah, we’re all good, thanks! :+1:

I hope is this so you can solve it.

Have a good time.

I have a solution for you.
Shameless self-plug incoming:

If icecream doesn’t work for you, I can help you with distcc. I had it working before switching to icecream.
Icecream requires one time setup and you can add more nodes dynamically.
It is used an developed by people at Suse

3 Likes

It’s too late to start on something like that tonight, and tomorrow I need to issue new images.
As soon as I finish that, I am definitely going to try icecream. I have two Ryzen computers sitting idle that both have 8 cpus & 16 threads plus a ton of RAM.

Thank you.

Pudge

EDIT:

MAKEFLAGS="-jN"

Would N be the number of cores or the number of threads?

Since it’s the number of jobs in parallel, I guess it’s the number of threads.

2 Likes

yes, its the number of threads

1 Like