Openssh question

Hi to all,
since yesterday I am no longer able to connect to several Arch based machines using my raspberry pi “Raspbian Jessie” based.
I am using key authentication, and all keys are rsa, 4096bit, sha256 generated. Until OpenSSH_8.8p1 the issue popped up. I do have many things between all my servers which are using ssh and rsa keys. Between Arch machines which are using the new OpenSSH version everything runs smoothely as before - using the same rsa keys on all machines as before. But I cannot connect to the raspberry using key authentication - I am dropped to password authentication (which is working). If I try to connect from my Raspberry to one of my Arch servers I get an “no hostkey alg” - error. It looks like host key (which resides in /etc/ssh/) is no longer usable. I already have disables rsa host key but the error persists. There is no possibility to upgrade openssh on the Raspberry to a newer version (it is OpenSSH_6.7p1) so I must find a solution which uses this version and does not have security issues.
Is it a problem with the sshd host keys? Or the keys for the users in ~/.ssh/ ? I have got working the direction Arch → Raspberry by adding “PubkeyAcceptedKeyTypes +ssh-rsa” but this should only be temporarily. The other direction (Raspberry → Arch I could not get working.

Does your “raspberry pi “Raspbian Jessie” based.” have a static IP address or DHCP?

With DHCP, most routers can and do change the IP address when the lease expires.

In your raspberry pi open a tty window, do:
$ ip addr

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp37s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 70:85:c2:8a:53:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.108/24 brd 192.168.0.255 scope global dynamic noprefixroute enp37s0
       valid_lft 5616sec preferred_lft 5616sec
    inet6 fe80::987c:e3fb:5da5:1570/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

under 2: you will see the device name, in this case enp37s0
go to the inet line

 inet 192.168.0.108/24 brd 192.168.0.255 scope global dynamic noprefixroute enp37s0

it lists the current active ethernet IP address and if it says “dynamic” in this line, then the IP address was assigned by DHCP which is subject to change. For your RPi server, you want a static IP address.

There are multiple things that could cause this problem, but this is the easiest to check and quite often is the problem.

Pudge

EDIT:
Just as a FYI, the latest and supposedly greatest key format is ed25519 as opposed to rsa, and others.

In your RPi server, open tty window and cd to /etc/ssh

 ls -l
total 600
-rw-r--r-- 1 root root 570321 Apr 20 11:01 moduli
-rw-r--r-- 1 root root   1531 Apr 20 11:01 ssh_config
-rw-r--r-- 1 root root   3009 Jun 11 20:18 sshd_config
-rw------- 1 root root   1373 Feb  3  2021 ssh_host_dsa_key
-rw-r--r-- 1 root root    600 Feb  3  2021 ssh_host_dsa_key.pub
-rw------- 1 root root    505 Feb  3  2021 ssh_host_ecdsa_key
-rw-r--r-- 1 root root    172 Feb  3  2021 ssh_host_ecdsa_key.pub
-rw------- 1 root root    399 Feb  3  2021 ssh_host_ed25519_key
-rw-r--r-- 1 root root     92 Feb  3  2021 ssh_host_ed25519_key.pub
-rw------- 1 root root   2590 Feb  3  2021 ssh_host_rsa_key
-rw-r--r-- 1 root root    564 Feb  3  2021 ssh_host_rsa_key.pub

This should be a list of key formats that the server recognizes. In this case
dsa
ecdsa
ed25519
rsa

It is an OpenSSH-Problem, but I do not fully understand. Pse look here: https://levelup.gitconnected.com/demystifying-ssh-rsa-in-openssh-deprecation-notice-22feb1b52acd

The latest OpenSSH-Version that is available for Raspian Jessie is OpenSSH_6.7p1.

Best regards

Yes. BUT: I am using RSA keys on all my machines. ssh conections between Arch machines are running fine, ssh connections from an to the older OpenSSH on the Raspberry do not. I do not think it is an issue with RSA keys in complete, because everything is working as before on all Arch machines conecting each other. Only the Raspberry OpenSSH is not working. So how to fix without switching to a different key system and swapping tons of keys (that are working on all Arch machines)? Building a recent OpenSSH out of the sources?

Did you check the RPi server for a static IP address?

In your client machines, the /etc/hosts file lists the IP address / hostname relationships such as

192.168.0.XXX   serverhostname.localdomain    serverhostname

When you ssh into your server,

ssh username@serverhostname

/etc/hosts makes the correlation between serverhostname and an IP address. If the IP address of the RPi server has changed it won’t work.

Please get into your RPi server and check the IP address using $ ip addr
When you determine the RPi’s IP address, try to ssh into the server as
ssh username@192.168.0.xxx -p whatever-port-number-you-use
if you still use the default port 22 you can skip the -p port-number part

Pudge

Please, stop ip related hints. It has definitely nothing to do with ip addresses.

OKAY, since you refuse to check the IP address to eliminate one possibility, I then went and did your work for you and did a quick duckduckgo search and found this.

SSH-RSA key rejected with message “no mutual signature algorithm”

There is a work around listed, but if rsa is deprecated the work around will be temporary before it raises it’s ugly head again.

My suggestion, if you care to take it, is to switch everything from rsa to ed25519 as mentioned in post # 2.

Good Luck.

Pudge

Thank you. I have found this page, too. But as I understand not rsa itself is deprecated. It is only the sha algorithm. Only sha1 is deprecated. Ifyou use the more recent sha256 or sha512 everything is ok. Is it possible to use these more recent sha algs with the OpenSSH_6.7p1? I have not found any answer in the web…

Could you post the output of ssh -vvv when trying to connect to the server? This should indicate exactly what is at fault (and I’d be surprised if those keys are no longer valid).

Here is a snippet from this web page

### **Modern** (OpenSSH 6.7+)

File: `/etc/ssh/sshd_config`

# Supported HostKey algorithms by order of preference.
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

This lists the supported HostKey algorithms by order of preference.

Since your RPi has openssh 6.7p1 and it supports ed25519 and is the 6.7p1 preference
And since your other computers have openssh 8.8p1 which has obviously gone with this philosophy

and 8.8p1 still supports ed25519 the obvious solution would be to switch everything in your network to ed25519. IMHO ed25519 is the best future proof solution.

Pudge

The 8.8 defaults include RSA, I don’t think this is the cause of the issue:

#       $OpenBSD: sshd_config,v 1.104 2021/07/02 05:11:21 dtucker Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/sbin:/usr/local/bin:/usr/bin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
....
1 Like

Do not misunderstand: There are keys which are produced in different ways (RSA, DSA, ECDSA…). But during handshake it seems there are not transferred the keys but a hash of the keys. The hash is produced with an algorithm, and this algorithm (SHA1, SHA256, SHA512) likely seems to be the problem. I do not think RSA is the main issue, too. I think it is the hash algo, which is SHA1 in the older ssh-rsa procedure - not the keys in RSA itself. But I am confused at what stage this bothers. Is it at the connection stage where the ssh-server sends its machine public key? Or is it later when client and server are handling login via the key or the user who wants to login? I already have found on different places that it is not neccessary to change all RSA keys - they can stay as before. I was digging for an detailed description how an ssh conection is started and how the handshaking between server and client is working. This is again an interesting point not only to solve the issue but also to learn “how it works” and why sha1 is a security risk…

So, e.g. ssh -vvv df8oe@archserver

ssh -vvv tells me all algs have failed and I am dropped back to password authentication when I try to connect from an Arch machine to Raspberry. If I reenable ssh-rsa on the Arch machine everything works as before. This is the output when I tried to connect from Raspberry to Arch:

ssh -vvv archserver -p 34567
OpenSSH_6.7p1 Raspbian-5+deb8u8, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to archserver [111.222.111.222] port 34567.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type 3
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Raspbian-5+deb8u8
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.8
debug1: match: OpenSSH_8.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: [archserver]:34567
debug3: load_hostkeys: loading entries for host "[archserver]:34567" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: kex_parse_kexinit: rsa-sha2-512,rsa-sha2-256
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug2: mac_setup: setup umac-64-etm@openssh.com
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
no hostkey alg

In debug2 I see rsa-sha2-512,rsa-sha2-256 which are supported by Arch, too. Do I need to rewrite the existing keys of my Raspberry to use sha256 or sha512 instead of sha1? How can this be done? Or is the machanism which hashes the keys inside OpenSSH and I do not have any other possibility than upgrading OpenSSH on Raspberry (which means building it from the scratch or - better - switch to Arch Arm)?

If I inspect the same pub key on both machines using ssh-keygen -l -f /path/to/key I get different informations on Arch and Raspberry:

Arch:
4096 SHA256:0/TpXI90+Eom0lQ3k1dbk+…

Raspberry:
4096 50:80:d5:87:ca…

What’s the output of ls -la /etc/ssh/ssh_host_* on the server?

I do have a slightly different path:

ls -la /etc/certs/ssh/ssh_host_*
-rw------- 1 root root  668 12. Okt 2008  /etc/certs/ssh/ssh_host_dsa_key
-rw-r--r-- 1 root root  603 12. Okt 2008  /etc/certs/ssh/ssh_host_dsa_key.pub
-rw------- 1 root root 1675 12. Okt 2008  /etc/certs/ssh/ssh_host_rsa_key
-rw-r--r-- 1 root root  395 12. Okt 2008  /etc/certs/ssh/ssh_host_rsa_key.pub

Why do you have a different path on your Arch box? Did you move the files?

If sshd can’t find the host key files then it won’t work; you might want to try removing and reinstalling openssh.

Yes they are moved and everything is corrected. This is because of the server administration tool which allows customers to configure their webspace. mail etc. But everything is working and ssh was working up to the day it was upgraded from 8.7p1 to 8.8p1, too. It is definitely no misconfiguration.

EDIT:
I have running 7 Arch boxes where two have same path as above and the others do have “standard path”. Behaviour is exactly the same on all machines.

So…

Given Jessie went EOL in June 2020 it might be time to update to a supported OS version?

1 Like

So it is. I am already working at an Arch Arm installation and that will be the last machine I own where an OS with an EOL is installed. All other machines are “rolling” and it was a very wise decision to choose that way. It is not an easy transfer because the Raspberry has a Homematic server running, also a pihole, a sdr server and many ore “specials”. That is the reason it is not updated but now the time has definitely come. Thank you for this link - I have not found it before…

1 Like