No Internet / DNS issues after update (NetworkManager & systemd-resolved on KDE)

General system Kernel, boot, graphics & hardware
1

IMPORTANT INFO IN UPDATE 2

Hello everyone,

After doing a -Syu and reboot, I no longer have access to the internet system-wide. I can connect to the Wi-Fi but KDE shows a limited connectivity error.

Some observations:

  • I can ping 8.8.8.8 but not ping google.com (same results with other IPs and domains).
  • I can ping 192.168.1.1, my default gateway.
  • The issue persists with different networks, so it’s not related to a specific network or router.
  • My /etc/resolv.conf is generated by NetworkManager and is a regular file (not symlinked). So I assume NetworkManager is my DNS resolver.
  • Both /etc/resolv.conf and nmcli are pointing correctly to the DNS that I have set up (Adguard 94.140.14.14).

In general, it seems that the issue is DNS-related.

What I have tried so far:

  • Changing my current DNS to another public DNS (e.g. Cloudfare, Google, etc.) through KDE’s network settings.
    :cross_mark: did not work

  • Deleting my /etc/resolv.conf and restarting NetworkManager.service to regenerate the file.
    :cross_mark: did not work

  • Making systemd-resolved my DNS resolver by deleting my /etc/resolv.conf and symlinking ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf and restarting systemd-resolved.service.
    :check_mark: this worked! but… after I connect to any VPNs through any protocols succesfully, I no longer have access to the internet (e.g. can’t ping google). I also have to restart systemd-resolved after every boot. So this is not a good solution.

Network/DNS info:

ip a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 70:8b:cd:92:ee:cb brd ff:ff:ff:ff:ff:ff
    altname enx708bcd92eecb
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:72:00:65:00 brd ff:ff:ff:ff:ff:ff permaddr 54:8c:a0:0f:49:4f
    inet 192.168.1.134/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0
       valid_lft 85720sec preferred_lft 85720sec
nmcli device show | grep DNS 
IP4.DNS[1]:                             94.140.14.14
IP4.DNS[2]:                             94.140.15.15
Output of /etc/resolv.conf 
# Generated by NetworkManager
nameserver 94.140.14.14
nameserver 94.140.15.15
resolvectl status 
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: foreign
  Current DNS Server: 94.140.14.14
         DNS Servers: 94.140.14.14 94.140.15.15
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net
                      2001:4860:4860::8888#dns.google

Link 2 (enp2s0)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
     Default Route: no

Link 3 (wlan0)
    Current Scopes: none
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
       DNS Servers: 94.140.14.14 94.140.15.15
     Default Route: yes

General info:

  • Output for inxi -Fxxc0z: inxi link
  • Output for journalctl -k -b: journalctl link
  • NetworkManager version 1.52.0-1
  • systemd version 257.4-1

NOTE: Everything works fine when I revert the update with a snapshot. The issue only happens after system update.

Please feel free to ask If more info is needed.
Thank you in advance for your help!

UPDATE:

I disabled/stopped systemd-resolved.service and symlinked ln -s /run/NetworkManager/resolv.conf /etc/resolv.conf and restarted NetworkManager. Now I have access to the internet, but I still have the same VPN issue. Also I’m not sure if disabling systemd-resolved is a good idea…

UPDATE 2:

I JUST FOUND OUT SOMETHING! When I boot up my system (and have no internet access), I just need to restart systemd-resolved to connect to the internet. No need to tinker with any file or NetworkManager. Still can’t ping anything with VPN though.

The Problem is now narrowed down to this:

  1. Why do I have to manually restart systemd-resolved after booting up in order to access the internet?
  2. How do I fix the issue with my VPN connecting but not working? (this issue is most likely related to my system settings not the VPN itself)
  3. Why has this happened after updating my system?
2

I appreciate the way you stepped through your post @Sphinx212. You set it what has and hasn’t worked clearly.

When connected to a VPN, it’s not uncommon for DNS to be resolved via the VPN using VPN assigned servers, as a means of preventing DNS leakage. This would need to be factored in when performing any of the previously mentioned tests.

Were you connected to a VPN at the time, and does anything change when you are/aren’t?

3

Thanks for the reply.

Good point! I actually checked this. After symlinking the systemd-resolved file (as I mentioned in the third bullet point) and connecting to the VPN, the DNS changes to the VPN’s own internal DNS. So it seems that VPN is doing its job as it should.

Below is the output of resolvectl after connecting to the VPN. tun0 is the VPN connection.

resolvectl status 
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net
                      2001:4860:4860::8888#dns.google

Link 2 (enp2s0)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
     Default Route: no

Link 3 (wlan0)
    Current Scopes: DNS LLMNR/IPv4 mDNS/IPv4
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 94.140.15.15
       DNS Servers: 94.140.14.14 94.140.15.15
     Default Route: yes

Link 4 (tun0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.255.255.3
       DNS Servers: 10.255.255.3
     Default Route: yes

Please note that any of the steps and infos I provided above are without a VPN connection and on a regular network.
I just wanted to test my VPN after switching to systemd-resolved (but it was unsuccessful).

EDIT: I just posted two updates to narrow down what’s going on.

4

This is counter to the preferred method by some, but this updating posts actually confuses me, I guess I’m a linear thinker :slight_smile:
One thing to add into your data gathering is ‘ip route’ if you haven’t. I’ve been there with regards to VPN :slight_smile:
I will ASSUME? that if you don’t involve the VPN your system works normally?

5

Regarding your systemd-resolved issue -please have a read of this systemd issue:

It hit me with systemd 257.0 already. My workaround solution since then is once after boot sudo systemctl restart systemd-resolved.service (just a click on a kde menu entry).

6

Spot on.
I’ve had this issue ever since December of last year (which matches the 257.0 version release date).
But because I didn’t have the time to find out the cause of it back then, I kept rolling back my upgrades. Until now.

As an experiment, I just now downgraded only my systemd packages from 257.4-1 to 256.9-1 to see If the issue persists.
It worked! I didn’t have to restart systemd-resolved after boot this time. The VPN issue is still there though, but atleast I now know it’s systemd-related.

No VPN is a deal-breaker though. What should I do now? Should I roll back my whole system to November and wait until systemd fixes itself? Do you think not upgrading my system for 4+ months would be a security concern or not?

7

A couple suggestions/questions concerning your VPN issue:

  • Make sure you have systemd-networkd disabled (for readers: sudo systemctl status systemd-networkd.service). I have experienced that a recent systemd update all the sudden enabled it, which creates issues with NetworkManager (since they can’t coexist).

  • Make sure systemd-resolved is set up the right way with NetworkManager:

/etc/NetworkManager/conf.d/
rc-manager.conf
    [main]
    rc-manager=auto
dns.conf
    [main]
    dns=systemd-resolved

  • What is in your /run/systemd/resolve/stub-resolv.conf ?

  • I guess,since DNS servers IP show up in your global connection in resolve status (in your first post), you must have set it up in /etc/systemd/resolved.conf.d/dns_servers.conf? If so, please show content.

To avoid confusion, please show resolvectl status for both VPN disabled and VPN enabled with you most recent set up..

DNS resolving is so tricky… I use the app dnsleaktest to check what is really going on:

yay -Ss dnsleaktest                                                                                                                          
aur/dnsleaktest 1.3-2