Systemd service degraded at some time after boot

pebcak · December 31, 2021, 4:39pm

At one time or another, a systemd service gets “degraded” on my system:

htop-degraded-service

systemctl --failed shows the failed service to be shadow.service:

sudo dmesg | grep shadow

Dec 31 00:00:00 arch-gnome audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=shadow comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 31 00:00:00 arch-gnome kernel: audit: type=1130 audit(1640905200.178:722): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=shadow comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 31 00:00:00 arch-gnome sh[29782]: user 'netdata': directory '/var/cache/netdata' does not exist
Dec 31 00:00:00 arch-gnome sh[29782]: pwck: no changes
Dec 31 00:00:00 arch-gnome systemd[1]: shadow.service: Main process exited, code=exited, status=1/FAILURE
Dec 31 00:00:00 arch-gnome systemd[1]: shadow.service: Failed with result 'exit-code'.
Dec 31 00:00:00 arch-gnome audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=shadow comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Dec 31 00:00:00 arch-gnome kernel: audit: type=1131 audit(1640905200.207:723): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=shadow comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

From joutnalctl:

Dec 31 00:18:59 arch-gnome systemd[1]: Started Verify integrity of password and group files.
Dec 31 00:18:59 arch-gnome sh[37124]: user 'netdata': directory '/var/cache/netdata' does not exist
Dec 31 00:18:59 arch-gnome sh[37124]: pwck: no changes
Dec 31 00:18:59 arch-gnome systemd[1]: shadow.service: Main process exited, code=exited, status=1/FAILURE
Dec 31 00:18:59 arch-gnome systemd[1]: shadow.service: Failed with result 'exit-code'.

systemctl --all | grep shadow
  shadow.service                                                                                   loaded    inactive dead      Verify integrity of password and group files
  shadow.timer                                                                                     loaded    active   waiting   Daily verification of password and group files

sudo systemctl restart shadow 

× shadow.service - Verify integrity of password and group files
     Loaded: loaded (/usr/lib/systemd/system/shadow.service; static)
     Active: failed (Result: exit-code) since Fri 2021-12-31 00:18:59 CET; 12s ago
TriggeredBy: ● shadow.timer
    Process: 37123 ExecStart=/bin/sh -c /usr/bin/pwck -r || r=1; /usr/bin/grpck -r && exit $r (code=exited, status=1/FAILURE)
   Main PID: 37123 (code=exited, status=1/FAILURE)
        CPU: 13ms

systemctl status  shadow.timer     
● shadow.timer - Daily verification of password and group files
     Loaded: loaded (/usr/lib/systemd/system/shadow.timer; static)
     Active: active (waiting) since Fri 2021-12-31 05:59:27 CET; 11h ago
      Until: Fri 2021-12-31 05:59:27 CET; 11h ago
    Trigger: Sat 2022-01-01 00:00:00 CET; 6h left
   Triggers: ● shadow.service

Dec 31 05:59:27 arch-gnome systemd[1]: Started Daily verification of password and group files.

Do you guys have any idea why this happens and if it can be fixed?
Looking forward your comments, suggestions, ideas…

inxi -Fxxxaz

System:
  Kernel: 5.15.12-zen1-1-zen x86_64 bits: 64 compiler: gcc v: 11.1.0
    parameters: initrd=\arch-gnome\intel-ucode.img
    initrd=\arch-gnome\initramfs-linux-zen.img rw
    root=UUID=1d9646ea-0edf-423b-85a7-88a429c6314b
    rootflags=subvol=@arch-gnome-root lsm=landlock,lockdown,yama,apparmor,bpf
    nowatchdog zswap.enabled=0
  Desktop: GNOME 41.2 tk: GTK 3.24.31 wm: gnome-shell dm: GDM 41.0
    Distro: Arch Linux
Machine:
  Type: Laptop System: Dell product: XPS 13 9380 v: N/A
    serial: <superuser required> Chassis: type: 10 serial: <superuser required>
  Mobo: Dell model: 0KTDY6 v: A00 serial: <superuser required> UEFI: Dell
    v: 1.16.1 date: 10/07/2021
Battery:
  ID-1: BAT0 charge: 42.0 Wh (100.0%) condition: 42.0/52.0 Wh (80.7%)
    volts: 8.6 min: 7.6 model: LGC-LGC6.73 DELL H754V8C type: Li-ion
    serial: <filter> status: Full
CPU:
  Info: model: Intel Core i7-8565U bits: 64 type: MT MCP arch: Whiskey Lake
    family: 6 model-id: 0x8E (142) stepping: 0xB (11) microcode: 0xEA
  Topology: cpus: 1x cores: 4 tpc: 2 threads: 8 smt: enabled cache:
    L1: 256 KiB desc: d-4x32 KiB; i-4x32 KiB L2: 1024 KiB desc: 4x256 KiB
    L3: 8 MiB desc: 1x8 MiB
  Speed (MHz): avg: 919 high: 2452 min/max: 400/4600 scaling:
    driver: intel_pstate governor: powersave cores: 1: 700 2: 2452 3: 707
    4: 700 5: 700 6: 700 7: 700 8: 700 bogomips: 31999
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Vulnerabilities:
  Type: itlb_multihit status: KVM: VMX disabled
  Type: l1tf status: Not affected
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable
  Type: meltdown status: Not affected
  Type: spec_store_bypass
    mitigation: Speculative Store Bypass disabled via prctl
  Type: spectre_v1
    mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional,
    IBRS_FW, STIBP: conditional, RSB filling
  Type: srbds mitigation: Microcode
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel WhiskeyLake-U GT2 [UHD Graphics 620] vendor: Dell
    driver: i915 v: kernel bus-ID: 00:02.0 chip-ID: 8086:3ea0 class-ID: 0300
  Device-2: CN09357GLOG008CLACSJA01 Integrated_Webcam_HD type: USB
    driver: uvcvideo bus-ID: 1-5:2 chip-ID: 0c45:6723 class-ID: 0e02
  Display: wayland server: X.org 1.21.1.2 compositor: gnome-shell driver:
    loaded: modesetting alternate: fbdev,intel,vesa display-ID: 0
    resolution: <missing: xdpyinfo>
  Message: Unable to show advanced data. Required tool glxinfo missing.
Audio:
  Device-1: Intel Cannon Point-LP High Definition Audio vendor: Dell
    driver: snd_hda_intel v: kernel
    alternate: snd_soc_skl,snd_sof_pci_intel_cnl bus-ID: 00:1f.3
    chip-ID: 8086:9dc8 class-ID: 0403
  Sound Server-1: ALSA v: k5.15.12-zen1-1-zen running: yes
  Sound Server-2: JACK v: 1.9.19 running: no
  Sound Server-3: PulseAudio v: 15.0 running: no
  Sound Server-4: PipeWire v: 0.3.42 running: yes
Network:
  Device-1: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter
    vendor: Rivet Networks Killer 1435 Wireless-AC driver: ath10k_pci v: kernel
    bus-ID: 02:00.0 chip-ID: 168c:003e class-ID: 0280
  IF: wlan0 state: down mac: <filter>
  IF-ID-1: enp0s20f0u1 state: unknown speed: -1 duplex: half mac: <filter>
  IF-ID-2: tun0 state: unknown speed: 10 Mbps duplex: full mac: N/A
Bluetooth:
  Device-1: SAMSUNG_Android type: USB driver: rndis_host v: kernel
    bus-ID: 1-1:6 chip-ID: 04e8:6863 class-ID: 0a00 serial: <filter>
  Report: This feature requires one of these tools: hciconfig/bt-adapter
  Device-2: N/A type: USB driver: btusb v: 0.8 bus-ID: 1-7:3
    chip-ID: 0489:e0a2 class-ID: e001
  Report: ID: hci0 rfk-id: 0 state: up address: N/A
Drives:
  Local Storage: total: 465.76 GiB used: 180.61 GiB (38.8%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Samsung
    model: SSD 970 EVO 500GB size: 465.76 GiB block-size: physical: 512 B
    logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter>
    rev: 2B2QEXE7 temp: 34.9 C scheme: GPT
Partition:
  ID-1: / raw-size: 456.76 GiB size: 456.76 GiB (100.00%)
    used: 180.26 GiB (39.5%) fs: btrfs dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-2: /home raw-size: 456.76 GiB size: 456.76 GiB (100.00%)
    used: 180.26 GiB (39.5%) fs: btrfs dev: /dev/nvme0n1p2 maj-min: 259:2
  ID-3: /var/log raw-size: 456.76 GiB size: 456.76 GiB (100.00%)
    used: 180.26 GiB (39.5%) fs: btrfs dev: /dev/nvme0n1p2 maj-min: 259:2
Swap:
  Kernel: swappiness: 10 (default 60) cache-pressure: 100 (default)
  ID-1: swap-1 type: file size: 8 GiB used: 0 KiB (0.0%) priority: -2
    file: /swap/swapfile
  ID-2: swap-2 type: zram size: 7.41 GiB used: 819 MiB (10.8%)
    priority: 100 dev: /dev/zram0
Sensors:
  System Temperatures: cpu: 25.0 C pch: 37.0 C mobo: N/A
  Fan Speeds (RPM): cpu: 0 fan-2: 0
Info:
  Processes: 314 Uptime: 11h 41m wakeups: 8489 Memory: 7.41 GiB
  used: 5.36 GiB (72.3%) Init: systemd v: 250 tool: systemctl Compilers:
  gcc: 11.1.0 Packages: 902 pacman: 880 lib: 228 flatpak: 22 Shell: Bash
  v: 5.1.12 running-in: gnome-terminal inxi: 3.3.11

dalto · December 31, 2021, 4:45pm

It looks like there is an issue with the user netdata

pebcak · December 31, 2021, 4:48pm

Right. I saw this:

user 'netdata': directory '/var/cache/netdata' does not exist

in the output of dmesg. Honestly I haven’t dug any deeper into it. Do you mind pointing me to where and what to look for?

Edit: There is this Wiki page:

https://wiki.archlinux.org/title/Netdata

but I am not sure what I am looking at and where to go from there.

Edit 2:

systemctl status netdata
○ netdata.service - Real time performance monitoring
     Loaded: loaded (/usr/lib/systemd/system/netdata.service; disabled; vendor preset: disabled)
     Active: inactive (dead)

Do I need to enable this?

dalto · December 31, 2021, 5:00pm

There is probably something wrong with the entry for netdata in /etc/passwd or /etc/shadow

Can see the entry for netdata from /etc/passwd

ricklinux · December 31, 2021, 5:02pm

On mine i don’t have that loaded?

[ricklinux@eos-kde ~]$ systemctl status netdata
Unit netdata.service could not be found.
[ricklinux@eos-kde ~]$

[ricklinux@eos-kde ~]$ systemctl status  shadow.timer    
● shadow.timer - Daily verification of password and group files
     Loaded: loaded (/usr/lib/systemd/system/shadow.timer; static)
     Active: active (waiting) since Fri 2021-12-31 10:48:08 EST; 1h 14min ago
      Until: Fri 2021-12-31 10:48:08 EST; 1h 14min ago
    Trigger: Sat 2022-01-01 00:00:00 EST; 11h left
   Triggers: ● shadow.service

Dec 31 10:48:08 eos-kde systemd[1]: Started Daily verification of password and group files.
[ricklinux@eos-kde ~]$

Stagger_Lee · December 31, 2021, 5:03pm

Do you have netdata installed?

pebcak · December 31, 2021, 5:04pm

In /etc/passwd:

netdata:x:134:134::/var/cache/netdata:/usr/bin/nologin

In /etc/shadow:

netdata:!*:18908::::::

ricklinux · December 31, 2021, 5:05pm

No …i don’t.

Stagger_Lee · December 31, 2021, 5:05pm

Well maybe that’s why you don’t have the service.

ricklinux · December 31, 2021, 5:05pm

@pebcak
See why you have these issues. You keep installing stuff!

pebcak · December 31, 2021, 5:07pm

Frankly, I am perplexed why it is installed in this system and on top of it “explicitly”

pacman -Qi netdata
Name            : netdata
Version         : 1.32.1-2
Description     : Real-time performance monitoring, in the greatest possible detail, over the web
Architecture    : x86_64
URL             : https://github.com/netdata/netdata
Licenses        : GPL
Groups          : None
Provides        : None
Depends On      : libmnl  libnetfilter_acct  zlib  judy  libuv  json-c  libcap  lz4  openssl  which  snappy  protobuf  libwebsockets  mongo-c-driver
Optional Deps   : nodejs: for monitoring named and SNMP devices
                  lm_sensors: for monitoring hardware sensors [installed]
                  iproute2: for monitoring Linux QoS [installed]
                  python: for most of the external plugins [installed]
                  python-psycopg2: for monitoring PostgreSQL databases
                  python-mysqlclient: for monitoring MySQL/MariaDB databases
                  python-requests: for monitoring elasticsearch [installed]
                  hddtemp: for monitoring hhd temperature
                  fping: for for fping module
                  apcupsd: for monitoring APC UPS
                  cups: for CUPS plugin
                  iw: for monitoring Linux as access point
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 30.65 MiB
Packager        : Felix Yan <felixonmars@archlinux.org>
Build Date      : Wed 22 Dec 2021 10:02:47 PM CET
Install Date    : Fri 24 Dec 2021 01:28:30 AM CET
Install Reason  : Explicitly installed
Install Script  : Yes
Validated By    : Signature

ricklinux · December 31, 2021, 5:08pm

I didn’t do it!

pebcak · December 31, 2021, 5:09pm

I take that as confession of guilt!

dalto · December 31, 2021, 5:09pm

I would either remove the netdata package or create the directory /var/cache/netdata

Stagger_Lee · December 31, 2021, 5:12pm

Well, a user mentioned it on Oct 8, and you thanked and gave a “thumbs up” to the post…I’m guessing this is maybe related .

pebcak · December 31, 2021, 5:12pm

Since nothing seems to be dependent on it, I guess I could remove it.
However I don’t understand its rapport with shadow.service

dalto · December 31, 2021, 5:13pm

I believe shadow.service is not happy that the user has a home directory which doesn’t exist.

ricklinux · December 31, 2021, 5:15pm

I can understand having Gnome that you’d want to monitor it with a shadow service. It just want’s to know where to dump the data.

pebcak · December 31, 2021, 5:16pm

Right you are!
This was one of those no longer remembered memorable moments of 2021

pebcak · December 31, 2021, 5:17pm

Yeah! We Gnomies operate in the shadows!

Sorry @dalto!
The post was meant for @ricklinux