I could use some help finding these monkeys

Could it be malware?

You haven’t let your Linux hang around in a bad crowd?

A lightweight web server

Pudge

It does…

/usr/bin/sudo. Although, it isn’t sudo because it also happens with pkexec

I don’t see anything abnormal and it happens with a new clean user

Not that I can find

But not with root? Strange.

One of the strangest issues I’ve ever seen!

mon keys aren’t attracted to root!. :rofl:

OK, let’s dive deep.

Install strace and trace the execution, e.g.

sudo strace -f pacman -S ....

(-f means “follow” so it will follow any forked subprocesses)

It will output a lot of text so you will want to pipe to grep or narrow down the system calls with e.g.

sudo strace -f -e open pacman -S ....

which limits it to the open syscall. Others may be useful, like stat, statx, read, and write. It’s going to take a little while to find the specific call, but this should show the exact execution trace and identify exactly where the monkey is coming from.

For example:

$ sudo strace -f -eopen,stat,statx,access pacman -Syu
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
stat("/", {st_mode=S_IFDIR|0755, st_size=284, ...}) = 0
stat("/var/lib/pacman/", {st_mode=S_IFDIR|0755, st_size=18, ...}) = 0
access("/var/lib/pacman/sync/core.db", R_OK) = 0
access("/var/lib/pacman/sync/core.db", R_OK) = 0
access("/var/lib/pacman/sync/core.db.sig", R_OK) = -1 ENOENT (No such file or directory)
access("/var/lib/pacman/sync/extra.db", R_OK) = 0
access("/var/lib/pacman/sync/extra.db", R_OK) = 0
access("/var/lib/pacman/sync/extra.db.sig", R_OK) = -1 ENOENT (No such file or directory)
access("/var/lib/pacman/sync/community.db", R_OK) = 0
access("/var/lib/pacman/sync/community.db", R_OK) = 0
access("/var/lib/pacman/sync/community.db.sig", R_OK) = -1 ENOENT (No such file or directory)
access("/var/lib/pacman/sync/multilib.db", R_OK) = 0
access("/var/lib/pacman/sync/multilib.db", R_OK) = 0
access("/var/lib/pacman/sync/multilib.db.sig", R_OK) = -1 ENOENT (No such file or directory)
access("/var/lib/pacman/sync/endeavouros.db", R_OK) = 0
access("/var/lib/pacman/sync/endeavouros.db", R_OK) = 0
access("/var/lib/pacman/sync/endeavouros.db.sig", R_OK) = -1 ENOENT (No such file or directory)
access("/var/lib/pacman/sync/repo-ck.db", R_OK) = 0
access("/var/lib/pacman/sync/repo-ck.db", R_OK) = 0
access("/var/lib/pacman/sync/repo-ck.db.sig", R_OK) = -1 ENOENT (No such file or directory)
access("/etc/pacman.d/gnupg/pubring.gpg", R_OK) = 0
access("/etc/pacman.d/gnupg/trustdb.gpg", R_OK) = 0
access("/home/jonathon/perl5/perlbrew/bin/gpgconf", X_OK) = -1 ENOENT (No such file or directory)
access("/home/jonathon/bin/gpgconf", X_OK) = -1 ENOENT (No such file or directory)
access("/usr/local/bin/gpgconf", X_OK)  = -1 ENOENT (No such file or directory)
access("/usr/bin/gpgconf", X_OK)        = 0
access("/usr/bin/gpgconf", F_OK)        = 0
strace: Process 460917 attached
strace: Process 460918 attached
[pid 460917] +++ exited with 0 +++
[pid 460916] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=460917, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 460918] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 460918] access("/etc/gcrypt/fips_enabled", F_OK) = -1 ENOENT (No such file or directory)
[pid 460918] stat("/run/user/0", 0x7ffc01e92d50) = -1 ENOENT (No such file or directory)
[pid 460918] stat("/var/run/user/0", 0x7ffc01e92d50) = -1 ENOENT (No such file or directory)
[pid 460918] +++ exited with 0 +++
strace: Process 460919 attached
strace: Process 460920 attached
[pid 460919] +++ exited with 0 +++
[pid 460916] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=460919, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 460920] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 460920] access("/etc/gcrypt/fips_enabled", F_OK) = -1 ENOENT (No such file or directory)
[pid 460920] access("/usr/bin/pinentry", F_OK) = 0
...

Here is what I think it the relevant output: https://pastebin.com/qNzMapGZ

Hopefully we don’t need the ~17,000 lines after that. :slight_smile:

It does use the locate ‘updatedb’ for filename searching - but I don’t think impacts file content searches - and it still seems relatively quick.

I can’t remember if I had to install it, or if it came with EnOS - but it does’t take long either way! (nor does the sudo updatedb, or its GUI equivalent).

OK,

stat("/var/lib/pacman/local/", {st_mode=S_IFDIR|0755, st_size=1652, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2310, ...}) = 0
access("/var/lib/pacman/local/glibc-2.32-4/install", R_OK) = 0
access("/tmp/", F_OK)                   = 0
strace: Process 2106182 attached
[pid 2106182] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 2106087] stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2310, ...}) = 0
monkey

So, check the contents of /var/lib/pacman/local/glibc-2.32-4/install .

If that’s normal then we need to go deeper - you’ll have to try some other syscalls and e.g. check what is using read to read “monkey” from a file and/or write “monkey” to an output.

e.g.:

$ sudo strace -f -e write pacman -Syu
...
write(10, "\26\3\1\2\0\1\0\1\374\3\3\346\305:\202\341\227\332@\276\0204\207\301\356\306$jo\336\224\201"..., 517) = 517
...
write(1, "\33[0m", 4)                   = 4
write(3, "[2020-09-09T23:18:28+0100] [PACM"..., 65) = 65
write(2, "\33[1;33mwarning: \33[0m", 20warning: ) = 20
...

The first number tells us the device handle, and while it’s still open:

$ sudo lsof -p $PACMAN_PID
...
pacman  35786 root    1u   CHR              136,0      0t0        3 /dev/pts/0
pacman  35786 root    2u   CHR              136,0      0t0        3 /dev/pts/0
pacman  35786 root    3w   REG               0,56 12872787     3378 /var/log/pacman.log
...
pacman  35786 root   10u  IPv6             395060      0t0      TCP localhost:60844->server6.alpix.eu:https (ESTABLISHED)
...

will tell us which handle maps to which device.

Remember that you can use CTRL+Z to background (and pause) a process when you see something interesting, and fg to resume it.

If you want to go still deeper, then gdb will be the next step.

Man, these are some subterranean monkeys… :joy:

Perhaps we’ll need to put them on display when they finally located - a zoo next to the wiki? :grin:

What’s in /usr/share/libalpm/scripts ?

There’s always one bad monkey in the group.

But which one ???

OK, this seems relevant:

[pid 2225925] read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 `\1\0\0\0\0\0"..., 832) = 832
[pid 2225925] read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\22\0\0\0\0\0\0"..., 832) = 832
[pid 2225925] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\202\2\0\0\0\0\0"..., 832) = 832
[pid 2225925] read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 p\1\0\0\0\0\0"..., 832) = 832
[pid 2225925] read(3, "# GNU libc iconv configuration.\n"..., 8192) = 8192
[pid 2225925] read(3, "\tISO-IR-110//\t\tISO-8859-4//\nalia"..., 8192) = 8192
[pid 2225925] read(3, "DIC-ES//\nalias\tEBCDICES//\t\tEBCDI"..., 8192) = 8192
[pid 2225925] read(3, "\t\tIBM863//\nalias\tOSF1002035F//\t\t"..., 8192) = 8192
[pid 2225925] read(3, "UJIS//\t\t\tEUC-JP//\nmodule\tEUC-JP/"..., 8192) = 8192
[pid 2225925] read(3, "O_5427-EXT//\nalias\tISO_5427EXT//"..., 8192) = 8192
[pid 2225925] read(3, "112//\t\tINTERNAL\t\tIBM1112\t\t1\nmodu"..., 8192) = 7201
[pid 2225925] read(3, "", 8192)         = 0
[pid 2225925] read(3, "#\n# /etc/bash.bashrc\n#\n\n# If not"..., 618) = 618
[pid 2225925] read(3, "echo monkey\n", 12) = 12

@jonathon, maybe I am reading this wrong but I don’t see anything to indicate which file maps to the above. My fear would be it was closed before I had a chance to stop the process

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
pacman  2222573 root  cwd    DIR               0,26       22      34 /
pacman  2222573 root  rtd    DIR               0,26       22      34 /
pacman  2222573 root  txt    REG               0,26   145344  723340 /usr/bin/pacman
pacman  2222573 root  DEL    REG               0,26          1228567 /usr/lib/locale/locale-archive
pacman  2222573 root  DEL    REG               0,26          1227733 /usr/lib/libresolv-2.32.so
pacman  2222573 root  mem    REG               0,26    22200  727651 /usr/lib/libkeyutils.so.1.10
pacman  2222573 root  mem    REG               0,26    55264 1281250 /usr/lib/libkrb5support.so.0.1
pacman  2222573 root  mem    REG               0,26  1574704   18025 /usr/lib/libunistring.so.2.1.0
pacman  2222573 root  mem    REG               0,26    22216 1024642 /usr/lib/libattr.so.1.1.2448
pacman  2222573 root  DEL    REG               0,26          1221707 /usr/lib/libdl-2.32.so
pacman  2222573 root  DEL    REG               0,26          1221741 /usr/lib/libpthread-2.32.so
pacman  2222573 root  mem    REG               0,26    18112 1025036 /usr/lib/libcom_err.so.2.1
pacman  2222573 root  mem    REG               0,26   194456 1281227 /usr/lib/libk5crypto.so.3.1
pacman  2222573 root  mem    REG               0,26   936256 1281247 /usr/lib/libkrb5.so.3.3
pacman  2222573 root  mem    REG               0,26   344088 1281221 /usr/lib/libgssapi_krb5.so.2.2
pacman  2222573 root  mem    REG               0,26   585320  926575 /usr/lib/libssl.so.1.1
pacman  2222573 root  mem    REG               0,26    71272  729990 /usr/lib/libpsl.so.5.3.3
pacman  2222573 root  mem    REG               0,26   256144     812 /usr/lib/libssh2.so.1.0.1
pacman  2222573 root  mem    REG               0,26   128696   23532 /usr/lib/libidn2.so.0.3.7
pacman  2222573 root  mem    REG               0,26   173848  970800 /usr/lib/libnghttp2.so.14.20.0
pacman  2222573 root  mem    REG               0,26   100096   16486 /usr/lib/libz.so.1.2.11
pacman  2222573 root  mem    REG               0,26    74440   16606 /usr/lib/libbz2.so.1.0.8
pacman  2222573 root  mem    REG               0,26   132784   17179 /usr/lib/liblz4.so.1.9.2
pacman  2222573 root  mem    REG               0,26   853696  932973 /usr/lib/libzstd.so.1.4.5
pacman  2222573 root  mem    REG               0,26   157456 1095508 /usr/lib/liblzma.so.5.2.5
pacman  2222573 root  mem    REG               0,26   190128   21662 /usr/lib/libexpat.so.1.6.11
pacman  2222573 root  mem    REG               0,26    38704 1024682 /usr/lib/libacl.so.1.1.2253
pacman  2222573 root  DEL    REG               0,26          1221713 /usr/lib/libm-2.32.so
pacman  2222573 root  mem    REG               0,26  2982648  926355 /usr/lib/libcrypto.so.1.1
pacman  2222573 root  mem    REG               0,26   583856  723551 /usr/lib/libcurl.so.4.6.0
pacman  2222573 root  mem    REG               0,26   153560  727800 /usr/lib/libgpg-error.so.0.30.0
pacman  2222573 root  mem    REG               0,26    79776   26478 /usr/lib/libassuan.so.0.8.3
pacman  2222573 root  mem    REG               0,26   322128 1127845 /usr/lib/libgpgme.so.11.23.0
pacman  2222573 root  DEL    REG               0,26          1227728 /usr/lib/libc-2.32.so
pacman  2222573 root  mem    REG               0,26   788312  121404 /usr/lib/libarchive.so.13.4.3
pacman  2222573 root  mem    REG               0,26   223512  723353 /usr/lib/libalpm.so.12.0.2
pacman  2222573 root  DEL    REG               0,26          1227717 /usr/lib/ld-2.32.so
pacman  2222573 root    0u   CHR              136,1      0t0       4 /dev/pts/1
pacman  2222573 root    1u   CHR              136,1      0t0       4 /dev/pts/1
pacman  2222573 root    2u   CHR              136,1      0t0       4 /dev/pts/1
pacman  2222573 root    3w   REG               0,26  1270856     642 /var/log/pacman.log
pacman  2222573 root    4w   REG               0,26        0 1201784 /var/lib/pacman/db.lck
pacman  2222573 root    5r   DIR               0,54      102       2 /home/evan
pacman  2222573 root    6u  unix 0x0000000026568c7e      0t0 1767239 type=STREAM

Bingo.

Now it’s a case of finding which file has echo monkey in it. Maybe something related to /etc/bash.bashrc somewhere along the line…

I’m trying to find something with # GNU libc iconv configuration in it as that would seem obvious, but I haven’t found it yet.

Yup, if it’s not present then it was already closed (doing this in gdb lets you step through one call at a time, and that takes ages).

Probably /usr/lib/gconv/gconv-modules but there are no monkeys in it.

That make sense, looking again, this bit shows the file has 0 bytes read, so the /etc/bash.bashrc is in a different file.

The monkeys just jumped to another tree. :unamused:

Did you use a filter wider than just -o read here? If you use -o open,stat,access,read it might point to a file.

I noticed that the output didn’t have any open calls. A little research revealed that openat is what is needed.

That produced this:

[pid 2316086] openat(AT_FDCWD, "/root/.bashrc", O_RDONLY) = 3
[pid 2316086] read(3, "echo monkey\n", 12) = 12

Which then led to this:

$ sudo cat /root/.bashrc                                                                                                                          
echo monkey

Next came this:

$ sudo rm /root/.bashrc

Finally we have:

$ /usr/bin/yay -S glibc                                                                                                                           
warning: glibc-2.32-4 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (1) glibc-2.32-4

Total Installed Size:  46.16 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                                                                 [#######################################################] 100%
(1/1) checking package integrity                                                               [#######################################################] 100%
(1/1) loading package files                                                                    [#######################################################] 100%
(1/1) checking for file conflicts                                                              [#######################################################] 100%
:: Processing package changes...
(1/1) reinstalling glibc                                                                       [#######################################################] 100%
Generating locales...
  en_US.UTF-8... done
Generation complete.
:: Running post-transaction hooks...
(1/5) Reloading system manager configuration...
(2/5) Creating temporary files...
(3/5) Arming ConditionNeedsUpdate...
(4/5) Restarting cronie for libc upgrade...
(5/5) Updating the info directory file...

Thanks @jonathon!

The monkeys have been found!

So, any reason your root .bashrc was echo-ing monkey? Is it something you added while testing something? :thinking:

You know what happens when you cage a monkey!

Are you saying that doesn’t come in normally with the install? :face_with_monocle:

Well, I am sure there was a reason. It probably wasn’t a good reason though. It has been doing it for quite a while now so I have long since forgotten whatever I was trying to test with that. :innocent:

The most interesting part is that root’s shell isn’t set to bash which is why it worked when you logged in as root.