Kernel Out of Memory invoked, cannot find the reason

I’m getting oom-killer invoked quite a bit, starting since my last update. I’ve searched around for some guides and looked into the logs, but nothing is showing up as an obvious cause.

free shows

               total        used        free      shared  buff/cache   available
Mem:         7185120     4363816      474488      967100     3624144     2821304
Swap:              0           0           0

and logs at the time oom-killer is invoked show a total memory use (sum of rss*4kb)[1] of 4796800, which is way less than 7185120 (but way more than 3624144 if it’s ‘available’ I should be using)

I’m not clear how to about checking the following;

  1. is the problem caused by a normal ‘too many applications wanting memory’ problem, or is it something more insidious?
  2. if the problem is a normal ‘too many applications wanting memory’ problem, how do I interpret the logs to find out where the problem lies?

to clarify (2) I’ve got logs from journalctl which show me memory usage at the time of crash, columns like ‘total_vm’ and ‘rss’, and I’ve got the output of things like free and less/proc/meminfo, but the numbers don’t seem to relate.

I (perhaps naively) assumed that if the system is out of memory, I should be able to look up somewhere how much memory it has, also look up how much memory it was using at the time, and see for myself that one number is bigger than the other.

But none of the numbers in the tables I’ve accessed so far seem to relate to one another.

[1] from here https://stackoverflow.com/a/43611576

Add some swap. It doesn’t matter what kind, a swapfile, partition, zswap or zram. Just make sure this number is not 0.

The oom-killer behaves poorly when there is no swap available.

I have 64GB of RAM and the oom-killer will start killing applications when I don’t have swap after long period of uptime.

2 Likes

I concur. @dalto’s advice is sound. I’ve experienced the same oom-killer problems on my desktop, which disappeared after getting a swap file up and running.

More info at [1] and [2] <edit: and [3]. /edit>

[1] https://wiki.archlinux.org/title/Swap#Swap_file_creation
[2] https://discovery.endeavouros.com/storage-and-partitions/adding-swap-after-installation/2021/03/
[3] https://chrisdown.name/2018/01/02/in-defence-of-swap.html

Can oom-killer be modified to behave less aggressively when there is no swap?

Never tried it myself, but the oom-killer can be influenced it seems [1].
Have not found comparable info on the arch wiki yet.

[1] https://last9.io/blog/understanding-the-linux-oom-killer/

1 Like

Probably but I have never looked into it in depth since there is very little reason not to have swap and the oom-killer isn’t the only thing that benefits from having swap.

2 Likes

Thanks for the advice, I’ll do as you suggest.

And thanks @SemLraug for the links. I can’t even recall why I have no swap, I’m sure I used to have… so a reminder of how to do it was going to be my next question.

1 Like

No problem.

I hope adding swap solves your problem.

Follow up question, if you don’t mind.

Arch wiki https://wiki.archlinux.org/title/Swap#Swap_file_creation suggests running

# mkswap -U clear --size 4G --file /swapfile

the man for mkswap https://man.archlinux.org/man/mkswap.8, says

-U, –uuid UUID

Specify the UUID to use. The default is to generate a UUID. The format of the UUID is a series of hex digits separated by hyphens, like this: “c1b9d5a2-f162-11cf-9ece-0020afc76f16”. The UUID parameter may also be one of the following:

clear

clear the filesystem UUID

‘clear’ the filesystem UUID?? Is that really what I want to do - it doesn’t sound good. The man doesn’t mention it any further.

Sorry, i can’t advise you in this case. Perhaps @dalto can clarify.

Do you need your swapfile to have a real UUID for some reason?

It is referring to the swap filesystem it is creating inside the file.

Honestly, I don’t know. Which probably means no, I don’t.

It was the expression “clear the filesystem UUID” that worried me. It just doesn’t sound like the sort of thing one wants to do.

As far as my (limited) understanding goes the ‘filesystem’ is all my files and the UUID is how fstab identifies it. So clearing that identification sounds, to the un-knowledgeable, like a terrible idea.

I was worried that the Wiki might be assuming an empty system in the process of being built rather than an already up and running system that happens to be lacking a swap.

1 Like

That’s the UUID for your swap file system. No worries. Just follow the arch wiki to create, enable and mount it.

@chroot
@dalto

Thanks. Running it now

From what I read, mkswap treats swapfile in a similar way it treats swap partitions in generating an UUID for them but this UUID for swapfile is not used as identifier in /etc/fstab or any other practical purpose from what I gather.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.