How to actually get an accurate rundown of where your ram is going to?

I’m just about at my wits end trying to figure out how to get an accurate representation of where my RAM is being used.

Free gives an accurate number for how much ram is used.

Top and Htop don’t show everything that’s using your ram, and even if they did, parsing it is hell anyhow.

PS gives better results for something like say the top 10 most RAM using applications on my system, but only by using residential memory for reference, which will if tallied up show a bigger number than the actual total ram use due to duplicate data that shares teh same space on the RAM.

If I tally up the percentage of RAM use in ps however, it comes up short. I use this command see:

ps -o %mem ghx | awk '{total += $1} END {print total}'

And it spits out 4.7%(2.8GB), but i’m actually using 5%(3.4GB); and even if I account for tmpfs usage(~160mb), it still doesn’t add up!

Why is this so damn convoluted and how am I supposed to actually do this?

ps_mem gives the clearest view in my opinion.

However, keep in mind. Shared memory and buffers can make it hard to get a number that matches.

Also, it doesn’t matter much because of how memory allocation works. Applications will use more memory when memory is available and less when it isn’t

You can never have too much. Then you don’t need to worry about it.

1 Like

it’s almost an entire gigabyte short, even if you add up all the shared and all the tmpfs ram, that comes down to under 600MB, where’s the rest of it?

@fbodymechanic yeah, i just upgraded to 64gb ram (still doesn’t seem like enough sometimes), honestly this has been my philosophy for a while too. But I have a friend on linux who had about 20Gb of RAM unaccounted for, and we couldn’t figure out where it was coming from.

And well, I don’t really have to worry about it, usually, but there will probably come times when I do occasionally, and at those points i’d like to actually be able to.

You are comparing the total amount of RAM to the amount used by processes. The kernel also consumes RAM.

20Gb seems like too much unless you have something with a lot of reserved memory. For example, some filesystems will reserve a lot of RAM for caching unless you ask them not to.

Take a look at sudo slabtop

my kernel would be using 400mb of ram?

Also he’s using ext4, and we tried literally closing everything, including X and his display manager, and it still showed 20GB RAM in use entirely unaccounted for by top and htop and ps_mem

Mine is currently using 1.1GB

How do you see this?

sudo slabtop

1 Like

the total size used? part at the top is what tells you how much ur kernel is using?

Edit: found out the total size used field in slabtop must be inuaccurate, you can see slab use in /proc/meminfo, and it never matches up, but particularly the applicable field here for me (e.g .ram that is unreclaimable)

Is discoverable with this command

cat /proc/meminfo | grep SUnreclaim

And it’s a fair bit lower than the value in slabtop. Which itself is a fair bit lower than the Slab value in /proc/meminfo thus clearly is not accounting for everything.

slabtop captures data in real time. In other words, its value is updated once after a certain amount of time has passed. The value you saw in the output of slabtop may not match the value you obtain at the moment you ran cat /proc/meminfo because the invocations of slabtop and cat /proc/meminfo aren’t synchonized to begin with.

If you want, you can use the watch command on cat /proc/meminfo, but I highly doubt it’s going to make any practical difference.

Also, you might want to take a look at the Notes section in slabtop’s man page.

Ok, so it’s not inaccurate, it’s just a different number, in the end i was right, the numbers in meminfo are more relevant to me in this case.

If I want to see my situation in one command, I use smem :

smem -R 16G -K /boot/vmlinuz-linux-lts -wk
Area                           Used      Cache   Noncache 
firmware/hardware            468.0M          0     468.0M 
kernel image                  12.3M          0      12.3M 
kernel dynamic memory          6.6G       5.6G       1.1G 
userspace memory               3.4G     555.3M       2.8G 
free memory                    5.5G       5.5G          0 

Smem is good, but i feel like the numbers don’t quite add up in it.

see my actual RAM use is at 7.3GB

But smem says 5.9…

Although, at a glance it seems liek i can get an accurate number by adding userspace used memory with kernel noncache memory, it seems like together they match the RAM use… But why isn’t it the noncache value that adds up? Because according to just about everything i’ve read that’s literally what it should, is the noncache/cache number for userspace just inaccurate or something?

According to the manual for free, used is calculated from available as the difference between total and available. available, on the other hand, corresponds to MemAvailable in /proc/meminfo. Now, according to the kernel documentation for /proc/meminfo:

MemAvailable: An estimate of how much memory is available for starting new applications, without swapping. Calculated from MemFree, SReclaimable, the size of the file LRU lists, and the low watermarks in each zone. The estimate takes into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use. The impact of those factors will vary from system to system.

MemAvailable is an estimate, which, by extension, indicate that the value for used returned by free is also an estimate.

I’m not sure how you did the math, but the numbers shown in your screenshot seemed pretty consistent to me. Here is my interpretation of those numbers:

The developers of smem most likely defined used as total - free as opposed to total - available (the latter of which being the definition used in the free command). If you look across the board, the values under the free and buffer/cache columns are consistent between the free and smem commands.

        free    buff/cache
smem    48.4           7.8
free    48.0           7.8

The apparent difference in the outputs of the two programs is the used column.

smem     13.7
free      7.3

This “discrepancy” is most likely due to different definitions being applied to used in both programs like I mentioned earlier. smem defines used as the amount of memory that is “not free” (if you use the values from the free command, total - free = 62 - 48 = 14, which is approximately 13.7) whereas free defines used as an estimate of how much memory is in use after performing additional calculations to estimate the amount of memory that is reclaimable and then subtracting that amount from 14G (the amount of memory that is not free). I do not have any knowledge on the exact algorithm used to perform the estimate, but the kernel documentation suggests that it will “take into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use…” and that the impact of the aforementioned factors will “vary from system to system.”

1 Like

That’s what I don’t like in free, to me, available should be free+buffers/cache.

So what you’re saying is… no matter what we do, the best we can get are estimates?

No wonder this is difficult… But i don’t know, if we assume the memavailable in /proc/meminfo is mostly accurate, it seems like free’s memused should be just as accurate. I guess it depeneds on your definition of used, free and available, free has a pretty clear definition at least, and total, total has a clear definition too, available has a clear definition but it seems like there’s not a whole lot of agreement on how it should be calculated…

I mean if vazicebon is correct, i currently have 57GB available and 5GB used.

If free is right, i have 6GB used and 56GB available

If smem is correct, i have 4.6GB used and 57.5GB free/cached

If I tally the percentage of RAM used according to ps by all applications and estimate RAM used that way, i get 7GB, which is oddly high (I kinda would’ve expected it to match the value given by smem which is based in PSS, but instead it’s right between PSS and RSS)

If that’s correct then I will be running into OOM issues long before smem or free would be making it look like i should :thinking: which now that I think of it did seem to happen on my old laptop, it had 32gb ram but i was running into oom shit at like 24GB use according to free…

I have all these different contradictory numbers, how am i supposed to know which ones to believe?

This is extremely confusing, why is it like this?

Because in practice, estimates are more than enough to do the job. You don’t need to track the memory usage of your system down to the very last byte to know that your system is running out of memory. If applications become less responsive, you might be running low on memory. If swap is being used, you might be running low on memory. If free says that your available memory is approximately 100MiB, then you’re running low on memory. Taking an overly pedantic approach in the tracking of memory usage serves no practical purpose at all (not to mention the fact that you have more than 60 Gigs of RAM installed). Even if you’re trying to diagnose memory leaks, there are other specialized tools for that.

Because memory management, at the lowest level, is not a simple and straight-forward task like you probably expect.

If you really want to know precisely how the Linux kernel manages memory, you can either: A) read the kernel’s source code; or B) reach out to the developers responsible for the kernel’s memory module.

Again, even if you did delve deep into the kernel’s memory management code, I highly doubt that the information you learn will be of any practical value because the way the kernel manages memory might change upon the next release.

Another approach (the more practical one, in my opinion) is to just play around in a VM? Set up a VM, allocate a small amount of RAM (e.g 1GB), install an OS, and then experiment—observe the patterns of change in those numbers as you try to exhaust the system’s memory in the VM.

Yeah, this seems like the best answer to my original question. There have been good ones along the way, I particularly like smem, ps_mem was a bit of a miss tbh, smem seems better.

ps_mem requires root, and only shows you the pss, smem does need root to show pss for all applications in a list when you run it with -tk arg, but if you use -w as non-root it’ll show the same number for pss as it would if you were running smem -tk as root, which is quite useful. smem also shows many other useful things, between free, smem and ps a lot of different estimates can be gathered but there seems to be no consensus to which estimates are the most accurate, so doing as you said, making a vm and just actually testing when you hit OOM seems to be the only way to really confirm it.

Estimates are close enough.

If you want to see a more direct breakdown of each item, bottom right of btop shows what each process is running specifically.