Pi 5 generic kernel help

gentoo · February 16, 2025, 1:57am

Tldr:

What I am trying to do → enable hugepages, possibly use the generic kernel instead of the rpi one.

Hardware: pi5 8gb, argon40 M2 case latest model

What I have tried

I started by just sudo pacman -S linux-aarch64 linux-aarch64-headers and this broke my pi in a spectacular way. I have a argon40 nvme case with endeavoros installed to the nvme. I first had to install to a sd card then dd the image to the nvme then remove the SD card. So I have a working setup SD card on me.

I install to nvme. Remove SD card. Install generic kernel, package manager tells me it’s going to remove rpi kernel and headers since those are in conflict. Cool sounds good. I reboot.

This broke everything. Impressively badly.

When I rebooted I got the standard pi bootloader screen as though you don’t have an os installed. Ok fair, I created a unbootable kernel. Let me plug in the SD card since that has a known working system and I’ll chroot and remove the generic kernel and bring back the rpi kernel.

Nope.avi.mkv.exe

Although the pi boot priority is the SD card HAVEING THE M2 PLUGGED IN causes it to hang in the boot process before it loads gpu driver and you continue booting with the smaller font before it gives you a welcome screen.

Basically now I need a USB m.2 or plug the M2 into another PC unless there is a way to make the SD card ignore the M2. Not a super big issue but was surprising I was able to break this so badly. XD.

Ok so generic kernel is out even though I see patches from 6mo ago that added pi 5 boot support.

https://lore.kernel.org/linux-arm-kernel/20240510-overreact-snare-05566341fd37@spud/T/

I’m confused is this not merged? If so is it merged in 6.14rc or Linux-next? I tried to find aur packages for those kernels so I could build and install those but I don’t see a aur package for these.

So now I’m thinking ok maybe I need to compile the rpi kernel with hugepages support since I can’t boot the generic kernel. How would I do that? The rpi kernel is compiled with hugepages configured out. It also has 8 numa nodes which is kind of a problem because I need my numa aware application not to try to allocate 2GB x 8 numa nodes and get killed by the system since it (tried) to allocate 16Gb of ram when I have 8Gb of ram (~7GB free). I either need to figure out how to make that app ignore numa or stop the device from creating 8 fake numa nodes since dmesg says fakenuma=8.

I can read the directions on rpi and adapt them to arch a bit but I’m concerned I’ll not have the .dtbs and boot setup properly again.

Any advise is appreciated

Tldr I’m confused as to if mainline kernel booting support works with pi and how to have pacman not wreck my boot partition when installing the generic kernel OR I need to figure out how to compile the rpi kernel with hugepages on.

Pudge · February 16, 2025, 3:03am

Your title is “Pi 5 generic kernel help”. As far as I know, there is no generic Pi 5 kernel because there are too many RPi modules and patches not included in the aarch64 kernel.

Just out of curiosity, why do you want huge page sizes?
If you use this kernel, you have 16 K page size.

Here is the description of the linux-rpi-16k kernel

Linux kernel and modules (RPi Foundation fork) with 16k pagesize for bcm2712/RPi5 ONLY

the key being “and modules (RPi Foundation fork) with 16k pagesize” meaning you would have to include these RPi Foundation modules and patches in the PKGBUILD to compile a working kernel for RPi 5.

AFAIK you would still have to include these packages
firmware-raspberrypi
uboot-raspberrypi
and possibly others.

Why re-invent the wheel when Archlinux ARM supplies an excellent kernel?

The way I understand it, when you run off a M.2 storage device, argon40 M2 cases requires you to modify the boot loader in the eeprom. That modification changes the boot seek to pretty much the M.2

If you want to go back to uSD cards, you will probably need to get the eeprom back to default.

Pudge

gentoo · February 16, 2025, 3:51am

I want 2Mb and 1GB hugepages for performance with xmrig.

Ok so generic kernel is pretty much a no go then.

I want to just do “CONFIG_HUGETLBFS” on the rpi kernel then since the rpi kernel has that flag either off or set to false. It used to be on / true but that changed in early 2024 and all new kernels they supply have it off or not existing.

Is it just

Pull rpi kernel source
Menu config the “CONFIG_HUGETLBFS” to yes/true or just have that in the compile flags (not sure exactly what it needs)
compile

Or am I overthinking this and could just tell the kernel to use hugepages with a boot param in cmdline.txt or something? That would be significantly easier. Like append CONFIG_HUGETLBFS=true or something?

If I need to custom compile the rpi 16k pages kernel then how would I install it? Just drop the .img file it spits out in /boot/ and change the cmdline.txt to point to that kernel?

gentoo · February 16, 2025, 6:15am

pkgname=rpi5-kernel-hugepages
   pkgver=6.13.2 # Replace with the latest kernel version
   pkgrel=1
   pkgdesc="Raspberry Pi 5 kernel with CONFIG_HUGETLBFS enabled"
   arch=('aarch64')
   url="https://github.com/raspberrypi/linux"
   license=('GPL2')
   makedepends=('git' 'bc' 'bison' 'flex' 'make')
   source=("git+https://github.com/raspberrypi/linux.git#branch=rpi-6.13.y")
   sha256sums=('SKIP')

   prepare() {
       cd "$srcdir/linux"
       # Use the default Raspberry Pi 5 configuration
       make bcm2712_defconfig
       # Enable CONFIG_HUGETLBFS
       sed -i 's/# CONFIG_HUGETLBFS is not set/CONFIG_HUGETLBFS=y/' .config
   }

   build() {
       cd "$srcdir/linux"
       make -j$(nproc) Image modules dtbs
   }

   package() {
       cd "$srcdir/linux"
       # Install kernel image
       install -Dm644 arch/arm64/boot/Image "$pkgdir/boot/kernel8.img"
       # Install device tree blobs
       install -Dm644 arch/arm64/boot/dts/broadcom/*.dtb -t "$pkgdir/boot/"
       install -Dm644 arch/arm64/boot/dts/overlays/*.dtb* -t "$pkgdir/boot/overlays/"
       install -Dm644 arch/arm64/boot/dts/overlays/README -t "$pkgdir/boot/overlays/"
       # Install kernel modules
       make INSTALL_MOD_PATH="$pkgdir/usr" modules_install
   }

This brrr’ed for an hour compiling but failed to install since all the dtb files exist in /boot already.

I tried just copying over the kernel8.img and recursively copying over the boot folder it compiled witb the dtb files in it.

It… Booted… Kinda.

Then it hung on “reached graphical target wants” cursor blinking.

It didnt boot far enough for ssh so i cant check if this worked.

Rip. I tried to get the example pkgbuild from archlinuxarm but their webserver has 500 errors as an example so I asked an LLM.

ricklinux · February 16, 2025, 8:06pm

As far as i know you’re going to have to use a kernel parameter by modifying your boot configuration. You also need to enable huge pages.

Edit: I would be trying this on the kernel supplied.

gentoo · February 16, 2025, 11:58pm

This kernel param is ignored on cmdline.txt

I got hugepages working but now i am trying to disable fakenuma=8 since i want to allocate hugepages but not bind them to numa nodes. idk who decided to give 2 numa nodes per cpu to iMpRoVe pErFoRmAnCe https://pcper.com/2024/07/numa-arrives-on-raspberry-pi-5/ but it doesnt for my usecase since my app will either bind to ALL numa nodes 2GB dataset (causing OOM since 8 numa x 2GB = 16GB and my pi only has 8GB of ram) or ignore numa and the kernel will bind it for me to the first numa node / preferred numa node giving me 1/8th memory bandwidth. I do not want fake numa.

Here is how i eventually got the kernel compiled with the configuration i wanted.

Pull the actual kernel PKGBUILD, git clone https://github.com/archlinuxarm/PKGBUILDs.git
cd PKGBUILDs/core/linux-rpi
nano PKGBUILD and add # Enable CONFIG_HUGETLBFS sed -i 's/# CONFIG_HUGETLBFS is not set/CONFIG_HUGETLBFS=y/' .config
in the build section before make -s kernelrelease > version
Compile it for 2 hours and install

This worked great because /proc/meminfo showed hugepages working. And i djdnt have boot issues!

I tries to cat config8 | grep -i numa and add those as =n but they were overridden somehow on compile, likely by config8

I edited config8 directly and we will see if this works. It told me do not edit but im not sure how to get the PKGBUILD to override config8 when something is set already. I may even be in the wrong place because i want to turn fakenuma off but none of the configs in config8 had fake and numa in the name. So idk.

It’s really annoying since a compile of the kernel takes ~ 2 hours and then check and see its still enabling numa. Idont want numa OR i want one numa node. I dont want fake numa that artificially split up processes so they to more banks since I need each 2GB dataset to have acsess to all banks. Artificially splitting them up wrecks my performance or makes what im trying to do impossible since i dont have 24GB of ram to supply each numa node with the dataset.

Anyway i promise ive tried basic stuff like

numa=off in cmdline.txt but it doesnt work since its set by the kernel as default on, then off doesnr overrride it being on or delete the fake nodes.

gentoo · February 17, 2025, 2:13am

Ok i figured it out, sort of. Couldnt remove the numa crap from the kernel no matter what i did. If anyone knows please let me know.

The eeprom was exposing the fake nodes to the kernel though and if i could get it to not do that then it would work.

Im currently the world record holder for pi 5 xmrig https://xmrig.com/benchmark/6ngFxx

How to enable hugepages and limit numa to 1

1. Pull the actual kernel PKGBUILD, git clone https://github.com/archlinuxarm/PKGBUILDs.git
cd PKGBUILDs/core/linux-rpi
nano PKGBUILD and add # Enable CONFIG_HUGETLBFS sed -i 's/# CONFIG_HUGETLBFS is not set/CONFIG_HUGETLBFS=y/' .config
in the build section before make -s kernelrelease > version
Compile it for 2 hours and install with makepkg - si
yay -S rpi-eeprom-git
git pull https://github.com/raspberrypi/rpi-eeprom
Copy the rpi-eeprom-digest into /usr/bin and chmod +x it
sudo rpi-eeprom-config -e
Add SDRAM_BANKLOW=4, save and reboot
1 numa node + huge pages enabled.

Pudge · February 17, 2025, 8:00pm

Thank you for your feedback on the solution. Good luck with your xmrig.

Pudge

system · February 19, 2025, 8:01pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.