So here I am with the report from some more fiddling and testing. By chance I went by a “brick and mortar” computer shop and picked up
this tiny little fan

https://www.adafruit.com/product/3368
It looked quite frail but I was curios to see how it performs. It was also very cheap so not a hard decision to make.
Anyways, I have been watching some videos and reading a bit of the documentation about overclocking and the options in the /boot/config.txt. In one of the videos it was said that according to the manufacturer the cortex-a72 could run with as high a clockspeed as 2500 Mhz (something I couldn’t corroborate doing some half-hearted searching). However, he had only succeeded to get the system stable at 2300 Mhz with the following values in the config.txt:
over_voltage=10
arm_freq=2300
Since I didn’t want to set that high of a value for over_voltage (the recommended range from Raspberry being -16 to 8), I went ahead with he following values:
over_voltage=8
arm_freq=2300
The system came up fine after reboot but 15 to 20 seconds into running the stress test, the system choked up and I had to power it down and re-edit the config file to read:
over_voltage=8
arm_freq=2250
This seems to have worked. After reboot and running the test, no sign of choking or freezing. Of course the temperature goes up but given the fact that the default value is set to 85 C before the processor starts throttling to protect itself, so I found the temp reasonable.
I know this is a rather lame set up compared with what some of you guys are using but here is how mine looks like for now:

And here is the part of the /boot/config.txt with my modifications:
# Overclocking
over_voltage=8
arm_freq=2250
gpu_freq=700
####
initial_turbo=60
####
temp_limit=70
Htop showing the CPU usage, clockspeed and temp at idle:

Running the stress test (stress -c 4 ) for 10 minutes:

Now running the Pi at 100% CPU usage clocked at 2300 Mhz doesn’t by far correspond to my current usage.
Since using the Pi is quite new to me, I am in a process of trials and errors to learn its capacities and limitations.
I use cpupower
to set the cpu power scaling:
$ cpupower frequency-info
analyzing CPU 0:
driver: cpufreq-dt
CPUs which run at the same hardware frequency: 0 1 2 3
CPUs which need to have their frequency coordinated by software: 0 1 2 3
maximum transition latency: Cannot determine or is not supported.
hardware limits: 600 MHz - 2.30 GHz
available frequency steps: 600 MHz, 700 MHz, 800 MHz, 900 MHz, 1000 MHz, 1.10 GHz, 1.20 GHz, 1.30 GHz, 1.40 GHz, 1.50 GHz, 1.60 GHz, 1.70 GHz, 1.80 GHz, 1.90 GHz, 2.00 GHz, 2.10 GHz, 2.20 GHz, 2.30 GHz
available cpufreq governors: conservative ondemand userspace powersave performance schedutil
current policy: frequency should be within 600 MHz and 2.30 GHz.
The governor "schedutil" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 700 MHz (asserted by call to kernel)
I monitor constantly htop and I can see that the CPU oscillates between all the above values depending on the load and actually bursts at its max value of 2300 Mhz when I launch an application like Firefox or stream videos.
I have the impression that the system is quite stable. No sign of freezing or lagging. No issues with streaming videos at 1080. The temp is rather stable between 40-55, seldom above 55 and I have yet to see temps beyond 60 for my current usage which consists of browsing, watching videos locally or streaming, listening to music; average, casual daily usage.
With all that said and done, I would prefer to be in the side of the caution and will be getting one of those nice Noctua fans very soon.
Thanks for reading! Please tell me if I have omitted some information for you to get a more complete picture. Looking forward to your comments, advice and suggestions. Any tips and tricks for a smoother usage of my RPi are highly welcome.