I learned something about the AM5 platform today

Lounge EndeavourOS pub
1

My main workstation has a failing mainboard. The cost to replace it is far too high for a 5 year old setup so I am going be replacing the majority of it. Because of this, I have been researching what to replace it with.

I was reading tons of reviews on various boards from multiple manufacturers and kept seeing complaints about speeds of the m.2 slots when using certain slots. I started digging deeper and learned something interesting.

There are only 4 PCIe Gen4 lanes connecting the chipset to the CPU on the AM5 platform. That means that everything hanging off the chipset is sharing that bandwidth. That shared bandwidth will throttle a fast Gen4 nvme drive. To make matters worse, many of these boards have 3 M.2 slots hanging off the chipset. If those slots are used at the same time it will kill performance.

Normally, when something is sharing bandwidth this is disclosed in the documentation. i.e. PCIe_4 shares bandwidth with Sata 1+2 so those ports will be disabled when it is in use. However, in this case, since it is a chipset limitation, it isnā€™t disclosed in any meaningful way.

I guess there a few learnings here:

  • If you have a fast nvme Gen4 drive, ensure it is connected to an M.2 slot that is connected to the CPU. Unfortunately, in most AM5 boards this is only the Gen5 slot.
  • If you are planning to use multiple fast nvme drives consider the boards topology.
  • You may need to look into hanging your nvme drives off a card in a PCIe slot. However, this opens a whole separate can of worms.
9 Likes
2

my first reaction was ā€˜so much for QCā€™ and kind a lamebrained engineering shortcut & executive decision that seems disingenuous not to disclose.

sata 1 and 2 donā€™t need to be sharing like thatā€¦holy dracula.
seems like AM5 niche or ā€˜general purposeā€™ board when I read that. I can think of about a dozen scenarios (nas, server, etc) where choke points like that will render it unusable.

what did you conclude, as far as board, was more in line with your needs?

3

That was just a random example. I was describing how things are typically disclosed. However, I did see one board that disabled SATA ports when one of PCIe slots was in use.

That really isnā€™t the problem. Almost all boards with all chipsets have some limitations. There are only so many lanes available to the platform.

The problem is that this is an implicit limitation in the platform itself that isnā€™t widely discussed for some reason.

Still working on it. All I have determined so far is that there isnā€™t one that meets my full needs so I will need to decide where I want to compromise.

1 Like
4

I almost have a phobia of brand new hardware as I harbor a belief (right or wrong) that stuff is built to last less and less.
As far as lanes, I get it. Where you need to compromise is the sticking point.

2 Likes
5

Thanks for sharing Dalto. I am in the market for a new mobo too and planned to use four m2 drives in a RAID6 and was looking for the fastest that I could afford. It appears AM5 architecture would only compound speed issues associated with RAID6. Even if I buy a dedicated pcie discrete raid controller, iā€™d be sharing bandwidth with any other pcie16 device such as a good gaming video card, which I also planned. this was a bad move with AMDā€™s latest tech. I have an I9 running on an older 600 series chipset and it runs like a raped ape! Maybe I should just stay in the Intel playground until next time. Thanks again, brother!

6

I thought that gen5 slot is backward compatible with gen4 drive - so no problem there except for the unused potential. Or is there some performance hit as well?

I would argue that ā€œmainstreamā€ computers may be equiped with only one drive (or 2 - one to cpu and one to chipset) and therefore you would not notice it. I might be completely worng.

Is that fast or slow? I lack the experience in this fieldā€¦ :rofl:

1 Like
7

well, thats basically the case on ALL boards and it is not widely discussed because it is common knowledge on people that build systems at least every 2-3 years. PCIe lanes are extremely limited and have been for as long as I can remember.

At least on AM5 with an ā€œEā€ - chipset like X670E, one can now actually use 2-3 full speed PCIe 4 m.2 SSDā€™s on the board without needing extension cards or limiting other slots on the board.
With AM4, only 1 full speed PCIe 4 m.2 SSD was possible, everything else limited the GPU speed xD

On Intel, same restrictions apply, but I havenā€™t taken closer looks as there are other reasons why Intel is no option in my builds.

Personally, I have an X670E board and have 2 m.2 SSDā€™s on it, and even there I feel like that is something that most people wonā€™t do. I am kind of an enthusiast when it comes to hardware.

AM* is neither designed for servers nor is it designed for NAS. For servers, SP* and TR* are the platforms designed for that. For NAS, it highly depends on the design of the NAS, could be either SP*, TR* or an embedded platform.

2 Likes
8

Contemporary nvme / PCIe lanes - is architectural cringeā€¦and not without itā€™s reasons, but whole thing is kinda glue taped to work :rofl:

honka_memes-128px-50

9

There are trade-offs over there as well. The current Intel CPUs have twice as many Gen4 lanes connecting the CPU to the chipset but only have 16 Gen5 lanes in total. For comparison, the AMD CPUs have 24 Gen5 lanes.

The important part is to understand the limitations of each setup and make your choices accordingly.

That isnā€™t a problem it is a recommended solution :slight_smile:

Even with only a single drive on the chipset, if it is fast enough, you will see some throttling. Also, if you donā€™t know about this issue, there is no guarantee you wouldnā€™t use two slots that are both connected to the chipset and not use the one connected to the CPU.

The difference is that as devices are getting faster, it is becoming easier to break these limitations.

But the connection to the chipset from the CPU is still limited to 4 lanes. No matter how many lanes you hang off that chipset, they are bottled up into those 4 lanes connecting to the CPU.

Most of the AM5 boards(regardless of chipset) only have a single M.2 slot tied to the CPU and the rest hanging off the chipset. That means that all those devices hanging off the chipset are bottlenecked down to 4 lanes so if you put even a single fast Gen4 drive, in practice it loses a percentage of itā€™s performance. If you fill all 3-4 M.2 slots it may be a lot worse depending on how you use those drives.

That means that basically most AM5 boards only allow one Gen5 M.2 slot to run at a full speed and a second drive to run at near full Gen4 speed. There are exceptions, some boards have a second M.2 slot hanging off the CPU. Some even have have a 3rd slot on the CPU that steals bandwidth from the main PCIe slot.

Also, you need to understand the performance characteristics of the device in question. Most Gen4 drives arenā€™t that fast. Only a handful of the fastest Gen4 drives will be impacted by that in a single drive scenario. Even then the impact wouldnā€™t be all that much. The bigger problem is when you want to do something like use multiple M.2 slots off the chipsets in a scenario where all the drives are accessed in parallel.

To be clear, I am not trying to criticize AMD or the AM5 platform. All platforms have limitations. I do, however, think that these things should be more clearly disclosed by the board partners in their marketing materials and documentation.

My goal with this post was only to spread awareness. I think it could help for those who were planning to use all of the available slots for their application or who need to know which slots to prioritize when building their system.

2 Likes
10

Iā€™m sitting on a 2015 i7-6700k 4Ghz CPU running on a Gigabyte GA-Z170X-UD3. 32GB of Corsair DDR4 memory and a handful of Samsung EVO SSDs. Iā€™ve been looking at AM5 as the next step for another decades worth of computing, and potentially my last ever build. Researching boards and builds has clearly shown AM5 perhaps needs a little more time for manufacturers to smoothen out the rough edges.

AMD are literally releasing two more CPUs for AM4 next year, so thereā€™s a rock-solid platform out there at great prices for those who may not want to build on Gen-5. Iā€™m in two minds about which way to go, quietly Iā€™m hoping my current rig can last a couple more years. The 6700XT has been a revelation as far as GPUs go. Flawless for games and overall stability on both X and Wayland.

4 Likes
11

That is a good argument. My current motherboard has 1 M.2 slot to CPU, 1 to chipset + 4 SATA. There is no mention that 4 SATA running at the same time may interfere with the performance of that second M.2 drive.
And honestly I would not notice it anyway unless the drive slows down to HDD performace. :rofl:

It is easier to get the information about how many RGB the board support than about anything more technical. Even the physical dimension of my bord is listed as 170x170 (not even the vertical dimension, cooler size compatibility - a drawing with dimensions would be nice).

But beggars cannot be chosers. I am lucky that there was at least some option for purchase (it it mini-ITX since I wanted some smaller PC).

1 Like
12

Depending on AM4 mainboard.

  • B550 supports only one M.2 nvme with PCIe 4.0.
  • X570 supports two M.2 nvme.
13

You are missing the point. No matter how many the board supports, the ones hanging off the chipset wonā€™t run at full speed.

So in the case of x570, the second and/or third nvme drive will not be capable of full speed because of the bottleneck between the CPU and the chipset.

The only way that wouldnā€™t be true is if the second nvme slot was stealing bandwidth from the primary PCIe slot. However, that would mean a reduction in performance of that PCIe slot when the nvme drive was in use.

2 Likes
14

I believe that a unidirectional read or write speed limit on AM4 X570 in the test result of RAID0 would be ~15000 MB/s maximum.
Therefore AM4 could handle two nvme drives in RAID0 at full speed.

15

That benchmark looks like it is from bifurcating the PCIe slot directly connected to the CPU. That traffic isnā€™t passing over the chipset uplink which is where the throttling occurs.

16

I checked the manual of my future B550 motherboard, the first M.2 SSD use the PCIe 4.0 x4 lanes, the second one use PCIe 3.0 x4 lanes wich means half the bandwith, but it also share it with 2 SATA ports (disabled when M.2 SSD is in use).

Donā€™t know if PCIe 3.0 x4 + 2 SATA6 = PCIe 4.0 x4

17

That seems fine to me. Donā€™t forget that it is likely some of your USB ports are also sharing that same bandwidth. Of course, how often do you use full bandwidth from your SATA SSDs, your fast nvme drive and all your USB ports at the same time. Probably never unless you have a crazy specific use case.

18

Iā€™m confused after comparing two different details: X570 vs B550.

Any B550 mainboard only support one native nvme PCIe 4.0 that connects directly to CPU, this nvme does not use the chipset by design:

image
image1901Ɨ1062 135 KB

Any X570 mainboard supports two native nvme PCIE4.0 which connect directly to CPU by design:

image
image1809Ɨ1125 143 KB

19

I think that diagram is misleading. I think it is listing the same thing in 2 different boxes. As far as I know, there are 24 PCIe lanes off those CPUs. 4 of those lanes go to the chipset.

Generally, that leaves 16x for one slot and 4x more for something else that will usually be used for an nvme socket. Of course, some manufacturers will do other things.

But I donā€™t think X570 gives you more lanes off the CPU than B550. I am not an AMD board expert though so someone else may have a different perspective.

1 Like