Poor transfer rate despite 10g NIC

General system Newbie
1

I am trying to transfer data to my server (Unraid) with my desktop (Endeavour OS) using 10g NIC. I do this via NFS shares. But from desktop to server I get a maximum of 270 MiB/s and from server to desktop a maximum of 500 MiB/s. A test with Iperf shows ~9.5Gbits/s. I can exclude problems with the network card, cabling or switch, since I have already replaced all components at least once for testing.
I copy from NVME (Samsung Evo Plus 970 250GB) to NVME (Samsung Evo Plus 970 2TB) and vice versa.

Unfortunately, those are my only two devices that have a 10g NIC. Therefore, I can’t say for sure if it’s actually the desktop PC or the server that’s causing problems. Since I am still a newbie it is hard for me to find a starting point apart from exchanging the hardware. Is there anything on the desktop side that I can check?

Directly after a data transfer, sudo dmesg | grep atlantic gives me the following output on my Endeavour OS desktop:

[    4.017648] atlantic 0000:03:00.0: enabling device (0000 -> 0002)
[    4.526373] atlantic 0000:03:00.0 enp3s0: renamed from eth0
[   11.799253] atlantic 0000:03:00.0 enp3s0: atlantic: link change old 0 new 10000
[ 1460.386194] atlantic 0000:03:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1460.386197] atlantic 0000:03:00.0:   device [1d6a:07b1] error status/mask=00000041/0000a000
[ 1460.386201] atlantic 0000:03:00.0:    [ 0] RxErr                  (First)
[ 1460.386204] atlantic 0000:03:00.0:    [ 6] BadTLP                
[ 1534.282376] atlantic 0000:03:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 1534.282380] atlantic 0000:03:00.0:   device [1d6a:07b1] error status/mask=00000041/0000a000
[ 1534.282383] atlantic 0000:03:00.0:    [ 0] RxErr                  (First)
[ 1534.282386] atlantic 0000:03:00.0:    [ 6] BadTLP

Client inxi -Fxxx output:

System:
  Host: sin Kernel: 6.3.5-zen2-1-zen arch: x86_64 bits: 64 compiler: gcc
    v: 13.1.1 Desktop: Qtile v: 0.22.1 vt: 7 dm: LightDM v: 1.32.0
    Distro: EndeavourOS base: Arch Linux
Machine:
  Type: Desktop Mobo: BIOSTAR model: Z690GTA serial: <superuser required>
    UEFI: American Megatrends LLC. v: 5.24 date: 03/10/2022
CPU:
  Info: 12-core (8-mt/4-st) model: 12th Gen Intel Core i7-12700KF bits: 64
    type: MST AMCP smt: enabled arch: Alder Lake rev: 2 cache: L1: 1024 KiB
    L2: 12 MiB L3: 25 MiB
  Speed (MHz): avg: 3219 high: 3600 min/max: 800/8500 cores: 1: 3600 2: 3600
    3: 3600 4: 3600 5: 1056 6: 3600 7: 1100 8: 1042 9: 3600 10: 3600 11: 3600
    12: 3600 13: 3600 14: 3600 15: 3600 16: 3600 17: 3600 18: 3600 19: 3600
    20: 3600 bogomips: 144383
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3
Graphics:
  Device-1: NVIDIA GA104 [GeForce RTX 3070 Ti] vendor: Palit Microsystems
    driver: nvidia v: 530.41.03 arch: Ampere pcie: speed: 2.5 GT/s lanes: 16
    ports: active: none off: DP-1 empty: DP-2,DP-3,HDMI-A-1 bus-ID: 01:00.0
    chip-ID: 10de:2482 class-ID: 0300
  Display: x11 server: X.Org v: 21.1.8 compositor: Picom v: git-b700a
    driver: X: loaded: nvidia gpu: nvidia,nvidia-nvswitch display-ID: :0
    screens: 1
  Screen-1: 0 s-res: 1920x1080 s-dpi: 90 s-size: 542x301mm (21.34x11.85")
    s-diag: 620mm (24.41")
  Monitor-1: DP-1 mapped: DP-0 note: disabled model: VG259QM
    serial: M3LMQS288239 res: 1920x1080 hz: 280 dpi: 90
    size: 544x303mm (21.42x11.93") diag: 623mm (24.5") modes: max: 1920x1080
    min: 640x480
  API: OpenGL v: 4.6.0 NVIDIA 530.41.03 renderer: NVIDIA GeForce RTX 3070
    Ti/PCIe/SSE2 direct-render: Yes
Audio:
  Device-1: Intel Alder Lake-S HD Audio vendor: Biostar Microtech Intl Corp
    driver: snd_hda_intel v: kernel bus-ID: 00:1f.3 chip-ID: 8086:7ad0
    class-ID: 0403
  Device-2: NVIDIA GA104 High Definition Audio vendor: Palit Microsystems
    driver: snd_hda_intel v: kernel pcie: speed: 16 GT/s lanes: 16
    bus-ID: 01:00.1 chip-ID: 10de:228b class-ID: 0403
  Device-3: Creative Sound Blaster GC7
    driver: cdc_acm,hid-generic,snd-usb-audio,usbhid type: USB rev: 2.0
    speed: 480 Mb/s lanes: 1 bus-ID: 1-3.2:4 chip-ID: 041e:3271 class-ID: 0102
    serial: 744AD220DDF30288
  API: ALSA v: k6.3.5-zen2-1-zen status: kernel-api
  Server-1: PipeWire v: 0.3.71 status: active with: 1: pipewire-pulse
    status: active 2: wireplumber status: active 3: pipewire-alsa type: plugin
    4: pw-jack type: plugin
Network:
  Device-1: Aquantia AQC107 NBase-T/IEEE 802.3bz Ethernet [AQtion]
    driver: atlantic v: kernel pcie: speed: 8 GT/s lanes: 4 port: N/A
    bus-ID: 03:00.0 chip-ID: 1d6a:07b1 class-ID: 0200 temp: 52.0 C
  IF: enp3s0 state: up speed: 10000 Mbps duplex: full mac: 78:8c:b5:91:54:f3
Drives:
  Local Storage: total: 2.5 TiB used: 1.14 TiB (45.6%)
  ID-1: /dev/nvme0n1 vendor: Western Digital model: WDS500G1X0E-00AFY0
    size: 465.76 GiB speed: 63.2 Gb/s lanes: 4 tech: SSD serial: 215171447012
    fw-rev: 614600WD temp: 40.9 C scheme: GPT
  ID-2: /dev/nvme1n1 vendor: Samsung model: SSD 970 EVO Plus 250GB
    size: 232.89 GiB speed: 31.6 Gb/s lanes: 4 tech: SSD serial: S4EUNM0R134102M
    fw-rev: 2B2QEXM7 temp: 41.9 C scheme: GPT
  ID-3: /dev/sda vendor: Western Digital model: WD10JPVT-24A1YT0
    size: 931.51 GiB speed: 3.0 Gb/s tech: HDD rpm: 5400 serial: WD-WX81A72D5230
    fw-rev: 1A01 scheme: GPT
  ID-4: /dev/sdb vendor: HGST (Hitachi) model: HTS721010A9E630
    size: 931.51 GiB speed: 6.0 Gb/s tech: HDD rpm: 7200 serial: JR100X6P3JT8XE
    fw-rev: A3J0 scheme: GPT
Partition:
  ID-1: / size: 464.78 GiB used: 20.7 GiB (4.5%) fs: btrfs dev: /dev/nvme0n1p2
  ID-2: /boot/efi size: 998 MiB used: 576 KiB (0.1%) fs: vfat
    dev: /dev/nvme0n1p1
  ID-3: /home size: 464.78 GiB used: 20.7 GiB (4.5%) fs: btrfs
    dev: /dev/nvme0n1p2
  ID-4: /var/log size: 464.78 GiB used: 20.7 GiB (4.5%) fs: btrfs
    dev: /dev/nvme0n1p2
Swap:
  ID-1: swap-1 type: zram size: 11.72 GiB used: 256 KiB (0.0%) priority: 100
    dev: /dev/zram0
Sensors:
  System Temperatures: cpu: 28.0 C mobo: N/A gpu: nvidia temp: 48 C
  Fan Speeds (RPM): N/A gpu: nvidia fan: 0%
Info:
  Processes: 388 Uptime: 14m wakeups: 0 Memory: available: 62.64 GiB
  used: 2.5 GiB (4.0%) Init: systemd v: 253 default: graphical Compilers:
  gcc: 13.1.1 Packages: pm: pacman pkgs: 1172 Shell: Bash v: 5.1.16
  running-in: xfce4-terminal inxi: 3.3.27

As indicated, I have already replaced some hardware. I had in the desktop PC first an Intel X520-DA1, then Mellanox-X3 and now the Aquantia card, have all cables (Cat.7 and Cat.8) in the chain 3 times exchanged and the Zyxel XGS1250-12 against a Miktrotik CRS305-1G-4S+IN exchanged. So I can actually be sure here that it can not be due to the devices.

2

I have the same Ethernet card. (chip) Are you using Ethernet for your internet? Are you plugged into a 1 GB port or the 10 GB port on the Router? This board supports 2.5 GbE LAN. In order to get more than 1 GB speed you need to be plugged into the 10 GB port on the router and you also need to use a CAT 6A Ethernet cable. Not sure if the transfer issue is due to something else in your set up but just wanted to make sure you understood this first.

Edit: Sorry i didn’t notice you stated that you had CAT 7 and 8 cables.

Edit: Category 7 is not an actual IEEE cabling standard, but a proprietary design that does not have an official blessing from the networking industry. It does not use RJ45 connectors, but a proprietary connector. Without official IEEE or EIA standardization, any “Category 7” cabling you purchase will be different from manufacturer to manufacturer.

CAT 8 is official successor to Cat 6A

1 Like
3

the desktop (10g port) is connected to the Mikrotik CRS305-1G-4S+IN with a Cat.8 cable of 5m length. The Mikrotik 10g SFP+ port contains a 10g RJ 45 module. The Mikrotik switch is connected to the Unraid server via a 10g DAC cable (0,5m). The Cat.8 cable was replaced 3 times, the DAC cable 2 times. Both NICs have been replaced several times. As I said, I have tried a number of NICs.
Instead of the Mikrotik I had also tried the connection via the 10g ports of the Zyxel XGS1250-12 and directly via the 10g Mellanox ports of my Ipfire router. Also I replaced the 10g RJ 45 module.

4

I’m not familair with Mikrotik CRS305-1G-4S+IN or Unraid server via a 10g DAC cable

Edi: Don’t get me wrong. I know what it is, I just have no experience using that hardware.

Edit2: Also have no experience with ipfire router.

5

Yes, I honestly didn’t assume that :slightly_smiling_face:. It would be too good if someone could be found who knows exactly the combination of hardware and software :slightly_smiling_face:. But I did not have this idea with my post. I would like to try to rule out a problem with the desktop pc which is running Endeavour OS. But since I’m a newbie as I said, I do not know where I can start.

6

What is your normal Internet speed for downloads and uploads?

Screenshot_20230606_113201
Screenshot_20230606_113201701×134 12.6 KB

7

First, it appears your BIOS is not the most current version. It’s up to you if you want to update that. https://www.biostar.com.tw/app/en/mb/introduction.php?S_ID=1038#download

I found there were some “optimization recommendations” for “Aquantia AQtion AQC107” for “maximum performance”: https://sebvance.medium.com/aquantia-aqtion-aqc107-advanced-driver-properties-tweaks-for-maximum-performance-d005b5786236

Since we’re in Linux, right clicking the device in Device Manager to access the advanced driver settings isn’t possible. So it appears these settings can be accessed/adjusted with the ‘ethtool’ command: https://www.kernel.org/doc/html/latest/networking/device_drivers/ethernet/aquantia/atlantic.html

These may get you started:

# View your adapter settings with ethtool <ethX>
ethtool enp3s0

# Vew your adapter information with ethtool -i <ethX>
ethtool -i enp3s0

# Disable Interrupt coalescing (as mentioned in the blog post linked above at medium.com) use:
# ethtool -C <ethX> tx-usecs 0 rx-usecs 0 tx-max-frames 1 tx-max-frames 1
ethtool -C enp3s0 tx-usecs 0 rx-usecs 0 tx-max-frames 1 tx-max-frames 1

There seems to be a lot of other interesting tinkery you could do with ethtool to perhaps further investigate the issue. I also see the NIC has a temperature sensor on it. Hopefully, the NIC overheating isn’t causing stability issues? :confused:

8

531 Mbit/s download and 111 upload. Which corresponds to what I booked with the ISP.

9

I updated the bios. Have also performed the adjustments with Ethtool unfortunately no success. :confused:

Have also tied with cable ties a Noctua fan on the Nic. :sweat_smile:

1 Like
10

A few more random thoughts:

What is your IP configuration for your LAN and what is the IP configuration for your desktop to server NIC connection? The point is to make sure that (especially from desktop to server) the traffic is definitely being routed over the 10g route. IMHO, it would make best sense to ensure that this direct desktop to server 10g subnet is DIFFERENT than the main LAN subnet. That makes it easier to ensure the traffic isn’t getting randomly shunted out the main (slower) LAN port(s). Further, I would double check if these interfaces are configured for BOTH the same ip4 AS WELL AS ip6 subnet. For funsies, you could disable one or the other for testing purposes. If configured properly, they should be ok both enabled, and you can specify which one is default. Lastly, maybe put an entry in /etc/hosts for your server on your desktop, and vice-versa on the server for your desktop. I’m not sure what protocols you’re using to test speeds, but if it’s file-share based, hostname resolution could be important.

11

One more thought (and it’s WAY out there), you might try putting a crossover cable connection between your desktop and server. I know that Ethernet protocol standards have evolved quite a bit over the past 20 years including fanciness in the link negotiation so that crossover cables have become pretty irrelevant. I’m just wondering if MAYBE the auto-negotiation of having 2 like devices directly connected with a straight-through cable (PC NIC to PC NIC in this case) isn’t sufficing in the sense that these higher frequencies/voltages of 10g are somehow still creating some crosstalk/EFI. It should be easy enough to procure or even make a Cat6a crossover. I’m not sure about Cat8 (theoretically should be do-able, just more expensive I’m guessing). The insulation in the cables should be sufficient for this to not actually be an issue, but if you run out of ideas and get bored, maybe give that a whack. :wink: