I have been having an issue for a while now, and decided to come here for help.
I cannot figure out why usb speeds slow to a crawl. I was trying to download Snappy Driver Origin (to get driverpacks to install on a windows machine that is offline) which I have used for probably 2 decades and it is a 40~GB download. When transferring it to the usb drive, the transfer drops to an abysmal <1Mbps transfer speed. Doesnt matter if its 2.0 or 3.0 usb devices or ports.
I have seen this before, MANY times. Thats why Im asking here. I asked once on Manjaro, and lets just say they removed my solution… (I just used windows as I was dual booting at the time, it was a 200GB hard drive copy I was moving via usb)
I’m not bashing manjaro, they did try to help. This issue crops up on every major distro I’ve come across. Doesnt matter the hardware, or the usb drive whether is flash or HDD. They couldn’t replicate the issue, and somehow also did not have any similar equipment to test with. (2.0 or 3.0 usb flash/HDD drives)
When I say slow, I mean hours for a few GBs to actual days if its a few more GBs. I’ve searched elsewhere on google, it pops up everywhere and no one can find the answer. There is a linux mint post here:https://forums.linuxmint.com/viewtopic.php?t=387874 They eventually decide its not a linux issue but busted hardware, despite the user clearing the tests–they do hint that it -may- be a bug in the kernel, which I also suggested in my manjaro forum post. It gets repeated as far back as 2014 on ubuntu forums, so the issue is not new.
THE TEST: Transfer a large amount of data to a usb drive. Doesn’t matter if its a flash or HDD drive. It’s not a cache issue or a “reporting the wrong speed issue” or a thermal issue. My 40GBs of data ran for 5 hours before I unmounted the drive, to find it had wrote about 6GB. Just 6. Its not thousands of small files either, just a few that a 1-3GB in size.
Can anyone else replicate this? I am a newbie when it comes to linux, but i’ve had this problem off and on for as far back as I can remember.
Edit: forgot to mention it seems to occur after 30 seconds to a minute or two of transferring. Sorry for the blogpost but I had to add some backstory.
Having done fairly large transfers to USB drives on Linux, this isn’t a problem of Linux, but more likely a problem of specific hardware, at least from my experiences in the past.
That isn’t to say that it’s impossible that the current kernels might not have a problem, but they worked fine back in the 5. and early 6. eras.
A lot more information is needed. I mean first What DE is this happening in? How are you doing the copy? What other processes are running at the time? What filesystem?
I regually copy large files between the computer and USB thumb drives and hard drives. While the speed is always faster going to another hard drive the thumb drives can take much longer to write.
When ever I copy large files I go straight to the command line. Doing it this way for years and never an issue. Much less on the resources and its faster.
If copying from a Linux Device to a NTFS device can take quite a long time itself. I think this is more because of the way Microsoft does things vs the way linux does things (data storage).
It has spanned back for over a decade that i can recall. Thats the problem, unless someone can see it for themselves its impossible to diagnose.
Any DE, though I mostly use KDE. Copy method doesnt seem to matter, happens in dolphin file manager or through terminal with copy command. What ever other processes exist during runtime, I can’t possibly list that. As far as filesystems go, this is one that seems it might have something to do with it (as a lot of people immediately jump on the NTFS/windows hate wagon,) but it doesnt seem to matter what filesystem. I’ve tried dozens of variables, ext3-4, ntfs, exfat, etc on usb 2.0 and 3.0, flash drive or HDD, to the same avail. Works at first, followed by slowdown, followed by a crawl.
The only OS i tested that worked was FerenOS, and that test even spanned multiple machines both desktop and laptop (from 2009ish to a 2022 dell laptop) I have been thorough doing tests like this before, the problem is getting someone with more knowledge than me to investigate it with similar hardware. The current two pc’s im having an issue with are a ryzen 1800x with x370 asrock board and an ancient inspiron 560 with a broke ethernet adapter. I tried 4 different 128GB 3.0 sticks and a 3.0 HDD before i finally just opened Snappy drivers using WINE (i love WINE ) and downloaded the drivers into a folder and shared the folder over a network, which worked fine of course. Hilariously enough, using a usb to ethernet adapter.. so once again im not sure why the problem affects drives and not a usb to ethernet adapter which functioned up to a continuous 8MBps (100Mbps) which took about a reasonable hour.
I first noticed the issue when creating live iso files on usb drives using linux. I had a 4GB flash drive that i regularly reinstalled windows with, and knew it took a solid 20 minutes to copy over to the drive. But when using linux as a daily driver on another pc, I could never get it to copy and would just pull the drive and reformat it in frustration and use another windows pc to make the live iso again. That was 10 years ago at least, I remember slipstreaming drivers onto the windows iso lol as it wouldnt boot via usb.
I’m resurrecting this post because it hit me again.
I was in the middle of a nightmare situation thinking my pc had begun to kick the bucket, and decided to reformat a usb drive with ventoy on it so that i knew the data wasnt corrupted, and once again hit this slow usb speed issue. I copied over EndeavourOS’s iso file and a few others (archcraft, manjaro, xubuntu, and a couple of necessary boot/repair iso’s) for a collection of about 35GB, and it took about 7 hours to transfer. I actually used the corrupted pc in that forum post initially, but after about an hour thought it would be better to use a more trustworthy machine (a spare laptop I have) based on the circumstances. I went to town and did a few errands and got food, only to come back to see it still transferring.
I’m not sure if i made this clear in the original post (i didn’t reread it since i posted it a while back) but the first few GB’s transfer perfectly fine at the correct speeds. It drops MASSIVELY in speed after that, all the way down to a few hundred kilobytes per second, and the time it takes to complete the entire transfer backs up such speed reports like those listed in KDE’s system tray. It goes up and down at times, but it stays in the 0.5-2MB range, even with usb 3.0 hardware. The most interesting part of this is if you transfer large files, but one at a time (say no more than 1.5GB) they transfer fine and at the correct speed and is easily verified by actually running the iso files or hashing them. The same 35GB of files will transfer in prob about 30-40 minutes depending on how fast you are at copy/pasting the files. It’s not specific to arch either, that transfer was done on a xubuntu install.
The transfer was done with mechanical HDD to usb 3.0 flash drive via a blue 3.0 port, on both computers. Both took hours, and i would’ve done the “small and fast” method but some of those iso files are >5GB.
No light to shed, I can say that I burned a 7 GB ISO this past week and it took (guestimate) 20 minutes to dd. Seemed like it took longer to use dd than to use Ventoy, but that was probably my imagination and impatience.
I copy Gigs of files over to external devices all the time and the only “slowness” i ever get is when writing to a thumb drive. (a limitation of thumbdrives)
It’s almost certainly harware-related. I’ve seen this any number of times on laptops across the years. You already eliminated USB protocol version as the cause. Things to look at:
HDD characteristics. Drive speed bus type, and bus cache will massively affect transfer rates, especially with files spread out over multiple platters. There is a world of difference between a 5000 rpm and a 7500 rpm HDD.
USB stick itself. Is the USB stick rated for USB 3.0? A 3.0 thumb drive is much faster and has a larger cache. If you’re using a cheap USB 2.0 stick in a USB 3.0 slot on your computer, chances are good you will hit the drive’s cache and overwhelm it. The USB memory speed is also a factor, as is the defect rate. Unsurprisingly, cheap USBs tend to use slower memory, have small caches, and have more defects than higher-quality sticks. Lastly, if the stick has been used a lot, the writes will have degraded the memory over time.
Computer data bus speed and cache. Same comment as above. Bus speed and cache speed will affect data transfers.
EXT4 journaling. It writes changes to the journal before it writes data to a file system.
The symptom of high transfer rate, dropping suddenly down to a crawl, strongly suggests a cache is being overwhelmed. Which cache is the question.
Interesting topic, Usb 1.0 to Usb 3.0 . . . quite a difference in speed and performance especially data transfer. Thumb drives are color coded to fit matching Usb ports for optimal transfer. Blue to Blue, Red to Red . . . speed on the bus is determined by what other factors? Memory buffer? I wouldn’t know.
I went through a bit of a research back then as I had a similar issue with thumbdrives, and I suspected my chipset (B550) was to blame. Turns out the thumbdrive was very, very slow, as most of them appear to be.
You say you see the same with HDDs, so your case is probably something else.
You also mentioned you are sure it’s not a caching issue. Do you mean you have already tried the writeback-cache tuning as shown here, for instance?
Nevertheless: I see no mention of the hardware involved. Can you provide the motherboard/laptop model? Also, the types of media you used.
I thought it was CMR to SMR that was causing the problem, then maybe the filesystem types, etc. I already cleared those avenues sadly.
Nah. This is persistent across sticks. I’ve checked to make sure it wasnt a 2.0/3.0 issue. Proper ports and drives.
Works fine under windows, and apparently FerenOS, though when i tested it it was 2 years ago.
Even happens with NTFS to NTFS drives. I tested a few others, I dont remember which now.
True, but a cache wouldnt turn a 30 minute 4gb transfer on 2.0 into a 4-8 hour endeavor, or a 200gb transfer over 3.0 into a few days. Maybe a slowdown, but thats not even a crawl
I do, the original post at the top that links to my issue when I had manjaro was between HDDs. One was an internal SATA CMR drive if i remember correctly, transferring to a usb 3.0 external HDD. But I did different tests and came across the same issue, whether stick or HDD. I never tested SSDs.
I think i did try that at one point, but it didnt help. One significant difference is I never had the system slow to a crawl or lock up, unlike that article you pointed me to. I’m not sure Ive ever experienced what theyre describing to be honest.
I skipped some of the probably irrelevent data. The interal drives are standard old seagate HDDs. I dont have access to the pc i used in the manjaro tests, someone bought it.
(I just used windows as I was dual booting at the time, it was a 200GB hard drive copy I was moving via usb)
) and downloaded the drivers into a folder and shared the folder over a network, which worked fine of course. Hilariously enough, using a usb to ethernet adapter.. so once again im not sure why the problem affects drives and not a usb to ethernet adapter which functioned up to a continuous 8MBps (100Mbps) which took about a reasonable hour.