In a unholy Conundrum

Good day ladies and gentlemen. I am a lowly chia farmer that is curious of an issue I am having with my system that kind of has me scratching my head. Firstly my system.
I am using a z840 with dual e5-2699 v4, 512gb 2400 mhz ddr4. Endeavor is installed on a 2 TB samsung sata ssd runs flawlessly but did and still does take a little learning on my part.

So I utilize the z840 for plot creation which is used to farm chia crypto. So using a 3rd party software called gigahorse the system utilizes the nvidia 3060ti I have installed to create the plots within 2 minutes compared to approx. 10 minutes using the cpus. Once created the program sends it to a u.2 ssd that is used as a buffer and sends the plots (approx. 56 gigs per) to the sata 18tb hard drives. I may have anywhere from 1 - 4 of these hard drives in the system depending on the price I can find them at.

And here is my issue. Plot creation takes me around 2 minutes. Once finished it is on the U.2.The u.2 begins to write it to the sata hard drives and simultaneously receiving data of the plot being created. So at any given time I am reading to write to let’s say 2 drives while writing to the u.2.

Using mission center to get a visual representation of what my drives are doing, it appears I am writing to the hard drives in small bursts, instead of big writes. (long continuous writes). I do not know if I am pushing the u.2 too hard or if I have a wonky file scheduler. I see sporadic as needed by the software writes to the u.2 of 1.5 gigs per second, but small blips and bursts of writes to the sata hard drives of 250mbs. I feel like the u.2 has the bandwidth necessary to do the job as I used Ubuntu in a Windows dual boot and it was basically a 1 for 1 exchange on the u. 2 one written to and one moved out to the hard drives. I checkedy scheduler and it shows me deadline, the same that I had on Ubuntu. I have tried different file systems. Previously it was xfs and that was under performing on mint and manjaro. Right now my fs are ext4 for endeavor install with noatime, f2fs on the u.2 (recommendation from another chia nut) and ntfs on the sata hard drives as I use windows to farm (old spare pc I had lying around). All drives are in a netapp.
Any help with the writes to the hard drives would be greatly appreciated as it fills my u.2 which slows down my whole process.
Thank you and thanks for reading my book.

Hi! :partying_face:

Likely scheduler is to blame, try this and see if it helps:

To list currently used schedulers:

cat /sys/class/block/sd*/queue/scheduler

Also you did good on choosing ext4 for storage drives - i’ve seen quite a few of drives with btrfs completely f*cked after simple power outage :clown_face:

Thank you very much for the response. Well I played a little scheduler roulette and added another u.2 into the mix and everything smoothed out .no scheduler for the u.2s . And yes I know NTFS sucks! That is why I am using linux to create plots .Linux 2 minutes , Winderz 47 10 minutes. Ridiculous. Gotta hand it to Bill Gates though . Who else could have gotten people to use a patched together nightmare piece of software , got us all to pay to beta test it . I mean half of the security and av/anti-malware software industry owe their very existence to MS LOL. How well would Endeavor support 600 tb of ntfs drives? If it isn’t a problem winderz may well disappear. Chia is a hobby I do not sit and read whitepapers , strategize this and that etc. I realized linux kicked arse in plotting (dabbled with many flavors in the past). Tried doing a stand alone install of ubuntu which would not install. I downloaded the .iso 4 times tried several usb ISO writing softs and it would not even begin to install. Tried mint -meh , new manjaro -someone was drinking when they created that one. I use to be a fan and stepped away after my laptop died.And now NOPE. Ran across Endeavor and found it appealing and enjoy it so far . Gotta remember how to install this and that etc. Arch was always intimidating as I am not a command line guru . But Endeavor is kicking arse and taking names , not a single issue for me. Thanks again . Have a great day.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.