Grsync works but it gets slower the longer it runs?

General system Applications
1

Hi,

I have a Grsync task between 2x ntfs drives

Both 12TB Seagate Iron Wolfs with about 9TB of data on them

1 local

1 remote

The remote was empty when I started the sync

The sync has crashed about 4 times so far-reboot the remote PC and start over and it picks up where it left off and goes until it crashes again.

When I started the sync it said it would take approx. 8 hours (this is about what I experienced on my Windows PC’s setup before I switch to Linux 8~11hours or so.

After a bit I noticed it was up to 20 some hours and I thought weird, never took that long before-I watched it and was still copying so let it be.

That rose to 40 some hours, then to 70 then to 80 I was up to about 182 hours when I notice it had stopped copying.

I stopped the copy using the stop button in Grsync.

Everything on the source PC seemed fine.

I went to the remote PC and checked it and it was not responsive-I had to use the power button to shut it down.

This only happens when copying-I ran the PC for days serving up media on the same drive and it was fine.I decided to format the remote drive because it was really out of sync with the source due to not having a backup process for a few months–figured I’d just start over and have a completely sycn’d drive once done.

I have not yet looked into this much but thought I’d ask for any hints/tips/direction anyone with experience with Grsycn/rsync

I started the process again today. About 2.5hrs ago.

When I started it, it said remaining time was 1.39hrs.

I look not it says remaining time is 33hrs.

I’ll grab the args when it crash’s.

BUT, if memory serves– delete on destination, show transfer progress and windows compatibility, on advanced which I cannot see right now) I believe i have hard/soft links being copied as such, compression and that’s about it. After the initial copy is done I am planning on enabling checksums and possibly skip newer (not sure about skip newer-only I maintain this drive so not sure if this is necessary-no one else will copy any “older” files to it)

Thanks All.

2

I don’t know anything about Grsync but why not just use rsync to do the transfer. Its a Windows format so it will be naturally slower than if you were going to a native linux format.

something like
rsync -avuhP ./Path/To/Copy /Path/To/Copy/To

1 Like
3

Have you looked at unison? I used it extensively when syncing data on my dual-boot systems, mostly btrfs < – > ntfs.

1 Like
4

Well.

1 thing is, I now know that the time remaining and completed, are in minutes not hours…phew and…sigh…

So by the time it is done, it will have taken about as long as it would take in a windows environment. About 8-12 hours depending on how busy the network is.

I also ran another eos-update between crashes and there were more updates to install and this time, the Grsync backup has been running since 8AM or so today and it has not crashed. It was only lasting 2-4 hours and it would crash before that last update.

I am hopeful it will not crash again.

I’m a GUI driven ex-Windows user so that’s the answer to you question.

I AM doing a LOT more at the terminal than I ever did in Windows though, so that’s a good thing.

Who knows, maybe in a year or five when I’ve learned a lot more about Linux I’ll be giving the noobr’s the same advice as you.

BUT, for now, I AM the noobr :slight_smile:

6

I have not but tyvm for the heads up.

I will look at it for sure.

I have a dual boot.

However, I am edging closer to NOT having one than having one.

Or, at last having another OS but another Linux OS.

Currently, Windows is on that drive and I haven’t booted into it for a while now.

7

I just recently got rid of mine, too, after figuring out that my bad gaming performance in Linux was due to a memory clock setting mistake in BIOS.

8

Yeah I always hated coping to a windows formatted drive. I can copy the same thing in about a third the time over to ext4.

1 Like
9

grsync/rsync just a fancy bulk file mover. it’s faster than sudo cp copying but not by a hell of a lot.

furthermore, grsync/rsync has never done me wrong.

those are some gigantic TB numbers you are posting. I use grsync to back my data….like a folder at a time / few GBs at a time.

TLDR: this tool has always been flawless for me but I don’t throw too much at it. Incremental.

2 cents

1 Like
10

have you check status remote pc for the disk ?

11

I am not sure which status you are referring to Stephane?

BUT, the answer is yes. I checked the drive and it isn’t hot or throwing off any sata errors or disk r/w errors.

The drives are identical (Seagate Iron Wolf 12TB each @ 7200RPM).

Both are the exact data usage. 8.8TB currently, with 3.3TB available.

I believe the crashes were caused because I selected delete at destination (mirror disk) for my first few sync attempts and at that point BOTH drives had files on them but one of them had already been Jellyfin-afied…Meaning Jellfyfin had put all the meta data it is supposed to put in each videos folder and they were not on the source drive so it was deleting the files that were on the destination drive but not on the source drive and Jellfyfin was syncing them back just as fast so–after an hour or so of this = crash.

At first I thought the update resolved this crash issue but, in hindsight, probably shutting down the Jellyfin server had more to do with resolving this.

I ahve a post in the Jellyfin forums asking if we can have custom folders for this crap. I mean WHY, why do all media servers do this? Download the cover.jpeg, then thumb.jpeg, and whatever else and put it in a metadata/videos/videofolder1, videofolder2, videofolder3 etc directory and have a db or an xml file to link them.

The directories get SO messy the way it is now and if I want to have a mirrored drive I have to shut down the server when I am uploading to it?

Incremental backups to a media drive is OK but I do not want all that crap in the source drives folders.

There MUST be a way, I mean it would be the height of stupidity to hard code them into the binary….

Another year and I’ll need larger drives. Sigh.

I have a lifetime license for PLEX but HATE the direction plex went so I do not like to use PLEX anymore but, I think it does the same thing to your directories, makes them messy.

Either way I will pick someone as a resolution because I don’t want to pick myself and since TheFrog posted a reply first so, it’ll be him.

Thanks to everyone who posted their comments.

John

12

Another thought.

Now that all the files are on the Jellyfin server and they’ve been metadata’d only new directories and files will be written so I can probably sync them and it will only remove files that I’ve deleted from the source but it won’t fight with Grsync now to keep deleting metadata files Jellyfin is promptly restoring.

I also set Jellyfin to run the metadata maintenance on a schedule instead of on the fly.

AS time goes by I’ll learn more and we’ll see about custom folders.

13

Your situation reminds me of a thread that I made a few weeks ago.