Best way to copy large amounts of files, varying size

I’ve set my Eon up with 2x4tb in mirror, i’ve got a USB drive connected straight to the PI with roughly 300k files and 2.8tb data that i’d like to copy over.

Whats the best way to do this?

I’d also like to verify the data once complete as well as it’s the only copy that I have…

Further to this -

Is there a command i could use over SSH possibly to basically just copy

/dev/sda2 all folders/structure to /dev/md126 ?

Lots of different ways, you could use:

cp
scp
rsync

I’d use rsync -avhp source/ dest/

-a : archive mode
-v : verbose
-h : human readable
-p : preserve permissions.

so for your example:

rsync -avhp /dev/sda2/ /dev/md126/

Thanks, i will give that a go, would i be able to check the progress of that somehow in OMV?

No idea, I don’t use OMV, but I suspect you will see the amount of free space drop. I know the storage dsiplay on the OLED will change. :wink:

1 Like

$ rsync -avhp /dev/sda2/ /dev/md126/
sending incremental file list
rsync: [sender] change_dir “/dev/sda2” failed: Not a directory (20)
rsync: [Receiver] ERROR: cannot stat destination “/dev/md126/”: Not a directory (20)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(768) [Receiver=3.2.3]

It gave me this error message, what’s the error? it’s not possible to do the whole drive and i’d have to select individual folders maybe?

You need file systems on the destination as well as the source,

rsync source dest

1 Like

Ah, so i’d have to reference the individual folders on the source? and a folder on the destination as well? can’t just to “C:\ to D:\” in windows terminology

image

No one said that. Linux makes a distinction between storage, file systems and mount points.

In your case /dev/sda2 contains a file system, but you need the MOUNT point… i.e. where is that file system mounted.

Under Windows D:\ is the mount point for the file system, and \physicaldeviceX, is the where the storage is.

run the command:

df -H

That will show the connection between the mount point and the physical devices. Given that information we can probably construct a command line for you.

Since /dev/sda2 appears to be an NTFS partition (once properly mounted as @NHHiker mentioned), you will likely need to include rsync’s --size-only flag per the example directly below :

rsync -avhp --size-only /mnt/source/ /mnt/destination/

Also note that a “trailing” slash on the source path means copy the contents of this directory . Without a trailing slash means “copy the directory”.

Depending on the read/write speeds of your various drives, a 2.77TB rsync job will likely take several hours to complete.

/dev/md126 4.1T 4.3M 4.0T 1% /srv/dev-disk-by-uuid-73b27dec-e8bc-4b02-b bfc-7ef19208b344

/dev/sda2 8.1T 3.1T 5.0T 39% /srv/dev-disk-by-uuid-4A96FB8496FB6F37
tmpfs 403M 0 403M 0% /run/user/1000

So would the command be

rsync -avhp --size-only /srv/dev-disk-by-uuid-4A96FB8496FB6F37/ /srv/dev-disk-by-uuid-73b27dec-e8bc-4b02-b bfc-7ef19208b344/

?

@kuddlywuddly Looks like it from the output.

Thank you for the help, it did fail partly…?

image

Buuut it’s listing a bunch of the files now so seems like its copying, will just have to let it run overnight and see, I’m assuming i can terminate my putty session and it will keep going? :slight_smile: