Suggestion for better data integrity

[VirtualMe]

TLDR at the bottom.

I am a Vail refugee and am interested in Amahi and Greyhole. However, I really want data integrity and protection. I've been looking at ZFS and it sounds really good for catching corrupted files and automatically fixing them, except for the fact that it's not truly on Linux yet ( FUSE doesn't count! and I want to use programs in Linux like CrashPlan), you cannot EASILY add more drives to RAID-Z, and I cannot have the zpool selectively have redundancy for certain files. It's all or nothing.

I know it's still early in development but Greyhole gives me the above except for the data integrity part (beyond having multiple copies of the files). ZFS will checksum a file when it is read and written and users can do a scrub of the pool, which I understand to be verifying integrity of all files.
Can we do a poor man's checksum and data integrity with Greyhole?

I have been looking into md5sum to detect a corrupted file and par2 to repair corrupted files. I'm not sure yet if par2 will detect a corrupted file. If it will, then md5sum would not really be needed. From what little I've read of par2, I'm also trying to understand if it can repair any corruption whatsoever in a file if given enough space to create the parity information, without actually creating a duplicate of the entire file.

Then I got to thinking, par2 might not really be necessary if Greyhole is keeping multiple copies of files.

It would take me a while since I'm not familiar with the ins and outs of Linux but I figured I could write a Python script to do this on a periodic basis. Thinking more, I thought it might be better if Greyhole was involved, at least with the first 2 points below.

So I've come up with this idea:

1. When Greyhole writes a new file:
If the file is to have multiple copies, create an md5sum for each copy after it has been copied to each hard drive.
or
If there is only one copy of the file, create a par2 file to fix any corruption. (Create md5sum to detect it if necessary)

Maybe not necessary or not possible (I don't know how SAMBA works):
2. If SAMBA keeps a log of the files that are read:
Keep a list of all files read so we can process it later. Then on periodic basis X, maybe 12 or 24 hours, check for corruption and repair any errors (by using duplicate file or par2 information for single files) any file that was read in the last X hours.

3. Then on an even longer basis, maybe once a week, go over the entire Greyhole pool and check for corruption. With files that have multiple copies, a corrupted file detected with the md5sum can just be overwritten by another copy of the file if it is not corrupted. Or if the file is a single file, use the par2 information to fix the errors. Email a report of the files that were fixed and what hard drives they were on.
Give the users options on how often to run the check and maybe a percentage of the pool to check. Large pools could take a really long time, so maybe do 50% one week and the other 50% the next week.


That's my idea in a nutshell. Not being an expert on any of this I don't know if it would just be a waste of space, horrible to implement with a ton of corner cases, not very efficient and would not catch many errors, or a super awesome idea.

Any input would be appreciated.

Sean

TLDR - Use md5sum and par2 to detect and fix bit rot in, hopefully, a similar manner to ZFS while still retaining the advantages of an EASILY expandable storage pool.

[gboudreau]

The idea makes sense, but the implementation would add quite a bit of overhead to all file operations that GH currently do.

1. Greyhole don't read and write files itself; it uses external tools like rsync to do so. As such, if you want to calculate md5 hashes, the complete file would have to be read a second time. That's a lot more I/O than what Greyhole currently does.

2. Yes, Samba can log file operations. This is in fact the basis of Greyhole: Samba logs all writes/renames/deletes, and Greyhole eats that log, and process each operation in chronological order. Reads are currently not logged, but it would be trivial to add those to the Samba VFS module that GH uses. That could make the Samba log become quite big, depending on how much reads you have on your shares...

3. There is currently a weekly fsck maintenance that starts on Sunday mornings. It currently uses file sizes to identify bad files. Adding the calculation of md5 hashes, to compare with the last valid md5, would be almost trivial. But it would make that fsck window become a 2 days maintenance! And since GH isn't multi-threaded, that would mean all other file operations that you'd do on your shares during this time would be post-poned until the fsck completes. Not something you'd want really...

i.e. it's possible, but creates quite an overhead.

Why don't you mix ZFS and Greyhole to get the benefits of both?
Maybe you can format your drives using ZFS, then add them to the Greyhole pool..?
I think a single ZFS partition does offer protection against corruption, does it not?

[VirtualMe]

Ah yes, I didn't think about having to read the file a second time after it is written. That would mean essentially twice the number of bytes being processed.

It could take quite a while to do the md5 calculations over the whole storage pool if it is quite large. Might be worth it though...

I don't think ZFS will 'auto-heal' if each disk is a standalone partition. I could be wrong about that.
I think I will look into ZFS a bit more and also try running an md5 sum on my files and see how long it takes. If it's not too long, running it once a week wouldn't be too bad.

Thanks for the comments.