checksums

checksums

[folkert]

Hi,

Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?

Also: is it planned to add an option to add checksums to the data as
well?


Folkert van Heusden

-- 
www.vanheusden.com/multitail - multitail is tail on steroids. multiple
               windows, filtering, coloring, anything you can think of
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: checksums

[Theodore Ts'o]

On Tue, May 14, 2013 at 11:14:08AM +0200, folkert wrote:
> 
> Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?

When e2fsck does a full check, it verifies the contents of all of the
metadata blocks, and this includes verifying all of the checksums.

> Also: is it planned to add an option to add checksums to the data as
> well?

That's not currently on the development roadmap; I could imagine
someone deciding to design an extension to ext4 that would do this
probably by storing the checksums in the indirect blocks, but no one
is currently working on it.

Regards,

					- Ted

Re: checksums

[folkert]

> > Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?
> 
> When e2fsck does a full check, it verifies the contents of all of the
> metadata blocks, and this includes verifying all of the checksums.

Ok. But that would only when the filesystem is not mounted.
Maybe some on-line functionality for doing so would be nice. I'm not
totally aware of the filesystem structures in memory/on disk, but
reading meta-data from disk which has changes pending in memory/in the
journal would give at worst a verify of old(er) data. I don't think this
(checking occasional old data) is a bad thing - scrubbing a
raid-device/disk doesn't give you the situation for the whole disk(s) in
1 (!) point at time either. If that would be required, then the user
could still unmount the filesystem and do a check.

> > Also: is it planned to add an option to add checksums to the data as
> > well?
> That's not currently on the development roadmap; I could imagine
> someone deciding to design an extension to ext4 that would do this
> probably by storing the checksums in the indirect blocks, but no one
> is currently working on it.

Would crowdfunding such a feature be an option?
E.g. via https://www.catincan.com/ or http://www.kickstarter.com/

On other thing I was thinking of, was a switch that would enable a
read-after-write which would verify that the data is indeed on the
platters/NAND-chips.
(I don't care (too) much about speed - I just want to be really sure my
data doesn't get corrupt/lost without investing huge piles of money on
a san/nas with remote mirroring etc.).


Regards,

Folkert van Heusden

-- 
Winnen in de staatsloterij? Check htp://vanheusden.com/sl.php voor
alle staatsloterij-uitslag statistieken denkbaar en zelfs ook een paar
win-tips!
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: checksums

[Darrick J. Wong]

On Tue, May 14, 2013 at 04:40:34PM +0200, folkert wrote:
> > > Is it possible to "scrub" (check/verify) (the new-) checksums in ext4?
> > 
> > When e2fsck does a full check, it verifies the contents of all of the
> > metadata blocks, and this includes verifying all of the checksums.
> 
> Ok. But that would only when the filesystem is not mounted.
> Maybe some on-line functionality for doing so would be nice. I'm not
> totally aware of the filesystem structures in memory/on disk, but
> reading meta-data from disk which has changes pending in memory/in the
> journal would give at worst a verify of old(er) data. I don't think this
> (checking occasional old data) is a bad thing - scrubbing a
> raid-device/disk doesn't give you the situation for the whole disk(s) in
> 1 (!) point at time either. If that would be required, then the user
> could still unmount the filesystem and do a check.

Well... if you ran filefrag -v on every file on the disk and read all the
xattrs, you'd scrub nearly all the metadata.  The only things you'd miss are
unallocated parts of the disk, most of which e2fsck also skips.

Ofc e2fsck will also catch incorrect values in the metadata, whereas I think
the kernel only notices /insane/ values.

> > > Also: is it planned to add an option to add checksums to the data as
> > > well?
> > That's not currently on the development roadmap; I could imagine
> > someone deciding to design an extension to ext4 that would do this
> > probably by storing the checksums in the indirect blocks, but no one
> > is currently working on it.

sha256sum < file > file.sha256 ? :D

(If only there was disk space and brain-time to do something where you could
*reconstruct* data.)

--D

> Would crowdfunding such a feature be an option?
> E.g. via https://www.catincan.com/ or http://www.kickstarter.com/
> 
> On other thing I was thinking of, was a switch that would enable a
> read-after-write which would verify that the data is indeed on the
> platters/NAND-chips.
> (I don't care (too) much about speed - I just want to be really sure my
> data doesn't get corrupt/lost without investing huge piles of money on
> a san/nas with remote mirroring etc.).
> 
> 
> Regards,
> 
> Folkert van Heusden
> 
> -- 
> Winnen in de staatsloterij? Check htp://vanheusden.com/sl.php voor
> alle staatsloterij-uitslag statistieken denkbaar en zelfs ook een paar
> win-tips!
> ----------------------------------------------------------------------
> Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: checksums

[folkert]

> > Ok. But that would only when the filesystem is not mounted.
> > Maybe some on-line functionality for doing so would be nice. I'm not
> > totally aware of the filesystem structures in memory/on disk, but
> > reading meta-data from disk which has changes pending in memory/in the
> > journal would give at worst a verify of old(er) data. I don't think this
> > (checking occasional old data) is a bad thing - scrubbing a
> > raid-device/disk doesn't give you the situation for the whole disk(s) in
> > 1 (!) point at time either. If that would be required, then the user
> > could still unmount the filesystem and do a check.
> 
> Well... if you ran filefrag -v on every file on the disk and read all the
> xattrs, you'd scrub nearly all the metadata.  The only things you'd miss are
> unallocated parts of the disk, most of which e2fsck also skips.

Yes but that is, imho, a bit dirty method.
Because I assume the result will be a message in dmesg and the
filesystem being remounted r/o?
I think it would be better if a nice message on the user's terminal and
an exit code.

> > > That's not currently on the development roadmap; I could imagine
> > > someone deciding to design an extension to ext4 that would do this
> > > probably by storing the checksums in the indirect blocks, but no one
> > > is currently working on it.
> 
> sha256sum < file > file.sha256 ? :D

Then you would need to read the whole file. I think it would be better
to have this on e.g. block-level. 4KB so CRC32 suffices?

> (If only there was disk space and brain-time to do something where you could
> *reconstruct* data.)

ah yes.
These days everything is done by the gpu, maybe it can help with that :)


Folkert van Heusden

-- 
www.vanheusden.com/multitail - multitail is tail on steroids. multiple
               windows, filtering, coloring, anything you can think of
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: checksums

[Darrick J. Wong]

On Tue, May 14, 2013 at 08:57:54PM +0200, folkert wrote:
> > > Ok. But that would only when the filesystem is not mounted.
> > > Maybe some on-line functionality for doing so would be nice. I'm not
> > > totally aware of the filesystem structures in memory/on disk, but
> > > reading meta-data from disk which has changes pending in memory/in the
> > > journal would give at worst a verify of old(er) data. I don't think this
> > > (checking occasional old data) is a bad thing - scrubbing a
> > > raid-device/disk doesn't give you the situation for the whole disk(s) in
> > > 1 (!) point at time either. If that would be required, then the user
> > > could still unmount the filesystem and do a check.
> > 
> > Well... if you ran filefrag -v on every file on the disk and read all the
> > xattrs, you'd scrub nearly all the metadata.  The only things you'd miss are
> > unallocated parts of the disk, most of which e2fsck also skips.
> 
> Yes but that is, imho, a bit dirty method.
> Because I assume the result will be a message in dmesg and the
> filesystem being remounted r/o?
> I think it would be better if a nice message on the user's terminal and
> an exit code.

You should see "I/O Error" (or whatever -EIO becomes in the message catalog) on
the terminal running filefrag if you hit a checksum error, in addition to a
complaint in dmesg and a ro fs.

> > > > That's not currently on the development roadmap; I could imagine
> > > > someone deciding to design an extension to ext4 that would do this
> > > > probably by storing the checksums in the indirect blocks, but no one
> > > > is currently working on it.
> > 
> > sha256sum < file > file.sha256 ? :D
> 
> Then you would need to read the whole file. I think it would be better
> to have this on e.g. block-level. 4KB so CRC32 suffices?

block or bigalloc-cluster level, I suppose.

--D
> 
> > (If only there was disk space and brain-time to do something where you could
> > *reconstruct* data.)
> 
> ah yes.
> These days everything is done by the gpu, maybe it can help with that :)
> 
> 
> Folkert van Heusden
> 
> -- 
> www.vanheusden.com/multitail - multitail is tail on steroids. multiple
>                windows, filtering, coloring, anything you can think of
> ----------------------------------------------------------------------
> Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

Re: checksums...

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

Koen Kooi wrote:
> Hi,
> 
> Could people adding new recipe and people git-am'ing patches adding new 
> recipes please add the corresponding checksums to checksums.ini?
> 
> It's not hard:
> 
> 1) bitbake <recipe>
> 2) watch it fail
> 3) cd $TMPDIR ; cat /OE/org.openembedded.dev/conf/checksums.ini >> 
> checksums.ini ; python 
> /OE/org.openembedded.dev/contrib/source-checker/oe-checksums-sorter.py 
> checksums.ini > /OE/org.openembedded.dev/conf/checksums.ini
> 4) git commit --amend /OE/org.openembedded.dev/conf/checksums.ini

I screwed this up once (and committed a recipe without the checksum) 
when I left ALLOW_INSECURE_DOWNLOADS set in my local.conf. Now I have it 
commented out and am really careful only to use it only when hacking.

I also need to look up what "git commit --amend" does.

Philip

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

checksums...

[Koen Kooi]

Hi,

Could people adding new recipe and people git-am'ing patches adding new 
recipes please add the corresponding checksums to checksums.ini?

It's not hard:

1) bitbake <recipe>
2) watch it fail
3) cd $TMPDIR ; cat /OE/org.openembedded.dev/conf/checksums.ini >> 
checksums.ini ; python 
/OE/org.openembedded.dev/contrib/source-checker/oe-checksums-sorter.py 
checksums.ini > /OE/org.openembedded.dev/conf/checksums.ini
4) git commit --amend /OE/org.openembedded.dev/conf/checksums.ini

regards,

Koen