From: "Patrik Dahlström" <risca@powerlamerz.org>
To: Andreas Klauer <Andreas.Klauer@metamorpher.de>
Cc: Brad Campbell <lists2009@fnarfbargle.com>, linux-raid@vger.kernel.org
Subject: Re: Recover array after I panicked
Date: Tue, 25 Apr 2017 01:00:47 +0200 [thread overview]
Message-ID: <727eeaf5-b856-305d-99b6-e42f3383b699@powerlamerz.org> (raw)
In-Reply-To: <20170424133946.GA7057@metamorpher.de>
On 04/24/2017 03:39 PM, Andreas Klauer wrote:
> On Mon, Apr 24, 2017 at 02:54:50PM +0200, Patrik Dahlström wrote:
> Another thing you can check is, pick an arbitrary offset that does not
> have zeroes on the disks, and see if the XOR matches for 5, or 6 disks.
>
> It should match for 6 for however far the grow progressed,
> and match for 5 afterwards.
I got some success!
I started experimenting with this and realised that about 1 TB into
/dev/sdf, I started getting all zeros from the disk. Incidentally this
will also make the xor pass for both 5 and 6 disk arrays. Here's some
examples:
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0x8000000000 "$f"; done
8000000000 96 6c 5d 2c a4 03 7a 62 9c 7d 67 b9 55 24 aa 84 |.l],..zb.}g.U$..|
8000000010
8000000000 59 10 a8 e9 a7 5e fa e9 cd 8c 16 a2 7c 06 60 f6 |Y....^......|.`.|
8000000010
8000000000 7d ca 8e ea cc fe 2a 36 be b8 a8 b6 77 f9 fa 87 |}.....*6....w...|
8000000010
8000000000 4d 42 49 1e 0b ae a7 3b 42 50 68 bb c1 d5 89 96 |MBI....;BPh.....|
8000000010
8000000000 fc 3d c7 b7 82 39 06 a7 ad fc 00 81 05 5b 52 e1 |.=...9.......[R.|
8000000010
8000000000 03 c9 f5 86 46 34 0b 21 00 e5 b1 97 9a 55 eb 82 |....F4.!.....U..|
8000000010
5 disk raid xor check: 84 ^ f6 ^ 87 ^ 96 ^ e1 == 82, NOK
6 disk raid xor check: 84 ^ f6 ^ 87 ^ 96 ^ e1 ^ 82 == 0, OK
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0x10000000000 "$f"; done
10000000000 03 33 0a 04 d9 7a 44 4b dc 8f be 58 0b 80 8f 46 |.3...zDK...X...F|
10000000010
10000000000 79 5f 18 51 f7 44 03 59 7f aa ce 9d f1 a9 3d 73 |y_.Q.D.Y......=s|
10000000010
10000000000 67 f4 9e b2 34 b6 c2 43 b5 8d 2f 0d 5f 80 a4 6d |g...4..C../._..m|
10000000010
10000000000 4f ec f4 c9 da 04 64 db dc e9 e1 72 7f e4 74 06 |O.....d....r..t.|
10000000010
10000000000 52 74 78 2e c0 8c e1 8a ca 41 be ba da 4d 62 5e |Rtx......A...Mb^|
10000000010
10000000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
10000000010
5 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e == 0, OK
6 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e ^ 00 == 0, OK
So I did a binary search from 0x8000000000 to 0x10000000000 to see
where I start getting all zeros from the newest drive. I ended up at
offset 0xfaa2880000 (~0.98 TB or ~18 % into the disk). After that, it's
all zeros.
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0xfaa287fff8 "$f"; done
faa287fff8 e5 4d 6b e6 ef 7b 1f b0 2e 30 82 8e 5b 4b e0 30 |.Mk..{...0..[K.0|
faa2880008
faa287fff8 dd f7 ab 02 cf e8 a2 6d 93 a8 08 a7 d8 9e c7 b4 |.......m........|
faa2880008
faa287fff8 31 f8 7b d9 41 c7 72 13 f1 37 b6 4a 51 fc 46 74 |1.{.A.r..7.JQ.Ft|
faa2880008
faa287fff8 87 29 59 58 97 05 87 1b 1a 8d 83 84 83 b0 21 4a |.)YX..........!J|
faa2880008
faa287fff8 04 2b 32 6d 0f ab fb b7 56 22 bf e7 51 99 40 ba |.+2m....V"..Q.@.|
faa2880008
faa287fff8 f4 06 c8 81 00 00 03 ae 00 00 00 00 00 00 00 00 |................|
faa2880008
Now, at offset 0xfaa2880000 the 5 disk raid xor sums are OK:
0xfaa2880000: 2e ^ 93 ^ f1 ^ 1a ^ 56 = 0, OK
But immediately before that, I can't get the xor sums to line up:
0xfaa287ffff: b0 ^ 6d ^ 13 ^ 1b ^ b7 != ae (62 actually), NOK
This would mean that it's incorrect for both 5 and 6 disk raids.
So I did a binary search backwards to find out where the checksums match
again and came to offset 0xed90332000 (~0.93 TB or ~17 % into the disk).
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0xed90331ff8 "$f"; done
ed90331ff8 69 1a b1 fb 9f 80 cf fa 6f 97 dc b7 26 40 38 6f |i.......o...&@8o|
ed90332008
ed90331ff8 e3 58 c4 e7 7a 9a b5 a4 74 37 5a de d8 24 a6 e5 |.X..z...t7Z..$..|
ed90332008
ed90331ff8 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
ed90332008
ed90331ff8 53 0b 7c bb 49 d6 63 55 ff fb b2 40 35 00 03 39 |S.|.I.cU...@5..9|
ed90332008
ed90331ff8 9b 49 25 23 ec ad 89 0c ce a0 29 29 a2 86 bc c2 |.I%#......))....|
ed90332008
ed90331ff8 bd ff d3 7b bf 9e 6f f8 17 bc f3 8d e7 9b d0 65 |...{..o........e|
ed90332008
0xed90331fff: fa ^ a4 ^ ff ^ 55 ^ 0c == f8, OK for 6 disk raid
0xed90332000: 6f ^ 74 ^ ff ^ ff ^ ce != 17 (d5 actually), NOK
That is a span of ~52 GB where I presumably can't get the checksums
right. What does all this mean? What am I missing?
Best regards
// Patrik
next prev parent reply other threads:[~2017-04-24 23:00 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-04-23 9:47 Recover array after I panicked Patrik Dahlström
2017-04-23 10:16 ` Andreas Klauer
2017-04-23 10:23 ` Patrik Dahlström
2017-04-23 10:46 ` Andreas Klauer
2017-04-23 11:12 ` Patrik Dahlström
2017-04-23 11:36 ` Wols Lists
2017-04-23 11:47 ` Patrik Dahlström
2017-04-23 11:53 ` Reindl Harald
2017-04-23 11:58 ` Roman Mamedov
2017-04-23 12:11 ` Wols Lists
2017-04-23 12:15 ` Patrik Dahlström
2017-04-24 21:04 ` Phil Turmel
2017-04-24 21:56 ` Patrik Dahlström
2017-04-24 23:35 ` Phil Turmel
2017-04-23 13:16 ` Andreas Klauer
2017-04-23 13:49 ` Patrik Dahlström
2017-04-23 14:36 ` Andreas Klauer
2017-04-23 14:45 ` Patrik Dahlström
2017-04-23 12:32 ` Patrik Dahlström
2017-04-23 12:45 ` Andreas Klauer
2017-04-23 12:57 ` Patrik Dahlström
2017-04-23 14:06 ` Brad Campbell
2017-04-23 14:09 ` Patrik Dahlström
2017-04-23 14:20 ` Patrik Dahlström
2017-04-23 14:25 ` Brad Campbell
2017-04-23 14:48 ` Andreas Klauer
2017-04-23 15:11 ` Patrik Dahlström
2017-04-23 15:24 ` Patrik Dahlström
2017-04-23 15:42 ` Andreas Klauer
2017-04-23 16:29 ` Patrik Dahlström
2017-04-23 19:21 ` Patrik Dahlström
2017-04-24 2:09 ` Brad Campbell
2017-04-24 7:34 ` Patrik Dahlström
2017-04-24 11:04 ` Andreas Klauer
2017-04-24 12:13 ` Patrik Dahlström
2017-04-24 12:37 ` Andreas Klauer
2017-04-24 12:54 ` Patrik Dahlström
2017-04-24 13:39 ` Andreas Klauer
2017-04-24 14:05 ` Patrik Dahlström
2017-04-24 14:21 ` Andreas Klauer
2017-04-24 16:00 ` Patrik Dahlström
2017-04-24 23:00 ` Patrik Dahlström [this message]
2017-04-25 0:16 ` Andreas Klauer
2017-04-25 8:44 ` Patrik Dahlström
2017-04-25 9:01 ` Andreas Klauer
2017-04-25 10:40 ` Patrik Dahlström
2017-04-25 10:51 ` Patrik Dahlström
2017-04-25 11:08 ` Andreas Klauer
2017-04-25 11:37 ` Patrik Dahlström
2017-04-25 12:41 ` Andreas Klauer
2017-04-25 18:22 ` Wols Lists
2017-04-27 19:57 ` Patrik Dahlström
2017-04-27 23:12 ` Andreas Klauer
2017-04-28 7:11 ` Patrik Dahlström
2017-04-28 9:52 ` Andreas Klauer
2017-04-28 10:31 ` Patrik Dahlström
2017-04-28 11:39 ` Andreas Klauer
2017-04-28 22:46 ` Patrik Dahlström
2017-04-29 9:56 ` Andreas Klauer
2017-05-02 13:08 ` Patrik Dahlström
2017-05-02 13:11 ` Brad Campbell
2017-05-02 15:49 ` Anthony Youngman
2017-04-25 23:01 ` Patrik Dahlström
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=727eeaf5-b856-305d-99b6-e42f3383b699@powerlamerz.org \
--to=risca@powerlamerz.org \
--cc=Andreas.Klauer@metamorpher.de \
--cc=linux-raid@vger.kernel.org \
--cc=lists2009@fnarfbargle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.