From: Willy TARREAU <willy@w.ods.org>
To: Stephan von Krawczynski <skraw@ithnet.com>
Cc: Willy Tarreau <willy@w.ods.org>,
marcelo@conectiva.com.br, kpfleming@cox.net, stoffel@lucent.com,
gibbs@scsiguy.com, linux-kernel@vger.kernel.org,
green@namesys.com
Subject: Re: Undo aic7xxx changes (now rc7+aic20030603)
Date: Sat, 21 Jun 2003 12:50:19 +0200 [thread overview]
Message-ID: <20030621105019.GA834@pcw.home.local> (raw)
In-Reply-To: <20030621014828.0420b74f.skraw@ithnet.com>
On Sat, Jun 21, 2003 at 01:48:28AM +0200, Stephan von Krawczynski wrote:
> Well, in fact I am a bit lost in the case, because of the shere data volume, I
> have space for several sets on disk, but it takes a damn long time to produce
> one cycle write/verify. Anyway I will do if that helps. The big problem with
> tar is that I have (to my knowledge) no chance to let it somewhere save the
> verify-failing data parts. I guess this could help a lot, because we could then
> see what the corruption looks like, how long (in bytes) it is and so on.
> If anybody has an idea how to achieve this goal let me know.
I wanted to implement a compare-and-capture feature in my check tool, but
realized that it would certainly be of no help if you get duplicated blocks or
so, because you'll have no way to tell *where* the captured block should have
been. That's why I suggested the checksum instead : if you get a pattern such
as :
check1 check2
0: 1234 1234
1: 4567 4567
3: 789a 4567
4: bcde 789a
5: f012 bcde
... it will mean than block 1 was duplicated in check2. If you see :
check1 check2
0: 1234 1234
1: 4567 4567
3: 789a 4567
4: bcde bcde
5: f012 f012
... it will mean than block 1 was repeated instead of block 2 in check2.
If you see 0000, it probably means that you got a block full of zeros, since
the algorithm is only additive.
The resulting files will be 1/512 of the input, I think you'll find some space
on your disk for such a file.
It may be interesting to do regular checks during the second read, so that you
can abort after the first error, and not have to get a second full read.
> Ok, weekend is here, I see what can be done.
Here is my proposed program. I tried it on my local hard disk, it took 5 min
to check the full 8 GB (30 MB/s), and I reached 123 MB/s on a 4 disks software
raid5 array with an AHA29160. It outputs the current offset every 64 MB.
Here it is running on a DDS3 :
[root@alpha /root]# ~willy/c/chkblk.alpha /dev/nst0 > nst0.chk
At offset 603979776...
I hope it can help.
Cheers,
Willy
/*
* chkblk - computes block checksums - 2003/06/21 - Willy Tarreau <w@w.ods.org>
*
* This program is free, do what you want with it, I will not be responsible if
* it trashes all your data.
*
* Reads a file and outputs a binary 16 bit checksum for each 1KB block.
* Useful to check for data corruption. Eg :
*
* # chkblk /dev/tape > test1.chk
* # chkblk /dev/tape > test2.chk
* # cmp -l test[12].chk
*
* or :
* # chkblk /dev/sda2 |od -tx2 -Ax > test1.txt
* # chkblk /dev/sda2 |od -tx2 -Ax > test2.txt
* # diff -u test[12].txt
*
* To be able to read files bigger than 2GB, you should compile it
* with "-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64".
*
*
*/
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#define BLOCKSIZE 1024
#if _FILE_OFFSET_BITS == 64
#define OFF_T_FMT "%ll"
#else
#define OFF_T_FMT "%l"
#endif
void usage() {
fprintf(stderr,
"Usage: chkblk input > output\n"
" - input is a file, device, ...\n"
" - output will be a binary file 1/512th the size of input\n"
);
exit(1);
}
main(int argc, char **argv) {
int fd;
int len;
off_t inp_off;
unsigned long *buffer;
if (argc != 2)
usage();
buffer = (void *)malloc(BLOCKSIZE);
if (buffer == NULL) {
fprintf(stderr,"Out of memory\n");
exit(2);
}
fd = open(argv[1], O_RDONLY);
if (fd < 0) {
perror("open");
exit(3);
}
inp_off = 0;
while ((len = read(fd, buffer, BLOCKSIZE)) > 0) {
unsigned long sum = 0;
int off;
inp_off += len;
/* displays the offset every 64 MB */
if ((inp_off & 0x3ffffff) == 0)
fprintf(stderr,"At offset " OFF_T_FMT "u...\r", inp_off);
for (off = 0; off < len/sizeof(*buffer); off++)
sum += buffer[off];
while (sum >= (1<<16)) {
sum = (sum & 0xffff) + (sum >> 16);
}
putchar(sum);
putchar(sum >> 8);
}
fprintf(stderr,"At offset " OFF_T_FMT "u", inp_off);
if (len < 0) {
fprintf(stderr, ", read returned : \n");
perror("");
close(fd);
exit(4);
}
else {
fprintf(stderr, ", check completed without error\n");
}
close(fd);
exit(0);
}
next prev parent reply other threads:[~2003-06-21 10:36 UTC|newest]
Thread overview: 110+ messages / expand[flat|nested] mbox.gz Atom feed top
2003-05-07 20:22 Undo aic7xxx changes Marcelo Tosatti
2003-05-09 0:45 ` Justin T. Gibbs
2003-05-09 10:06 ` Stephan von Krawczynski
2003-05-09 12:06 ` Willy Tarreau
2003-05-09 13:02 ` Stephan von Krawczynski
2003-05-09 13:27 ` Willy Tarreau
2003-05-09 13:46 ` Stephan von Krawczynski
2003-05-09 14:56 ` Willy Tarreau
2003-05-09 15:08 ` Arjan van de Ven
2003-05-09 16:27 ` Willy Tarreau
2003-05-09 15:18 ` Andreas Schwab
2003-05-09 15:19 ` William Lee Irwin III
2003-05-09 14:11 ` Stephan von Krawczynski
2003-05-09 14:57 ` Willy Tarreau
2003-05-12 9:02 ` Stephan von Krawczynski
2003-05-12 15:43 ` Marc-Christian Petersen
2003-05-12 17:25 ` Willy Tarreau
2003-05-23 10:38 ` Stephan von Krawczynski
2003-05-23 12:58 ` Justin T. Gibbs
2003-05-23 13:11 ` Stephan von Krawczynski
2003-05-23 19:57 ` Willy Tarreau
2003-05-24 10:52 ` Stephan von Krawczynski
2003-05-24 11:16 ` Willy Tarreau
2003-05-25 10:58 ` Stephan von Krawczynski
2003-05-25 12:35 ` Willy TARREAU
2003-05-25 12:47 ` Marc-Christian Petersen
2003-05-25 13:50 ` Stephan von Krawczynski
2003-05-25 14:01 ` Marc-Christian Petersen
2003-05-25 14:03 ` Geller Sandor
2003-05-26 15:00 ` Stephan von Krawczynski
2003-05-26 16:44 ` Willy Tarreau
2003-05-30 8:09 ` Stephan von Krawczynski
2003-05-30 8:19 ` Marc-Christian Petersen
2003-05-30 8:21 ` Arjan van de Ven
2003-05-30 8:51 ` Stephan von Krawczynski
2003-05-30 13:34 ` Jeff Garzik
2003-05-30 13:59 ` Stephan von Krawczynski
2003-05-30 13:35 ` Jeff Garzik
2003-05-25 18:30 ` Justin T. Gibbs
2003-06-05 15:05 ` Undo aic7xxx changes (now rc7+aic20030603) Stephan von Krawczynski
2003-06-05 18:14 ` Willy Tarreau
2003-06-06 8:17 ` Oleg Drokin
2003-06-06 9:04 ` Stephan von Krawczynski
2003-06-06 9:17 ` Oleg Drokin
2003-06-06 15:24 ` short freezing while file re-creation Stephan von Krawczynski
2003-06-06 16:02 ` Oleg Drokin
2003-06-06 19:00 ` Chris Mason
2003-06-06 19:10 ` Oleg Drokin
2003-06-06 19:20 ` Chris Mason
2003-06-08 10:15 ` Undo aic7xxx changes (now rc7+aic20030603) Stephan von Krawczynski
2003-06-08 11:19 ` Stephan von Krawczynski
2003-06-08 11:49 ` Stephan von Krawczynski
2003-06-08 16:07 ` Stephan von Krawczynski
2003-06-09 15:10 ` Stephan von Krawczynski
2003-06-09 15:32 ` Justin T. Gibbs
2003-06-10 10:23 ` Stephan von Krawczynski
2003-06-10 15:38 ` Justin T. Gibbs
2003-06-10 17:11 ` Stephan von Krawczynski
2003-06-10 18:07 ` Justin T. Gibbs
2003-06-11 0:51 ` Stephan von Krawczynski
2003-06-11 4:39 ` Justin T. Gibbs
2003-06-11 20:23 ` Stephan von Krawczynski
2003-06-11 21:01 ` John Stoffel
2003-06-13 9:45 ` Stephan von Krawczynski
2003-06-15 12:56 ` Stephan von Krawczynski
2003-06-15 13:26 ` John Stoffel
2003-06-17 20:47 ` Marcelo Tosatti
2003-06-18 11:05 ` Stephan von Krawczynski
2003-06-18 14:21 ` John Stoffel
2003-06-18 14:54 ` Stephan von Krawczynski
2003-06-20 19:59 ` Marcelo Tosatti
2003-06-20 20:59 ` Kevin P. Fleming
2003-06-20 21:13 ` Marcelo Tosatti
2003-06-20 22:03 ` Willy Tarreau
2003-06-20 23:48 ` Stephan von Krawczynski
2003-06-21 10:50 ` Willy TARREAU [this message]
2003-06-22 19:00 ` Stephan von Krawczynski
2003-06-23 11:30 ` Stephan von Krawczynski
2003-06-24 11:11 ` Stephan von Krawczynski
2003-06-24 17:43 ` Willy Tarreau
2003-06-24 21:26 ` Stephan von Krawczynski
2003-06-24 22:03 ` Willy Tarreau
2003-06-24 23:43 ` Stephan von Krawczynski
2003-06-25 19:16 ` Willy Tarreau
2003-06-25 19:42 ` Stephan von Krawczynski
2003-06-25 20:30 ` John Stoffel
2003-06-26 9:36 ` Stephan von Krawczynski
2003-06-26 11:34 ` Stephan von Krawczynski
2003-06-30 10:10 ` Stephan von Krawczynski
2003-06-30 11:39 ` Marcelo Tosatti
2003-06-30 12:08 ` Stephan von Krawczynski
2003-06-25 23:04 ` Bernd Eckenfels
2003-06-25 2:22 ` Valdis.Kletnieks
2003-06-24 18:31 ` Bill Davidsen
2003-06-12 13:54 ` Stephan von Krawczynski
2003-06-10 1:38 ` Zwane Mwaikambo
2003-06-10 10:30 ` Stephan von Krawczynski
2003-06-10 12:51 ` Zwane Mwaikambo
2003-06-10 13:38 ` Stephan von Krawczynski
2003-06-10 13:51 ` Zwane Mwaikambo
2003-06-10 15:55 ` Stephan von Krawczynski
2003-06-10 16:23 ` Oleg Drokin
2003-06-10 17:44 ` Stephan von Krawczynski
2003-06-10 18:15 ` Zwane Mwaikambo
2003-06-10 23:55 ` Stephan von Krawczynski
2003-06-10 18:20 ` Zwane Mwaikambo
2003-05-23 18:30 ` Undo aic7xxx changes Marcelo Tosatti
2003-05-23 19:25 ` Stephan von Krawczynski
[not found] <20030507203025$6f60@gated-at.bofh.it>
[not found] ` <20030509005011$6cee@gated-at.bofh.it>
[not found] ` <20030509101012$732a@gated-at.bofh.it>
[not found] ` <20030509122007$758f@gated-at.bofh.it>
[not found] ` <20030509131009$00f3@gated-at.bofh.it>
[not found] ` <20030611045008$03cf@gated-at.bofh.it>
[not found] ` <20030611203031$12de@gated-at.bofh.it>
[not found] ` <20030611211012$34cf@gated-at.bofh.it>
[not found] ` <20030613095017$1680@gated-at.bofh.it>
[not found] ` <20030617210022$3e37@gated-at.bofh.it>
[not found] ` <20030618111010$154f@gated-at.bofh.it>
2003-06-18 12:46 ` Undo aic7xxx changes (now rc7+aic20030603) Pascal Schmidt
2003-06-18 12:49 ` Stephan von Krawczynski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20030621105019.GA834@pcw.home.local \
--to=willy@w.ods.org \
--cc=gibbs@scsiguy.com \
--cc=green@namesys.com \
--cc=kpfleming@cox.net \
--cc=linux-kernel@vger.kernel.org \
--cc=marcelo@conectiva.com.br \
--cc=skraw@ithnet.com \
--cc=stoffel@lucent.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).