* Large-file corruption. ReiserFS? VFS?
@ 2003-09-11 20:28 Kyle Rose
[not found] ` <20030911144732.S18851@schatzie.adilger.int>
` (2 more replies)
0 siblings, 3 replies; 17+ messages in thread
From: Kyle Rose @ 2003-09-11 20:28 UTC (permalink / raw)
To: linux-kernel
I ran mkisofs (2.01a16) this morning on a 2.6.0-test4 machine with the
target on a 60GB ReiserFS partition. The resulting file was to be
just over 4GB (around 4,360,000,000 bytes), and during most of the
course of writing the file, a small prefix looked as it should through
a hexdump: that is, just part of the iso9660 directory.
However, just as the write completed, the beginning of the file became
corrupted. I considered a 4GB problem to be likely, and re-tested
with fewer source files such that the result would be smaller than
4GB; lo and behold, no corruption. The same result occurs whether
mkisofs is given the -o flag, or output is redirected to a file from
stdout using the shell's redirection facility, suggesting the problem
is probably at the kernel level.
I don't have a large enough ext2/3 filesystem to compare with, so
there's no easy way for me to tell whether this is a Reiser-only
problem or not. Can anyone confirm that they see the same problem, or
whether they see a similar problem on another file system? Please
feel free to ask me for any other information you think might be
illuminating.
Cheers,
Kyle
^ permalink raw reply [flat|nested] 17+ messages in thread[parent not found: <20030911144732.S18851@schatzie.adilger.int>]
* Re: Large-file corruption. ReiserFS? VFS?
[not found] ` <20030911144732.S18851@schatzie.adilger.int>
@ 2003-09-11 20:56 ` Kyle Rose
2003-09-11 21:00 ` Andreas Dilger
0 siblings, 1 reply; 17+ messages in thread
From: Kyle Rose @ 2003-09-11 20:56 UTC (permalink / raw)
To: Andreas Dilger; +Cc: linux-kernel
> I would guess that mkisofs isn't opening the file with O_LARGEFILE.
> It probably only expected to write 600MB output files. Purely a
> guess though.
Thanks for the suggestion, but it is, in fact, opening with
O_LARGEFILE:
open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
Cheers,
Kyle
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Large-file corruption. ReiserFS? VFS?
2003-09-11 20:56 ` Kyle Rose
@ 2003-09-11 21:00 ` Andreas Dilger
2003-09-13 6:50 ` Rogier Wolff
2003-09-13 6:51 ` Rogier Wolff
0 siblings, 2 replies; 17+ messages in thread
From: Andreas Dilger @ 2003-09-11 21:00 UTC (permalink / raw)
To: Kyle Rose; +Cc: linux-kernel
On Sep 11, 2003 16:56 -0400, Kyle Rose wrote:
> > I would guess that mkisofs isn't opening the file with O_LARGEFILE.
> > It probably only expected to write 600MB output files. Purely a
> > guess though.
>
> Thanks for the suggestion, but it is, in fact, opening with
> O_LARGEFILE:
>
> open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
True, and O_LARGEFILE would have bit you at 2GB and not 4GB... If you are
doing output redirected from the shell, then it can't be a seek issue
either.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Large-file corruption. ReiserFS? VFS?
2003-09-11 21:00 ` Andreas Dilger
@ 2003-09-13 6:50 ` Rogier Wolff
2003-09-13 6:51 ` Rogier Wolff
1 sibling, 0 replies; 17+ messages in thread
From: Rogier Wolff @ 2003-09-13 6:50 UTC (permalink / raw)
To: Kyle Rose, linux-kernel
On Thu, Sep 11, 2003 at 03:00:17PM -0600, Andreas Dilger wrote:
> True, and O_LARGEFILE would have bit you at 2GB and not 4GB... If you are
> doing output redirected from the shell, then it can't be a seek issue
> either.
Nah! Some programs just "try to seek" and if it works, will.
dd if=/dev/zero count=200 | strace sox -t raw -r 44100 -s -w - -t wav - > /tmp/test
dd if=/dev/zero count=200 | sox -t raw -r 44100 -s -w - -t wav - | dd of=/tmp/test
sox: Length in output .wav header will be wrong since can't seek to fix it
Roger.
--
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Large-file corruption. ReiserFS? VFS?
2003-09-11 21:00 ` Andreas Dilger
2003-09-13 6:50 ` Rogier Wolff
@ 2003-09-13 6:51 ` Rogier Wolff
1 sibling, 0 replies; 17+ messages in thread
From: Rogier Wolff @ 2003-09-13 6:51 UTC (permalink / raw)
To: Kyle Rose, linux-kernel
On Thu, Sep 11, 2003 at 03:00:17PM -0600, Andreas Dilger wrote:
> On Sep 11, 2003 16:56 -0400, Kyle Rose wrote:
> > > I would guess that mkisofs isn't opening the file with O_LARGEFILE.
> > > It probably only expected to write 600MB output files. Purely a
> > > guess though.
> >
> > Thanks for the suggestion, but it is, in fact, opening with
> > O_LARGEFILE:
> >
> > open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
>
> True, and O_LARGEFILE would have bit you at 2GB and not 4GB... If you are
> doing output redirected from the shell, then it can't be a seek issue
> either.
... but mkisofs will NOT be seeking, as we commonly burn CDs while
mkisofs is providing the data through a pipe.
Roger.
--
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Large-file corruption. ReiserFS? VFS?
2003-09-11 20:28 Large-file corruption. ReiserFS? VFS? Kyle Rose
[not found] ` <20030911144732.S18851@schatzie.adilger.int>
@ 2003-09-12 9:36 ` Matthias Schniedermeyer
2003-09-12 11:47 ` Kyle Rose
2003-09-12 15:39 ` Oleg Drokin
2 siblings, 1 reply; 17+ messages in thread
From: Matthias Schniedermeyer @ 2003-09-12 9:36 UTC (permalink / raw)
To: Kyle Rose; +Cc: linux-kernel
On Thu, Sep 11, 2003 at 04:28:58PM -0400, Kyle Rose wrote:
>
> I don't have a large enough ext2/3 filesystem to compare with, so
> there's no easy way for me to tell whether this is a Reiser-only
> problem or not. Can anyone confirm that they see the same problem, or
> whether they see a similar problem on another file system? Please
> feel free to ask me for any other information you think might be
> illuminating.
I can only speak for 2.4-Kernel
I burn DVDs since more than 2 years now. Starting with 2.4.4 and i had
NEVER problems with the size, i never used the split-option of mkisofs
(and i don't even know the correct(tm) name of it. I only know that it
exists).
The maximum is 4489MB, (>4.700.000.000 bytes) and the biggest images i
had are only a few KB smaller that that.
Since about 2 month i use reiserfs on the "source" and on the
"target"-HDD. No changes, except that reiserfs is slower then ext2.
Bis denn
--
Real Programmers consider "what you see is what you get" to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated,
cryptic, powerful, unforgiving, dangerous.
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Large-file corruption. ReiserFS? VFS?
2003-09-11 20:28 Large-file corruption. ReiserFS? VFS? Kyle Rose
[not found] ` <20030911144732.S18851@schatzie.adilger.int>
2003-09-12 9:36 ` Matthias Schniedermeyer
@ 2003-09-12 15:39 ` Oleg Drokin
2003-09-12 17:59 ` Mike Fedyk
2003-09-14 12:13 ` Kyle Rose
2 siblings, 2 replies; 17+ messages in thread
From: Oleg Drokin @ 2003-09-12 15:39 UTC (permalink / raw)
To: Kyle Rose; +Cc: linux-kernel
Hello!
On Thu, Sep 11, 2003 at 04:28:58PM -0400, Kyle Rose wrote:
> However, just as the write completed, the beginning of the file became
> corrupted. I considered a 4GB problem to be likely, and re-tested
You are absolutely right.
Ther is a reiserfs problem that I just found based on your description.
The patch below should help. Please confirm that it works for you too.
Thanks a lot for the report.
Bye,
Oleg
===== fs/reiserfs/file.c 1.22 vs edited =====
--- 1.22/fs/reiserfs/file.c Fri Aug 15 05:17:00 2003
+++ edited/fs/reiserfs/file.c Fri Sep 12 19:18:53 2003
@@ -779,7 +779,7 @@
/* Now if all the write area lies past the file end, no point in
maping blocks, since there is none, so we just zero out remaining
parts of first and last pages in write area (if needed) */
- if ( (pos & ~(PAGE_CACHE_SIZE - 1)) > inode->i_size ) {
+ if ( (pos & ~((loff_t)PAGE_CACHE_SIZE - 1)) > inode->i_size ) {
if ( from != 0 ) {/* First page needs to be partially zeroed */
char *kaddr = kmap_atomic(prepared_pages[0], KM_USER0);
memset(kaddr, 0, from);
@@ -801,9 +801,9 @@
we need to allocate (calculated above) */
/* Mask write position to start on blocksize, we do it out of the
loop for performance reasons */
- pos &= ~(inode->i_sb->s_blocksize - 1);
+ pos &= ~((loff_t) inode->i_sb->s_blocksize - 1);
/* Set cpu key to the starting position in a file (on left block boundary)*/
- make_cpu_key (&key, inode, 1 + ((pos) & ~(inode->i_sb->s_blocksize - 1)), TYPE_ANY, 3/*key length*/);
+ make_cpu_key (&key, inode, 1 + ((pos) & ~((loff_t) inode->i_sb->s_blocksize - 1)), TYPE_ANY, 3/*key length*/);
reiserfs_write_lock(inode->i_sb); // We need that for at least search_by_key()
for ( i = 0; i < num_pages ; i++ ) {
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 15:39 ` Oleg Drokin
@ 2003-09-12 17:59 ` Mike Fedyk
2003-09-12 18:40 ` Oleg Drokin
2003-09-14 12:13 ` Kyle Rose
1 sibling, 1 reply; 17+ messages in thread
From: Mike Fedyk @ 2003-09-12 17:59 UTC (permalink / raw)
To: Oleg Drokin; +Cc: Kyle Rose, linux-kernel
On Fri, Sep 12, 2003 at 07:39:35PM +0400, Oleg Drokin wrote:
> Hello!
>
> On Thu, Sep 11, 2003 at 04:28:58PM -0400, Kyle Rose wrote:
>
> > However, just as the write completed, the beginning of the file became
> > corrupted. I considered a 4GB problem to be likely, and re-tested
>
> You are absolutely right.
> Ther is a reiserfs problem that I just found based on your description.
> The patch below should help. Please confirm that it works for you too.
> Thanks a lot for the report.
>
Yow, I guess large files on reiserfs in 2.6 isn't very common...
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 17:59 ` Mike Fedyk
@ 2003-09-12 18:40 ` Oleg Drokin
2003-09-12 20:54 ` Mike Fedyk
0 siblings, 1 reply; 17+ messages in thread
From: Oleg Drokin @ 2003-09-12 18:40 UTC (permalink / raw)
To: Kyle Rose, linux-kernel
Hello!
On Fri, Sep 12, 2003 at 10:59:17AM -0700, Mike Fedyk wrote:
> > > However, just as the write completed, the beginning of the file became
> > > corrupted. I considered a 4GB problem to be likely, and re-tested
> > You are absolutely right.
> > Ther is a reiserfs problem that I just found based on your description.
> > The patch below should help. Please confirm that it works for you too.
> > Thanks a lot for the report.
> Yow, I guess large files on reiserfs in 2.6 isn't very common...
Or may be nobody noticed the corruption.
Bye,
Oleg
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 18:40 ` Oleg Drokin
@ 2003-09-12 20:54 ` Mike Fedyk
2003-09-12 21:05 ` Kyle Rose
2003-09-13 7:00 ` Oleg Drokin
0 siblings, 2 replies; 17+ messages in thread
From: Mike Fedyk @ 2003-09-12 20:54 UTC (permalink / raw)
To: Oleg Drokin; +Cc: Kyle Rose, linux-kernel
On Fri, Sep 12, 2003 at 10:40:01PM +0400, Oleg Drokin wrote:
> Hello!
>
> On Fri, Sep 12, 2003 at 10:59:17AM -0700, Mike Fedyk wrote:
> > > > However, just as the write completed, the beginning of the file became
> > > > corrupted. I considered a 4GB problem to be likely, and re-tested
> > > You are absolutely right.
> > > Ther is a reiserfs problem that I just found based on your description.
> > > The patch below should help. Please confirm that it works for you too.
> > > Thanks a lot for the report.
> > Yow, I guess large files on reiserfs in 2.6 isn't very common...
>
> Or may be nobody noticed the corruption.
Possible.
Does this affect 2.4 also? If not, then that will narrow the possible
number of people who could have hit this bug.
Luckily when I was using reiserfs, I used mostly small files.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 20:54 ` Mike Fedyk
@ 2003-09-12 21:05 ` Kyle Rose
2003-09-12 21:56 ` Mike Fedyk
2003-09-13 7:00 ` Oleg Drokin
1 sibling, 1 reply; 17+ messages in thread
From: Kyle Rose @ 2003-09-12 21:05 UTC (permalink / raw)
To: Oleg Drokin; +Cc: linux-kernel
> Does this affect 2.4 also?
As I said, this did not affect 2.4, at least not uniformly: I
regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2}
without problems.
Cheers,
Kyle
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 21:05 ` Kyle Rose
@ 2003-09-12 21:56 ` Mike Fedyk
2003-09-13 0:46 ` Hans Reiser
0 siblings, 1 reply; 17+ messages in thread
From: Mike Fedyk @ 2003-09-12 21:56 UTC (permalink / raw)
To: Kyle Rose; +Cc: Oleg Drokin, linux-kernel
On Fri, Sep 12, 2003 at 05:05:24PM -0400, Kyle Rose wrote:
> > Does this affect 2.4 also?
>
> As I said, this did not affect 2.4, at least not uniformly: I
> regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2}
> without problems.
Ok, thanks. I'm glad it wasn't in 2.4
EOST
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 21:56 ` Mike Fedyk
@ 2003-09-13 0:46 ` Hans Reiser
0 siblings, 0 replies; 17+ messages in thread
From: Hans Reiser @ 2003-09-13 0:46 UTC (permalink / raw)
To: Mike Fedyk; +Cc: Kyle Rose, Oleg Drokin, linux-kernel
Mike Fedyk wrote:
>On Fri, Sep 12, 2003 at 05:05:24PM -0400, Kyle Rose wrote:
>
>
>>>Does this affect 2.4 also?
>>>
>>>
>>As I said, this did not affect 2.4, at least not uniformly: I
>>regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2}
>>without problems.
>>
>>
>
>Ok, thanks. I'm glad it wasn't in 2.4
>
>EOST
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
>
>
>
>
It was a bug in Oleg's large write patch, which it seems was truly for
2.5 only.....
--
Hans
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 20:54 ` Mike Fedyk
2003-09-12 21:05 ` Kyle Rose
@ 2003-09-13 7:00 ` Oleg Drokin
1 sibling, 0 replies; 17+ messages in thread
From: Oleg Drokin @ 2003-09-13 7:00 UTC (permalink / raw)
To: Kyle Rose, linux-kernel
Hello!
On Fri, Sep 12, 2003 at 01:54:46PM -0700, Mike Fedyk wrote:
> > > > > However, just as the write completed, the beginning of the file became
> > > > > corrupted. I considered a 4GB problem to be likely, and re-tested
> > > > You are absolutely right.
> > > > Ther is a reiserfs problem that I just found based on your description.
> > > > The patch below should help. Please confirm that it works for you too.
> > > > Thanks a lot for the report.
> > > Yow, I guess large files on reiserfs in 2.6 isn't very common...
> > Or may be nobody noticed the corruption.
> Possible.
> Does this affect 2.4 also? If not, then that will narrow the possible
> number of people who could have hit this bug.
No, 2.4 is not affected.
Bye,
Oleg
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Large-file corruption. ReiserFS? VFS?
2003-09-12 15:39 ` Oleg Drokin
2003-09-12 17:59 ` Mike Fedyk
@ 2003-09-14 12:13 ` Kyle Rose
1 sibling, 0 replies; 17+ messages in thread
From: Kyle Rose @ 2003-09-14 12:13 UTC (permalink / raw)
To: Oleg Drokin; +Cc: linux-kernel
> You are absolutely right.
> Ther is a reiserfs problem that I just found based on your description.
> The patch below should help. Please confirm that it works for you too.
> Thanks a lot for the report.
This indeed solves the problem. Thanks, Oleg!
Cheers,
Kyle
^ permalink raw reply [flat|nested] 17+ messages in thread
[parent not found: <uGoz.5NC.7@gated-at.bofh.it>]
* Re: Large-file corruption. ReiserFS? VFS?
[not found] <uGoz.5NC.7@gated-at.bofh.it>
@ 2003-09-11 20:49 ` Pascal Schmidt
0 siblings, 0 replies; 17+ messages in thread
From: Pascal Schmidt @ 2003-09-11 20:49 UTC (permalink / raw)
To: Kyle Rose; +Cc: linux-kernel
On Thu, 11 Sep 2003 22:40:11 +0200, you wrote in linux.kernel:
> However, just as the write completed, the beginning of the file became
> corrupted. I considered a 4GB problem to be likely, and re-tested
> with fewer source files such that the result would be smaller than
> 4GB; lo and behold, no corruption. The same result occurs whether
> mkisofs is given the -o flag, or output is redirected to a file from
> stdout using the shell's redirection facility, suggesting the problem
> is probably at the kernel level.
Could also be that your mkisofs is not compiled with large file
support.
--
Ciao,
Pascal
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2003-09-14 12:13 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-09-11 20:28 Large-file corruption. ReiserFS? VFS? Kyle Rose
[not found] ` <20030911144732.S18851@schatzie.adilger.int>
2003-09-11 20:56 ` Kyle Rose
2003-09-11 21:00 ` Andreas Dilger
2003-09-13 6:50 ` Rogier Wolff
2003-09-13 6:51 ` Rogier Wolff
2003-09-12 9:36 ` Matthias Schniedermeyer
2003-09-12 11:47 ` Kyle Rose
2003-09-12 15:39 ` Oleg Drokin
2003-09-12 17:59 ` Mike Fedyk
2003-09-12 18:40 ` Oleg Drokin
2003-09-12 20:54 ` Mike Fedyk
2003-09-12 21:05 ` Kyle Rose
2003-09-12 21:56 ` Mike Fedyk
2003-09-13 0:46 ` Hans Reiser
2003-09-13 7:00 ` Oleg Drokin
2003-09-14 12:13 ` Kyle Rose
[not found] <uGoz.5NC.7@gated-at.bofh.it>
2003-09-11 20:49 ` Pascal Schmidt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).