Large-file corruption. ReiserFS? VFS?

Large-file corruption. ReiserFS? VFS?

[Kyle Rose]

I ran mkisofs (2.01a16) this morning on a 2.6.0-test4 machine with the
target on a 60GB ReiserFS partition.  The resulting file was to be
just over 4GB (around 4,360,000,000 bytes), and during most of the
course of writing the file, a small prefix looked as it should through
a hexdump: that is, just part of the iso9660 directory.
 
However, just as the write completed, the beginning of the file became
corrupted.  I considered a 4GB problem to be likely, and re-tested
with fewer source files such that the result would be smaller than
4GB; lo and behold, no corruption.  The same result occurs whether
mkisofs is given the -o flag, or output is redirected to a file from
stdout using the shell's redirection facility, suggesting the problem
is probably at the kernel level.
 
I don't have a large enough ext2/3 filesystem to compare with, so
there's no easy way for me to tell whether this is a Reiser-only
problem or not.  Can anyone confirm that they see the same problem, or
whether they see a similar problem on another file system?  Please
feel free to ask me for any other information you think might be
illuminating.

Cheers, 
Kyle 

Re: Large-file corruption. ReiserFS? VFS?

[Kyle Rose]

> I would guess that mkisofs isn't opening the file with O_LARGEFILE.
> It probably only expected to write 600MB output files.  Purely a
> guess though.

Thanks for the suggestion, but it is, in fact, opening with
O_LARGEFILE:

open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3

Cheers,
Kyle

Re: Large-file corruption. ReiserFS? VFS?

[Andreas Dilger]

On Sep 11, 2003  16:56 -0400, Kyle Rose wrote:
> > I would guess that mkisofs isn't opening the file with O_LARGEFILE.
> > It probably only expected to write 600MB output files.  Purely a
> > guess though.
> 
> Thanks for the suggestion, but it is, in fact, opening with
> O_LARGEFILE:
> 
> open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3

True, and O_LARGEFILE would have bit you at 2GB and not 4GB...  If you are
doing output redirected from the shell, then it can't be a seek issue
either.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Large-file corruption. ReiserFS? VFS?

[Matthias Schniedermeyer]

On Thu, Sep 11, 2003 at 04:28:58PM -0400, Kyle Rose wrote:
>  
> I don't have a large enough ext2/3 filesystem to compare with, so
> there's no easy way for me to tell whether this is a Reiser-only
> problem or not.  Can anyone confirm that they see the same problem, or
> whether they see a similar problem on another file system?  Please
> feel free to ask me for any other information you think might be
> illuminating.

I can only speak for 2.4-Kernel

I burn DVDs since more than 2 years now. Starting with 2.4.4 and i had
NEVER problems with the size, i never used the split-option of mkisofs
(and i don't even know the correct(tm) name of it. I only know that it
exists).
The maximum is 4489MB, (>4.700.000.000 bytes) and the biggest images i
had are only a few KB smaller that that.

Since about 2 month i use reiserfs on the "source" and on the
"target"-HDD. No changes, except that reiserfs is slower then ext2.





Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

Re: Large-file corruption. ReiserFS? VFS?

[Kyle Rose]

> I can only speak for 2.4-Kernel

Yes, I never had a problem doing this on 2.4.  It started when I
upgraded to 2.6.

Cheers,
Kyle

Re: Large-file corruption. ReiserFS? VFS?

[Oleg Drokin]

Hello!

On Thu, Sep 11, 2003 at 04:28:58PM -0400, Kyle Rose wrote:

> However, just as the write completed, the beginning of the file became
> corrupted.  I considered a 4GB problem to be likely, and re-tested

You are absolutely right.
Ther is a reiserfs problem that I just found based on your description.
The patch below should help. Please confirm that it works for you too.
Thanks a lot for the report.

Bye,
    Oleg

===== fs/reiserfs/file.c 1.22 vs edited =====
--- 1.22/fs/reiserfs/file.c	Fri Aug 15 05:17:00 2003
+++ edited/fs/reiserfs/file.c	Fri Sep 12 19:18:53 2003
@@ -779,7 +779,7 @@
     /* Now if all the write area lies past the file end, no point in
        maping blocks, since there is none, so we just zero out remaining
        parts of first and last pages in write area (if needed) */
-    if ( (pos & ~(PAGE_CACHE_SIZE - 1)) > inode->i_size ) {
+    if ( (pos & ~((loff_t)PAGE_CACHE_SIZE - 1)) > inode->i_size ) {
 	if ( from != 0 ) {/* First page needs to be partially zeroed */
 	    char *kaddr = kmap_atomic(prepared_pages[0], KM_USER0);
 	    memset(kaddr, 0, from);
@@ -801,9 +801,9 @@
        we need to allocate (calculated above) */
     /* Mask write position to start on blocksize, we do it out of the
        loop for performance reasons */
-    pos &= ~(inode->i_sb->s_blocksize - 1);
+    pos &= ~((loff_t) inode->i_sb->s_blocksize - 1);
     /* Set cpu key to the starting position in a file (on left block boundary)*/
-    make_cpu_key (&key, inode, 1 + ((pos) & ~(inode->i_sb->s_blocksize - 1)), TYPE_ANY, 3/*key length*/);
+    make_cpu_key (&key, inode, 1 + ((pos) & ~((loff_t) inode->i_sb->s_blocksize - 1)), TYPE_ANY, 3/*key length*/);
 
     reiserfs_write_lock(inode->i_sb); // We need that for at least search_by_key()
     for ( i = 0; i < num_pages ; i++ ) { 

Re: Large-file corruption. ReiserFS? VFS?

[Mike Fedyk]

On Fri, Sep 12, 2003 at 07:39:35PM +0400, Oleg Drokin wrote:
> Hello!
> 
> On Thu, Sep 11, 2003 at 04:28:58PM -0400, Kyle Rose wrote:
> 
> > However, just as the write completed, the beginning of the file became
> > corrupted.  I considered a 4GB problem to be likely, and re-tested
> 
> You are absolutely right.
> Ther is a reiserfs problem that I just found based on your description.
> The patch below should help. Please confirm that it works for you too.
> Thanks a lot for the report.
> 

Yow, I guess large files on reiserfs in 2.6 isn't very common...

Re: Large-file corruption. ReiserFS? VFS?

[Oleg Drokin]

Hello!

On Fri, Sep 12, 2003 at 10:59:17AM -0700, Mike Fedyk wrote:
> > > However, just as the write completed, the beginning of the file became
> > > corrupted.  I considered a 4GB problem to be likely, and re-tested
> > You are absolutely right.
> > Ther is a reiserfs problem that I just found based on your description.
> > The patch below should help. Please confirm that it works for you too.
> > Thanks a lot for the report.
> Yow, I guess large files on reiserfs in 2.6 isn't very common...

Or may be nobody noticed the corruption.

Bye,
    Oleg

Re: Large-file corruption. ReiserFS? VFS?

[Mike Fedyk]

On Fri, Sep 12, 2003 at 10:40:01PM +0400, Oleg Drokin wrote:
> Hello!
> 
> On Fri, Sep 12, 2003 at 10:59:17AM -0700, Mike Fedyk wrote:
> > > > However, just as the write completed, the beginning of the file became
> > > > corrupted.  I considered a 4GB problem to be likely, and re-tested
> > > You are absolutely right.
> > > Ther is a reiserfs problem that I just found based on your description.
> > > The patch below should help. Please confirm that it works for you too.
> > > Thanks a lot for the report.
> > Yow, I guess large files on reiserfs in 2.6 isn't very common...
> 
> Or may be nobody noticed the corruption.

Possible.

Does this affect 2.4 also?  If not, then that will narrow the possible
number of people who could have hit this bug.

Luckily when I was using reiserfs, I used mostly small files.

Re: Large-file corruption. ReiserFS? VFS?

[Kyle Rose]

> Does this affect 2.4 also?

As I said, this did not affect 2.4, at least not uniformly: I
regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2}
without problems.

Cheers,
Kyle

Re: Large-file corruption. ReiserFS? VFS?

[Mike Fedyk]

On Fri, Sep 12, 2003 at 05:05:24PM -0400, Kyle Rose wrote:
> > Does this affect 2.4 also?
> 
> As I said, this did not affect 2.4, at least not uniformly: I
> regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2}
> without problems.

Ok, thanks.  I'm glad it wasn't in 2.4

EOST

Re: Large-file corruption. ReiserFS? VFS?

[Hans Reiser]

Mike Fedyk wrote:

>On Fri, Sep 12, 2003 at 05:05:24PM -0400, Kyle Rose wrote:
>  
>
>>>Does this affect 2.4 also?
>>>      
>>>
>>As I said, this did not affect 2.4, at least not uniformly: I
>>regularly created DVD ISO images larger than 4GB under 2.4.2{0,1,2}
>>without problems.
>>    
>>
>
>Ok, thanks.  I'm glad it wasn't in 2.4
>
>EOST
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
It was a bug in Oleg's large write patch, which it seems was truly for 
2.5 only.....

-- 
Hans

Re: Large-file corruption. ReiserFS? VFS?

[Rogier Wolff]

On Thu, Sep 11, 2003 at 03:00:17PM -0600, Andreas Dilger wrote:
> True, and O_LARGEFILE would have bit you at 2GB and not 4GB...  If you are
> doing output redirected from the shell, then it can't be a seek issue
> either.

Nah! Some programs just "try to seek" and if it works, will. 

dd if=/dev/zero count=200 | strace sox -t raw -r 44100 -s -w - -t wav - > /tmp/test

dd if=/dev/zero count=200 | sox -t raw -r 44100 -s -w - -t wav - | dd of=/tmp/test
sox: Length in output .wav header will be wrong since can't seek to fix it

		Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****

Re: Large-file corruption. ReiserFS? VFS?

[Rogier Wolff]

On Thu, Sep 11, 2003 at 03:00:17PM -0600, Andreas Dilger wrote:
> On Sep 11, 2003  16:56 -0400, Kyle Rose wrote:
> > > I would guess that mkisofs isn't opening the file with O_LARGEFILE.
> > > It probably only expected to write 600MB output files.  Purely a
> > > guess though.
> > 
> > Thanks for the suggestion, but it is, in fact, opening with
> > O_LARGEFILE:
> > 
> > open("/mnt/angband/krose/tmp/862839.img", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
> 
> True, and O_LARGEFILE would have bit you at 2GB and not 4GB...  If you are
> doing output redirected from the shell, then it can't be a seek issue
> either.

... but mkisofs will NOT be seeking, as we commonly burn CDs while
mkisofs is providing the data through a pipe. 

		Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****

Re: Large-file corruption. ReiserFS? VFS?

[Oleg Drokin]

Hello!

On Fri, Sep 12, 2003 at 01:54:46PM -0700, Mike Fedyk wrote:
> > > > > However, just as the write completed, the beginning of the file became
> > > > > corrupted.  I considered a 4GB problem to be likely, and re-tested
> > > > You are absolutely right.
> > > > Ther is a reiserfs problem that I just found based on your description.
> > > > The patch below should help. Please confirm that it works for you too.
> > > > Thanks a lot for the report.
> > > Yow, I guess large files on reiserfs in 2.6 isn't very common...
> > Or may be nobody noticed the corruption.
> Possible.
> Does this affect 2.4 also?  If not, then that will narrow the possible
> number of people who could have hit this bug.

No, 2.4 is not affected.

Bye,
    Oleg

Re: Large-file corruption. ReiserFS? VFS?

[Kyle Rose]

> You are absolutely right.
> Ther is a reiserfs problem that I just found based on your description.
> The patch below should help. Please confirm that it works for you too.
> Thanks a lot for the report.

This indeed solves the problem.  Thanks, Oleg!

Cheers,
Kyle

Re: Large-file corruption. ReiserFS? VFS?

[Pascal Schmidt]

On Thu, 11 Sep 2003 22:40:11 +0200, you wrote in linux.kernel:

> However, just as the write completed, the beginning of the file became
> corrupted.  I considered a 4GB problem to be likely, and re-tested
> with fewer source files such that the result would be smaller than
> 4GB; lo and behold, no corruption.  The same result occurs whether
> mkisofs is given the -o flag, or output is redirected to a file from
> stdout using the shell's redirection facility, suggesting the problem
> is probably at the kernel level.

Could also be that your mkisofs is not compiled with large file
support.

-- 
Ciao,
Pascal