[PATCH v2] create_inode: fix copying large files

[PATCH v2] create_inode: fix copying large files

[Ross Burton]

When copying large files into a ext filesystem at mkfs time the copy fails at
2^31 bytes in.  There are two problems:

copy_file_chunk() passes an offset (off_t, 64-bit typically) to
ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
instead.

try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
boundaries, but the calculation gets truncated to 32-bits.  Solve by casting the
32-bit blocksize to off_t to ensure this doesn't happen.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 misc/create_inode.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/misc/create_inode.c b/misc/create_inode.c
index 05aa6363..1b35c769 100644
--- a/misc/create_inode.c
+++ b/misc/create_inode.c
@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
 				ptr += blen;
 				continue;
 			}
-			err = ext2fs_file_lseek(e2_file, off + bpos,
+			err = ext2fs_file_llseek(e2_file, off + bpos,
 						EXT2_SEEK_SET, NULL);
 			if (err)
 				goto fail;
@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
 		if (hole < 0)
 			return EXT2_ET_UNIMPLEMENTED;
 
-		data_blk = data & ~(fs->blocksize - 1);
-		hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
+		data_blk = data & ~(off_t)(fs->blocksize - 1);
+		hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
 		err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
 				      zerobuf);
 		if (err)
-- 
2.11.0

Re: [PATCH v2] create_inode: fix copying large files

[Darrick J. Wong]

On Fri, Feb 01, 2019 at 06:32:47PM +0000, Ross Burton wrote:
> When copying large files into a ext filesystem at mkfs time the copy fails at
> 2^31 bytes in.  There are two problems:
> 
> copy_file_chunk() passes an offset (off_t, 64-bit typically) to
> ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
> is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
> instead.
> 
> try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
> boundaries, but the calculation gets truncated to 32-bits.  Solve by casting the
> 32-bit blocksize to off_t to ensure this doesn't happen.
> 
> Signed-off-by: Ross Burton <ross.burton@intel.com>

Looks ok,
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>

--D

> ---
>  misc/create_inode.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/misc/create_inode.c b/misc/create_inode.c
> index 05aa6363..1b35c769 100644
> --- a/misc/create_inode.c
> +++ b/misc/create_inode.c
> @@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
>  				ptr += blen;
>  				continue;
>  			}
> -			err = ext2fs_file_lseek(e2_file, off + bpos,
> +			err = ext2fs_file_llseek(e2_file, off + bpos,
>  						EXT2_SEEK_SET, NULL);
>  			if (err)
>  				goto fail;
> @@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
>  		if (hole < 0)
>  			return EXT2_ET_UNIMPLEMENTED;
>  
> -		data_blk = data & ~(fs->blocksize - 1);
> -		hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
> +		data_blk = data & ~(off_t)(fs->blocksize - 1);
> +		hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
>  		err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
>  				      zerobuf);
>  		if (err)
> -- 
> 2.11.0
> 

Re: [PATCH v2] create_inode: fix copying large files

[Burton, Ross]

Obviously this was a problem that the test suite should have spotted.
Would it be simple enough to write a test suite to make a 4.1GB file,
put in in a ext with -d, and the md5 the file in the file system?  I
was using sudo and loopback mounts which isn't really ideal for the
test suite.

Ross

On Fri, 1 Feb 2019 at 18:59, Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Fri, Feb 01, 2019 at 06:32:47PM +0000, Ross Burton wrote:
> > When copying large files into a ext filesystem at mkfs time the copy fails at
> > 2^31 bytes in.  There are two problems:
> >
> > copy_file_chunk() passes an offset (off_t, 64-bit typically) to
> > ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
> > is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
> > instead.
> >
> > try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
> > boundaries, but the calculation gets truncated to 32-bits.  Solve by casting the
> > 32-bit blocksize to off_t to ensure this doesn't happen.
> >
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
>
> Looks ok,
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
>
> --D
>
> > ---
> >  misc/create_inode.c | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/misc/create_inode.c b/misc/create_inode.c
> > index 05aa6363..1b35c769 100644
> > --- a/misc/create_inode.c
> > +++ b/misc/create_inode.c
> > @@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
> >                               ptr += blen;
> >                               continue;
> >                       }
> > -                     err = ext2fs_file_lseek(e2_file, off + bpos,
> > +                     err = ext2fs_file_llseek(e2_file, off + bpos,
> >                                               EXT2_SEEK_SET, NULL);
> >                       if (err)
> >                               goto fail;
> > @@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
> >               if (hole < 0)
> >                       return EXT2_ET_UNIMPLEMENTED;
> >
> > -             data_blk = data & ~(fs->blocksize - 1);
> > -             hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
> > +             data_blk = data & ~(off_t)(fs->blocksize - 1);
> > +             hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
> >               err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
> >                                     zerobuf);
> >               if (err)
> > --
> > 2.11.0
> >

Re: [PATCH v2] create_inode: fix copying large files

[Burton, Ross]

Gentle reminder to the maintainers that this patch hasn't been merged yet.

Ross

On Fri, 1 Feb 2019 at 18:59, Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Fri, Feb 01, 2019 at 06:32:47PM +0000, Ross Burton wrote:
> > When copying large files into a ext filesystem at mkfs time the copy fails at
> > 2^31 bytes in.  There are two problems:
> >
> > copy_file_chunk() passes an offset (off_t, 64-bit typically) to
> > ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
> > is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
> > instead.
> >
> > try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
> > boundaries, but the calculation gets truncated to 32-bits.  Solve by casting the
> > 32-bit blocksize to off_t to ensure this doesn't happen.
> >
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
>
> Looks ok,
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
>
> --D
>
> > ---
> >  misc/create_inode.c | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/misc/create_inode.c b/misc/create_inode.c
> > index 05aa6363..1b35c769 100644
> > --- a/misc/create_inode.c
> > +++ b/misc/create_inode.c
> > @@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
> >                               ptr += blen;
> >                               continue;
> >                       }
> > -                     err = ext2fs_file_lseek(e2_file, off + bpos,
> > +                     err = ext2fs_file_llseek(e2_file, off + bpos,
> >                                               EXT2_SEEK_SET, NULL);
> >                       if (err)
> >                               goto fail;
> > @@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
> >               if (hole < 0)
> >                       return EXT2_ET_UNIMPLEMENTED;
> >
> > -             data_blk = data & ~(fs->blocksize - 1);
> > -             hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
> > +             data_blk = data & ~(off_t)(fs->blocksize - 1);
> > +             hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
> >               err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
> >                                     zerobuf);
> >               if (err)
> > --
> > 2.11.0
> >

Re: [PATCH v2] create_inode: fix copying large files

[Theodore Y. Ts'o]

On Fri, Feb 01, 2019 at 10:59:50AM -0800, Darrick J. Wong wrote:
> On Fri, Feb 01, 2019 at 06:32:47PM +0000, Ross Burton wrote:
> > When copying large files into a ext filesystem at mkfs time the copy fails at
> > 2^31 bytes in.  There are two problems:
> > 
> > copy_file_chunk() passes an offset (off_t, 64-bit typically) to
> > ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
> > is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
> > instead.
> > 
> > try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
> > boundaries, but the calculation gets truncated to 32-bits.  Solve by casting the
> > 32-bit blocksize to off_t to ensure this doesn't happen.
> > 
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
> 
> Looks ok,
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>

Thanks, applied.

					- Ted