Files truncate on vfat filesystem

Files truncate on vfat filesystem

[Peder Stray]

I have a 250GB usb-storage disk i use to transport large files between
work and home, I uses vfat (since I haven't found any other good
filesystems that don't require me to either be root or have all files
worldreadable). Anyways...

Some files get its size truncated to 0 after a while (usually a few
minuts, or when i umount the disk). They seem to be the correct size
when I do ls -l immediately after i have transfered the files (with cp
or rsync, doesn't really matter). Moving files on the disk with mv also
seems to trigger the problem sometimes.

I see that blocks get allocated, but the filesizes are 0. Currently
there is a difference of 17GB in the output of df and du.

I have also noticed that the size of some directories get truncated too,
thus all files copied or moved into those directories dissappear. ls -l
in those directories doesn't even show . and ..

it seem very inconsistent which files are affected, both in size, length
of filename, unusual characters or depth in the filestructure. No
messages from the kernel logs.

A check of the filesystem from XP reports no errors in the
filestructure, and it work 100% there.

kernel versions used are are amongst 2.4.18 and 2.4.20, selfcompiled or
stock from RH.

More details can of course be supplied if anyone have any ideas what to
check and how.

any comments or help would be much appreciated as the loss of data i
experience is more than a little annyoing.

-- 
  Peder Stray

Re: Files truncate on vfat filesystem

[OGAWA Hirofumi]

Peder Stray <peder@ifi.uio.no> writes:

> I have a 250GB usb-storage disk i use to transport large files between
> work and home, I uses vfat (since I haven't found any other good
> filesystems that don't require me to either be root or have all files
> worldreadable). Anyways...

What partition size do you use? And does that FAT use what logical
sector size?  The directory entry pointer may be overflowed...
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: Files truncate on vfat filesystem

[Peder Stray]

On Wed, 7 May 2003, OGAWA Hirofumi wrote:

> Peder Stray <peder@ifi.uio.no> writes:
>
> > I have a 250GB usb-storage disk i use to transport large files between
> > work and home, I uses vfat (since I haven't found any other good
> > filesystems that don't require me to either be root or have all files
> > worldreadable). Anyways...
>
> What partition size do you use? And does that FAT use what logical
> sector size?  The directory entry pointer may be overflowed...

250GB partition, FAT32 LBA (partition type 0x0c i think).

The problem is verry inconsistent as i said earlier, so the number of
files in a directory doesn't seem to matter, nor do the depth in the
directory structure. Some files i manage to copy yo the disk, some files i
don't... I havent managed to find any pattern in what files this affects.

-- 
  Peder Stray

Re: Files truncate on vfat filesystem

[OGAWA Hirofumi]

Peder Stray <peder@ifi.uio.no> writes:

> > What partition size do you use? And does that FAT use what logical
> > sector size?  The directory entry pointer may be overflowed...
> 
> 250GB partition, FAT32 LBA (partition type 0x0c i think).
> 
> The problem is verry inconsistent as i said earlier, so the number of
> files in a directory doesn't seem to matter, nor do the depth in the
> directory structure. Some files i manage to copy yo the disk, some files i
> don't... I havent managed to find any pattern in what files this affects.

I meant, "directory entry pointer" is position of the directory entry
in the _partition_. like the following,

in fs/fat/misc.c:fat__get_entry()

		offset &= sb->s_blocksize - 1;
		*de = (struct msdos_dir_entry *) ((*bh)->b_data + offset);
		*ino = (sector << sbi->dir_per_block_bits) + (offset >> MSDOS_DIR_BITS);
                 ^^^

This is used for _updates_ (not create) of the directory entry. And
there is a possibility of cause of the problem which you said.  If my
guess is right and a partition will be splited small, a problem will
not occur.

In short, I think it's the bug of fat driver and it's needed the fix.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: Files truncate on vfat filesystem

[Peder Stray]

On Wed, 7 May 2003, OGAWA Hirofumi wrote:

> Peder Stray <peder@ifi.uio.no> writes:
>
> > The problem is verry inconsistent as i said earlier, so the number of
> > files in a directory doesn't seem to matter, nor do the depth in the
> > directory structure. Some files i manage to copy yo the disk, some files i
> > don't... I havent managed to find any pattern in what files this affects.
>
> I meant, "directory entry pointer" is position of the directory entry
> in the _partition_. like the following,

aha, that was what you ment...

> in fs/fat/misc.c:fat__get_entry()
>
>	offset &= sb->s_blocksize - 1;
>	*de = (struct msdos_dir_entry *) ((*bh)->b_data + offset);
>	*ino = (sector << sbi->dir_per_block_bits) + (offset >> MSDOS_DIR_BITS);
>        ^^^
>
> This is used for _updates_ (not create) of the directory entry. And
> there is a possibility of cause of the problem which you said.  If my
> guess is right and a partition will be splited small, a problem will
> not occur.
>
> In short, I think it's the bug of fat driver and it's needed the fix.

So effeivly all directories located over a given boundry on the disk would
be affected? Would removing a file written to the disk early and making a
few directories insted fix some of the problems... Have to try that...

Need to find a way to get back the 17G of allocated but unsuable data
too...

-- 
  Peder Stray

Re: Files truncate on vfat filesystem

[Sumit Narayan]

This probably couldnt be the problem, but I have faced the similar problem on my XP machine, while transferring files ranging in 600MB+ from CD to HD. The transfer would be complete, and when I remove the CD, it showed the file size 0. I just assumed that the same problem existed with you also.
Sorry for wrong info..
Sumit 
--

On Tue, 6 May 2003 18:12:07   
 Peder Stray wrote:
>On Tue, 6 May 2003, Sumit Narayan wrote:
>
>> I think the problem that you are facing is because you umount the disk
>> immediately after transferring the file. Though ls indicates that the
>> file has been transferred, and the size match, the transfer would still
>> be in progress behind the scene, and once you umount, and remove the USB
>> disk, the data is lost, since the transfer was not complete.
>
>That is definitly not the problem... i have mounted both with and without
>the sync option, and umount usually hangs until all buffers are flushed to
>the disk.
>
>> You are facing this problem randomly, depending on how long you wait
>> after the transfer to remove the disk.
>
>yes, i usually have the disk mounted all the time it is connected to my
>computer, and umount/mount is just somthing i use to check if the files
>got transferred correctly
>
>> This happens when you are transferring large files, and not while using
>> small files.
>
>size doesn't matter at all with this problem... files ranging from under
>1k in size to well up under 1GiB have been truncated
>
>> This will happen even when you are transferring large files from your
>> CDs.
>
>never tried to copy from my cd.
>
>> I hope this helps. Just wait for few seconds after the transfer is done,
>> and you wont lose your data anymore.
>
>sorry to report that your input didn't help at all.
>
>> Regards,
>> Sumit
>> --
>>
>> On 06 May 2003 17:29:12 +020
>>  Peder Stray wrote:
>> >
>> >I have a 250GB usb-storage disk i use to transport large files between
>> >work and home, I uses vfat (since I haven't found any other good
>> >filesystems that don't require me to either be root or have all files
>> >worldreadable). Anyways...
>> >
>> >Some files get its size truncated to 0 after a while (usually a few
>> >minuts, or when i umount the disk). They seem to be the correct size
>> >when I do ls -l immediately after i have transfered the files (with cp
>> >or rsync, doesn't really matter). Moving files on the disk with mv also
>> >seems to trigger the problem sometimes.
>> >
>> >I see that blocks get allocated, but the filesizes are 0. Currently
>> >there is a difference of 17GB in the output of df and du.
>> >
>> >I have also noticed that the size of some directories get truncated too,
>> >thus all files copied or moved into those directories dissappear. ls -l
>> >in those directories doesn't even show . and ..
>> >
>> >it seem very inconsistent which files are affected, both in size, length
>> >of filename, unusual characters or depth in the filestructure. No
>> >messages from the kernel logs.
>> >
>> >A check of the filesystem from XP reports no errors in the
>> >filestructure, and it work 100% there.
>> >
>> >kernel versions used are are amongst 2.4.18 and 2.4.20, selfcompiled or
>> >stock from RH.
>> >
>> >More details can of course be supplied if anyone have any ideas what to
>> >check and how.
>> >
>> >any comments or help would be much appreciated as the loss of data i
>> >experience is more than a little annyoing.
>> >
>> >--
>> >  Peder Stray
>> >
>> >-
>> >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> >the body of a message to majordomo@vger.kernel.org
>> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >Please read the FAQ at  http://www.tux.org/lkml/
>> >
>>
>>
>> ____________________________________________________________
>> Get advanced SPAM filtering on Webmail or POP Mail ... Get Lycos Mail!
>> http://login.mail.lycos.com/r/referral?aid=27005
>>
>
>-- 
>  Peder Stray
>


____________________________________________________________
Get advanced SPAM filtering on Webmail or POP Mail ... Get Lycos Mail!
http://login.mail.lycos.com/r/referral?aid=27005

Re: Files truncate on vfat filesystem

[Peder Stray]

On Tue, 6 May 2003, Sumit Narayan wrote:

> I think the problem that you are facing is because you umount the disk
> immediately after transferring the file. Though ls indicates that the
> file has been transferred, and the size match, the transfer would still
> be in progress behind the scene, and once you umount, and remove the USB
> disk, the data is lost, since the transfer was not complete.

That is definitly not the problem... i have mounted both with and without
the sync option, and umount usually hangs until all buffers are flushed to
the disk.

> You are facing this problem randomly, depending on how long you wait
> after the transfer to remove the disk.

yes, i usually have the disk mounted all the time it is connected to my
computer, and umount/mount is just somthing i use to check if the files
got transferred correctly

> This happens when you are transferring large files, and not while using
> small files.

size doesn't matter at all with this problem... files ranging from under
1k in size to well up under 1GiB have been truncated

> This will happen even when you are transferring large files from your
> CDs.

never tried to copy from my cd.

> I hope this helps. Just wait for few seconds after the transfer is done,
> and you wont lose your data anymore.

sorry to report that your input didn't help at all.

> Regards,
> Sumit
> --
>
> On 06 May 2003 17:29:12 +020
>  Peder Stray wrote:
> >
> >I have a 250GB usb-storage disk i use to transport large files between
> >work and home, I uses vfat (since I haven't found any other good
> >filesystems that don't require me to either be root or have all files
> >worldreadable). Anyways...
> >
> >Some files get its size truncated to 0 after a while (usually a few
> >minuts, or when i umount the disk). They seem to be the correct size
> >when I do ls -l immediately after i have transfered the files (with cp
> >or rsync, doesn't really matter). Moving files on the disk with mv also
> >seems to trigger the problem sometimes.
> >
> >I see that blocks get allocated, but the filesizes are 0. Currently
> >there is a difference of 17GB in the output of df and du.
> >
> >I have also noticed that the size of some directories get truncated too,
> >thus all files copied or moved into those directories dissappear. ls -l
> >in those directories doesn't even show . and ..
> >
> >it seem very inconsistent which files are affected, both in size, length
> >of filename, unusual characters or depth in the filestructure. No
> >messages from the kernel logs.
> >
> >A check of the filesystem from XP reports no errors in the
> >filestructure, and it work 100% there.
> >
> >kernel versions used are are amongst 2.4.18 and 2.4.20, selfcompiled or
> >stock from RH.
> >
> >More details can of course be supplied if anyone have any ideas what to
> >check and how.
> >
> >any comments or help would be much appreciated as the loss of data i
> >experience is more than a little annyoing.
> >
> >--
> >  Peder Stray
> >
> >-
> >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >Please read the FAQ at  http://www.tux.org/lkml/
> >
>
>
> ____________________________________________________________
> Get advanced SPAM filtering on Webmail or POP Mail ... Get Lycos Mail!
> http://login.mail.lycos.com/r/referral?aid=27005
>

-- 
  Peder Stray

Re: Files truncate on vfat filesystem

[Sumit Narayan]

Peder,

I think the problem that you are facing is because you umount the disk immediately after transferring the file. Though ls indicates that the file has been transferred, and the size match, the transfer would still be in progress behind the scene, and once you umount, and remove the USB disk, the data is lost, since the transfer was not complete. You are facing this problem randomly, depending on how long you wait after the transfer to remove the disk. This happens when you are transferring large files, and not while using small files. This will happen even when you are transferring large files from your CDs.
I hope this helps. Just wait for few seconds after the transfer is done, and you wont lose your data anymore.

Regards,
Sumit 
--

On 06 May 2003 17:29:12 +020  
 Peder Stray wrote:
>
>I have a 250GB usb-storage disk i use to transport large files between
>work and home, I uses vfat (since I haven't found any other good
>filesystems that don't require me to either be root or have all files
>worldreadable). Anyways...
>
>Some files get its size truncated to 0 after a while (usually a few
>minuts, or when i umount the disk). They seem to be the correct size
>when I do ls -l immediately after i have transfered the files (with cp
>or rsync, doesn't really matter). Moving files on the disk with mv also
>seems to trigger the problem sometimes.
>
>I see that blocks get allocated, but the filesizes are 0. Currently
>there is a difference of 17GB in the output of df and du.
>
>I have also noticed that the size of some directories get truncated too,
>thus all files copied or moved into those directories dissappear. ls -l
>in those directories doesn't even show . and ..
>
>it seem very inconsistent which files are affected, both in size, length
>of filename, unusual characters or depth in the filestructure. No
>messages from the kernel logs.
>
>A check of the filesystem from XP reports no errors in the
>filestructure, and it work 100% there.
>
>kernel versions used are are amongst 2.4.18 and 2.4.20, selfcompiled or
>stock from RH.
>
>More details can of course be supplied if anyone have any ideas what to
>check and how.
>
>any comments or help would be much appreciated as the loss of data i
>experience is more than a little annyoing.
>
>-- 
>  Peder Stray
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>


____________________________________________________________
Get advanced SPAM filtering on Webmail or POP Mail ... Get Lycos Mail!
http://login.mail.lycos.com/r/referral?aid=27005