ANN: NTFS new release available (1.1.15)

ANN: NTFS new release available (1.1.15)

[Anton Altaparmakov]

Announcing a new NTFS patch (1.1.15). It is generated against kernel 
2.4.4-ac16 but also applies cleanly to -ac17. Quite probably it will apply 
fine to any 2.4.4 kernel (at least 2.4.4-pre5 I think).

How to get the patch
====================

- Wait for one of the next -ac series kernel patches (patch is submitted to 
Alan Cox).
- For the impatient: patch is available (for a while only, for some value 
of while) for download as bzip2 or gzip compressed file from:
         http://www-stu.christs.cam.ac.uk/~aia21/ntfs/
(To apply patch, uncompress it, cd into your kernel directory and type:
         patch -p1 < 
/path/to/patch/ntfs-kernel-2.4.4-ac16-to-ntfs-1.1.15.diff.gz
Then you will need to recompile your kernel and reboot into it.)

NTFS 1.1.15 - ChangeLog
=======================

- New mount option show_sys_files=<bool> to show all system files as normal 
files.
- Support for files and in general any attributes up to the full 2TiB size 
supported by the NTFS filesystem. Note we only support up to 32-bits worth 
of inodes/clusters at this point.
- Support for more than 128kiB sized runlists (using vmalloc_32() instead 
of kmalloc()).
- Fixed races in allocation of clusters and mft records.
- Fixed major bugs in attribute handling / searching / collation.
- Fixed major bugs in compressing a run list into a mapping pairs array.
- Fixed major bugs in inode allocation. Especially file create and mkdir.
- Fixed memory leaks.
- Fixed major bug in inode layout assignment of sequence numbers.
- Lots of other bug fixes I can't think of right now...
- Fixed NULL bug found by the Stanford checker in ntfs_dupuni2map().
- Convert large stack variable to dynamically allocated one in 
ntfs_get_free_cluster_count() (found by Stanford checker).
- New versioning scheme as I was too confused by the date based one... 
(versions prior to this are internal only, never been released to public, 
so you haven't missed the numbers before 1.1.15).

Known bugs and (mis-)features
=============================

- Do not use the driver for writing as it corrupts the file system. If you 
do use it, get the Linux-NTFS tools and use the ntfsfix utility after 
dismounting a partition you wrote to.

- Use "ls -l" instead of just "ls", otherwise the last entry in the 
directory is not displayed for some directories. (?!?)

- Use the show_sys_files mount option which should make things work 
generally better. (It results in both the short and long file names being 
shown as well as the sytem files.)

- Special characters are not treated correctly. For example if the file 
name contains an apostrophe you will not be able to open it.

- Writing of extension records is not supported properly.

Please send bug reports/comments/feed back/abuse to the Linux-NTFS 
development list at sourceforge: linux-ntfs-dev@lists.sourceforge.net

For daring people, write support now works ok for simple files and 
directories. For example it is relatively safe to create a directory inside 
a not too big directory, and create a few relatively small files in it. 
umount, run ntfsfix, reboot in Windows, chkdsk will run and (hopefully) not 
detect any problems at all!

Thank you for your attention.

Best regards,

         Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: ANN: NTFS new release available (1.1.15)

[Jeff Garzik]

Anton Altaparmakov wrote:
> NTFS 1.1.15 - ChangeLog
> =======================
> - Support for more than 128kiB sized runlists (using vmalloc_32() instead
> of kmalloc()).

If you are running into kmalloc size limit, please consider some
alternative method of allocation.
Can you map it into the page cache, as Al Viro has done in recent
patches?
Can you break your allocation into an array of pages, obtained via
get_free_page?
If runlists are on-disk structures, can you look at bh->b_data instead
of keeping them in memory?

-- 
Jeff Garzik      | Disbelief, that's why you fail.
Building 1024    |
MandrakeSoft     |

Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Anton Altaparmakov]

At 00:59 26/05/2001, Jeff Garzik wrote:
>Anton Altaparmakov wrote:
> > NTFS 1.1.15 - ChangeLog
> > =======================
> > - Support for more than 128kiB sized runlists (using vmalloc_32() instead
> > of kmalloc()).
>
>If you are running into kmalloc size limit, please consider some 
>alternative method of allocation.

Why? I was told that's what vmalloc is for. And anyway, Windows NT/2k 
operate on a 256kB sized memory allocation basis and NTFS has been designed 
with this in mind. Breaking away from that is a PITA. I was very happy to 
learn about vmalloc as it solved a lot of my headaches with regards to 
structures being >128kb.

btw. Maybe you, or someone else, can explain me whether vmalloc() will 
always work or whether vmalloc_32() is required? I am thinking of highmem 
pages which vmalloc could return. Would I have to make sure the pages were 
accessible by calling some function or is this done transparently?

>Can you map it into the page cache, as Al Viro has done in recent patches?

If someone would document how the page cache actually works from a kernel 
point of view, perhaps...<hint, hint> Sorry, but I haven't had the time to 
go through all the related source to understand it properly. I have an idea 
of how it works conceptually (well, kind of...) but not a deep enough 
understanding to start implementing anything that goes anywhere near the 
page cache.

I would love to move inode metadata to the page cache. That would simplify 
so much in the NTFS driver and would result in an incredible speed boost 
due to getting rid of all the silly copying of data back and forth inside 
the driver as we could just pass pointers around instead (to locked pages 
or other form of internal locking).

>Can you break your allocation into an array of pages, obtained via 
>get_free_page?

No thank you. I would rather have it contiguous or at least logically 
contiguous. I don't care if the pages are physically contiguous as long as 
I can see them as one piece... I mean, ok, I could break up run lists, as 
they are arrays of fixed size structs and the boundary cases would be 
predictable (but still would slow down the code with boundary checks) but 
this doesn't solve the generic problem later on as more things get 
implemented where we encounter variable width structures. It is much 
cleaner and easier to implement using a logically contiguous memory 
allocation IMHO. It's a shame Linux kernel doesn't support this kind of 
stuff (last time I read somewhere about this it said it doesn't anyway. I 
would be happy to be told wrong here).

>If runlists are on-disk structures, can you look at bh->b_data instead of 
>keeping them in memory?

They are not. They are derived from compressed on-disk structures (which 
are re-compressed when updating the inode). Runlists of such large size are 
only needed for A) huge files/directories, B) extremely fragmented 
files/directories, or C) a combination of A and B. (Remembering that 
metadata is stored as files, so the same applies for the metadata itself.)

Best regards,

Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Martin von Loewis]

> I would love to move inode metadata to the page cache. That would simplify 
> so much in the NTFS driver and would result in an incredible speed boost 
> due to getting rid of all the silly copying of data back and forth inside 
> the driver as we could just pass pointers around instead (to locked pages 
> or other form of internal locking).
[...]
> No thank you. I would rather have it contiguous or at least logically 
> contiguous. I don't care if the pages are physically contiguous as long as 
> I can see them as one piece... I mean, ok, I could break up run lists, as 
> they are arrays of fixed size structs and the boundary cases would be 
> predictable

When you talk about avoiding copying, are you thinking of not copying
the runlists, as well? If so, how can you avoid uncompressing them?
That seems very complicated, plus you have to concatenate the pieces
of the runlist if they span multiple MFT records, anyway.

> They are not. They are derived from compressed on-disk structures (which 
> are re-compressed when updating the inode). Runlists of such large size are 
> only needed for A) huge files/directories, B) extremely fragmented 
> files/directories, or C) a combination of A and B. (Remembering that 
> metadata is stored as files, so the same applies for the metadata itself.)

I guess this answers my question - you will continue to uncompress the
runlist when opening the inode, right?

Regards,
Martin

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Anton Altaparmakov]

At 12:23 27/05/2001, Martin von Loewis wrote:
> > I would love to move inode metadata to the page cache. That would simplify
> > so much in the NTFS driver and would result in an incredible speed boost
> > due to getting rid of all the silly copying of data back and forth inside
> > the driver as we could just pass pointers around instead (to locked pages
> > or other form of internal locking).
>[...]
> > No thank you. I would rather have it contiguous or at least logically
> > contiguous. I don't care if the pages are physically contiguous as long as
> > I can see them as one piece... I mean, ok, I could break up run lists, as
> > they are arrays of fixed size structs and the boundary cases would be
> > predictable
>
>When you talk about avoiding copying, are you thinking of not copying
>the runlists, as well? If so, how can you avoid uncompressing them?
>That seems very complicated, plus you have to concatenate the pieces
>of the runlist if they span multiple MFT records, anyway.

No, you can't avoid uncompressing them. But you can avoid copying all 
attributes out of the record into separate memory locations and then back. 
The run list is not part of the attribute so obviously has to be kept 
separate. Also, it is pointless to parse all attributes and their run lists 
on opening an inode, chaces are some of the attributes will never be 
touched at all and maybe we are just running stat on the file but don't 
want to start reading/writing the data. My idea involves a different run 
list format (as I already use in the linux-ntfs project on SF) which keeps 
[VCN, LCN, Length] records thus allowing you to not have the whole run 
list, i.e. if user accesses only beginning of the file, we only need to 
decompress the run list for the first extent. Also, this format allows out 
of line attribute extent handling, so if we access data at a certain offset 
we only get that extent and later as required we get the others and insert 
them into the run list. (This is not implemented yet, but I have been 
thinking about it.) Although I have to say I am not sure I want to have the 
run list extents fused into one. That only causes headaches when writing 
back the inode. But perhaps we can fuse them and then discard them without 
write back. Only if the attribute size changes do we need to deal with the 
problem but still it remains a problem.

> > They are not. They are derived from compressed on-disk structures (which
> > are re-compressed when updating the inode). Runlists of such large size 
> are
> > only needed for A) huge files/directories, B) extremely fragmented
> > files/directories, or C) a combination of A and B. (Remembering that
> > metadata is stored as files, so the same applies for the metadata itself.)
>
>I guess this answers my question - you will continue to uncompress the
>runlist when opening the inode, right?

Yes and no. They will be uncompressed but not when opening the inode. It 
will be "uncompress required extent's run list on demand". The current 
driver with uncompressing everything and keeping copies of all attributes 
is A) very slow and B) eats incredible amounts of memory when you are 
operating on large partitions (talking tens of Gb partitions).

Best regards,

Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Martin von Loewis]

> Yes and no. They will be uncompressed but not when opening the inode. It 
> will be "uncompress required extent's run list on demand".

Are you sure this can work? Initially, I thought I could use the
attribute list to only uncompress the extend that has the VCN I'm
interested in.

That would not work: NT would split individual runs across extends
(i.e. split them in the middle). Did I misunderstand, or do you have a
solution for that as well.

Regards,
Martin

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Anton Altaparmakov]

At 13:53 27/05/2001, Martin von Loewis wrote:
> > Yes and no. They will be uncompressed but not when opening the inode. It
> > will be "uncompress required extent's run list on demand".
>
>Are you sure this can work?

No.

>Initially, I thought I could use the attribute list to only uncompress the 
>extend that has the VCN I'm interested in.

Same idea here.

>That would not work: NT would split individual runs across extends
>(i.e. split them in the middle).

Argh! That seems really stupid thing to do, as it makes it difficult to 
interpret what the highest_vcn/lowest_vcn field of attribute extents is 
supposed to mean!?!

This does however explain some of the code uglyness I have seen (and chosen 
to ignore) in the ntfs.sys disassembly run list handling...

>Did I misunderstand, or do you have a solution for that as well.

No solution. I wasn't aware this could happen. I knew a compression block 
could be split in it halves but I didn't realize this braindamaged 
complication.

I guess we will have to decode the whole run list in one go then. Anything 
else would be slower over all. - What we could do of course is to walk the 
mapping pairs array real quick not reading the numbers only to get the 
correct starting offset into the next extent and then only decode that one 
but that would mean walking the mapping pairs array repeatedly (once to get 
each extent) which would be overall slower than just getting the lot at 
once. - I maintain that we should do this on demand and not on inode open, 
though. - I will have another think about this, but if it is true that NT 
splits the records anywhere, then it would be impossible to start at any 
extent other than the first one.

Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Yuri Per]

Martin von Loewis wrote:

>That would not work: NT would split individual runs across extends
>(i.e. split them in the middle). Did I misunderstand, or do you have a
>solution for that as well.
>
Are you sure that it's true? My NTFS resizer interprets parts of runlist 
stored in different FILE records independently and I never experienced 
any problems with that.


Yuri

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Martin von Loewis]

> >That would not work: NT would split individual runs across extends
> >(i.e. split them in the middle). Did I misunderstand, or do you have a
> >solution for that as well.
> >
> Are you sure that it's true? My NTFS resizer interprets parts of runlist 
> stored in different FILE records independently and I never experienced 
> any problems with that.

I'm sure it could happen on NT 3.1. Maybe MS has fixed it since.

Regards,
Martin

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Anton Altaparmakov]

At 12:10 28/05/2001, Martin von Loewis wrote:
> > >That would not work: NT would split individual runs across extends
> > >(i.e. split them in the middle). Did I misunderstand, or do you have a
> > >solution for that as well.
> > >
> > Are you sure that it's true? My NTFS resizer interprets parts of runlist
> > stored in different FILE records independently and I never experienced
> > any problems with that.
>
>I'm sure it could happen on NT 3.1. Maybe MS has fixed it since.

Hm, in that case I think I will implement it the way I was thinking and I 
will assume that it can't happen any more. - I will add some kind of sanity 
check at the beginning to catch problems and scream to the syslog which we 
can remove later.

Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I know 
I can make sure we don't mount such beasts considering we know the driver 
would fail on them... - I am aware of only one person stil using NT 3.51 
and he doesn't believe in the NTFS Linux driver any more, so I guess we can 
just say we support NT 4.0 and above only.

Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Yuri Per]

Anton Altaparmakov wrote:

> Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I 
> know  I can make sure we don't mount such beasts considering we know 
> the driver  would fail on them... - I am aware of only one person stil 
> using NT 3.51  and he doesn't believe in the NTFS Linux driver any 
> more, so I guess we can  just say we support NT 4.0 and above only.

NT 3.51 uses exactly the same version of NTFS as NT 4.0 does.


Yuri

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Anton Altaparmakov]

At 14:08 28/05/2001, Yuri Per wrote:
>Anton Altaparmakov wrote:
>
>>Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I 
>>know  I can make sure we don't mount such beasts considering we know the 
>>driver  would fail on them... - I am aware of only one person stil using 
>>NT 3.51  and he doesn't believe in the NTFS Linux driver any more, so I 
>>guess we can  just say we support NT 4.0 and above only.
>
>NT 3.51 uses exactly the same version of NTFS as NT 4.0 does.

Ok. Thanks.

Anyone know about 3.1?

Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Jeff V. Merkey]

On Mon, May 28, 2001 at 03:49:28PM +0100, Anton Altaparmakov wrote:
> At 14:08 28/05/2001, Yuri Per wrote:
> >Anton Altaparmakov wrote:
> >
> >>Does anyone know what NTFS version the NT 3.1 / 3.51 volumes had? If I 
> >>know  I can make sure we don't mount such beasts considering we know the 
> >>driver  would fail on them... - I am aware of only one person stil using 
> >>NT 3.51  and he doesn't believe in the NTFS Linux driver any more, so I 
> >>guess we can  just say we support NT 4.0 and above only.
> >
> >NT 3.51 uses exactly the same version of NTFS as NT 4.0 does.
> 
> Ok. Thanks.
> 
> Anyone know about 3.1?
> 
> Anton
> 

It's an HPFS variant.  Try the HPFS driver, and it may work.  The first cut
of NT was with a hacked up HPFS driver from Microsoft OS/2.  NTFS was 
designed internally by David G. and friends.

Jeff

> 
> -- 
>    "Nothing succeeds like success." - Alexandre Dumas
> -- 
> Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
> Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
> ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Martin von Loewis]

> > Anyone know about 3.1?
> > 
> 
> It's an HPFS variant.

No. NT was using NTFS right from the start.

Regards,
Martin

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Jeff V. Merkey]

On Tue, May 29, 2001 at 10:18:41AM +0200, Martin von Loewis wrote:
> > > Anyone know about 3.1?
> > > 
> > 
> > It's an HPFS variant.
> 
> No. NT was using NTFS right from the start.
> 
> Regards,
> Martin

No.  They were not.  Their first cuts of NT used an HPFS variant until 
NTFS could be completed.  The guy who wrote NTFS for MS was a consultant
for us for several years, and I got a very long briefing on the history
of NT file systems over the years.

I do not know which version was the cutover, but I do believe it was 
pre 3.X NT versions.  

Jeff