short term task list for Reiser4

short term task list for Reiser4

[Hans Reiser]

Please feel free to comment on this list and the order of its tasks:

0) fix all bugs as they arise

1) get batch_write into the -mm kernel --- small task

2) get read optimization code into the -mm kernel (coded and probably
debugged but not fully tested and not sent in yet) --- small task

3) get EVERYTHING into wiki (migration has started already, thanks flx).

4) review complaints of pauses while using reiser4 --- size of task
unknown, and it is also unknown how much we may have fixed it while
writing recent patches.

5) review crypt-compress code --- full code review --- substantive task

6) optimize fsync --- substantive task which requires using fixed area
for write twice logging, and using write twice logging for fsync'd
data.  It might require creating mount options to choose whether to
optimize for serialized sequential fsyncs vs. lazy fsyncs.

7) review all of our installation instructions --- I am already doing
that, but volunteers who will help out our wiki would be sorely
appreciated.  Installing reiser4 as the root for each distro needs
step-by-step instructions.

8) review our kernel documentation --- I should do that but when will I
have time?

Unfortunately, our code stability is going to decrease for a bit due to
all these changes to the read and write code --- no way to cure that but
passage of time.   On the other hand, our CPU usage went way down. 
Reiser4's only performance weakness now is fsync.  

Once the crypt-compress code is ready, we will release Reiser4.1-beta
(with plugins, releasing a beta means telling users that if they mount
-o reiser4.1-beta then cryptcompress will be their default plugin, and
if they don't, then they are using Reiser4.0 still).  Doubling our
performance and halving our disk usage is going to be fun.

Re: short term task list for Reiser4

[Clay Barnes]

On 15:04 Tue 11 Jul     , Hans Reiser wrote:
> Please feel free to comment on this list and the order of its tasks:
> 
> 0) fix all bugs as they arise
> 
> 1) get batch_write into the -mm kernel --- small task
> 
> 2) get read optimization code into the -mm kernel (coded and probably
> debugged but not fully tested and not sent in yet) --- small task
> 
> 3) get EVERYTHING into wiki (migration has started already, thanks flx).
> 
> 4) review complaints of pauses while using reiser4 --- size of task
> unknown, and it is also unknown how much we may have fixed it while
> writing recent patches.
> 
> 5) review crypt-compress code --- full code review --- substantive task
> 
> 6) optimize fsync --- substantive task which requires using fixed area
> for write twice logging, and using write twice logging for fsync'd
> data.  It might require creating mount options to choose whether to
> optimize for serialized sequential fsyncs vs. lazy fsyncs.
With the serialized sequential fsync, is that essentially what I was
talking about earlier with slowly streaming dirty writes to disk when
the HDD is idle?  If that's the case, I don't see the advantage in having
lazy fsyncs except in situations where you want to keep the HDD spun down
as much as possible.  If you keep as much of the writes in RAM as you
would have if you used lazy fsyncs, then you get the same temporal
locality speed up, with the added advantages that you can clear the RAM
under memory pressure immediately and crashes result in lower likelyhood
of data loss than with lazy fsync.  I suppose it isn't a bad idea to give
people more options (unless you're a GNOME UI developer :-P), but at the
very least I think that slow streaming to disk would be a very wise
default option.

My CS focus is Human Interfaces, so I may be way out of my league here
with FSs, but I thought I'd still throw in my two cents.
> 
> 7) review all of our installation instructions --- I am already doing
> that, but volunteers who will help out our wiki would be sorely
> appreciated.  Installing reiser4 as the root for each distro needs
> step-by-step instructions.
I've been meaning to hose my laptop (assuming I fix one problem with my
desktop), so I am willing to help write Gentoo install docs (or possibly
Arch Linux).  I can also test exsiting instructions.
> 
> 8) review our kernel documentation --- I should do that but when will I
> have time?
> 
> Unfortunately, our code stability is going to decrease for a bit due to
> all these changes to the read and write code --- no way to cure that but
> passage of time.   On the other hand, our CPU usage went way down. 
> Reiser4's only performance weakness now is fsync.  
> 
> Once the crypt-compress code is ready, we will release Reiser4.1-beta
> (with plugins, releasing a beta means telling users that if they mount
> -o reiser4.1-beta then cryptcompress will be their default plugin, and
> if they don't, then they are using Reiser4.0 still).  Doubling our
> performance and halving our disk usage is going to be fun.


--Clay

Re: short term task list for Reiser4

[Hans Reiser]

Clay Barnes wrote:

>On 15:04 Tue 11 Jul     , Hans Reiser wrote:
>  
>
>>
>>6) optimize fsync --- substantive task which requires using fixed area
>>for write twice logging, and using write twice logging for fsync'd
>>data.  It might require creating mount options to choose whether to
>>optimize for serialized sequential fsyncs vs. lazy fsyncs.
>>    
>>
>With the serialized sequential fsync, is that essentially what I was
>talking about earlier with slowly streaming dirty writes to disk when
>the HDD is idle?  If that's the case, I don't see the advantage in having
>lazy fsyncs
>
if you are optimizing throughput rather than latency, then you let
things get to disk whenever they get there, and you let the app hang
while it waits. A mailer processing many requests in parallel might find
30 seconds of latency to be just fine but a database might find 3
seconds of latency to be too much. (I make up these examples, mailer
programmers please correct me.)

> except in situations where you want to keep the HDD spun down
>as much as possible.
>
No, that is not when you do it.

>I've been meaning to hose my laptop (assuming I fix one problem with my
>desktop), so I am willing to help write Gentoo install docs (or possibly
>Arch Linux).  I can also test exsiting instructions.
>  
>
That would be way cool.
<http://wiki.namesys.com/Reiser4-GettingStarted#preview>

Re: short term task list for Reiser4

[Clay Barnes]

On 16:11 Tue 11 Jul     , Hans Reiser wrote:
> Clay Barnes wrote:
> >On 15:04 Tue 11 Jul     , Hans Reiser wrote:
> >>6) optimize fsync --- substantive task which requires using fixed area
> >>for write twice logging, and using write twice logging for fsync'd
> >>data.  It might require creating mount options to choose whether to
> >>optimize for serialized sequential fsyncs vs. lazy fsyncs.
> >With the serialized sequential fsync, is that essentially what I was
> >talking about earlier with slowly streaming dirty writes to disk when
> >the HDD is idle?  If that's the case, I don't see the advantage in having
> >lazy fsyncs
> if you are optimizing throughput rather than latency, then you let
> things get to disk whenever they get there, and you let the app hang
> while it waits. A mailer processing many requests in parallel might find
> 30 seconds of latency to be just fine but a database might find 3
> seconds of latency to be too much. (I make up these examples, mailer
> programmers please correct me.)
I see your point, but here's where I'm still uncertain:

If you have a lazy write policy, what exactly is gained by intentionally
delaying writes (beyond a certain size that is necessary to make things
like dancing trees actually effecient)?  If you trickle some data to
disk, then when memory pressure causes (or an app calls) a big sync,
then you have less to actually write.  What I'm suggesting, now, is not
a major write policy change, but rather a light process that is limited
to extremely low resource use (I/O, CPU, etc.).  It would take some of
the edge off of major syncs, and for many (most?) non-server users, it
could wholly eliminate memory pressure-induced heavy syncs.

--Clay

Re: short term task list for Reiser4

[Hans Reiser]

Reiser4 generally could use some knobs controlling things like whether
we trickle data slowly and continuously

>
>If you have a lazy write policy, what exactly is gained by intentionally
>delaying writes (beyond a certain size that is necessary to make things
>like dancing trees actually effecient)?  If you trickle some data to
>disk, then when memory pressure causes (or an app calls) a big sync,
>then you have less to actually write.  What I'm suggesting, now, is not
>a major write policy change, but rather a light process that is limited
>to extremely low resource use (I/O, CPU, etc.).  It would take some of
>the edge off of major syncs, and for many (most?) non-server users, it
>could wholly eliminate memory pressure-induced heavy syncs.
>
>--Clay
>
>
>  
>

Re: short term task list for Reiser4

[Christoph Lameter]

Will there be any NUMA /SMP fixes? Reiserfs performance is severely 
impacted at higher processor counts by the mandatory centralized locking 
in both read and write paths in the filesystem.

Re: short term task list for Reiser4

[Hans Reiser]

Christoph Lameter wrote:

>Will there be any NUMA /SMP fixes? Reiserfs performance is severely 
>impacted at higher processor counts by the mandatory centralized locking 
>in both read and write paths in the filesystem.
>
>
>  
>
Reiserfs or Reiser4?

Re: short term task list for Reiser4

[Christoph Lameter]

On Thu, 13 Jul 2006, Hans Reiser wrote:

> Christoph Lameter wrote:
> 
> >Will there be any NUMA /SMP fixes? Reiserfs performance is severely 
> >impacted at higher processor counts by the mandatory centralized locking 
> >in both read and write paths in the filesystem.
> Reiserfs or Reiser4?

Either.
 

Re: short term task list for Reiser4

[rvalles]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

On Tue, Jul 11, 2006 at 03:04:20PM -0700, Hans Reiser wrote:
> Please feel free to comment on this list and the order of its tasks:
> 4) review complaints of pauses while using reiser4 --- size of task
> unknown, and it is also unknown how much we may have fixed it while
> writing recent patches.
>
> 6) optimize fsync --- substantive task which requires using fixed area
> for write twice logging, and using write twice logging for fsync'd
> data.  It might require creating mount options to choose whether to
> optimize for serialized sequential fsyncs vs. lazy fsyncs.
> 
I believe those two are related. I'm having the "pauses" (of many
minutes at times!) when writing to reiser4. It seems it is triggered
mostly by the use of fsync(); NFS in sync mode manages to trigger it
way often: I mount my old desktop's home from my new computer via synced
NFS.
The pauses consist in the application being frozen and the other
applications being slowed down on their IO operations, while the disks
writes data continously during an interval that usually last a minute or
so, but may last many times that. It often happens with small files
(like, when sending a mail, as it passes through the MTA), so I believe
it probably (re)writes to disk lots of stuff that doesn't need to be
written to disk at all.
It only happens on reiser4 patches against 2.6.13 or newer, 2.6.12.x is
fine.
-- 
- Do you study theories? - Oh yes! Theories are fun. - How do you use them? - When I must act, theories are too slow. To act, I must know.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: short term task list for Reiser4

[Hans Reiser]

rvalles wrote:

>On Tue, Jul 11, 2006 at 03:04:20PM -0700, Hans Reiser wrote:
>  
>
>>Please feel free to comment on this list and the order of its tasks:
>>4) review complaints of pauses while using reiser4 --- size of task
>>unknown, and it is also unknown how much we may have fixed it while
>>writing recent patches.
>>
>>6) optimize fsync --- substantive task which requires using fixed area
>>for write twice logging, and using write twice logging for fsync'd
>>data.  It might require creating mount options to choose whether to
>>optimize for serialized sequential fsyncs vs. lazy fsyncs.
>>
>>    
>>
>I believe those two are related. I'm having the "pauses" (of many
>minutes at times!) when writing to reiser4. It seems it is triggered
>mostly by the use of fsync(); NFS in sync mode manages to trigger it
>way often: I mount my old desktop's home from my new computer via synced
>NFS.
>The pauses consist in the application being frozen and the other
>applications being slowed down on their IO operations, while the disks
>writes data continously during an interval that usually last a minute or
>so, but may last many times that. It often happens with small files
>(like, when sending a mail, as it passes through the MTA), so I believe
>it probably (re)writes to disk lots of stuff that doesn't need to be
>written to disk at all.
>It only happens on reiser4 patches against 2.6.13 or newer, 2.6.12.x is
>fine.
>  
>
So the pauses are experienced by a process waiting on fsync to finish?
If yes, then the problem is a very different issue from what I thought....

Hans

Re: short term task list for Reiser4

[Hans Reiser]

Christoph Lameter wrote:

>On Thu, 13 Jul 2006, Hans Reiser wrote:
>
>  
>
>>Christoph Lameter wrote:
>>
>>    
>>
>>>Will there be any NUMA /SMP fixes? Reiserfs performance is severely 
>>>impacted at higher processor counts by the mandatory centralized locking 
>>>in both read and write paths in the filesystem.
>>>      
>>>
>>Reiserfs or Reiser4?
>>    
>>
>
>Either.
> 
>
>
>  
>
What big locks are you running into in V4?  Please tell me more.

Hans

Re: short term task list for Reiser4

[Jindrich Makovicka]

On Fri, 14 Jul 2006 00:01:49 -0700
Hans Reiser <reiser@namesys.com> wrote:
> rvalles wrote:
> >I believe those two are related. I'm having the "pauses" (of many
> >minutes at times!) when writing to reiser4. It seems it is triggered
> >mostly by the use of fsync(); NFS in sync mode manages to trigger it
> >way often: I mount my old desktop's home from my new computer via
> >synced NFS.
> >The pauses consist in the application being frozen and the other
> >applications being slowed down on their IO operations, while the
> >disks writes data continously during an interval that usually last a
> >minute or so, but may last many times that. It often happens with
> >small files (like, when sending a mail, as it passes through the
> >MTA), so I believe it probably (re)writes to disk lots of stuff that
> >doesn't need to be written to disk at all.
> >It only happens on reiser4 patches against 2.6.13 or newer, 2.6.12.x
> >is fine.
> >  
> >
> So the pauses are experienced by a process waiting on fsync to finish?
> If yes, then the problem is a very different issue from what I
> thought....

I am pretty sure I triggered the same a couple of times with aMule,
which was downloading and uploading about 20-30kB per second in each
direction, and randomly accessing larger files. Doing a sync then
suddenly caused the disk seek like crazy and write about 500kB per
second constantly. I waited about a minute, and then better killed
aMule. Could it be possible that sync tries to sync the writes which
arrive during the operation and cannot catch up?

-- 
Jindrich Makovicka

Re: short term task list for Reiser4

[Hans Reiser]

Jindrich Makovicka wrote:

>On Fri, 14 Jul 2006 00:01:49 -0700
>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>rvalles wrote:
>>    
>>
>>>I believe those two are related. I'm having the "pauses" (of many
>>>minutes at times!) when writing to reiser4. It seems it is triggered
>>>mostly by the use of fsync(); NFS in sync mode manages to trigger it
>>>way often: I mount my old desktop's home from my new computer via
>>>synced NFS.
>>>The pauses consist in the application being frozen and the other
>>>applications being slowed down on their IO operations, while the
>>>disks writes data continously during an interval that usually last a
>>>minute or so, but may last many times that. It often happens with
>>>small files (like, when sending a mail, as it passes through the
>>>MTA), so I believe it probably (re)writes to disk lots of stuff that
>>>doesn't need to be written to disk at all.
>>>It only happens on reiser4 patches against 2.6.13 or newer, 2.6.12.x
>>>is fine.
>>> 
>>>
>>>      
>>>
>>So the pauses are experienced by a process waiting on fsync to finish?
>>If yes, then the problem is a very different issue from what I
>>thought....
>>    
>>
>
>I am pretty sure I triggered the same a couple of times with aMule,
>which was downloading and uploading about 20-30kB per second in each
>direction, and randomly accessing larger files. Doing a sync then
>suddenly caused the disk seek like crazy and write about 500kB per
>second constantly. I waited about a minute, and then better killed
>aMule. Could it be possible that sync tries to sync the writes which
>arrive during the operation and cannot catch up?
>
>  
>
zam, can you answer this, because he is right that if the answer is yes,
then we have a bit of a problem we need to address.  I worry that,
assuming the answer is yes, that one process doing a lot of fsync, could
muck up the performance of everyone its atoms fuse with if we expire the
atom with every fsync, but if we don't then.....

Jindrich, don't let us forget this issue before we fix it, ok?  Thanks,

Hans

Re: short term task list for Reiser4

[Gary Wilson Jr.]

Just wanted to mention that I have experienced pauses also.  I use gentoo, and
have triggered the problem when saving a small text file in vim while emerging
in the background.  Upon save and exit (:wq) vim will just hang there anywhere
from 5-30 seconds.  Let me know if/how I can help.