linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* 2.5.66-mm2
@ 2003-04-01  8:01 Andrew Morton
  2003-04-01  8:48 ` 2.5.66-mm2 Andrew Morton
                   ` (2 more replies)
  0 siblings, 3 replies; 11+ messages in thread
From: Andrew Morton @ 2003-04-01  8:01 UTC (permalink / raw)
  To: linux-kernel, linux-mm


ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.66/2.5.66-mm2/


. Just lots of little fixes, cleanups, late-but-promised minor features,
  etc.

. There is a small patch from Ingo here against the CPU scheduler which we
  hope will fix the new starvation problems which people have been reporting.
  I this is you, please test and report.

. It turns out that a recent change to the anticipatory scheduler
  accidentally made it quite ineffective on SCSI (performance is similar to
  deadline).  The patch which did that has been disabled, but we expect that
  the seeky OLTP loads will suffer until this is fixed up for real.



Changes since 2.5.66-mm1:


 linus.patch

 Latest from Linus

-initcall-debug.patch
-posix-timers-64-bit-fix.patch
-slab-off-by-one-fix.patch
-install_page-flush_cache_page.patch
-ext3-fsync-speedup.patch
-pcmcia-2.patch
-pcmcia-3b.patch
-pcmcia-3.patch
-pcmcia-4.patch
-pcmcia-5.patch
-pcmcia-6.patch
-pcmcia-7b.patch
-pcmcia-7.patch
-pcmcia-8.patch
-pcmcia-9.patch
-pcmcia-10.patch
-posix-timer-double-expiration-fix.patch
-hugh-01-no-SWAP_ERROR.patch
-hugh-02-try_to_unmap-CONFIG_SWAP.patch
-hugh-03-add_to_swap_cache.patch
-hugh-09-tmpfs-truncation.patch
-hugh-10-tmpfs-atomics.patch
-hugh-12-vm_enough_memory-double-counts.patch
-ext3-max-file-size-fix.patch
-linear-oops-fix-1.patch
-dev_t-drm-warnings.patch
-x86-clock-override-option.patch
-tty_io-cleanup.patch
-page_to_pfn-in-blk_queue_bounce.patch
-init_inode_once-bloat-fix.patch
-VM_DONTEXPAND-fix.patch
-compound-page-warning-fix.patch
-cdevname-irq-safety-fix.patch
-register_chrdev_region-leak-fix.patch
-slab-cache-sizes-cleanup.patch
-sync_blockdev-on-final-close.patch

 Merged

+as-queue_notready-cleanup.patch

 Anticipatory scheduler cleanups.

+as-disable-thinktime.patch

 Anticipatory scheduler simply doesn't work on scsi.  Disable the thinktime
 heuristic to make it go again (this will cause slowdowns for OLTP loads).

+cdrom-stack-usage.patch

 Reduced stack usage

+sched-interactivity-backboost-revert.patch

 Revert the CPU scheduler backboost heuristic.

+tmpfs-blk_congestion_wait-fix.patch

 tmpfs fixlet.

+page_convert_anon-locking-fix.patch

 Fix locking for objrmap.

+monotonic-clock-hangcheck.patch

 get_cycles() clock source, used in the hangcheck timer.

+module_load_notification.patch

 Provide notification of module load/unload to other places in the kernel.

+put_task_struct-debug.patch

 Try to find out who is doing put_task_struct() on a freed task_struct.

+remove-kdev_name.patch

 Remove kdevname()

+percpu_counter.patch
+blockgroup_lock.patch

 Broken out from the ext2 speedup patches.

+kill-TIOCTTYGSTRUCT.patch

 Remove the TIOCTTYGSTRUCT ioctl.

+misc.patch

 Misc fixes

+sony-apm-fix.patch

 Fix APM for Sony notebooks.

+init-sections-in-kallsyms.patch

 Make kallsyms aware of __init symbols.

+3c59x-980-support.patch

 Extra PCI IDs for 3c59x

+fadvise-flush-data.patch

 Teach fadvise(FADV_DONTNEED) to gently flush out dirty pages.

+console-scrollback.patch

 Another TIOCLINUX command for scrolling the console down.

+usb-disconnect-crash-fix.patch

 Fix some USB crash (this isn't right)

+devfs-rescan_partitions-fix.patch

 Fix a devfs problem

+umsdos-fixes.patch

 Fixes

+exp_parent-locking-fix.patch

 Fix nfsd locking

+real_lookup-race-fix.patch

 Fix a fixed race differently

+remove-dparent_lock.patch

 Remove the global dparent_lock

+PCI-aliases-fix.patch

 Stuff from Rusty.

+jbd_expect.patch
+jbd-assert-io-failure-fix.patch

 Forward-port from Stephen's recent batch of ext3 patches.

+jbd-handle-journal-io-errors.patch
+jbd-handle-journal-io-errors-fix.patch

 Propagate IO errors against the ext3 journal back to user syscalls, mainly
 fsync().

+ext3-concurrent-block-allocation-hashed.patch

 Change the ext3 BKL-removal code to use the hashed locking and
 percpu_counters from ext2.



All 99 patches:

linus.patch

mm.patch
  add -mmN to EXTRAVERSION

kgdb-ga.patch
  kgdb stub for ia32 (George Anzinger's one)

ppa-null-pointer-fix.patch

config_spinline.patch
  uninline spinlocks for profiling accuracy.

ppc64-reloc_hide.patch

ppc64-pci-patch.patch
  Subject: pci patch

ppc64-aio-32bit-emulation.patch
  32/64bit emulation for aio

ppc64-scruffiness.patch
  Fix some PPC64 compile warnings

sym-do-160.patch
  make the SYM driver do 160 MB/sec

config-PAGE_OFFSET.patch
  Configurable kenrel/user memory split

ptrace-flush.patch
  cache flushing in the ptrace code

buffer-debug.patch
  buffer.c debugging

warn-null-wakeup.patch

ext3-truncate-ordered-pages.patch
  ext3: explicitly free truncated pages

reiserfs_file_write-5.patch

rcu-stats.patch
  RCU statistics reporting

ext3-journalled-data-assertion-fix.patch
  Remove incorrect assertion from ext3

nfs-speedup.patch

nfs-oom-fix.patch
  nfs oom fix

sk-allocation.patch
  Subject: Re: nfs oom

nfs-more-oom-fix.patch

rpciod-atomic-allocations.patch
  Make rcpiod use atomic allocations

linux-isp.patch

isp-update-1.patch

kblockd.patch
  Create `kblockd' workqueue

as-iosched.patch
  anticipatory I/O scheduler

as-np-reads-1.patch
  AS: read-vs-read fixes

as-np-reads-2.patch
  AS: more read-vs-read fixes

as-predict-data-direction.patch
  as: predict direction of next IO

as-remove-frontmerge.patch
  AS: remove frontmerge tunable

as-misc-cleanups.patch
  AS: misc cleanups

as-minor-tweaks.patch
  AS: tuning and tweaks

as-remove-stats.patch
  AS: remove statistics

as-queue_notready-cleanup.patch
  don't dispatch request on elv_queue_empty

as-disable-thinktime.patch

cfq-2.patch
  CFQ scheduler, #2

unplug-use-kblockd.patch
  Use kblockd for running request queues

fremap-all-mappings.patch
  Make all executable mappings be nonlinear

cdrom-stack-usage.patch
  reduce stack in cdrom/optcd.c

objrmap-2.5.62-5.patch
  object-based rmap

sched-2.5.64-D3.patch
  sched-2.5.64-D3, more interactivity changes

scheduler-tunables.patch
  scheduler tunables

show_task-free-stack-fix.patch
  show_task() fix and cleanup

yellowfin-set_bit-fix.patch
  yellowfin driver set_bit fix

htree-nfs-fix.patch
  Fix ext3 htree / NFS compatibility problems

task_prio-fix.patch
  simple task_prio() fix

slab_store_user-large-objects.patch
  slab debug: perform redzoning against larger objects

htree-nfs-fix-2.patch
  htree nfs fix

hugh-04-page_convert_anon-ENOMEM.patch
  swap 04/13 page_convert_anon -ENOMEM

hugh-05-page_convert_anon-unlocking.patch
  swap 05/13 page_convert_anon unlocking

hugh-06-wrap-below-vm_start.patch
  swap 06/13 wrap below vm_start

hugh-07-objrmap-page_table_lock.patch
  swap 07/13 objrmap page_table_lock

hugh-08-rmap-comments.patch
  swap 08/13 rmap comments

hugh-11-fix-unuse_pmd-fixme.patch
  swap 11/13 fix unuse_pmd fixme

tmpfs-blk_congestion_wait-fix.patch
  tmpfs blk_congestion_wait fix

page_convert_anon-locking-fix.patch
  page_convert_anon locking fix

monotonic-clock-hangcheck.patch
  monotonic clock source for hangcheck timer

module_load_notification.patch
  module load notification

put_task_struct-debug.patch

remove-kdev_name.patch
  remove kdevname() before someone starts using it again

percpu_counter.patch
  percpu_counters: approximate but scalabel counters

blockgroup_lock.patch
  blockgroup_lock: hashed spinlocks for ext2 and ext3 blockgroup locking

ext2-no-lock_super-ng.patch

ext2-ialloc-no-lock_super-ng.patch

stat_t-larger-dev_t.patch
  struct stat - support larger dev_t

kill-TIOCTTYGSTRUCT.patch
  kill TIOCTTYGSTRUCT

dev_t-32-bit.patch
  [for playing only] change type of dev_t

dev_t-remove-B_FREE.patch
  dev_t: eliminate B_FREE

sg-dev_t-fix.patch
  32-bit dev_t fix for sg

misc.patch
  misc fixes

sony-apm-fix.patch
  fix ec_read using wrong #define's in sonypi driver.

init-sections-in-kallsyms.patch
  Put all functions in kallsyms

3c59x-980-support.patch
  Additional 3c980 device support

fadvise-flush-data.patch

oops-dump-preceding-code.patch
  i386 oops output: dump preceding code

console-scrollback.patch
  add vt console scrollback ioctl

usb-disconnect-crash-fix.patch
  Subject: Re: [linux-usb-devel] timer hang with current 2.5 BK

devfs-rescan_partitions-fix.patch
  Fix devfs' partition handling

umsdos-fixes.patch
  umsdos fixes

exp_parent-locking-fix.patch
  exp_parent locking fixes

real_lookup-race-fix.patch
  real_lookup race fix

remove-dparent_lock.patch
  remove dparent_lock

conntrack-use-after-free-fix.patch
  fix use-after-free in ip_conntrack

PCI-aliases-fix.patch
  Fix PCI aliases.

jbd_expect.patch
  Add less-severe assert-failure form for ext3.

jbd-assert-io-failure-fix.patch
  Fix jbd assert failure on IO error.

acpi-build-fix.patch
  ACPI build fix

ext3_mark_inode_dirty-speedup.patch
  ext3_mark_inode_dirty() speedup

ext3_mark_inode_dirty-less-calls.patch
  ext3_commit_write speedup

ext3-handle-cache.patch
  ext3: create a slab cache for transaction handles

jbd-handle-journal-io-errors.patch
  ext3 journal commit I/O error fix

jbd-handle-journal-io-errors-fix.patch

ext3-no-bkl.patch

journal_dirty_metadata-speedup.patch

journal_get_write_access-speedup.patch

ext3-concurrent-block-inode-allocation.patch
  Subject: [PATCH] concurrent block/inode allocation for EXT3

ext3-concurrent-block-allocation-fix-1.patch

ext3-concurrent-block-allocation-hashed.patch
  Subject: Re: [PATCH] concurrent block/inode allocation for EXT3




^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-01  8:01 2.5.66-mm2 Andrew Morton
@ 2003-04-01  8:48 ` Andrew Morton
  2003-04-01  9:15   ` 2.5.66-mm2 Andrew Morton
  2003-04-02  1:10 ` 2.5.66-mm2 Martin J. Bligh
  2003-04-03 17:53 ` 2.5.66-mm2 Maciej Soltysiak
  2 siblings, 1 reply; 11+ messages in thread
From: Andrew Morton @ 2003-04-01  8:48 UTC (permalink / raw)
  To: linux-kernel, linux-mm

Andrew Morton <akpm@digeo.com> wrote:
>
> 
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.66/2.5.66-mm2/

oops.  2.5.66-mm2.gz is missing lots of patches.

Please use 2.5.66-mm2-1.gz from that directory.



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-01  8:48 ` 2.5.66-mm2 Andrew Morton
@ 2003-04-01  9:15   ` Andrew Morton
  0 siblings, 0 replies; 11+ messages in thread
From: Andrew Morton @ 2003-04-01  9:15 UTC (permalink / raw)
  To: linux-kernel, linux-mm

Andrew Morton <akpm@digeo.com> wrote:
>
> Andrew Morton <akpm@digeo.com> wrote:
> >
> > 
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.66/2.5.66-mm2/
> 
> oops.  2.5.66-mm2.gz is missing lots of patches.
> 
> Please use 2.5.66-mm2-1.gz from that directory.

And now kernel.org mirroring is having another sulk.

There's a copy at
	http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.66-mm2-1.gz

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-01  8:01 2.5.66-mm2 Andrew Morton
  2003-04-01  8:48 ` 2.5.66-mm2 Andrew Morton
@ 2003-04-02  1:10 ` Martin J. Bligh
  2003-04-02  1:34   ` 2.5.66-mm2 Andrew Morton
  2003-04-03 17:53 ` 2.5.66-mm2 Maciej Soltysiak
  2 siblings, 1 reply; 11+ messages in thread
From: Martin J. Bligh @ 2003-04-02  1:10 UTC (permalink / raw)
  To: Andrew Morton, Alex Tomas; +Cc: linux-kernel, linux-mm


> +ext3-concurrent-block-allocation-hashed.patch
> 
>  Change the ext3 BKL-removal code to use the hashed locking and
>  percpu_counters from ext2.

Ho hum. All very strange. Kernbench seems to be really behaving itself
quite well now, but SDET sucks worse than ever. The usual 16x NUMA-Q 
machine .... 

Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)
                              Elapsed      System        User         CPU
               2.5.66-mm2       44.04       81.12      569.40     1476.75
          2.5.66-mm2-ext3       44.43       84.10      568.82     1469.00

Kernbench: (make -j N vmlinux, where N = 16 x num_cpus)
                              Elapsed      System        User         CPU
               2.5.66-mm2       44.36       89.64      575.45     1499.25
          2.5.66-mm2-ext3       44.79       91.15      575.48     1488.25

Kernbench: (make -j vmlinux, maximal tasks)
                              Elapsed      System        User         CPU
               2.5.66-mm2       44.06       87.03      574.31     1501.00
          2.5.66-mm2-ext3       44.35       89.33      573.47     1495.75


DISCLAIMER: SPEC(tm) and the benchmark name SDET(tm) are registered
trademarks of the Standard Performance Evaluation Corporation. This 
benchmarking was performed for research purposes only, and the run results
are non-compliant and not-comparable with any published results.

Results are shown as percentages of the first set displayed

SDET 1  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         1.9%
          2.5.66-mm2-ext3        92.6%         1.8%

SDET 2  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         0.0%
          2.5.66-mm2-ext3        88.4%         5.1%

SDET 4  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         1.9%
          2.5.66-mm2-ext3        26.3%         7.8%

SDET 8  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         1.0%
          2.5.66-mm2-ext3         8.0%         3.1%

SDET 16  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         1.0%
          2.5.66-mm2-ext3         5.6%         1.6%

SDET 32  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         0.7%
          2.5.66-mm2-ext3         4.7%         1.5%

SDET 64  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         0.7%
          2.5.66-mm2-ext3         3.6%         1.9%

SDET 128  (see disclaimer)
                           Throughput    Std. Dev
               2.5.66-mm2       100.0%         0.6%
          2.5.66-mm2-ext3         3.9%         0.4%

diffprofile for SDET 128 (+ worse with ext3, - better)

  10410044  1594.1% total
   8799596  2479.9% default_idle
    649656  15845.3% __down
    347530     0.0% .text.lock.sched
    218233  8660.0% schedule
    183618  9984.7% __wake_up
    117605     0.0% .text.lock.transaction
     30896     0.0% do_get_write_access
     18162  1319.9% .text.lock.attr
     14815     0.0% journal_dirty_metadata
     13655     0.0% journal_get_write_access
     12762  79762.5% cpu_idle
     11023  7252.0% __blk_queue_bounce
      9404     0.0% start_this_handle
      9310     0.0% journal_add_journal_head
      5566     0.0% block_write_full_page
      5478     0.0% journal_stop
      4210     0.0% ext3_get_inode_loc
      4127   668.9% __find_get_block_slow
      3852  1965.3% .text.lock.sem
      3660     0.0% ext3_mark_iloc_dirty
      3630   184.5% __find_get_block
      3360   239.8% __brelse
      2911     0.0% ext3_do_update_inode
      2672  16700.0% scsi_request_fn
      2565     0.0% journal_start
      2432     0.0% journal_commit_transaction
      1945     0.0% journal_unlock_journal_head
      1937     0.0% ext3_new_inode
      1820     0.0% ext3_orphan_del
      1731  13315.4% __make_request
      1606  1784.4% .text.lock.ioctl
      1591   103.9% .text.lock.base
      1575     0.0% find_next_usable_block
      1563   142.2% __block_prepare_write
      1483     0.0% ext3_orphan_add
      1428     0.0% __journal_unfile_buffer
      1367     0.0% journal_get_undo_access
      1365   156.7% kmap_atomic
      1360  1766.2% default_wake_function
      1344    40.5% do_anonymous_page
      1232     0.0% ext3_reserve_inode_write
      1180    64.7% current_kernel_time
      1172     0.0% journal_invalidatepage
      1139   136.4% kmalloc
      1121  37366.7% mempool_alloc
      1017     0.0% __journal_file_buffer
...
     -1080   -55.4% d_alloc
     -1105   -67.4% filemap_nopage
     -1153  -100.0% ext2_new_inode
     -1182   -37.3% pte_alloc_one
     -1186   -46.4% copy_process
     -1470   -98.4% .text.lock.highmem
     -1496   -95.2% .text.lock.file_table
     -1555   -64.2% file_move
     -1596   -94.4% __read_lock_failed
     -1601   -61.8% path_release
     -1602  -100.0% grab_block
     -1649   -51.0% copy_mm
     -2413   -57.4% remove_shared_vm_struct
     -2540   -78.9% free_pages_and_swap_cache
     -3236   -93.1% .text.lock.namei
     -3600   -47.5% page_add_rmap
     -4671   -72.9% path_lookup
     -4720   -95.8% .text.lock.dcache
     -4921   -42.9% __d_lookup
     -5140   -86.9% follow_mount
     -6005   -39.7% copy_page_range
     -6534   -72.0% release_pages
     -6905   -94.2% .text.lock.dec_and_lock
     -7158   -74.2% atomic_dec_and_lock
     -7307   -47.3% page_remove_rmap
     -9399   -48.2% zap_pte_range

Looks just like horrific semaphore contention to me.

I dumped the full results to:
ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/benchmarks/2.5.66-mm2/
ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/benchmarks/2.5.66-mm2-ext3/
as you probably want the gory detail, knowing you ;-)

M.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-02  1:10 ` 2.5.66-mm2 Martin J. Bligh
@ 2003-04-02  1:34   ` Andrew Morton
  2003-04-02 15:34     ` 2.5.66-mm2 Martin J. Bligh
  0 siblings, 1 reply; 11+ messages in thread
From: Andrew Morton @ 2003-04-02  1:34 UTC (permalink / raw)
  To: Martin J. Bligh; +Cc: bzzz, linux-kernel, linux-mm

"Martin J. Bligh" <mbligh@aracnet.com> wrote:
>
> Ho hum. All very strange. Kernbench seems to be really behaving itself
> quite well now, but SDET sucks worse than ever. The usual 16x NUMA-Q 
> machine .... 
> 
> Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)
>                               Elapsed      System        User         CPU
>                2.5.66-mm2       44.04       81.12      569.40     1476.75
>           2.5.66-mm2-ext3       44.43       84.10      568.82     1469.00

Is this ext2 versus ext3?  If so, that's a pretty good result isn't it?  I
forget what kernbench looked like for stock ext3.

> SDET 32  (see disclaimer)
>                            Throughput    Std. Dev
>                2.5.66-mm2       100.0%         0.7%
>           2.5.66-mm2-ext3         4.7%         1.5%

Yes, this is presumably a lot more metadata-intensive, so we're just
hammering the journal semaphore to death.  We're working on it.

> ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/benchmarks/2.5.66-mm2-ext3/

Offtopic, a raw sdet64 profile says:

5392317 total
4478683 default_idle
307163 __down
169770 .text.lock.sched
106769 schedule
88092 __wake_up
57280 .text.lock.transaction

I'm slightly surprised that the high context switch rate is showing up so
much contention in sched.c.  I'm assuming that it's on the sleep/wakeup path
and not in the context switch path.  It would be interesting to inline the
spinlock code and reprofile.

We really should be using the waker-removes-wakee facility in the semaphore
code, but that's not completely trivial.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-02  1:34   ` 2.5.66-mm2 Andrew Morton
@ 2003-04-02 15:34     ` Martin J. Bligh
  0 siblings, 0 replies; 11+ messages in thread
From: Martin J. Bligh @ 2003-04-02 15:34 UTC (permalink / raw)
  To: Andrew Morton; +Cc: bzzz, linux-kernel, linux-mm

>> Ho hum. All very strange. Kernbench seems to be really behaving itself
>> quite well now, but SDET sucks worse than ever. The usual 16x NUMA-Q 
>> machine .... 
>> 
>> Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)
>>                               Elapsed      System        User         CPU
>>                2.5.66-mm2       44.04       81.12      569.40     1476.75
>>           2.5.66-mm2-ext3       44.43       84.10      568.82     1469.00
> 
> Is this ext2 versus ext3?  If so, that's a pretty good result isn't it?  I
> forget what kernbench looked like for stock ext3.

Yes, it's splendid. Used to look more like this:

Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)
                              Elapsed      System        User         CPU
            2.5.61-mjb0.1       46.04      115.46      563.07     1472.25
       2.5.61-mjb0.1-ext3       48.45      143.79      564.14     1459.00

That was before I had noatime though (I think) ...

Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)
                              Elapsed      System        User         CPU
              2.5.65-mjb1       43.73       81.69      563.54     1475.00
         2.5.65-mjb1-ext3       44.13       79.77      564.56     1460.25

So I think after noatime, SDET was really the big problem.

>> SDET 32  (see disclaimer)
>>                            Throughput    Std. Dev
>>                2.5.66-mm2       100.0%         0.7%
>>           2.5.66-mm2-ext3         4.7%         1.5%
> 
> Yes, this is presumably a lot more metadata-intensive, so we're just
> hammering the journal semaphore to death.  We're working on it.

Ah, that makes sense, thanks.
 
>> ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/benchmarks/2.5.66-mm2-ext3/
> 
> Offtopic, a raw sdet64 profile says:
> 
> 5392317 total
> 4478683 default_idle
> 307163 __down
> 169770 .text.lock.sched
> 106769 schedule
> 88092 __wake_up
> 57280 .text.lock.transaction
> 
> I'm slightly surprised that the high context switch rate is showing up so
> much contention in sched.c.  I'm assuming that it's on the sleep/wakeup path
> and not in the context switch path.  It would be interesting to inline the
> spinlock code and reprofile.

OK, done. diffprofile with an without spinlines below:

    350487   158.8% schedule
     67691     0.6% total
     49184     0.5% default_idle
     43095  1680.1% journal_start
     42048   136.1% do_get_write_access
     34988   638.7% journal_stop
     16813  1527.1% inode_change_ok
      6752     3.6% __wake_up
      5960     0.9% __down
      3942  3718.9% sem_exit
      3323   725.5% inode_setattr
      3191  1470.5% proc_pid_readlink
      1646  2743.3% sys_ioctl
       418    16.8% atomic_dec_and_lock
       408     4.4% journal_add_journal_head
       303   841.7% proc_root_lookup
       258     4.6% __find_get_block
       256    33.0% follow_mount
       245     1.7% journal_dirty_metadata
       223     4.7% __find_get_block_slow
       213  2366.7% chrdev_open
       209  1492.9% proc_root_readdir
       202     2.6% find_get_page
       171     8.8% journal_unlock_journal_head
       146     6.2% kmem_cache_free
       126     2.7% do_anonymous_page
       120     1.3% copy_page_range
       111  1387.5% sys_sysctl
       107    93.9% journal_get_create_access
       106     0.8% cpu_idle
       106   963.6% __posix_lock_file
       101   280.6% put_filp
       100  1666.7% de_put
...
      -102   -13.1% fget
      -118  -100.0% .text.lock.char_dev
      -120    -2.2% block_write_full_page
      -123   -20.0% block_prepare_write
      -127   -93.4% remove_from_page_cache
      -142  -100.0% .text.lock.sysctl
      -167    -1.5% __blk_queue_bounce
      -183   -27.8% do_generic_mapping_read
      -203   -11.7% free_hot_cold_page
      -205  -100.0% .text.lock.dcache
      -211   -13.0% buffered_rmqueue
      -237   -98.8% .text.lock.namei
      -426  -100.0% .text.lock.dec_and_lock
      -458  -100.0% .text.lock.root
      -516    -8.8% __copy_to_user_ll
      -781  -100.0% .text.lock.journal
     -1696  -100.0% .text.lock.ioctl
     -3123  -100.0% .text.lock.base
     -4048  -100.0% .text.lock.sem
    -19538  -100.0% .text.lock.attr
   -117523   -99.9% .text.lock.transaction
   -347530  -100.0% .text.lock.sched

Thanks, 

M.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-01  8:01 2.5.66-mm2 Andrew Morton
  2003-04-01  8:48 ` 2.5.66-mm2 Andrew Morton
  2003-04-02  1:10 ` 2.5.66-mm2 Martin J. Bligh
@ 2003-04-03 17:53 ` Maciej Soltysiak
  2003-04-03 21:22   ` 2.5.66-mm2 Andrew Morton
  2 siblings, 1 reply; 11+ messages in thread
From: Maciej Soltysiak @ 2003-04-03 17:53 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel

Hi,

remember my post about the machine locking up for a few seconds?
This time i've a vmstat report with 2.5.66-mm2, it's just to let you know
the stats during the lockup, i think they differ, there is no significant
swapping, so i think it's not a swapping issue.
I will try running the machine with mm3, with my next reboot.

Here are the printouts, 3 minutes of the machine stalling and when it was
becoming more responsive it was very sluggish, abnormally, some apps where
responding, some not.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   4432   1840  45420    2    2    81    70  547    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   4312   1840  45548    2    2    81    70  547    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   4192   1840  45676    2    2    81    70  547    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   4048   1856  45804    2    2    81    70  547    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   3928   1856  45932    2    2    81    70  547    17 59  2 38  1
-----
czw kwi  3 17:11:06 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   3928   1856  45936    2    2    81    70  547    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   4816   1832  45064    2    2    81    70  547    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   4576   1832  45320    2    2    81    70  547    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49220   4408   1848  45448    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2  49216   4688   1900  44840    2    2    81    70  548    17 59  2 38  1
-----
czw kwi  3 17:11:11 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49216   4816   1996  43852    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  49216   4328   2032  44184    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2  49216   4208   2100  44196    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1  49216   4728   2128  43252    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  49216   4608   2128  43380    2    2    81    70  548    17 59  2 38  1
-----
czw kwi  3 17:11:16 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  50168   1992    664  37472    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  52760   3532    476  35304    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  53476   1936    304  27436    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  53796   3216    296  27264    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  53796   3032    296  27392    2    2    81    70  548    17 59  2 38  1
-----
czw kwi  3 17:11:23 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1  53796   3824    296  27068    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  53820   3848    312  27152    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  53820   3632    316  27288    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  53820   3448    316  27496    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  53820   3284    316  27624    2    2    81    70  548    17 59  2 38  1
-----
czw kwi  3 17:11:28 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  53820   3192    316  27764    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  54088   3984    320  27068    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  54088   3904    320  27156    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  54088   3784    320  27284    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  54088   3592    320  27412    2    2    81    70  548    17 59  2 38  1
-----
czw kwi  3 17:11:33 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  54088   3232    320  27680    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  54088   3880    324  26632    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  54088   3880    324  26664    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  54088   3688    324  26852    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  54088   3376    324  27108    2    2    81    70  548    17 59  2 38  1
-----
czw kwi  3 17:11:38 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  54088   3088    332  27388    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  54656   2712    360  28632    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  55528   3480    360  28876    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  55528   3192    360  29136    2    2    81    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2  55528   3248    484  27980    2    2    81    70  548    17 59  2 38  1
-----
czw kwi  3 17:11:44 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  56284   3540    484  28532    2    2    82    70  548    17 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  56844   3208    488  28516    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3472    664  28564    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3232    664  28820    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3592    604  28532    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:11:49 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3304    624  28784    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3432    600  28716    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3144    600  29008    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3552    600  28592    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3384    600  28768    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:11:54 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0  56972   3488    616  28536    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3344    616  28664    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3272    616  28748    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3560    608  28472    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3356    608  28680    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:11:59 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3236    624  28784    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3404    608  28644    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3236    608  28768    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3336    608  28656    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3600    612  28580    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:12:04 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3336    616  28820    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3216    616  28948    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3384    584  28708    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3264    584  28816    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3520    596  28580    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:12:09 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3304    596  28792    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3472    572  28656    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3328    572  28812    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3576    560  28564    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3340    576  28788    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:12:15 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3464    572  28668    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3320    572  28820    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  56972   3568    572  28568    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  56972   3376    572  28756    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  56972   3164    592  28680    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:12:20 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  57216   3404    616  28720    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  4  58184   4412    628  28896    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  4  58988   3356    696  29104    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  59736   3836    780  29044    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  59736   3636    860  29600    2    2    82    70  548    18 59  2 38  1
-----
czw kwi  3 17:12:46 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  3  62760   3836    888  30400    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  64092   3808    792  31200    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  64824   2652    496  31884    2    2    82    70  548    18 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  4  67032   3808    496  31156    2    2    82    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  4  68156   3332    488  31000    2    2    82    70  549    19 59  2 38  1
-----
czw kwi  3 17:12:55 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  69872   3124    568  31200    2    2    82    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  70688   3128    644  30792    2    2    82    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  4  70688   2160    884  30676    2    2    82    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  4  72976   3272    960  28928    2    2    82    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  73384   3328   1036  28020    2    2    82    70  549    19 59  2 38  1
-----
czw kwi  3 17:13:03 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  5  75020   1948   1132  26556    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  6  75092   2076    916  26848    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  75284   3416    980  24744    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  75284   3536   1024  23324    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  75896   3776    976  22896    2    2    83    70  549    19 59  2 38  1
-----
czw kwi  3 17:13:11 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  75896   2880   1012  22136    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  4  76840   2628   1004  22100    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  77528   3372    964  21948    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  78160   3552    976  21572    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  78160   3312   1000  21368    2    2    83    70  549    19 59  2 38  1
-----
czw kwi  3 17:13:16 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  78940   2572   1164  21460    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  78940   2476   1524  21552    2    2    83    70  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  80016   2716   1676  20952    2    2    83    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  1  80640   3156   1516  22236    2    2    83    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0  78448  19228   1556  27388    2    2    83    71  549    19 59  2 38  1
-----
czw kwi  3 17:13:23 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1  78448   5072   1564  32820    2    2    83    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  79300   4104   1668  33264    2    2    83    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  79336   3120   1792  34352    2    2    83    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3  80256  19800   1692  35328    2    2    83    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  0  79604  17372   1816  36912    2    2    84    71  549    19 59  2 38  1
-----
czw kwi  3 17:13:29 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  1  79604   4748   1848  42228    2    2    84    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  5  79688   4404   1640  40964    2    2    84    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  8  79688   3300   1532  42208    2    2    84    71  549    19 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  2  79688  19016   1440  36264    2    2    84    71  549    20 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  1  79688  17352   1548  37636    2    2    84    71  549    20 59  2 38  1
-----
czw kwi  3 17:13:55 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  1  79688  15888   1932  38468    2    2    84    71  549    20 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2  79688  15572   2072  39068    2    2    84    71  549    20 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  79688  15140   2076  39252    2    2    84    71  549    20 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  79688  14988   2092  39380    2    2    84    71  549    20 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0  79680  20928   2112  38160    2    2    84    71  549    20 59  2 38  1
-----
czw kwi  3 17:14:00 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0  79680  20640   2116  38456    2    2    84    71  549    20 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  3  79660  12328   2156  42468    2    2    84    71  549    20 59  2 38  1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  4  79660   3672   1900  50732    2    2    84    71  549    20 59  2 38  1

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-03 17:53 ` 2.5.66-mm2 Maciej Soltysiak
@ 2003-04-03 21:22   ` Andrew Morton
  2003-04-04  8:00     ` 2.5.66-mm2 Maciej Soltysiak
  2003-04-07 16:40     ` 2.5.66-mm2 Maciej Soltysiak
  0 siblings, 2 replies; 11+ messages in thread
From: Andrew Morton @ 2003-04-03 21:22 UTC (permalink / raw)
  To: Maciej Soltysiak; +Cc: linux-kernel

Maciej Soltysiak <solt@dns.toxicfilms.tv> wrote:
>
> Hi,
> 
> remember my post about the machine locking up for a few seconds?

Could you try 2.5.66-mm3?  It has a CPU scheduler fix which might well help here.



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-03 21:22   ` 2.5.66-mm2 Andrew Morton
@ 2003-04-04  8:00     ` Maciej Soltysiak
  2003-04-07 16:40     ` 2.5.66-mm2 Maciej Soltysiak
  1 sibling, 0 replies; 11+ messages in thread
From: Maciej Soltysiak @ 2003-04-04  8:00 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel

> > remember my post about the machine locking up for a few seconds?
>
> Could you try 2.5.66-mm3?  It has a CPU scheduler fix which might well help here.
It's still there.
Using the following script to get vmstat output...
#!/bin/sh
rm vmlog
while (true);
do
        echo "-----" >> vmlog
        date >> vmlog
        for i in `seq 1 5`; do
                vmstat >> vmlog
                sleep 1;
        done;
done

I managed to get the following. The lockup occured between 9:53:00, and
9:53:20.

pią kwi  4 09:52:58 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  1  65724   4676   7256  48936    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  2  65724   4636   8184  49048    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  2  65724   4636   8184  49048    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  2  65716   5452   7456  49028    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  2  65716   5452   7456  49028    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 7 10  65716   4148   7476  49040    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 6 10  65716   4148   7476  49040    8    6    42    44  224   472 98  1  0  0
-----
pią kwi  4 09:53:20 CEST 2003
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 5  6  65736   4532   7344  48804    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 4  6  65736   4532   7348  48804    8    6    42    44  224   472 98  1  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
-----
pią kwi  4 09:53:21 CEST 2003

Running apps: [debian] xmms + opera + setiathome + x-terminal-emulator

It just stopped responding suddenly for 20 seconds and went on.
I think i need some other method of measuring what is going on.

Notice, that there are only 7 results between these 20 seconds, and there
should be 20. Earlier in the logs i also get these irregularities, the
script can not manage to get it on time, or the script is getting skewed
by some other factors i am unaware of.


Regards,
Maciej

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
  2003-04-03 21:22   ` 2.5.66-mm2 Andrew Morton
  2003-04-04  8:00     ` 2.5.66-mm2 Maciej Soltysiak
@ 2003-04-07 16:40     ` Maciej Soltysiak
  1 sibling, 0 replies; 11+ messages in thread
From: Maciej Soltysiak @ 2003-04-07 16:40 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel

> Could you try 2.5.66-mm3?  It has a CPU scheduler fix which might well
> help here.
Hi,

i am using mm3 now.

i think i know what my problem really is, i have 128MB of ram and about
160MB of swap space. Should not i have 2.5 times more that RAM?  Like
320MB. That could lead to enormous swapping. But still the lockups are
curious - even with ram outages and little swap space i should be getting
lots of continuos hdd load, which i do, but why would the system lockup
for 10 seconds with no disk activity?

I tried turning /proc/sys/vm/swappines to 0, and other values between 0
and 60. Also tried /proc/sys/vm/dirty_ratio 15 like you suggested, with
the same effects.

I will try to get more swap space, or maybe ram too and see if it helps.

Regards,
Maciej


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: 2.5.66-mm2
       [not found] <20030401081012$4c1b@gated-at.bofh.it>
@ 2003-04-01 16:38 ` Michal Schmidt
  0 siblings, 0 replies; 11+ messages in thread
From: Michal Schmidt @ 2003-04-01 16:38 UTC (permalink / raw)
  To: akpm; +Cc: linux-kernel

Andrew Morton wrote:
 >
 >   There is a small patch from Ingo here against the CPU scheduler 
which we
 >   hope will fix the new starvation problems which people have been 
reporting.
 >   I this is you, please test and report.


I patched 2.5.66 with:
http://www.zip.com.au/~akpm/linux/patches/2.5/2.5.66-mm2-1.gz

I can still easily reproduce my starvation problem with:

cat cedo.iso | bzip2 > /tmp/cedo.iso.bz2
(cedo.iso is a 700MB CD image)

On another virtual console, I run the following script to demonstrate
the starvation:

#!/bin/sh
while true; do
   sleep 30
   date
   ps x
   date
   echo -----
done


At first, everything is OK:

Tue Apr  1 18:00:03 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     S      0:00 cat cedo.iso
   648 tty1     R      0:54 bzip2
   651 tty2     S      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
   655 tty2     R      0:00 ps x
Tue Apr  1 18:00:03 CEST 2003
-----
Tue Apr  1 18:00:33 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     S      0:00 cat cedo.iso
   648 tty1     R      1:24 bzip2
   651 tty2     S      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
   659 tty2     R      0:00 ps x
Tue Apr  1 18:00:33 CEST 2003
-----


But after a while the problem appears:

-----
Tue Apr  1 18:05:09 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     D      0:01 cat cedo.iso
   648 tty1     S      5:55 bzip2
   651 tty2     S      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
  1043 tty2     R      0:00 ps x
Tue Apr  1 18:05:11 CEST 2003
-----
Tue Apr  1 18:05:49 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     S      0:01 cat cedo.iso
   648 tty1     R      6:37 bzip2
   651 tty2     S      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
  1047 tty2     R      0:00 ps x
Tue Apr  1 18:05:51 CEST 2003
-----
Tue Apr  1 18:06:21 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     D      0:01 cat cedo.iso
   648 tty1     S      7:23 bzip2
   651 tty2     S      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
  1051 tty2     R      0:00 ps x
Tue Apr  1 18:06:38 CEST 2003
-----
Tue Apr  1 18:07:08 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     D      0:01 cat cedo.iso
   648 tty1     S      7:59 bzip2
   651 tty2     S      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
  1055 tty2     R      0:00 ps x
Tue Apr  1 18:07:14 CEST 2003
-----
Tue Apr  1 18:07:53 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     S      0:01 cat cedo.iso
   648 tty1     R      8:37 bzip2
   651 tty2     R      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
  1059 tty2     R      0:00 ps x
Tue Apr  1 18:08:03 CEST 2003
-----
Tue Apr  1 18:08:33 CEST 2003
   PID TTY      STAT   TIME COMMAND
   301 tty1     S      0:00 -bash
   302 tty2     S      0:00 -bash
   647 tty1     D      0:01 cat cedo.iso
   648 tty1     S      9:38 bzip2
   651 tty2     S      0:00 /bin/sh ./test-interactivity.sh
   652 tty2     S      0:00 tee inter-2.5.66mm2.potreti
  1066 tty2     R      0:00 ps x
Tue Apr  1 18:08:54 CEST 2003
-----

I use Debian Woody on Athlon 800MHz, Asus A7V, 384MB RAM, disk WDC
WD800JB-00CRA1, nVidia Geforce2 MX, Realtek RTL-8139C, SB Live.
GCC is 2.95.4.

Michal


^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2003-04-07 16:28 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-04-01  8:01 2.5.66-mm2 Andrew Morton
2003-04-01  8:48 ` 2.5.66-mm2 Andrew Morton
2003-04-01  9:15   ` 2.5.66-mm2 Andrew Morton
2003-04-02  1:10 ` 2.5.66-mm2 Martin J. Bligh
2003-04-02  1:34   ` 2.5.66-mm2 Andrew Morton
2003-04-02 15:34     ` 2.5.66-mm2 Martin J. Bligh
2003-04-03 17:53 ` 2.5.66-mm2 Maciej Soltysiak
2003-04-03 21:22   ` 2.5.66-mm2 Andrew Morton
2003-04-04  8:00     ` 2.5.66-mm2 Maciej Soltysiak
2003-04-07 16:40     ` 2.5.66-mm2 Maciej Soltysiak
     [not found] <20030401081012$4c1b@gated-at.bofh.it>
2003-04-01 16:38 ` 2.5.66-mm2 Michal Schmidt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).