linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
@ 2004-08-16 13:23 Lawrence E. Freil
  0 siblings, 0 replies; 12+ messages in thread
From: Lawrence E. Freil @ 2004-08-16 13:23 UTC (permalink / raw)
  To: linux-kernel

Iteresting idea, here is the /proc/mtrr output:

reg00: base=0x00000000 (   0MB), size= 512MB: write-back, count=1
reg01: base=0x20000000 ( 512MB), size= 256MB: write-back, count=1
reg02: base=0x30000000 ( 768MB), size= 128MB: write-back, count=1
reg03: base=0x38000000 ( 896MB), size=  64MB: write-back, count=1
reg04: base=0x3c000000 ( 960MB), size=  32MB: write-back, count=1
reg05: base=0x3e000000 ( 992MB), size=  16MB: write-back, count=1
reg06: base=0xf0000000 (3840MB), size=   2MB: write-combining, count=1

The memory in question is the block from 896 to 1Gig.  Looks pretty
normal.

At this point I need to recompile "ls" with full profiling and time it
with and without extended memory.  I have a suspicion that it is related
to "mmap" on data in a high memory buffer.  It does not effect actual
file read/write.  Performance on that is pretty much the same.

>On Sat, Aug 14, 2004 at 12:43:11PM -0400, Lawrence E. Freil wrote:
>> Though there is a consistant sys component that is slightly higher.  I ran
>> profiles three times for "fast" and three times for "slow".  Here they are
>
>Try cat /proc/mtrr.  I bet your system isn't marking some of its RAM 
>cachable in the MTRRs.
>
>		-ben

-- 
        Lawrence Freil                      Email:lef@freil.com
        1768 Old Country Place              Phone:(770) 667-9274
        Woodstock, GA 30188



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
@ 2004-08-16 18:43 Mikael Pettersson
  0 siblings, 0 replies; 12+ messages in thread
From: Mikael Pettersson @ 2004-08-16 18:43 UTC (permalink / raw)
  To: lef, linux-kernel

On Mon, 16 Aug 2004 09:23:40 -0400, Lawrence E. Freil wrote:
>Iteresting idea, here is the /proc/mtrr output:
>
>reg00: base=0x00000000 (   0MB), size= 512MB: write-back, count=1
>reg01: base=0x20000000 ( 512MB), size= 256MB: write-back, count=1
>reg02: base=0x30000000 ( 768MB), size= 128MB: write-back, count=1
>reg03: base=0x38000000 ( 896MB), size=  64MB: write-back, count=1
>reg04: base=0x3c000000 ( 960MB), size=  32MB: write-back, count=1
>reg05: base=0x3e000000 ( 992MB), size=  16MB: write-back, count=1
>reg06: base=0xf0000000 (3840MB), size=   2MB: write-combining, count=1
>
>The memory in question is the block from 896 to 1Gig.  Looks pretty
>normal.

How does the E820 memory map look? (At the top of the dmesg log.)

Your BIOS left the [1008MB,1024MB[ range uncached. Try booting
with "mem=1008M". If that fixes the performance problem,
complain to your mainboard vendor that their BIOS is broken.

/Mikael

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
@ 2004-08-16 17:40 Lawrence E. Freil
  0 siblings, 0 replies; 12+ messages in thread
From: Lawrence E. Freil @ 2004-08-16 17:40 UTC (permalink / raw)
  To: linux-kernel

Problem solved.

I found a reference to a very similar (turns out same) problem with the
2.4 series kernels.  The problem is because the mtrr maps do not quite
cover all the memory the system was reporting.   Memory that does not
show up in the mtrr must be being mapped as non-cacheable.  Is there
a reason the system defaults to this as oppose to cacheable?  It obviously
sees the holes?  I can work around the problem by adding "mem=1008" instead
of letting it default (or I suppose I could patch the mtrr after boot).

Thanks for all those with suggestions and what to look for.

-- 
        Lawrence Freil                      Email:lef@freil.com
        1768 Old Country Place              Phone:(770) 667-9274
        Woodstock, GA 30188



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
  2004-08-14 16:43 Lawrence E. Freil
@ 2004-08-15 15:04 ` Benjamin LaHaise
  0 siblings, 0 replies; 12+ messages in thread
From: Benjamin LaHaise @ 2004-08-15 15:04 UTC (permalink / raw)
  To: Lawrence E. Freil; +Cc: linux-kernel

On Sat, Aug 14, 2004 at 12:43:11PM -0400, Lawrence E. Freil wrote:
> Though there is a consistant sys component that is slightly higher.  I ran
> profiles three times for "fast" and three times for "slow".  Here they are

Try cat /proc/mtrr.  I bet your system isn't marking some of its RAM 
cachable in the MTRRs.

		-ben

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
@ 2004-08-14 16:43 Lawrence E. Freil
  2004-08-15 15:04 ` Benjamin LaHaise
  0 siblings, 1 reply; 12+ messages in thread
From: Lawrence E. Freil @ 2004-08-14 16:43 UTC (permalink / raw)
  To: linux-kernel


On Sat Aug 14th, Martin J. Bligh wrote:
>Does time indicate more systime or user time? If it's systime (probably is)
>please take a kernel profile of each (see Documentation/basic_profiling.txt).
>That'll give us a much better clue what's going on.
>
>M.

Martin, the time is almost exclusively listed as "user" in the the "time"
results:

FAST (with mem=800M): time ls -laR /home/groups/sysmbase >/dev/null

      real    0m0.122s
      user    0m0.101s
      sys     0m0.021s

SLOW (without mem=800M): time ls -laR /home/groups/sysmbase >/dev/null
      real    0m5.175s
      user    0m5.145s
      sys     0m0.030s

Though there is a consistant sys component that is slightly higher.  I ran
profiles three times for "fast" and three times for "slow".  Here they are

FAST profile 1:
    33 __do_softirq                               0.2578
     1 groups_search                              0.0094
     1 in_group_p                                 0.0085
     1 do_mmap_pgoff                              0.0006
     5 __d_lookup                                 0.0208
     1 xfs_dir2_leaf_getdents                     0.0003
     1 xfs_trans_brelse                           0.0035
     6 strncpy_from_user                          0.0606
    49 total                                      0.0000

FAST profile 2:
     3 system_call                                0.0682
     1 do_page_fault                              0.0008
     1 groups_search                              0.0094
     1 sys_lstat64                                0.0175
     1 follow_mount                               0.0073
     1 do_lookup                                  0.0062
     3 link_path_walk                             0.0015
     5 __d_lookup                                 0.0208
     2 write_profile                              0.0312
     1 xfs_dir2_put_dirent64_direct               0.0067
     1 xfs_dir2_put_dirent64_uio                  0.0049
     1 xfs_dir2_leaf_getdents                     0.0003
     4 strncpy_from_user                          0.0404
     1 __copy_user_intel                          0.0058
     1 __copy_to_user_ll                          0.0093
    27 total                                      0.0000

FAST profile 3:
     2 system_call                                0.0455
     1 groups_search                              0.0094
     1 in_group_p                                 0.0085
     1 buffered_rmqueue                           0.0028
     1 find_vma_prev                              0.0123
     1 cp_new_stat64                              0.0037
     2 follow_mount                               0.0146
     4 link_path_walk                             0.0020
     2 dput                                       0.0058
     4 __d_lookup                                 0.0167
     1 de_put                                     0.0097
     3 write_profile                              0.0469
     2 strncpy_from_user                          0.0202
     1 __copy_user_intel                          0.0058
    26 total                                      0.0000

And here are the three "SLOW" profiles:

SLOW profile 1:
     4 system_call                                0.0909
     2 groups_search                              0.0189
     1 free_hot_cold_page                         0.0043
     1 kmem_cache_alloc                           0.0089
     1 kmem_cache_free                            0.0132
     1 __pagevec_lru_add_active                   0.0060
     1 do_wp_page                                 0.0013
     1 do_anonymous_page                          0.0027
     3 do_no_page                                 0.0040
     1 vfs_getattr                                0.0065
     2 follow_mount                               0.0146
     2 do_lookup                                  0.0124
     5 link_path_walk                             0.0024
     1 path_lookup                                0.0030
     1 dput                                       0.0029
     6 __d_lookup                                 0.0250
     3 write_profile                              0.0469
     1 xfs_bmapi                                  0.0002
     1 xfs_dir2_block_getdents                    0.0012
     2 strncpy_from_user                          0.0202
     2 __copy_user_intel                          0.0116
     7 __copy_to_user_ll                          0.0648
    49 total                                      0.0000

SLOW profile 2:
     2 system_call                                0.0455
     2 do_page_fault                              0.0015
     1 zap_pte_range                              0.0017
     1 do_anonymous_page                          0.0027
     2 do_no_page                                 0.0027
     1 follow_mount                               0.0073
     1 do_lookup                                  0.0062
     4 link_path_walk                             0.0020
     1 filldir64                                  0.0043
    10 __d_lookup                                 0.0417
     3 write_profile                              0.0469
     1 linvfs_getattr                             0.0159
     4 strncpy_from_user                          0.0404
     1 __copy_user_intel                          0.0058
     3 __copy_to_user_ll                          0.0278
     1 ata_interrupt                              0.0037
    38 total                                      0.0000

SLOW profile 3:
     4 system_call                                0.0909
     1 __might_sleep                              0.0063
     1 in_group_p                                 0.0085
     1 wake_up_page                               0.0175
     1 prep_new_page                              0.0123
     1 kmem_cache_free                            0.0132
     2 do_no_page                                 0.0027
     1 sys_llseek                                 0.0051
     1 vfs_getattr                                0.0065
     1 getname                                    0.0041
     3 link_path_walk                             0.0015
     1 path_lookup                                0.0030
    10 __d_lookup                                 0.0417
     2 write_profile                              0.0312
     1 xfs_da_brelse                              0.0034
     1 xfs_dir2_put_dirent64_direct               0.0067
     4 strncpy_from_user                          0.0404
     4 __copy_user_intel                          0.0233
     2 __copy_to_user_ll                          0.0185
    42 total                                      0.0000

Hope this helps.



-- 
        Lawrence Freil                      Email:lef@freil.com
        1768 Old Country Place              Phone:(770) 667-9274
        Woodstock, GA 30188



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
@ 2004-08-14 15:43 Lawrence E. Freil
  0 siblings, 0 replies; 12+ messages in thread
From: Lawrence E. Freil @ 2004-08-14 15:43 UTC (permalink / raw)
  To: linux-kernel

On Sat, Aug 14, 2004 at 8:51AM William Lee Irwin III wrote:
>Please try to reproduce this with CONFIG_HIGHMEM=y but using mem=700M
>This will tell me something useful beyond "boot with less RAM".
>
>-- wli

wli,

I did as you suggested and booted a kernel with HIMEM set but mem
set to 700M in boot params:

user-defined physical RAM map:
 user: 0000000000000000 - 000000000009fc00 (usable)
 user: 000000000009fc00 - 00000000000a0000 (reserved)
 user: 00000000000e8000 - 0000000000100000 (reserved)
 user: 0000000000100000 - 000000002bc00000 (usable)
0MB HIGHMEM available.
700MB LOWMEM available.

I get the same behaviour as if I had not specified HIMEM (fast directory
access)  The exact same kernel without the mem=700M runs slow.


-- 
        Lawrence Freil                      Email:lef@freil.com
        1768 Old Country Place              Phone:(770) 667-9274
        Woodstock, GA 30188



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
  2004-08-14  2:11 Lawrence E. Freil
  2004-08-14  4:18 ` Darren Williams
  2004-08-14  8:51 ` William Lee Irwin III
@ 2004-08-14 15:09 ` Martin J. Bligh
  2 siblings, 0 replies; 12+ messages in thread
From: Martin J. Bligh @ 2004-08-14 15:09 UTC (permalink / raw)
  To: Lawrence E. Freil, linux-kernel

> 2. I have discovered an issue with the Linux 2.6.7 kernel when HIMEM is
>    enabled which exhibits itself as a slowdown in directory access regardless
>    of filesystem used.  When HIMEM is disabled the performance returns to
>    normal.  The test I ran was a simple "/usr/bin/time ls -l" of a directory
>    with 3000 empty files.  With HIMEM enabled in the kernel this takes
>    approximately 1.5 seconds.  Without HIMEM it takes 0.03 seconds.  The
>    time is 100% CPU and no I/O operations are done to disk.  "time" reports
>    there are 460 "minor" page faults with zero "major" page faults.
>    I believe the issue here is the mapping of pages between high-mem and
>    lowmem in the kernel paging code.  This increase in time for directory
>    accesses doubles to triples times for applications using samba.
>    I have also tested this on another system which had only 512Meg of RAM
>    but with HIMEM set in the kernel and did not experience the problem.
>    I believe it only effects the performance when the paging buffers end
>    up in highmem.

Does time indicate more systime or user time? If it's systime (probably is)
please take a kernel profile of each (see Documentation/basic_profiling.txt).
That'll give us a much better clue what's going on.

M.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
  2004-08-14  4:18 ` Darren Williams
  2004-08-14 11:42   ` Lawrence E. Freil
@ 2004-08-14 15:09   ` Lawrence E. Freil
  1 sibling, 0 replies; 12+ messages in thread
From: Lawrence E. Freil @ 2004-08-14 15:09 UTC (permalink / raw)
  To: Darren Williams; +Cc: linux-kernel, lef, lef

Darren,
   I ran your test of "ls -laR of /etc >/dev/null" on our system and you
need to gen a kernel without himem and try it.  I get around .1 seconds
without himem and a whopping 4.45 with himem enabled.

I also tried without preemption and got the same results.  It is strictly
do to setting HIMEM in the kernel.

> Hi Lawrence
> 
> On Fri, 13 Aug 2004, Lawrence E. Freil wrote:
> 
> > Hello,
> >    
> > I'm following the kernel bug reporting format so:
> > 
> > 1. Linux 2.6.7 kernel slowdown in directory access with HIMEM on
> > 
> > 2. I have discovered an issue with the Linux 2.6.7 kernel when HIMEM is
> >    enabled which exhibits itself as a slowdown in directory access regardless
> >    of filesystem used.  When HIMEM is disabled the performance returns to
> >    normal.  The test I ran was a simple "/usr/bin/time ls -l" of a directory
> >    with 3000 empty files.  With HIMEM enabled in the kernel this takes
> >    approximately 1.5 seconds.  Without HIMEM it takes 0.03 seconds.  The
> >    time is 100% CPU and no I/O operations are done to disk.  "time" reports
> >    there are 460 "minor" page faults with zero "major" page faults.
> >    I believe the issue here is the mapping of pages between high-mem and
> >    lowmem in the kernel paging code.  This increase in time for directory
> >    accesses doubles to triples times for applications using samba.
> >    I have also tested this on another system which had only 512Meg of RAM
> >    but with HIMEM set in the kernel and did not experience the problem.
> >    I believe it only effects the performance when the paging buffers end
> >    up in highmem.
> > 
> > 3. Keywords: HIMEM, Performance
> > 
> Would you be running these in a gnome-terminal, I remember seeing a thread
> that discussed gnome-terminal problems though a quick search did not turn
> anything up. Here is what I get between a xterm and gnome-terminal.
> 
> xterm
> # time ls -lR /etc
> real    0m0.381s
> user    0m0.056s
> sys     0m0.130s
> 
> gnome-terminal
> # time ls -lR /etc
> real    0m0.869s
> user    0m0.057s
> sys     0m0.141s
> 
> I ran this twice in both teminals and reported the
> second result.
> 
> system info
> P4 3.06 HT
> 2.6.7 SMP/SMT/HIGHMEM
> 1GB ram
> 
> -------------------------------------------------
> Darren Williams <dsw AT gelato.unsw.edu.au>
> Gelato@UNSW <www.gelato.unsw.edu.au>
> --------------------------------------------------



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
  2004-08-14  4:18 ` Darren Williams
@ 2004-08-14 11:42   ` Lawrence E. Freil
  2004-08-14 15:09   ` Lawrence E. Freil
  1 sibling, 0 replies; 12+ messages in thread
From: Lawrence E. Freil @ 2004-08-14 11:42 UTC (permalink / raw)
  To: Darren Williams; +Cc: lef

Hello,
   The systems in question are embedded server systems with no GUI loaded.
The environment I use is a simple SSH into the system so minimal processes
are running and the load average is about as close to zero as you can get.
The exact command I am running is "/usr/bin/time ls -l MBRF >/dev/null".

The problem first appeared in relation to a foxpro database application run
off samba share from these systems (we have two identical systems that show 
this behaviour).  The customer was concerned because they upgraded from a 400Mz
celeron to a 2.5Gz P4-Xeon with double the RAM and Gigabit networking and
it was slower than before.  I traced the packets with tcpdump to both the
old and the new system and the slowdown was occuring on windows requests
to find files (mostly dll's).  On further investigation it also appeared
on simple commands like "ls" but I would not have noticed had I not been
looking.  Here are the exact timeings from one system (approx 900 files
in directory) without HIMEM and another with HIMEM on. 

yang (identical hardware and software to yin except HIMEM turned off)
0.01user 0.00system 0:00.01elapsed 87%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+350minor)pagefaults 0swaps

yin
0.70user 0.00system 0:00.71elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+350minor)pagefaults 0swaps

As you can see from this the additional time is entirely spent as user CPU 
time.  I have run these multiple times and checked for disk I/O and there
is no physical I/O operations between these two (everything is cached at this
point).  The time increase is occuring on every directory manipulation, but
read and update (as in my script that creates empty files), but does not
appear to effect sequential file read/write operations.  I have not setup
a test to see if it effects random file access which would be very similar
to the directory read/update case. 

I have run the same tests both on XFS and ext2 with identical results.  The
physical filesystem appears to have no bearing.  I also eliminated the disk
as the source of the problems.  The new systems have Serial ATA drives that
read approx 54MB/second (four times that of the old system) and in file
read tests over samba (and locally) the new systems much faster. I have also
run the tests on a a very similar system (slackware 10, but Athlon and only
512Meg of RAM) with 2.6.7 kernel and HIMEM on but did not see the problem.
I suspect it only occurs when the high memory is used for disk block buffering
but have not had time to attempt profiling the kernel to see where the CPU
time difference occurs.  I do have kernel preemption turned on.  This morning
I will generate a kernel with preemtion turned off and himem on and see if
there is an interaction with that.  I can send kernel configs if you believe
they are relevent but these are pretty much stock kernels with unused modules
turned off.

> Hi Lawrence
> 
> On Fri, 13 Aug 2004, Lawrence E. Freil wrote:
> 
> > Hello,
> >    
> > I'm following the kernel bug reporting format so:
> > 
> > 1. Linux 2.6.7 kernel slowdown in directory access with HIMEM on
> > 
> > 2. I have discovered an issue with the Linux 2.6.7 kernel when HIMEM is
> >    enabled which exhibits itself as a slowdown in directory access regardless
> >    of filesystem used.  When HIMEM is disabled the performance returns to
> >    normal.  The test I ran was a simple "/usr/bin/time ls -l" of a directory
> >    with 3000 empty files.  With HIMEM enabled in the kernel this takes
> >    approximately 1.5 seconds.  Without HIMEM it takes 0.03 seconds.  The
> >    time is 100% CPU and no I/O operations are done to disk.  "time" reports
> >    there are 460 "minor" page faults with zero "major" page faults.
> >    I believe the issue here is the mapping of pages between high-mem and
> >    lowmem in the kernel paging code.  This increase in time for directory
> >    accesses doubles to triples times for applications using samba.
> >    I have also tested this on another system which had only 512Meg of RAM
> >    but with HIMEM set in the kernel and did not experience the problem.
> >    I believe it only effects the performance when the paging buffers end
> >    up in highmem.
> > 
> > 3. Keywords: HIMEM, Performance
> > 
> Would you be running these in a gnome-terminal, I remember seeing a thread
> that discussed gnome-terminal problems though a quick search did not turn
> anything up. Here is what I get between a xterm and gnome-terminal.
> 
> xterm
> # time ls -lR /etc
> real    0m0.381s
> user    0m0.056s
> sys     0m0.130s
> 
> gnome-terminal
> # time ls -lR /etc
> real    0m0.869s
> user    0m0.057s
> sys     0m0.141s
> 
> I ran this twice in both teminals and reported the
> second result.
> 
> system info
> P4 3.06 HT
> 2.6.7 SMP/SMT/HIGHMEM
> 1GB ram
> 
> -------------------------------------------------
> Darren Williams <dsw AT gelato.unsw.edu.au>
> Gelato@UNSW <www.gelato.unsw.edu.au>
> --------------------------------------------------

-- 
        Lawrence Freil                      Email:lef@freil.com
        1768 Old Country Place              Phone:(770) 667-9274
        Woodstock, GA 30188


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
  2004-08-14  2:11 Lawrence E. Freil
  2004-08-14  4:18 ` Darren Williams
@ 2004-08-14  8:51 ` William Lee Irwin III
  2004-08-14 15:09 ` Martin J. Bligh
  2 siblings, 0 replies; 12+ messages in thread
From: William Lee Irwin III @ 2004-08-14  8:51 UTC (permalink / raw)
  To: Lawrence E. Freil; +Cc: linux-kernel

On Fri, Aug 13, 2004 at 10:11:23PM -0400, Lawrence E. Freil wrote:
> I'm following the kernel bug reporting format so:
> 1. Linux 2.6.7 kernel slowdown in directory access with HIMEM on
> 2. I have discovered an issue with the Linux 2.6.7 kernel when HIMEM is
>    enabled which exhibits itself as a slowdown in directory access regardless
>    of filesystem used.  When HIMEM is disabled the performance returns to
>    normal.  The test I ran was a simple "/usr/bin/time ls -l" of a directory
>    with 3000 empty files.  With HIMEM enabled in the kernel this takes
>    approximately 1.5 seconds.  Without HIMEM it takes 0.03 seconds.  The
>    time is 100% CPU and no I/O operations are done to disk.  "time" reports
>    there are 460 "minor" page faults with zero "major" page faults.
>    I believe the issue here is the mapping of pages between high-mem and
>    lowmem in the kernel paging code.  This increase in time for directory
>    accesses doubles to triples times for applications using samba.
>    I have also tested this on another system which had only 512Meg of RAM
>    but with HIMEM set in the kernel and did not experience the problem.
>    I believe it only effects the performance when the paging buffers end
>    up in highmem.

Please try to reproduce this with CONFIG_HIGHMEM=y but using mem=700M
This will tell me something useful beyond "boot with less RAM".

-- wli

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
  2004-08-14  2:11 Lawrence E. Freil
@ 2004-08-14  4:18 ` Darren Williams
  2004-08-14 11:42   ` Lawrence E. Freil
  2004-08-14 15:09   ` Lawrence E. Freil
  2004-08-14  8:51 ` William Lee Irwin III
  2004-08-14 15:09 ` Martin J. Bligh
  2 siblings, 2 replies; 12+ messages in thread
From: Darren Williams @ 2004-08-14  4:18 UTC (permalink / raw)
  To: linux-kernel; +Cc: lef

Hi Lawrence

On Fri, 13 Aug 2004, Lawrence E. Freil wrote:

> Hello,
>    
> I'm following the kernel bug reporting format so:
> 
> 1. Linux 2.6.7 kernel slowdown in directory access with HIMEM on
> 
> 2. I have discovered an issue with the Linux 2.6.7 kernel when HIMEM is
>    enabled which exhibits itself as a slowdown in directory access regardless
>    of filesystem used.  When HIMEM is disabled the performance returns to
>    normal.  The test I ran was a simple "/usr/bin/time ls -l" of a directory
>    with 3000 empty files.  With HIMEM enabled in the kernel this takes
>    approximately 1.5 seconds.  Without HIMEM it takes 0.03 seconds.  The
>    time is 100% CPU and no I/O operations are done to disk.  "time" reports
>    there are 460 "minor" page faults with zero "major" page faults.
>    I believe the issue here is the mapping of pages between high-mem and
>    lowmem in the kernel paging code.  This increase in time for directory
>    accesses doubles to triples times for applications using samba.
>    I have also tested this on another system which had only 512Meg of RAM
>    but with HIMEM set in the kernel and did not experience the problem.
>    I believe it only effects the performance when the paging buffers end
>    up in highmem.
> 
> 3. Keywords: HIMEM, Performance
> 
Would you be running these in a gnome-terminal, I remember seeing a thread
that discussed gnome-terminal problems though a quick search did not turn
anything up. Here is what I get between a xterm and gnome-terminal.

xterm
# time ls -lR /etc
real    0m0.381s
user    0m0.056s
sys     0m0.130s

gnome-terminal
# time ls -lR /etc
real    0m0.869s
user    0m0.057s
sys     0m0.141s

I ran this twice in both teminals and reported the
second result.

system info
P4 3.06 HT
2.6.7 SMP/SMT/HIGHMEM
1GB ram

-------------------------------------------------
Darren Williams <dsw AT gelato.unsw.edu.au>
Gelato@UNSW <www.gelato.unsw.edu.au>
--------------------------------------------------

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7
@ 2004-08-14  2:11 Lawrence E. Freil
  2004-08-14  4:18 ` Darren Williams
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Lawrence E. Freil @ 2004-08-14  2:11 UTC (permalink / raw)
  To: linux-kernel

Hello,
   
I'm following the kernel bug reporting format so:

1. Linux 2.6.7 kernel slowdown in directory access with HIMEM on

2. I have discovered an issue with the Linux 2.6.7 kernel when HIMEM is
   enabled which exhibits itself as a slowdown in directory access regardless
   of filesystem used.  When HIMEM is disabled the performance returns to
   normal.  The test I ran was a simple "/usr/bin/time ls -l" of a directory
   with 3000 empty files.  With HIMEM enabled in the kernel this takes
   approximately 1.5 seconds.  Without HIMEM it takes 0.03 seconds.  The
   time is 100% CPU and no I/O operations are done to disk.  "time" reports
   there are 460 "minor" page faults with zero "major" page faults.
   I believe the issue here is the mapping of pages between high-mem and
   lowmem in the kernel paging code.  This increase in time for directory
   accesses doubles to triples times for applications using samba.
   I have also tested this on another system which had only 512Meg of RAM
   but with HIMEM set in the kernel and did not experience the problem.
   I believe it only effects the performance when the paging buffers end
   up in highmem.

3. Keywords: HIMEM, Performance

4. No oops, kernel operates normally with the exception of the performance.

5. N/A

6. time "I=0; while test $I -lt 3000; do touch $I; I=`expr $I + 1`; done"
   With HIMEM set (and available!) this takes 20 seconds
   Without HIMEM it takes about 5

7. Slackware 10.0 with 2.6.7 kernel 1GB of RAM.

7.1 ver_linux
Gnu C                  3.3.4
Gnu make               3.80
binutils               2.15.90.0.3
util-linux             2.12a
mount                  2.12a
module-init-tools      3.0
e2fsprogs              1.35
jfsutils               1.1.6
xfsprogs               2.6.13
pcmcia-cs              3.2.7
quota-tools            3.12.
PPP                    2.4.2
nfs-utils              1.0.6
Linux C Library        2.3.2
Dynamic linker (ldd)   2.3.2
Linux C++ Library      5.0.6
Procps                 3.2.1
Net-tools              1.60
Kbd                    1.12
Sh-utils               5.2.1
Modules Loaded         ide_scsi

7.2 /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Pentium(R) 4 CPU 2.80GHz
stepping        : 9
cpu MHz         : 2793.765
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
bogomips        : 5521.40

7.3 /proc/modules
ide_scsi 13608 - - Live 0xf8cd9000

7.4 /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-005f : timer
0060-006f : keyboard
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : libata
01f0-01f7 : ide0
02f8-02ff : serial
0378-037a : parport0
03c0-03df : vesafb
03f6-03f6 : ide0
03f8-03ff : serial
0400-041f : 0000:00:1f.3
  0400-040f : i801-smbus
0480-04bf : 0000:00:1f.0
0800-087f : 0000:00:1f.0
0cf8-0cff : PCI conf1
d000-d0ff : 0000:01:09.0
  d000-d0ff : 8139too
d400-d4ff : 0000:01:0b.0
d800-d8ff : 0000:01:0b.0
de00-deff : 0000:01:0a.0
  de00-deff : SysKonnect SK-98xx
df00-df3f : 0000:01:08.0
  df00-df3f : eepro100
e800-e8ff : 0000:00:1f.5
ee80-eebf : 0000:00:1f.5
ef00-ef1f : 0000:00:1d.0
  ef00-ef1f : uhci_hcd
ef20-ef3f : 0000:00:1d.1
  ef20-ef3f : uhci_hcd
ef40-ef5f : 0000:00:1d.2
  ef40-ef5f : uhci_hcd
ef80-ef9f : 0000:00:1d.3
  ef80-ef9f : uhci_hcd
efe0-efe7 : 0000:00:02.0
fc00-fc0f : 0000:00:1f.2
  fc00-fc0f : libata

7.5 pciinfo
00:00.0 Host bridge: Intel Corp. 82865G/PE/P DRAM Controller/Host-Hub Interface (rev 02)
	Subsystem: Asustek Computer, Inc.: Unknown device 80a5
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
	Latency: 0
	Region 0: Memory at fe800000 (32-bit, prefetchable) [size=4M]
	Capabilities: [e4] #09 [0106]

00:02.0 VGA compatible controller: Intel Corp. 82865G Integrated Graphics Device (rev 02) (prog-if 00 [VGA])
	Subsystem: Asustek Computer, Inc.: Unknown device 80a5
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f0000000 (32-bit, prefetchable) [size=128M]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=512K]
	Region 2: I/O ports at efe0 [size=8]
	Capabilities: [d0] Power Management version 1
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-

00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Asustek Computer, Inc. P4P800 Mainboard
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 10
	Region 4: I/O ports at ef00 [size=32]

00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Asustek Computer, Inc. P4P800 Mainboard
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 10
	Region 4: I/O ports at ef20 [size=32]

00:1d.2 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #3 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Asustek Computer, Inc. P4P800 Mainboard
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin C routed to IRQ 5
	Region 4: I/O ports at ef40 [size=32]

00:1d.3 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #4 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Asustek Computer, Inc. P4P800 Mainboard
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 10
	Region 4: I/O ports at ef80 [size=32]

00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02) (prog-if 20 [EHCI])
	Subsystem: Asustek Computer, Inc. P4P800 Mainboard
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin D routed to IRQ 5
	Region 0: Memory at fe77bc00 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] #0a [20a0]

00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI Bridge (rev c2) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: fe400000-fe5fffff
	Prefetchable memory behind bridge: ee200000-ee2fffff
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-

00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02)
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0

00:1f.2 IDE interface: Intel Corp. 82801EB (ICH5) Serial ATA 150 Storage Controller (rev 02) (prog-if 8a [Master SecP PriP])
	Subsystem: Asustek Computer, Inc.: Unknown device 80a6
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 5
	Region 0: I/O ports at <unassigned>
	Region 1: I/O ports at <unassigned>
	Region 2: I/O ports at <unassigned>
	Region 3: I/O ports at <unassigned>
	Region 4: I/O ports at fc00 [size=16]

00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
	Subsystem: Asustek Computer, Inc. P4P800 Mainboard
	Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Interrupt: pin B routed to IRQ 5
	Region 4: I/O ports at 0400 [size=32]

00:1f.5 Multimedia audio controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) AC'97 Audio Controller (rev 02)
	Subsystem: Asustek Computer, Inc.: Unknown device 80b0
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 5
	Region 0: I/O ports at e800 [size=256]
	Region 1: I/O ports at ee80 [size=64]
	Region 2: Memory at fe77b800 (32-bit, non-prefetchable) [size=512]
	Region 3: Memory at fe77b400 (32-bit, non-prefetchable) [size=256]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-

01:08.0 Ethernet controller: Intel Corp. 82562EZ 10/100 Ethernet Controller (rev 02)
	Subsystem: Asustek Computer, Inc.: Unknown device 80f8
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (2000ns min, 14000ns max)
	Interrupt: pin A routed to IRQ 5
	Region 0: Memory at fe5ff000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at df00 [size=64]
	Capabilities: [dc] Power Management version 2
		Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=2 PME-

01:09.0 Ethernet controller: D-Link System Inc RTL8139 Ethernet (rev 10)
	Subsystem: D-Link System Inc DFE-530TX+ 10/100 Ethernet Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (8000ns min, 16000ns max)
	Interrupt: pin A routed to IRQ 5
	Region 0: I/O ports at d000 [size=256]
	Region 1: Memory at fe5fec00 (32-bit, non-prefetchable) [size=256]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-

01:0a.0 Ethernet controller: Linksys: Unknown device 1032 (rev 12)
	Subsystem: Linksys: Unknown device 0015
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (5750ns min, 7750ns max), cache line size 04
	Interrupt: pin A routed to IRQ 5
	Region 0: Memory at fe5f8000 (32-bit, non-prefetchable) [size=16K]
	Region 1: I/O ports at de00 [size=256]
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] Vital Product Data

01:0b.0 SCSI storage controller: Adaptec ASC-29320A U320 (rev 10)
	Subsystem: Adaptec: Unknown device 0060
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (10000ns min, 6250ns max), cache line size 04
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at d800 [disabled] [size=256]
	Region 1: Memory at fe5fc000 (64-bit, non-prefetchable) [size=8K]
	Region 3: I/O ports at d400 [disabled] [size=256]
	Expansion ROM at fe500000 [disabled] [size=512K]
	Capabilities: [dc] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [a0] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [94] PCI-X non-bridge device.
		Command: DPERE- ERO+ RBC=0 OST=4
		Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-

7.6 scsi info
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: Maxtor 6Y120M0   Rev: YAR5
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 01 Lun: 00
  Vendor: ATA      Model: Maxtor 6Y120M0   Rev: YAR5
  Type:   Direct-Access                    ANSI SCSI revision: 05

------
        Lawrence Freil                      Email:lef@freil.com
        1768 Old Country Place              Phone:(770) 667-9274
        Woodstock, GA 30188

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2004-08-16 18:46 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2004-08-16 13:23 Serious Kernel slowdown with HIMEM (4Gig) in 2.6.7 Lawrence E. Freil
  -- strict thread matches above, loose matches on Subject: below --
2004-08-16 18:43 Mikael Pettersson
2004-08-16 17:40 Lawrence E. Freil
2004-08-14 16:43 Lawrence E. Freil
2004-08-15 15:04 ` Benjamin LaHaise
2004-08-14 15:43 Lawrence E. Freil
2004-08-14  2:11 Lawrence E. Freil
2004-08-14  4:18 ` Darren Williams
2004-08-14 11:42   ` Lawrence E. Freil
2004-08-14 15:09   ` Lawrence E. Freil
2004-08-14  8:51 ` William Lee Irwin III
2004-08-14 15:09 ` Martin J. Bligh

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).