All of lore.kernel.org
 help / color / mirror / Atom feed
* [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-14  3:11 James Cloos
  2010-02-14 16:40 ` James Cloos
                   ` (3 more replies)
  0 siblings, 4 replies; 21+ messages in thread
From: James Cloos @ 2010-02-14  3:11 UTC (permalink / raw)
  To: linux-kernel; +Cc: linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
usually on the order of 60% sys, 40% user.

I've tried this with each of ext4, jfs and btrfs filesystems.  All show
the same issue.

Using dd(1) to read from the block specials directly works as well and
as fast as it always has; only reading or writing to mounted filesystems
is affected.

Box is 32-bit x86, PentiumIII-M; drives are ide using libata.

If the btrfs fs is mounted, the slowdown is enought to trigger the
hung_task call trace (120s) on the btrfs-transac process.

But the regression is just as apparent when only jfs and ext4 are mounted.

The only filesystems I've found which avoid the regression are tmpfs and
devtmpfs.

I didn't have time to write up a report when I noticed this in rc7 but
had to boot back into rc6 for work.

Some of the commits since rc7 looked like they might have addressed this
regression, but it persists in rc8.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-14  3:11 [Regression] Filesystem I/O is CPU-bound in rc7 and rc8 James Cloos
@ 2010-02-14 16:40 ` James Cloos
  2010-02-14 23:12 ` Rafael J. Wysocki
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-14 16:40 UTC (permalink / raw)
  To: linux-kernel; +Cc: linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

I forgot to say:

# CONFIG_SMP is not set
CONFIG_TREE_PREEMPT_RCU=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_PREEMPT_TRACER is not set

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-14  3:11 [Regression] Filesystem I/O is CPU-bound in rc7 and rc8 James Cloos
  2010-02-14 16:40 ` James Cloos
@ 2010-02-14 23:12 ` Rafael J. Wysocki
  2010-02-15 15:45 ` Chris Mason
  2010-02-19 21:22   ` Mike Fedyk
  3 siblings, 0 replies; 21+ messages in thread
From: Rafael J. Wysocki @ 2010-02-14 23:12 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

On Sunday 14 February 2010, James Cloos wrote:
> Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
> usually on the order of 60% sys, 40% user.
> 
> I've tried this with each of ext4, jfs and btrfs filesystems.  All show
> the same issue.
> 
> Using dd(1) to read from the block specials directly works as well and
> as fast as it always has; only reading or writing to mounted filesystems
> is affected.
> 
> Box is 32-bit x86, PentiumIII-M; drives are ide using libata.
> 
> If the btrfs fs is mounted, the slowdown is enought to trigger the
> hung_task call trace (120s) on the btrfs-transac process.
> 
> But the regression is just as apparent when only jfs and ext4 are mounted.
> 
> The only filesystems I've found which avoid the regression are tmpfs and
> devtmpfs.
> 
> I didn't have time to write up a report when I noticed this in rc7 but
> had to boot back into rc6 for work.
> 
> Some of the commits since rc7 looked like they might have addressed this
> regression, but it persists in rc8.

I have created the bug entry at http://bugzilla.kernel.org/show_bug.cgi?id=15306
for your report.

Thanks,
Rafael

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-14  3:11 [Regression] Filesystem I/O is CPU-bound in rc7 and rc8 James Cloos
  2010-02-14 16:40 ` James Cloos
  2010-02-14 23:12 ` Rafael J. Wysocki
@ 2010-02-15 15:45 ` Chris Mason
  2010-02-15 19:27   ` James Cloos
  2010-02-17 20:04     ` James Cloos
  2010-02-19 21:22   ` Mike Fedyk
  3 siblings, 2 replies; 21+ messages in thread
From: Chris Mason @ 2010-02-15 15:45 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

On Sat, Feb 13, 2010 at 10:11:07PM -0500, James Cloos wrote:
> Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
> usually on the order of 60% sys, 40% user.
> 
> I've tried this with each of ext4, jfs and btrfs filesystems.  All show
> the same issue.
> 
> Using dd(1) to read from the block specials directly works as well and
> as fast as it always has; only reading or writing to mounted filesystems
> is affected.
> 
> Box is 32-bit x86, PentiumIII-M; drives are ide using libata.
> 
> If the btrfs fs is mounted, the slowdown is enought to trigger the
> hung_task call trace (120s) on the btrfs-transac process.
> 
> But the regression is just as apparent when only jfs and ext4 are mounted.
> 
> The only filesystems I've found which avoid the regression are tmpfs and
> devtmpfs.

Could you please hit sysrq-l a few times during the high CPU usage and
send us the results?  Also, please send along the full dd command line
you are using and how much ram your system has.

(Even better would be running perf record <your dd command line>, if
you're willing to setup perf)

Thanks!

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-15 15:45 ` Chris Mason
@ 2010-02-15 19:27   ` James Cloos
  2010-02-17 20:04     ` James Cloos
  1 sibling, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-15 19:27 UTC (permalink / raw)
  To: Chris Mason
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

[-- Attachment #1: Type: text/plain, Size: 3170 bytes --]

>>>>> "Chris" == Chris Mason <chris.mason@oracle.com> writes:

>> Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
>> usually on the order of 60% sys, 40% user.

Chris> (Even better would be running perf record <your dd command line>, if
Chris> you're willing to setup perf)

The box has 512MB (the chipset's max).  (I did mention p3m, therefore OLD! :)

:; mount
/dev/root on / type jfs (rw,noatime)
/dev/sda3 on /var type ext4 (rw,noatime,barrier=1,data=ordered)
/dev/sdb2 on /media/bay type btrfs (rw,noatime,noacl)

: I only vaguely remember installing perf....
:; perf version
perf version 0.0.1.PERF

:; perf record dd if=/dev/sda4 of=/dev/null bs=4M count=16
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 14.7279 s, 4.6 MB/s
[ perf record: Captured and wrote 0.803 MB perf1.data (~35093 samples) ]

:; perf record dd if=/dev/sdb2 of=/dev/null bs=4M count=16
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 7.31642 s, 9.2 MB/s
[ perf record: Captured and wrote 0.938 MB perf2.data (~40980 samples) ]

:; perf record dd if=/dev/sdc2 of=/dev/null bs=4M count=16
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 60.9204 s, 1.1 MB/s
[ perf record: Captured and wrote 3.309 MB perf3.data (~144590 samples) ]

(I then mounted:
/dev/sdc2 on /media/usb type ext4 (rw,relatime,barrier=1,data=ordered)
)

:; perf record dd if=/dev/zero of=ZERO bs=4M count=16 # sda4
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 10.0027 s, 6.7 MB/s
[ perf record: Captured and wrote 1.634 MB perf4.data (~71386 samples) ]

:; perf record dd if=/dev/zero of=/var/tmp/ZERO bs=4M count=16 # sda3
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 1.712 s, 39.2 MB/s
[ perf record: Captured and wrote 0.364 MB perf5.data (~15915 samples) ]

:; perf record dd if=/dev/zero of=/media/bay/ZERO bs=4M count=16
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 4.51859 s, 14.9 MB/s
[ perf record: Captured and wrote 0.611 MB perf6.data (~26675 samples) ]

:; perf record dd if=/dev/zero of=/media/usb/ZERO bs=4M count=16
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 8.49166 s, 7.9 MB/s
[ perf record: Captured and wrote 1.523 MB perf.data (~66556 samples) ]

:; for ij in /media/usb/ZERO /media/bay/ZERO /var/tmp/ZERO /root/ZERO;do perf record dd of=/dev/null if=${ij} bs=4M count=16; mv -v perf.data perf$[kl++].data; done
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 49.1844 s, 1.4 MB/s
[ perf record: Captured and wrote 0.569 MB perf.data (~24858 samples) ]
`perf.data' -> `perf8.data'
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 10.0016 s, 6.7 MB/s
[ perf record: Captured and wrote 0.948 MB perf.data (~41438 samples) ]
`perf.data' -> `perf9.data'
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 5.72855 s, 11.7 MB/s
[ perf record: Captured and wrote 0.580 MB perf.data (~25355 samples) ]
`perf.data' -> `perf10.data'
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 7.32789 s, 9.2 MB/s
[ perf record: Captured and wrote 0.899 MB perf.data (~39278 samples) ]
`perf.data' -> `perf11.data'


[-- Attachment #2: perf1.report --]
[-- Type: text/plain, Size: 29694 bytes --]

# Samples: 35051
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    15.06%       dd  [kernel]                 [k] __copy_to_user_ll
    13.82%       dd  [kernel]                 [k] __make_request
     7.13%       dd  [kernel]                 [k] get_page_from_freelist
     4.60%       dd  [kernel]                 [k] scsi_dispatch_cmd
     4.44%       dd  [kernel]                 [k] block_read_full_page
     3.46%       dd  [kernel]                 [k] generic_make_request
     3.41%       dd  [kernel]                 [k] scsi_request_fn
     2.50%       dd  [kernel]                 [k] kmem_cache_alloc
     2.01%       dd  [kernel]                 [k] blkdev_get_block
     1.66%       dd  [kernel]                 [k] get_request
     1.63%       dd  [kernel]                 [k] generic_file_aio_read
     1.63%       dd  [kernel]                 [k] __alloc_pages_nodemask
     1.60%       dd  [kernel]                 [k] add_to_page_cache_locked
     1.41%       dd  [kernel]                 [k] __do_page_cache_readahead
     1.31%       dd  [kernel]                 [k] file_read_actor
     1.17%       dd  [kernel]                 [k] submit_bh
     1.03%       dd  [kernel]                 [k] blk_end_bidi_request
     0.97%       dd  [kernel]                 [k] bio_alloc_bioset
     0.88%       dd  [kernel]                 [k] __rcu_read_unlock
     0.85%       dd  [kernel]                 [k] T.1339
     0.84%     perf  [kernel]                 [k] generic_file_buffered_write
     0.84%       dd  [kernel]                 [k] zone_watermark_ok
     0.83%       dd  [kernel]                 [k] find_get_page
     0.81%       dd  [kernel]                 [k] put_page
     0.75%       dd  [kernel]                 [k] ata_sff_interrupt
     0.72%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.72%       dd  [kernel]                 [k] alloc_page_buffers
     0.63%       dd  [kernel]                 [k] submit_bio
     0.63%       dd  [kernel]                 [k] schedule
     0.63%       dd  [kernel]                 [k] blk_queue_bounce
     0.57%       dd  [kernel]                 [k] create_empty_buffers
     0.54%     perf  [kernel]                 [k] __generic_file_aio_write
     0.53%       dd  [kernel]                 [k] __lru_cache_add
     0.53%       dd  [kernel]                 [k] alloc_buffer_head
     0.53%       dd  [kernel]                 [k] next_zones_zonelist
     0.49%       dd  [kernel]                 [k] mempool_alloc
     0.48%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.39%       dd  [kernel]                 [k] bio_init
     0.39%       dd  [kernel]                 [k] handle_IRQ_event
     0.35%       dd  [kernel]                 [k] radix_tree_preload
     0.34%       dd  [kernel]                 [k] radix_tree_lookup
     0.31%     perf  [kernel]                 [k] generic_file_aio_write
     0.31%     perf  [kernel]                 [k] nobh_write_end
     0.30%       dd  [kernel]                 [k] __rcu_read_lock
     0.30%       dd  [kernel]                 [k] __do_softirq
     0.29%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.28%     perf  [kernel]                 [k] iov_iter_advance
     0.28%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     0.28%       dd  [kernel]                 [k] cfq_set_request
     0.26%     perf  [kernel]                 [k] __jfs_getxattr
     0.26%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.25%     perf  [kernel]                 [k] getnstimeofday
     0.25%     perf  [kernel]                 [k] nobh_write_begin
     0.25%       dd  [kernel]                 [k] do_page_fault
     0.24%     perf  [kernel]                 [k] sysenter_past_esp
     0.24%     perf  [kernel]                 [k] do_sync_write
     0.23%       dd  [kernel]                 [k] rcu_sched_qs
     0.23%       dd  [kernel]                 [k] bio_put
     0.22%       dd  [kernel]                 [k] unmap_vmas
     0.22%       dd  [kernel]                 [k] _cond_resched
     0.22%     perf  [kernel]                 [k] vfs_write
     0.22%       dd  [kernel]                 [k] mempool_alloc_slab
     0.21%       dd  [kernel]                 [k] mark_page_accessed
     0.20%       dd  [kernel]                 [k] req_bio_endio
     0.19%     perf  [kernel]                 [k] sys_write
     0.19%       dd  [kernel]                 [k] blk_rq_init
     0.19%     perf  [kernel]                 [k] rw_verify_area
     0.19%     perf  [kernel]                 [k] acpi_pm_read
     0.19%     perf  [kernel]                 [k] find_get_page
     0.19%       dd  [kernel]                 [k] blk_update_request
     0.17%     perf  [kernel]                 [k] generic_segment_checks
     0.17%       dd  [kernel]                 [k] preempt_schedule
     0.17%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.17%     perf  [kernel]                 [k] jfs_write_begin
     0.17%       dd  [kernel]                 [k] free_hot_cold_page
     0.17%       dd  [kernel]                 [k] bio_alloc
     0.16%     perf  [kernel]                 [k] copy_to_user
     0.15%       dd  [kernel]                 [k] blkdev_readpage
     0.15%       dd  [kernel]                 [k] set_bh_page
     0.15%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.14%     perf  [kernel]                 [k] T.1339
     0.14%     perf  [kernel]                 [k] sys_gettimeofday
     0.14%     perf  [kernel]                 [k] ea_get
     0.13%       dd  [kernel]                 [k] handle_mm_fault
     0.12%       dd  [kernel]                 [k] get_io_context
     0.11%     perf  [kernel]                 [k] fget_light
     0.11%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.11%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.11%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.11%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.10%     perf  [kernel]                 [k] __mark_inode_dirty
     0.10%     perf  [kernel]                 [k] fsnotify
     0.10%     perf  [kernel]                 [k] jfs_getxattr
     0.10%     perf  [kernel]                 [k] vsnprintf
     0.10%       dd  [kernel]                 [k] end_buffer_async_read
     0.10%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.09%     perf  [kernel]                 [k] put_page
     0.09%     perf  [kernel]                 [k] kfree
     0.09%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.09%     perf  [vdso]                   [.] 0x000000ffffe424
     0.09%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.09%     perf  [kernel]                 [k] number
     0.09%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.09%       dd  [kernel]                 [k] init_request_from_bio
     0.09%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.08%     perf  [kernel]                 [k] schedule
     0.08%       dd  [kernel]                 [k] kmem_cache_free
     0.08%     perf  [kernel]                 [k] should_remove_suid
     0.08%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.07%       dd  [kernel]                 [k] do_mpage_readpage
     0.07%     perf  [kernel]                 [k] unlock_page
     0.07%     perf  [kernel]                 [k] find_lock_page
     0.07%     perf  [kernel]                 [k] __rcu_read_unlock
     0.07%     perf  [kernel]                 [k] generic_write_sync
     0.07%     perf  [kernel]                 [k] dbAlloc
     0.07%     perf  [kernel]                 [k] extAlloc
     0.07%     perf  [kernel]                 [k] file_update_time
     0.07%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.07%       dd  [kernel]                 [k] current_io_context
     0.06%     perf  [kernel]                 [k] do_gettimeofday
     0.06%     perf  [kernel]                 [k] xtSearch
     0.06%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.06%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.06%       dd  [kernel]                 [k] bio_endio
     0.06%     perf  [kernel]                 [k] file_remove_suid
     0.06%     perf  [kernel]                 [k] mark_page_accessed
     0.06%     perf  [kernel]                 [k] __set_page_dirty
     0.06%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.06%     perf  [kernel]                 [k] format_decode
     0.06%       dd  [kernel]                 [k] release_pages
     0.06%       dd  [kernel]                 [k] cfq_cic_lookup
     0.06%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.05%       dd  [kernel]                 [k] kref_put
     0.05%     perf  [kernel]                 [k] ea_release
     0.05%       dd  [kernel]                 [k] __do_fault
     0.05%       dd  [kernel]                 [k] blk_recount_segments
     0.05%     perf  [kernel]                 [k] jfs_dirty_inode
     0.05%     perf  [kernel]                 [k] xtLookup
     0.05%     perf  [kernel]                 [k] dbAllocBits
     0.05%     perf  [kernel]                 [k] mutex_lock
     0.05%     perf  [kernel]                 [k] __wake_up_bit
     0.05%     perf  [kernel]                 [k] current_kernel_time
     0.04%     perf  [kernel]                 [k] current_fs_time
     0.04%     perf  [kernel]                 [k] copy_page_range
     0.04%     perf  [kernel]                 [k] strncmp
     0.04%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.04%     perf  [kernel]                 [k] dbAllocNext
     0.04%       dd  [kernel]                 [k] vm_normal_page
     0.04%       dd  [kernel]                 [k] get_request_wait
     0.04%     perf  [kernel]                 [k] get_page_from_freelist
     0.04%     perf  [kernel]                 [k] __fsnotify_parent
     0.04%     perf  [kernel]                 [k] strlen
     0.04%     perf  [kernel]                 [k] _cond_resched
     0.04%       dd  [kernel]                 [k] __wake_up
     0.04%       dd  [kernel]                 [k] ondemand_readahead
     0.04%       dd  [kernel]                 [k] bio_phys_segments
     0.04%       dd  [kernel]                 [k] bio_fs_destructor
     0.04%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.03%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.03%     perf  [kernel]                 [k] scsi_request_fn
     0.03%       dd  [kernel]                 [k] end_bio_bh_io_sync
     0.03%     perf  [kernel]                 [k] __wake_up
     0.03%     perf  [kernel]                 [k] timespec_trunc
     0.03%     perf  [kernel]                 [k] set_page_dirty
     0.03%     perf  [kernel]                 [k] kmem_cache_alloc
     0.03%     perf  [kernel]                 [k] extHint
     0.03%     perf  [kernel]                 [k] cap_file_permission
     0.03%       dd  [kernel]                 [k] __queue_work
     0.03%       dd  [kernel]                 [k] down_read_trylock
     0.03%       dd  [kernel]                 [k] filemap_fault
     0.03%       dd  [kernel]                 [k] page_cache_async_readahead
     0.03%       dd  [kernel]                 [k] xtSearch
     0.03%       dd  [kernel]                 [k] sha_transform
     0.03%     perf  [kernel]                 [k] down_read
     0.03%     perf  [kernel]                 [k] do_page_fault
     0.03%       dd  [kernel]                 [k] unlock_page
     0.03%       dd  [kernel]                 [k] free_hot_page
     0.03%       dd  [kernel]                 [k] find_vma
     0.03%       dd  [kernel]                 [k] generic_unplug_device
     0.03%       dd  [kernel]                 [k] blk_done_softirq
     0.03%       dd  [kernel]                 [k] scsi_finish_command
     0.03%       dd  [kernel]                 [k] scsi_put_command
     0.03%     perf  [kernel]                 [k] mutex_unlock
     0.03%       dd  [kernel]                 [k] elv_set_request
     0.02%     perf  [kernel]                 [k] sysenter_do_call
     0.02%     perf  [kernel]                 [k] __rcu_read_lock
     0.02%     perf  [kernel]                 [k] xtExtend
     0.02%       dd  [kernel]                 [k] test_clear_page_writeback
     0.02%       dd  [kernel]                 [k] blk_run_queue
     0.02%       dd  [kernel]                 [k] __scsi_put_command
     0.02%       dd  [kernel]                 [k] scsi_decide_disposition
     0.02%       dd  [kernel]                 [k] scsi_run_queue
     0.02%     perf  [kernel]                 [k] up_read
     0.02%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.02%     perf  [kernel]                 [k] __do_fault
     0.02%     perf  [kernel]                 [k] jfs_get_block
     0.02%     perf  [kernel]                 [k] security_file_permission
     0.02%     perf  [kernel]                 [k] blk_end_bidi_request
     0.02%       dd  [kernel]                 [k] __wake_up_bit
     0.02%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.02%     perf  [kernel]                 [k] handle_IRQ_event
     0.02%     perf  [kernel]                 [k] handle_mm_fault
     0.02%     perf  [kernel]                 [k] kmem_cache_free
     0.02%     perf  [kernel]                 [k] mnt_clone_write
     0.02%     perf  [kernel]                 [k] seq_printf
     0.02%     perf  [kernel]                 [k] spin_unlock
     0.02%     perf  [kernel]                 [k] txLock
     0.02%     perf  [kernel]                 [k] strchr
     0.02%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.02%       dd  [kernel]                 [k] up_read
     0.02%       dd  [kernel]                 [k] rcu_process_gp_end
     0.02%       dd  [kernel]                 [k] mempool_free
     0.02%       dd  [kernel]                 [k] ra_submit
     0.02%       dd  [kernel]                 [k] page_remove_rmap
     0.02%       dd  [kernel]                 [k] anon_vma_prepare
     0.02%       dd  [kernel]                 [k] __bio_add_page
     0.02%       dd  [kernel]                 [k] bio_free
     0.02%       dd  [kernel]                 [k] kobject_put
     0.02%       dd  [kernel]                 [k] scsi_device_unbusy
     0.02%       dd  [kernel]                 [k] scsi_io_completion
     0.02%       dd  [kernel]                 [k] sd_done
     0.01%     perf  [kernel]                 [k] rcu_sched_qs
     0.01%     perf  [kernel]                 [k] do_sys_poll
     0.01%     perf  [kernel]                 [k] free_buffer_head
     0.01%     perf  [kernel]                 [k] __find_get_block_slow
     0.01%     perf  [kernel]                 [k] dbSplit
     0.01%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.01%       dd  [kernel]                 [k] run_timer_softirq
     0.01%       dd  [kernel]                 [k] page_evictable
     0.01%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.01%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.01%       dd  [kernel]                 [k] end_buffer_async_write
     0.01%       dd  [kernel]                 [k] bio_add_page
     0.01%       dd  [kernel]                 [k] blk_unplug
     0.01%       dd  [kernel]                 [k] blk_update_bidi_request
     0.01%       dd  [kernel]                 [k] put_device
     0.01%       dd  [kernel]                 [k] __scsi_release_buffers
     0.01%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.01%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.01%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.01%     perf  [kernel]                 [k] mnt_drop_write
     0.01%     perf  [kernel]                 [k] ata_sff_interrupt
     0.01%       dd  [kernel]                 [k] lru_cache_add_lru
     0.01%       dd  [kernel]                 [k] mmap_region
     0.01%       dd  [kernel]                 [k] mpage_readpages
     0.01%       dd  [kernel]                 [k] __sg_free_table
     0.01%       dd  [kernel]                 [k] add_timer_randomness
     0.01%       dd  [kernel]                 [k] scsi_softirq_done
     0.01%       dd                 b713846f  [.] 0x000000b713846f
     0.01%     perf  [kernel]                 [k] up_write
     0.01%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.01%     perf  [kernel]                 [k] read_cache_page
     0.01%     perf  [kernel]                 [k] vm_normal_page
     0.01%     perf  [kernel]                 [k] do_wp_page
     0.01%     perf  [kernel]                 [k] anon_vma_link
     0.01%     perf  [kernel]                 [k] prepend
     0.01%     perf  [kernel]                 [k] d_path
     0.01%     perf  [kernel]                 [k] mangle_path
     0.01%     perf  [kernel]                 [k] show_map_vma
     0.01%     perf  [kernel]                 [k] dbAllocDmap
     0.01%     perf  [kernel]                 [k] __get_metapage
     0.01%       dd  [kernel]                 [k] try_to_wake_up
     0.01%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.01%       dd  [kernel]                 [k] __lock_page_killable
     0.01%       dd  [kernel]                 [k] mempool_free_slab
     0.01%       dd  [kernel]                 [k] shrink_page_list
     0.01%       dd  [kernel]                 [k] unlink_file_vma
     0.01%       dd  [kernel]                 [k] page_referenced
     0.01%       dd  [kernel]                 [k] jfs_get_block
     0.01%       dd  [kernel]                 [k] dtSearch
     0.01%       dd  [kernel]                 [k] metapage_readpage
     0.01%       dd  [kernel]                 [k] cap_file_permission
     0.01%       dd  [kernel]                 [k] blk_backing_dev_unplug
     0.01%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.01%       dd  [kernel]                 [k] kobject_get
     0.01%       dd  [kernel]                 [k] strnlen_user
     0.01%       dd  [kernel]                 [k] copy_to_user
     0.01%       dd  [kernel]                 [k] credit_entropy_bits
     0.01%       dd  [kernel]                 [k] scsi_log_send
     0.01%       dd  [kernel]                 [k] scsi_pool_free_command
     0.01%       dd  [kernel]                 [k] scsi_sg_free
     0.01%       dd  [kernel]                 [k] preempt_schedule_irq
     0.01%     perf  [kernel]                 [k] pgd_alloc
     0.01%     perf  [kernel]                 [k] copy_process
     0.01%     perf  [kernel]                 [k] __do_softirq
     0.01%     perf  [kernel]                 [k] ktime_get_ts
     0.01%     perf  [kernel]                 [k] perf_poll
     0.01%     perf  [kernel]                 [k] perf_mmap_fault
     0.01%     perf  [kernel]                 [k] do_read_cache_page
     0.01%     perf  [kernel]                 [k] read_cache_page_async
     0.01%     perf  [kernel]                 [k] activate_page
     0.01%     perf  [kernel]                 [k] vma_prio_tree_add
     0.01%     perf  [kernel]                 [k] next_zones_zonelist
     0.01%     perf  [kernel]                 [k] find_vma
     0.01%     perf  [kernel]                 [k] __d_path
     0.01%     perf  [kernel]                 [k] alloc_page_buffers
     0.01%     perf  [kernel]                 [k] bio_endio
     0.01%     perf  [kernel]                 [k] dbAdjTree
     0.01%     perf  [kernel]                 [k] txBeginAnon
     0.01%     perf  [kernel]                 [k] blk_update_request
     0.01%     perf  [kernel]                 [k] blk_done_softirq
     0.01%     perf  [kernel]                 [k] cfq_idle_slice_timer
     0.01%     perf  [kernel]                 [k] radix_tree_preload
     0.01%     perf  [kernel]                 [k] string
     0.01%     perf  [kernel]                 [k] preempt_schedule
     0.01%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.01%     perf  [kernel]                 [k] down_write
     0.01%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.01%       dd  [kernel]                 [k] sysenter_past_esp
     0.01%       dd  [kernel]                 [k] queue_work
     0.01%       dd  [kernel]                 [k] prepare_to_wait_exclusive
     0.01%       dd  [kernel]                 [k] ktime_get_ts
     0.01%       dd  [kernel]                 [k] __call_rcu
     0.01%       dd  [kernel]                 [k] rcu_process_callbacks
     0.01%       dd  [kernel]                 [k] shrink_inactive_list
     0.01%       dd  [kernel]                 [k] remove_vma
     0.01%       dd  [kernel]                 [k] do_sync_read
     0.01%       dd  [kernel]                 [k] rw_verify_area
     0.01%       dd  [kernel]                 [k] file_free_rcu
     0.01%       dd  [kernel]                 [k] __follow_mount
     0.01%       dd  [kernel]                 [k] do_lookup
     0.01%       dd  [kernel]                 [k] link_path_walk
     0.01%       dd  [kernel]                 [k] load_elf_binary
     0.01%       dd  [kernel]                 [k] prio_tree_next
     0.01%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_setup_hash
     0.00%     perf  [kernel]                 [k] task_nice
     0.00%     perf  [kernel]                 [k] wake_up_new_task
     0.00%     perf  [kernel]                 [k] dup_mm
     0.00%     perf  [kernel]                 [k] free_uid
     0.00%     perf  [kernel]                 [k] find_task_by_pid_ns
     0.00%     perf  [kernel]                 [k] remove_wait_queue
     0.00%     perf  [kernel]                 [k] __rcu_process_callbacks
     0.00%     perf  [kernel]                 [k] __lru_cache_add
     0.00%     perf  [kernel]                 [k] arch_get_unmapped_area_topdown
     0.00%     perf  [kernel]                 [k] inode_add_bytes
     0.00%     perf  [kernel]                 [k] path_init
     0.00%     perf  [kernel]                 [k] do_lookup
     0.00%     perf  [kernel]                 [k] poll_schedule_timeout
     0.00%     perf  [kernel]                 [k] __d_free
     0.00%     perf  [kernel]                 [k] mnt_want_write_file
     0.00%     perf  [kernel]                 [k] set_bh_page
     0.00%     perf  [kernel]                 [k] alloc_buffer_head
     0.00%     perf  [kernel]                 [k] end_buffer_async_read
     0.00%     perf  [kernel]                 [k] end_buffer_async_write
     0.00%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.00%     perf  [kernel]                 [k] m_start
     0.00%     perf  [kernel]                 [k] proc_pid_status
     0.00%     perf  [kernel]                 [k] release_metapage
     0.00%     perf  [kernel]                 [k] req_bio_endio
     0.00%     perf  [kernel]                 [k] __blk_recalc_rq_segments
     0.00%     perf  [kernel]                 [k] _atomic_dec_and_lock
     0.00%     perf  [kernel]                 [k] rb_insert_color
     0.00%     perf  [kernel]                 [k] put_dec_trunc
     0.00%     perf  [kernel]                 [k] put_dec
     0.00%     perf  [kernel]                 [k] strncpy
     0.00%     perf  [kernel]                 [k] credit_entropy_bits
     0.00%     perf  [kernel]                 [k] scsi_decide_disposition
     0.00%     perf  [kernel]                 [k] scsi_softirq_done
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_getline_internal
     0.00%     perf  /lib/ld-2.11.so          [.] check_match.8476
     0.00%     perf  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.00%       dd  [kernel]                 [k] destroy_context
     0.00%       dd  [kernel]                 [k] ___pte_free_tlb
     0.00%       dd  [kernel]                 [k] set_normalized_timespec
     0.00%       dd  [kernel]                 [k] prepare_exec_creds
     0.00%       dd  [kernel]                 [k] do_futex
     0.00%       dd  [kernel]                 [k] rcu_bh_qs
     0.00%       dd  [kernel]                 [k] check_for_new_grace_period
     0.00%       dd  [kernel]                 [k] generic_segment_checks
     0.00%       dd  [kernel]                 [k] sync_page
     0.00%       dd  [kernel]                 [k] do_read_cache_page
     0.00%       dd  [kernel]                 [k] read_cache_page_async
     0.00%       dd  [kernel]                 [k] wakeup_kswapd
     0.00%       dd  [kernel]                 [k] vma_prio_tree_remove
     0.00%       dd  [kernel]                 [k] free_pgd_range
     0.00%       dd  [kernel]                 [k] follow_page
     0.00%       dd  [kernel]                 [k] do_wp_page
     0.00%       dd  [kernel]                 [k] find_mergeable_anon_vma
     0.00%       dd  [kernel]                 [k] insert_vm_struct
     0.00%       dd  [kernel]                 [k] vma_adjust
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] sys_mmap_pgoff
     0.00%       dd  [kernel]                 [k] page_add_file_rmap
     0.00%       dd  [kernel]                 [k] kfree
     0.00%       dd  [kernel]                 [k] __kmalloc
     0.00%       dd  [kernel]                 [k] do_sys_open
     0.00%       dd  [kernel]                 [k] vfs_read
     0.00%       dd  [kernel]                 [k] sys_read
     0.00%       dd  [kernel]                 [k] fget
     0.00%       dd  [kernel]                 [k] __fput
     0.00%       dd  [kernel]                 [k] copy_strings
     0.00%       dd  [kernel]                 [k] kernel_read
     0.00%       dd  [kernel]                 [k] check_unsafe_exec
     0.00%       dd  [kernel]                 [k] getname
     0.00%       dd  [kernel]                 [k] do_follow_link
     0.00%       dd  [kernel]                 [k] do_filp_open
     0.00%       dd  [kernel]                 [k] prepend
     0.00%       dd  [kernel]                 [k] d_rehash
     0.00%       dd  [kernel]                 [k] __d_lookup
     0.00%       dd  [kernel]                 [k] dput
     0.00%       dd  [kernel]                 [k] alloc_fd
     0.00%       dd  [kernel]                 [k] mntput_no_expire
     0.00%       dd  [kernel]                 [k] block_sync_page
     0.00%       dd  [kernel]                 [k] bd_acquire
     0.00%       dd  [kernel]                 [k] mpage_end_io_read
     0.00%       dd  [kernel]                 [k] locks_remove_flock
     0.00%       dd  [kernel]                 [k] __get_metapage
     0.00%       dd  [kernel]                 [k] security_inode_permission
     0.00%       dd  [kernel]                 [k] security_vm_enough_memory
     0.00%       dd  [kernel]                 [k] blk_end_request
     0.00%       dd  [kernel]                 [k] prio_tree_replace
     0.00%       dd  [kernel]                 [k] __rb_rotate_left
     0.00%       dd  [kernel]                 [k] rb_insert_color
     0.00%       dd  [kernel]                 [k] strrchr
     0.00%       dd  [kernel]                 [k] __get_user_4
     0.00%       dd  [kernel]                 [k] strlen
     0.00%       dd  [kernel]                 [k] _copy_from_user
     0.00%       dd  [kernel]                 [k] half_md4_transform
     0.00%       dd  [kernel]                 [k] ioread8
     0.00%       dd  [kernel]                 [k] write_null
     0.00%       dd  [kernel]                 [k] account
     0.00%       dd  [kernel]                 [k] get_device
     0.00%       dd  [kernel]                 [k] scsi_next_command
     0.00%       dd  [kernel]                 [k] e100_exec_cmd
     0.00%       dd  [kernel]                 [k] acpi_pm_read
     0.00%       dd  [kernel]                 [k] __wait_on_bit_lock
     0.00%       dd  [kernel]                 [k] down_read
     0.00%       dd  [kernel]                 [k] syscall_call
     0.00%       dd  [kernel]                 [k] error_code
     0.00%       dd  /lib/libc-2.11.so        [.] 0x00000000021f5e
     0.00%       dd  /lib/libc-2.11.so        [.] _nl_load_locale_from_archive
     0.00%       dd  /lib/libc-2.11.so        [.] _IO_vfprintf_internal
     0.00%       dd  /lib/libc-2.11.so        [.] buffered_vfprintf
     0.00%       dd  /lib/libc-2.11.so        [.] _int_malloc
     0.00%       dd  /lib/libc-2.11.so        [.] index
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_strstr
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___libc_write
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___register_atfork
     0.00%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_relocate_object
     0.00%       dd  /lib/ld-2.11.so          [.] call_init
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_check_map_versions
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_discover_osversion
     0.00%       dd  /lib/ld-2.11.so          [.] __mmap
     0.00%       dd  /lib/ld-2.11.so          [.] strcmp
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #3: perf2.report --]
[-- Type: text/plain, Size: 31320 bytes --]

# Samples: 40938
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    20.58%       dd  [kernel]                 [k] __copy_to_user_ll
    15.62%       dd  [kernel]                 [k] __make_request
     7.73%       dd  [kernel]                 [k] get_page_from_freelist
     4.41%       dd  [kernel]                 [k] block_read_full_page
     3.60%       dd  [kernel]                 [k] generic_make_request
     2.60%       dd  [kernel]                 [k] kmem_cache_alloc
     2.00%       dd  [kernel]                 [k] generic_file_aio_read
     1.96%       dd  [kernel]                 [k] __alloc_pages_nodemask
     1.87%       dd  [kernel]                 [k] add_to_page_cache_locked
     1.82%       dd  [kernel]                 [k] scsi_dispatch_cmd
     1.77%       dd  [kernel]                 [k] blkdev_get_block
     1.71%       dd  [kernel]                 [k] __do_page_cache_readahead
     1.67%       dd  [kernel]                 [k] file_read_actor
     1.27%       dd  [kernel]                 [k] scsi_request_fn
     1.21%       dd  [kernel]                 [k] submit_bh
     1.01%       dd  [kernel]                 [k] put_page
     0.99%       dd  [kernel]                 [k] find_get_page
     0.98%       dd  [kernel]                 [k] T.1339
     0.95%       dd  [kernel]                 [k] zone_watermark_ok
     0.92%       dd  [kernel]                 [k] bio_alloc_bioset
     0.89%       dd  [kernel]                 [k] __rcu_read_unlock
     0.86%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.83%       dd  [kernel]                 [k] alloc_page_buffers
     0.71%       dd  [kernel]                 [k] __lru_cache_add
     0.68%       dd  [kernel]                 [k] blk_queue_bounce
     0.64%       dd  [kernel]                 [k] next_zones_zonelist
     0.63%       dd  [kernel]                 [k] submit_bio
     0.61%       dd  [kernel]                 [k] create_empty_buffers
     0.59%       dd  [kernel]                 [k] alloc_buffer_head
     0.53%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.52%       dd  [kernel]                 [k] mempool_alloc
     0.44%       dd  [kernel]                 [k] blk_end_bidi_request
     0.43%     perf  [kernel]                 [k] generic_file_buffered_write
     0.43%       dd  [kernel]                 [k] bio_init
     0.39%       dd  [kernel]                 [k] radix_tree_preload
     0.38%       dd  [kernel]                 [k] __rcu_read_lock
     0.36%       dd  [kernel]                 [k] schedule
     0.35%       dd  [kernel]                 [k] radix_tree_lookup
     0.34%       dd  [kernel]                 [k] get_request
     0.32%       dd  [kernel]                 [k] bio_put
     0.32%     perf  [kernel]                 [k] __generic_file_aio_write
     0.31%       dd  [kernel]                 [k] ata_sff_interrupt
     0.29%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.28%     perf  [kernel]                 [k] T.1339
     0.26%     perf  [kernel]                 [k] nobh_write_begin
     0.25%       dd  [kernel]                 [k] handle_mm_fault
     0.25%       dd  [kernel]                 [k] do_page_fault
     0.21%       dd  [kernel]                 [k] bio_alloc
     0.20%       dd  [kernel]                 [k] unmap_vmas
     0.20%       dd  [kernel]                 [k] mark_page_accessed
     0.19%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.19%       dd  [kernel]                 [k] req_bio_endio
     0.19%     perf  [kernel]                 [k] nobh_write_end
     0.19%       dd  [kernel]                 [k] mempool_alloc_slab
     0.18%     perf  [kernel]                 [k] vfs_write
     0.18%       dd  [kernel]                 [k] _cond_resched
     0.17%       dd  [kernel]                 [k] end_buffer_async_read
     0.17%       dd  [kernel]                 [k] blkdev_readpage
     0.17%     perf  [kernel]                 [k] do_sync_write
     0.17%       dd  [kernel]                 [k] handle_IRQ_event
     0.16%       dd  [kernel]                 [k] blk_update_request
     0.16%       dd  [kernel]                 [k] free_hot_cold_page
     0.15%     perf  [kernel]                 [k] extAlloc
     0.15%       dd  [kernel]                 [k] __do_softirq
     0.15%     perf  [kernel]                 [k] generic_segment_checks
     0.14%       dd  [kernel]                 [k] set_bh_page
     0.14%     perf  [kernel]                 [k] find_get_page
     0.14%     perf  [kernel]                 [k] generic_file_aio_write
     0.14%     perf  [kernel]                 [k] jfs_write_begin
     0.14%     perf  [kernel]                 [k] sysenter_past_esp
     0.14%     perf  [kernel]                 [k] __jfs_getxattr
     0.13%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.13%     perf  [kernel]                 [k] iov_iter_advance
     0.13%     perf  [kernel]                 [k] getnstimeofday
     0.13%       dd  [kernel]                 [k] __wake_up
     0.13%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.12%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     0.12%     perf  [kernel]                 [k] rw_verify_area
     0.12%     perf  [kernel]                 [k] number
     0.11%     perf  [kernel]                 [k] xtSearch
     0.11%     perf  [kernel]                 [k] ea_get
     0.11%       dd  [kernel]                 [k] cfq_set_request
     0.11%     perf  [kernel]                 [k] get_page_from_freelist
     0.11%     perf  [kernel]                 [k] dbAlloc
     0.10%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.10%     perf  [kernel]                 [k] acpi_pm_read
     0.09%     perf  [kernel]                 [k] __set_page_dirty
     0.09%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.09%     perf  [kernel]                 [k] sys_write
     0.09%     perf  [kernel]                 [k] file_update_time
     0.09%     perf  [kernel]                 [k] __mark_inode_dirty
     0.09%       dd  [kernel]                 [k] kmem_cache_free
     0.08%     perf  [kernel]                 [k] vsnprintf
     0.08%     perf  [kernel]                 [k] __rcu_read_unlock
     0.08%       dd  [kernel]                 [k] rcu_sched_qs
     0.08%       dd  [kernel]                 [k] ondemand_readahead
     0.08%       dd  [kernel]                 [k] find_vma
     0.08%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.07%     perf  [kernel]                 [k] sys_gettimeofday
     0.07%     perf  [kernel]                 [k] xtLookup
     0.07%       dd  [kernel]                 [k] __queue_work
     0.07%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.07%     perf  [kernel]                 [k] generic_write_sync
     0.07%     perf  [kernel]                 [k] copy_to_user
     0.07%       dd  [kernel]                 [k] kref_put
     0.07%       dd  [kernel]                 [k] preempt_schedule
     0.07%       dd  [kernel]                 [k] get_io_context
     0.07%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.06%       dd  [kernel]                 [k] end_bio_bh_io_sync
     0.06%     perf  [kernel]                 [k] fget_light
     0.06%     perf  [kernel]                 [k] format_decode
     0.06%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.06%     perf  [vdso]                   [.] 0x000000ffffe424
     0.06%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.06%     perf  [kernel]                 [k] fsnotify
     0.06%       dd  [kernel]                 [k] __wake_up_bit
     0.05%     perf  [kernel]                 [k] copy_page_range
     0.05%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.05%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.05%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.05%       dd  [kernel]                 [k] bio_endio
     0.05%       dd  [kernel]                 [k] generic_unplug_device
     0.05%     perf  [kernel]                 [k] unlock_page
     0.05%     perf  [kernel]                 [k] jfs_getxattr
     0.05%       dd  [kernel]                 [k] page_cache_async_readahead
     0.05%     perf  [kernel]                 [k] file_remove_suid
     0.05%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.05%       dd  [kernel]                 [k] __do_fault
     0.05%       dd  [kernel]                 [k] bio_fs_destructor
     0.05%       dd  [kernel]                 [k] blk_rq_init
     0.05%       dd  [kernel]                 [k] blk_recount_segments
     0.05%     perf  [kernel]                 [k] do_gettimeofday
     0.05%     perf  [kernel]                 [k] dbAllocBits
     0.05%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.05%       dd  [kernel]                 [k] filemap_fault
     0.05%       dd  [kernel]                 [k] release_pages
     0.04%     perf  [kernel]                 [k] __wake_up
     0.04%     perf  [kernel]                 [k] __fsnotify_parent
     0.04%       dd  [kernel]                 [k] init_request_from_bio
     0.04%     perf  [kernel]                 [k] __do_fault
     0.04%     perf  [kernel]                 [k] dbAllocNext
     0.04%     perf  [kernel]                 [k] do_page_fault
     0.04%       dd  [kernel]                 [k] unlock_page
     0.04%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.04%     perf  [kernel]                 [k] __rcu_read_lock
     0.04%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.04%     perf  [kernel]                 [k] mark_page_accessed
     0.04%     perf  [kernel]                 [k] put_page
     0.04%     perf  [kernel]                 [k] find_lock_page
     0.04%     perf  [kernel]                 [k] kfree
     0.04%     perf  [kernel]                 [k] xtExtend
     0.04%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.04%     perf  [kernel]                 [k] schedule
     0.04%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.03%     perf  [kernel]                 [k] timespec_trunc
     0.03%     perf  [kernel]                 [k] __wake_up_bit
     0.03%     perf  [kernel]                 [k] kmem_cache_alloc
     0.03%     perf  [kernel]                 [k] jfs_dirty_inode
     0.03%     perf  [kernel]                 [k] jfs_get_block
     0.03%       dd  [kernel]                 [k] vm_normal_page
     0.03%     perf  [kernel]                 [k] should_remove_suid
     0.03%     perf  [kernel]                 [k] set_page_dirty
     0.03%     perf  [kernel]                 [k] strncmp
     0.03%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.03%       dd  [kernel]                 [k] page_evictable
     0.03%       dd  [kernel]                 [k] bio_free
     0.03%       dd  [kernel]                 [k] sha_transform
     0.03%     perf  [kernel]                 [k] ea_release
     0.03%       dd  [kernel]                 [k] ra_submit
     0.03%       dd  [kernel]                 [k] strnlen_user
     0.03%     perf  [kernel]                 [k] __get_metapage
     0.03%     perf  [kernel]                 [k] mutex_unlock
     0.03%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.03%       dd  [kernel]                 [k] down_read_trylock
     0.03%       dd  [kernel]                 [k] get_request_wait
     0.03%       dd  [kernel]                 [k] kobject_put
     0.03%       dd  [kernel]                 [k] credit_entropy_bits
     0.02%     perf  [kernel]                 [k] current_fs_time
     0.02%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.02%     perf  [kernel]                 [k] extHint
     0.02%     perf  [kernel]                 [k] release_metapage
     0.02%       dd  [kernel]                 [k] free_hot_page
     0.02%       dd  [kernel]                 [k] bio_phys_segments
     0.02%     perf  [kernel]                 [k] __do_softirq
     0.02%     perf  [kernel]                 [k] handle_mm_fault
     0.02%     perf  [kernel]                 [k] dbAllocDmap
     0.02%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.02%       dd  [kernel]                 [k] up_read
     0.02%       dd  [kernel]                 [k] anon_vma_prepare
     0.02%       dd  [kernel]                 [k] elv_set_request
     0.02%       dd  [kernel]                 [k] blk_unplug
     0.02%     perf  [kernel]                 [k] sysenter_do_call
     0.02%     perf  [kernel]                 [k] zone_watermark_ok
     0.02%     perf  [kernel]                 [k] strlen
     0.02%     perf  [kernel]                 [k] _cond_resched
     0.02%     perf  [kernel]                 [k] mutex_lock
     0.02%     perf  [kernel]                 [k] down_read
     0.02%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.02%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.02%       dd  [kernel]                 [k] cfq_cic_lookup
     0.02%       dd  [kernel]                 [k] scsi_decide_disposition
     0.02%     perf  [kernel]                 [k] current_kernel_time
     0.02%     perf  [kernel]                 [k] mangle_path
     0.02%     perf  [kernel]                 [k] seq_printf
     0.02%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.02%     perf  [kernel]                 [k] blk_end_bidi_request
     0.02%       dd  [kernel]                 [k] rcu_process_gp_end
     0.02%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.02%       dd  [kernel]                 [k] lru_cache_add_lru
     0.02%       dd  [kernel]                 [k] add_timer_randomness
     0.02%       dd  [kernel]                 [k] scsi_finish_command
     0.02%       dd  [kernel]                 [k] scsi_run_queue
     0.01%     perf  [kernel]                 [k] up_read
     0.01%     perf  [kernel]                 [k] read_cache_page_async
     0.01%     perf  [kernel]                 [k] kmem_cache_free
     0.01%     perf  [kernel]                 [k] mnt_drop_write
     0.01%     perf  [kernel]                 [k] alloc_buffer_head
     0.01%     perf  [kernel]                 [k] __find_get_block_slow
     0.01%     perf  [kernel]                 [k] cap_file_permission
     0.01%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.01%       dd  [kernel]                 [k] run_timer_softirq
     0.01%       dd  [kernel]                 [k] dtSearch
     0.01%       dd  [kernel]                 [k] put_device
     0.01%       dd  [kernel]                 [k] scsi_put_command
     0.01%       dd  [kernel]                 [k] scsi_io_completion
     0.01%       dd  [kernel]                 [k] sd_done
     0.01%     perf  [kernel]                 [k] do_read_cache_page
     0.01%     perf  [kernel]                 [k] read_cache_page
     0.01%     perf  [kernel]                 [k] __lru_cache_add
     0.01%     perf  [kernel]                 [k] inode_add_bytes
     0.01%     perf  [kernel]                 [k] dbSplit
     0.01%     perf  [kernel]                 [k] txBeginAnon
     0.01%     perf  [kernel]                 [k] strchr
     0.01%     perf  [kernel]                 [k] ata_sff_interrupt
     0.01%       dd  [kernel]                 [k] mempool_free
     0.01%       dd  [kernel]                 [k] page_remove_rmap
     0.01%       dd  [kernel]                 [k] __d_lookup
     0.01%       dd  [kernel]                 [k] blk_update_bidi_request
     0.01%       dd  [kernel]                 [k] add_disk_randomness
     0.01%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.01%       dd  [kernel]                 [k] serio_interrupt
     0.01%       dd                 b770c0c0  [.] 0x000000b770c0c0
     0.01%     perf  [kernel]                 [k] dup_mm
     0.01%     perf  [kernel]                 [k] handle_IRQ_event
     0.01%     perf  [kernel]                 [k] perf_poll
     0.01%     perf  [kernel]                 [k] perf_mmap_fault
     0.01%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.01%     perf  [kernel]                 [k] do_wp_page
     0.01%     perf  [kernel]                 [k] __d_path
     0.01%     perf  [kernel]                 [k] free_buffer_head
     0.01%     perf  [kernel]                 [k] alloc_page_buffers
     0.01%     perf  [kernel]                 [k] show_map_vma
     0.01%     perf  [kernel]                 [k] dbAdjTree
     0.01%     perf  [kernel]                 [k] spin_unlock
     0.01%     perf  [kernel]                 [k] radix_tree_preload
     0.01%       dd  [kernel]                 [k] check_for_new_grace_period
     0.01%       dd  [kernel]                 [k] sys_write
     0.01%       dd  [kernel]                 [k] copy_strings
     0.01%       dd  [kernel]                 [k] blk_end_request
     0.01%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.01%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.01%       dd  [kernel]                 [k] __scsi_release_buffers
     0.01%       dd  [kernel]                 [k] scsi_device_unbusy
     0.01%       dd  [kernel]                 [k] system_call
     0.01%     perf  [kernel]                 [k] __page_cache_alloc
     0.01%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.01%     perf  [kernel]                 [k] lock_page
     0.01%     perf  [kernel]                 [k] find_vma
     0.01%     perf  [kernel]                 [k] mnt_clone_write
     0.01%     perf  [kernel]                 [k] end_buffer_async_read
     0.01%     perf  [kernel]                 [k] blk_update_request
     0.01%     perf  [kernel]                 [k] put_dec_trunc
     0.01%     perf  [kernel]                 [k] string
     0.01%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.01%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.01%       dd  [kernel]                 [k] perf_output_copy
     0.01%       dd  [kernel]                 [k] test_clear_page_writeback
     0.01%       dd  [kernel]                 [k] vma_adjust
     0.01%       dd  [kernel]                 [k] __split_vma
     0.01%       dd  [kernel]                 [k] path_get
     0.01%       dd  [kernel]                 [k] link_path_walk
     0.01%       dd  [kernel]                 [k] do_filp_open
     0.01%       dd  [kernel]                 [k] dnotify_flush
     0.01%       dd  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.01%       dd  [kernel]                 [k] current_io_context
     0.01%       dd  [kernel]                 [k] prio_tree_insert
     0.01%       dd  [kernel]                 [k] __sg_free_table
     0.01%       dd  [kernel]                 [k] scsi_next_command
     0.00%     perf  [kernel]                 [k] copy_process
     0.00%     perf  [kernel]                 [k] remove_wait_queue
     0.00%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.00%     perf  [kernel]                 [k] down_read_trylock
     0.00%     perf  [kernel]                 [k] rcu_sched_qs
     0.00%     perf  [kernel]                 [k] __rcu_process_callbacks
     0.00%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.00%     perf  [kernel]                 [k] activate_page
     0.00%     perf  [kernel]                 [k] vm_normal_page
     0.00%     perf  [kernel]                 [k] anon_vma_link
     0.00%     perf  [kernel]                 [k] generic_file_llseek_unlocked
     0.00%     perf  [kernel]                 [k] do_sys_poll
     0.00%     perf  [kernel]                 [k] prepend
     0.00%     perf  [kernel]                 [k] d_path
     0.00%     perf  [kernel]                 [k] mnt_want_write_file
     0.00%     perf  [kernel]                 [k] seq_read
     0.00%     perf  [kernel]                 [k] render_sigset_t
     0.00%     perf  [kernel]                 [k] txLock
     0.00%     perf  [kernel]                 [k] security_file_permission
     0.00%       dd  [kernel]                 [k] sysenter_past_esp
     0.00%       dd  [kernel]                 [k] rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] perf_output_begin
     0.00%       dd  [kernel]                 [k] wakeup_kswapd
     0.00%       dd  [kernel]                 [k] vma_prio_tree_remove
     0.00%       dd  [kernel]                 [k] free_pgtables
     0.00%       dd  [kernel]                 [k] do_wp_page
     0.00%       dd  [kernel]                 [k] __vma_link
     0.00%       dd  [kernel]                 [k] unlink_file_vma
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] rw_verify_area
     0.00%       dd  [kernel]                 [k] blk_backing_dev_unplug
     0.00%       dd  [kernel]                 [k] blk_run_queue
     0.00%       dd  [kernel]                 [k] blk_done_softirq
     0.00%       dd  [kernel]                 [k] rb_insert_color
     0.00%       dd  [kernel]                 [k] _copy_from_user
     0.00%       dd  [kernel]                 [k] scsi_log_send
     0.00%       dd  [kernel]                 [k] scsi_pool_free_command
     0.00%       dd  [kernel]                 [k] scsi_sg_free
     0.00%       dd  [kernel]                 [k] i8042_interrupt
     0.00%       dd  /lib/libc-2.11.so        [.] _int_malloc
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___libc_write
     0.00%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_setup_hash
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_relocate_object
     0.00%     perf  [kernel]                 [k] task_fork_fair
     0.00%     perf  [kernel]                 [k] __queue_work
     0.00%     perf  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.00%     perf  [kernel]                 [k] up_write
     0.00%     perf  [kernel]                 [k] rcu_bh_qs
     0.00%     perf  [kernel]                 [k] check_for_new_grace_period
     0.00%     perf  [kernel]                 [k] inherit_event
     0.00%     perf  [kernel]                 [k] end_page_writeback
     0.00%     perf  [kernel]                 [k] wait_on_page_read
     0.00%     perf  [kernel]                 [k] filemap_fault
     0.00%     perf  [kernel]                 [k] vma_prio_tree_add
     0.00%     perf  [kernel]                 [k] next_zones_zonelist
     0.00%     perf  [kernel]                 [k] spin_unlock
     0.00%     perf  [kernel]                 [k] unmap_vmas
     0.00%     perf  [kernel]                 [k] vm_stat_account
     0.00%     perf  [kernel]                 [k] vma_adjust
     0.00%     perf  [kernel]                 [k] anon_vma_prepare
     0.00%     perf  [kernel]                 [k] do_sys_open
     0.00%     perf  [kernel]                 [k] sys_read
     0.00%     perf  [kernel]                 [k] fget
     0.00%     perf  [kernel]                 [k] __fput
     0.00%     perf  [kernel]                 [k] poll_schedule_timeout
     0.00%     perf  [kernel]                 [k] poll_freewait
     0.00%     perf  [kernel]                 [k] __d_lookup
     0.00%     perf  [kernel]                 [k] alloc_fd
     0.00%     perf  [kernel]                 [k] mntput_no_expire
     0.00%     perf  [kernel]                 [k] seq_puts
     0.00%     perf  [kernel]                 [k] seq_path
     0.00%     perf  [kernel]                 [k] set_bh_page
     0.00%     perf  [kernel]                 [k] end_buffer_async_write
     0.00%     perf  [kernel]                 [k] bio_endio
     0.00%     perf  [kernel]                 [k] m_stop
     0.00%     perf  [kernel]                 [k] xtInsert
     0.00%     perf  [kernel]                 [k] security_vm_enough_memory_mm
     0.00%     perf  [kernel]                 [k] cap_task_to_inode
     0.00%     perf  [kernel]                 [k] cfq_idle_slice_timer
     0.00%     perf  [kernel]                 [k] _atomic_dec_and_lock
     0.00%     perf  [kernel]                 [k] kobject_put
     0.00%     perf  [kernel]                 [k] put_dec
     0.00%     perf  [kernel]                 [k] __sg_free_table
     0.00%     perf  [kernel]                 [k] credit_entropy_bits
     0.00%     perf  [kernel]                 [k] scsi_pool_free_command
     0.00%     perf  [kernel]                 [k] serio_interrupt
     0.00%     perf  [kernel]                 [k] down_write
     0.00%     perf  [kernel]                 [k] error_code
     0.00%     perf  /lib/libc-2.11.so        [.] __GI_strtol
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_file_doallocate_internal
     0.00%     perf  /lib/libc-2.11.so        [.] __fopen_internal
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_getline_info
     0.00%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.00%       dd  [kernel]                 [k] init_fpu
     0.00%       dd  [kernel]                 [k] sched_move_task
     0.00%       dd  [kernel]                 [k] try_to_wake_up
     0.00%       dd  [kernel]                 [k] unshare_files
     0.00%       dd  [kernel]                 [k] do_exit
     0.00%       dd  [kernel]                 [k] current_fs_time
     0.00%       dd  [kernel]                 [k] lock_timer_base
     0.00%       dd  [kernel]                 [k] sys_rt_sigaction
     0.00%       dd  [kernel]                 [k] drop_futex_key_refs
     0.00%       dd  [kernel]                 [k] module_put
     0.00%       dd  [kernel]                 [k] rcu_bh_qs
     0.00%       dd  [kernel]                 [k] __call_rcu
     0.00%       dd  [kernel]                 [k] perf_event_comm
     0.00%       dd  [kernel]                 [k] __lock_page
     0.00%       dd  [kernel]                 [k] free_pgd_range
     0.00%       dd  [kernel]                 [k] spin_unlock
     0.00%       dd  [kernel]                 [k] __pte_alloc
     0.00%       dd  [kernel]                 [k] __get_user_pages
     0.00%       dd  [kernel]                 [k] vma_wants_writenotify
     0.00%       dd  [kernel]                 [k] __vma_link_file
     0.00%       dd  [kernel]                 [k] __vma_link_rb
     0.00%       dd  [kernel]                 [k] __vm_enough_memory
     0.00%       dd  [kernel]                 [k] mmap_region
     0.00%       dd  [kernel]                 [k] anon_vma_unlink
     0.00%       dd  [kernel]                 [k] kfree
     0.00%       dd  [kernel]                 [k] vfs_write
     0.00%       dd  [kernel]                 [k] sys_llseek
     0.00%       dd  [kernel]                 [k] sys_read
     0.00%       dd  [kernel]                 [k] fget_light
     0.00%       dd  [kernel]                 [k] file_kill
     0.00%       dd  [kernel]                 [k] fput
     0.00%       dd  [kernel]                 [k] cp_new_stat64
     0.00%       dd  [kernel]                 [k] copy_strings_kernel
     0.00%       dd  [kernel]                 [k] set_task_comm
     0.00%       dd  [kernel]                 [k] setup_new_exec
     0.00%       dd  [kernel]                 [k] open_exec
     0.00%       dd  [kernel]                 [k] acl_permission_check
     0.00%       dd  [kernel]                 [k] path_init
     0.00%       dd  [kernel]                 [k] __follow_mount
     0.00%       dd  [kernel]                 [k] path_put
     0.00%       dd  [kernel]                 [k] __d_free
     0.00%       dd  [kernel]                 [k] d_alloc
     0.00%       dd  [kernel]                 [k] dput
     0.00%       dd  [kernel]                 [k] touch_atime
     0.00%       dd  [kernel]                 [k] mntput_no_expire
     0.00%       dd  [kernel]                 [k] fsnotify
     0.00%       dd  [kernel]                 [k] __fsnotify_parent
     0.00%       dd  [kernel]                 [k] inotify_inode_queue_event
     0.00%       dd  [kernel]                 [k] load_elf_binary
     0.00%       dd  [kernel]                 [k] removed_exe_file_vma
     0.00%       dd  [kernel]                 [k] jfs_lookup
     0.00%       dd  [kernel]                 [k] __get_metapage
     0.00%       dd  [kernel]                 [k] is_file_shm_hugepages
     0.00%       dd  [kernel]                 [k] cap_vm_enough_memory
     0.00%       dd  [kernel]                 [k] security_prepare_creds
     0.00%       dd  [kernel]                 [k] cap_inode_permission
     0.00%       dd  [kernel]                 [k] cap_file_free_security
     0.00%       dd  [kernel]                 [k] kobject_get
     0.00%       dd  [kernel]                 [k] kref_get
     0.00%       dd  [kernel]                 [k] prio_tree_replace
     0.00%       dd  [kernel]                 [k] __rb_rotate_left
     0.00%       dd  [kernel]                 [k] clear_user
     0.00%       dd  [kernel]                 [k] strncpy_from_user
     0.00%       dd  [kernel]                 [k] half_md4_transform
     0.00%       dd  [kernel]                 [k] tty_write
     0.00%       dd  [kernel]                 [k] n_tty_write
     0.00%       dd  [kernel]                 [k] tty_buffer_request_room
     0.00%       dd  [kernel]                 [k] scsi_log_completion
     0.00%       dd  [kernel]                 [k] __scsi_put_command
     0.00%       dd  [kernel]                 [k] scsi_softirq_done
     0.00%       dd  [kernel]                 [k] e100_watchdog
     0.00%       dd  [kernel]                 [k] mutex_lock
     0.00%       dd  [kernel]                 [k] error_code
     0.00%       dd  /lib/libpthread-2.11.so  [.] __pthread_mutex_lock_internal
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_setlocale
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_bsearch
     0.00%       dd  /lib/libc-2.11.so        [.] 0x000000000a1877
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___fxstat64
     0.00%       dd  /lib/libc-2.11.so        [.] __libc_pthread_init
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_start
     0.00%       dd  /lib/ld-2.11.so          [.] open_verify
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_map_object_from_fd
     0.00%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_lookup_symbol_x
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_new_object
     0.00%       dd  /lib/ld-2.11.so          [.] call_init
     0.00%       dd  /lib/ld-2.11.so          [.] access
     0.00%       dd  /bin/dd                  [.] dd_copy
     0.00%       dd  /bin/dd                  [.] human_readable
     0.00%       dd  /bin/dd                  [.] 0x0000000000854d
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #4: perf3.report --]
[-- Type: text/plain, Size: 36989 bytes --]

# Samples: 144548
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    19.08%       dd  [kernel]                 [k] __make_request
     4.56%       dd  [kernel]                 [k] __copy_to_user_ll
     3.96%     perf  [kernel]                 [k] generic_file_buffered_write
     3.87%       dd  [kernel]                 [k] generic_make_request
     2.84%     perf  [kernel]                 [k] __generic_file_aio_write
     2.74%       dd  [kernel]                 [k] kmem_cache_alloc
     2.53%       dd  [kernel]                 [k] block_read_full_page
     1.94%       dd  [kernel]                 [k] blkdev_get_block
     1.64%       dd  [kernel]                 [k] get_page_from_freelist
     1.50%       dd  [kernel]                 [k] kmem_cache_free
     1.44%     perf  [kernel]                 [k] vfs_write
     1.33%     perf  [kernel]                 [k] generic_file_aio_write
     1.33%     perf  [kernel]                 [k] iov_iter_advance
     1.33%     perf  [kernel]                 [k] do_sync_write
     1.29%     perf  [kernel]                 [k] nobh_write_end
     1.29%     perf  [kernel]                 [k] sysenter_past_esp
     1.25%       dd  [kernel]                 [k] submit_bh
     1.23%     perf  [kernel]                 [k] nobh_write_begin
     1.21%     perf  [kernel]                 [k] __jfs_getxattr
     1.18%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     1.05%       dd  [kernel]                 [k] bio_alloc_bioset
     1.04%     perf  [kernel]                 [k] getnstimeofday
     1.01%     perf  [kernel]                 [k] find_get_page
     0.97%     perf  [kernel]                 [k] rw_verify_area
     0.95%     perf  [kernel]                 [k] sys_write
     0.93%     perf  [kernel]                 [k] generic_segment_checks
     0.84%       dd  [kernel]                 [k] submit_bio
     0.82%     perf  [kernel]                 [k] jfs_write_begin
     0.80%     perf  [kernel]                 [k] ea_get
     0.75%       dd  [kernel]                 [k] T.1339
     0.72%       dd  [kernel]                 [k] blk_queue_bounce
     0.71%     perf  [kernel]                 [k] copy_to_user
     0.69%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.68%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.68%       dd  [kernel]                 [k] alloc_buffer_head
     0.61%     perf  [kernel]                 [k] sys_gettimeofday
     0.59%       dd  [kernel]                 [k] free_hot_cold_page
     0.59%     perf  [vdso]                   [.] 0x000000ffffe424
     0.55%     perf  [kernel]                 [k] acpi_pm_read
     0.55%     perf  [kernel]                 [k] generic_write_sync
     0.53%       dd  [kernel]                 [k] mempool_alloc
     0.52%       dd  [kernel]                 [k] add_to_page_cache_locked
     0.51%     perf  [kernel]                 [k] file_update_time
     0.50%     perf  [kernel]                 [k] __mark_inode_dirty
     0.49%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.48%       dd  [kernel]                 [k] alloc_page_buffers
     0.48%     perf  [kernel]                 [k] fget_light
     0.48%       dd  [kernel]                 [k] bio_init
     0.48%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.46%       dd  [kernel]                 [k] generic_file_aio_read
     0.45%       dd  [kernel]                 [k] remove_from_page_cache
     0.44%     perf  [kernel]                 [k] file_remove_suid
     0.41%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.41%       dd  [kernel]                 [k] put_page
     0.41%     perf  [kernel]                 [k] kfree
     0.41%     perf  [kernel]                 [k] fsnotify
     0.40%       dd  [kernel]                 [k] free_buffer_head
     0.39%       dd  [kernel]                 [k] file_read_actor
     0.39%     perf  [kernel]                 [k] put_page
     0.38%     perf  [kernel]                 [k] do_gettimeofday
     0.37%       dd  [kernel]                 [k] __do_page_cache_readahead
     0.37%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.37%     perf  [kernel]                 [k] T.1339
     0.36%     perf  [kernel]                 [k] jfs_getxattr
     0.35%       dd  [kernel]                 [k] __alloc_pages_nodemask
     0.35%     perf  [kernel]                 [k] __rcu_read_unlock
     0.35%       dd  [kernel]                 [k] block_invalidatepage
     0.34%     perf  [kernel]                 [k] find_lock_page
     0.32%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.32%       dd  [kernel]                 [k] try_to_free_buffers
     0.31%     perf  [kernel]                 [k] ea_release
     0.31%       dd  [kernel]                 [k] wake_up_bit
     0.30%     perf  [kernel]                 [k] __fsnotify_parent
     0.29%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.27%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.27%     perf  [kernel]                 [k] timespec_trunc
     0.27%     perf  [kernel]                 [k] strncmp
     0.26%     perf  [kernel]                 [k] unlock_page
     0.26%     perf  [kernel]                 [k] should_remove_suid
     0.25%       dd  [kernel]                 [k] bio_put
     0.24%     perf  [kernel]                 [k] current_fs_time
     0.23%       dd  [kernel]                 [k] __wake_up_bit
     0.23%       dd  [kernel]                 [k] bio_alloc
     0.23%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.23%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.23%     perf  [kernel]                 [k] current_kernel_time
     0.23%       dd  [kernel]                 [k] find_get_page
     0.22%     perf  [kernel]                 [k] strlen
     0.21%     perf  [kernel]                 [k] mark_page_accessed
     0.20%       dd  [kernel]                 [k] __rcu_read_unlock
     0.20%     perf  [kernel]                 [k] jfs_dirty_inode
     0.19%       dd  [kernel]                 [k] zone_watermark_ok
     0.19%       dd  [kernel]                 [k] get_request
     0.18%       dd  [kernel]                 [k] mempool_alloc_slab
     0.18%       dd  [kernel]                 [k] __lru_cache_add
     0.18%       dd  [kernel]                 [k] create_empty_buffers
     0.17%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.17%       dd  [kernel]                 [k] set_bh_page
     0.17%     perf  [kernel]                 [k] __wake_up_bit
     0.17%       dd  [kernel]                 [k] drop_buffers
     0.16%     perf  [kernel]                 [k] _cond_resched
     0.16%       dd  [kernel]                 [k] release_pages
     0.16%     perf  [kernel]                 [k] mutex_lock
     0.16%     perf  [kernel]                 [k] __rcu_read_lock
     0.16%     perf  [kernel]                 [k] cap_file_permission
     0.16%       dd  [kernel]                 [k] find_get_pages
     0.16%       dd  [kernel]                 [k] init_buffer_head
     0.15%     perf  [kernel]                 [k] set_page_dirty
     0.15%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.15%       dd  [kernel]                 [k] setup_object
     0.15%       dd  [kernel]                 [k] scsi_dispatch_cmd
     0.15%       dd  [kernel]                 [k] unlock_buffer
     0.14%     perf  [kernel]                 [k] down_read
     0.14%       dd  [kernel]                 [k] next_zones_zonelist
     0.13%       dd  [kernel]                 [k] __slab_alloc
     0.13%     perf  [kernel]                 [k] up_read
     0.12%       dd  [kernel]                 [k] cancel_dirty_page
     0.12%     perf  [kernel]                 [k] sysenter_do_call
     0.11%       dd  [kernel]                 [k] truncate_inode_page
     0.11%     perf  [kernel]                 [k] extAlloc
     0.11%       dd  [kernel]                 [k] schedule
     0.11%     perf  [kernel]                 [k] mutex_unlock
     0.11%     perf  [kernel]                 [k] dbAlloc
     0.11%     perf  [kernel]                 [k] security_file_permission
     0.11%       dd  [kernel]                 [k] __pagevec_free
     0.10%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.09%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.09%       dd  [kernel]                 [k] rcu_sched_qs
     0.09%       dd  [kernel]                 [k] truncate_inode_pages_range
     0.08%     perf  [kernel]                 [k] xtSearch
     0.08%       dd  [kernel]                 [k] radix_tree_preload
     0.08%     perf  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.08%       dd  [kernel]                 [k] radix_tree_lookup
     0.07%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.07%       dd  [kernel]                 [k] __rcu_read_lock
     0.07%       dd  [kernel]                 [k] handle_IRQ_event
     0.07%       dd  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.07%       dd  [kernel]                 [k] blkdev_releasepage
     0.07%       dd  [kernel]                 [k] preempt_schedule
     0.07%     perf  [kernel]                 [k] schedule
     0.07%       dd  [kernel]                 [k] _cond_resched
     0.07%       dd  [kernel]                 [k] __lookup
     0.07%     perf  [kernel]                 [k] __set_page_dirty
     0.07%       dd  [kernel]                 [k] try_to_release_page
     0.06%     perf  [kernel]                 [k] mnt_want_write_file
     0.06%       dd  [kernel]                 [k] unlock_page
     0.06%     perf  [kernel]                 [k] get_page_from_freelist
     0.06%       dd  [kernel]                 [k] do_invalidatepage
     0.06%       dd  [kernel]                 [k] blk_end_bidi_request
     0.05%     perf  [kernel]                 [k] mnt_clone_write
     0.05%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     0.05%       dd  [kernel]                 [k] handle_mm_fault
     0.05%     perf  [kernel]                 [k] mnt_drop_write
     0.05%       dd  [kernel]                 [k] ata_sff_interrupt
     0.05%       dd  [kernel]                 [k] do_page_fault
     0.05%       dd  [kernel]                 [k] unmap_vmas
     0.04%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.04%     perf  [kernel]                 [k] xtLookup
     0.04%       dd  [kernel]                 [k] mark_page_accessed
     0.04%       dd  [kernel]                 [k] ktime_get_ts
     0.04%       dd  [kernel]                 [k] scsi_request_fn
     0.04%       dd  [kernel]                 [k] __do_softirq
     0.04%       dd  [kernel]                 [k] blkdev_readpage
     0.04%     perf  [kernel]                 [k] dbAllocNext
     0.03%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.03%     perf  [kernel]                 [k] __wake_up
     0.03%     perf  [kernel]                 [k] jfs_get_block
     0.03%     perf  [kernel]                 [k] dbAllocBits
     0.03%     perf  [kernel]                 [k] xtExtend
     0.03%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.03%       dd  [kernel]                 [k] blk_rq_init
     0.03%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.03%       dd  [kernel]                 [k] cfq_set_request
     0.03%     perf  [kernel]                 [k] __get_metapage
     0.03%       dd  [kernel]                 [k] __wait_on_bit_lock
     0.03%       dd  [kernel]                 [k] __wake_up
     0.03%       dd  [kernel]                 [k] sha_transform
     0.02%     perf  [kernel]                 [k] number
     0.02%     perf  [kernel]                 [k] handle_IRQ_event
     0.02%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.02%     perf  [kernel]                 [k] mdio_ctrl_hw
     0.02%     perf  [kernel]                 [k] extHint
     0.02%       dd  [kernel]                 [k] prepare_to_wait_exclusive
     0.02%       dd  [kernel]                 [k] delayacct_end
     0.02%       dd  [kernel]                 [k] set_normalized_timespec
     0.02%     perf  [kernel]                 [k] vsnprintf
     0.02%       dd  [kernel]                 [k] yenta_interrupt
     0.02%     perf  [kernel]                 [k] __find_get_block_slow
     0.02%       dd  [kernel]                 [k] __lock_page_killable
     0.02%       dd  [kernel]                 [k] acpi_pm_read
     0.02%     perf  [kernel]                 [k] dbAllocDmap
     0.02%     perf  [kernel]                 [k] __do_softirq
     0.02%     perf  [kernel]                 [k] kmem_cache_alloc
     0.02%     perf  [kernel]                 [k] format_decode
     0.02%       dd  [kernel]                 [k] get_io_context
     0.02%       dd  [kernel]                 [k] serio_interrupt
     0.01%     perf  [kernel]                 [k] hrtimer_try_to_cancel
     0.01%     perf  [kernel]                 [k] txLock
     0.01%       dd  [kernel]                 [k] __do_fault
     0.01%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.01%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.01%     perf  [kernel]                 [k] do_sys_poll
     0.01%       dd  [kernel]                 [k] filemap_fault
     0.01%     perf  /lib/libpthread-2.11.so  [.] __write_nocancel
     0.01%       dd  [kernel]                 [k] find_vma
     0.01%       dd  [kernel]                 [k] blk_update_request
     0.01%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.01%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.01%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.01%       dd  [kernel]                 [k] cfq_cic_lookup
     0.01%       dd  [kernel]                 [k] radix_tree_gang_lookup_slot
     0.01%     perf  [kernel]                 [k] dbSplit
     0.01%     perf  [kernel]                 [k] txBeginAnon
     0.01%     perf  [kernel]                 [k] release_metapage
     0.01%       dd  [kernel]                 [k] lru_add_drain
     0.01%     perf  [kernel]                 [k] activate_page
     0.01%     perf  [kernel]                 [k] copy_page_range
     0.01%     perf  [kernel]                 [k] scsi_request_fn
     0.01%       dd  [kernel]                 [k] anon_vma_prepare
     0.01%     perf  [kernel]                 [k] kmem_cache_free
     0.01%     perf  [kernel]                 [k] inode_add_bytes
     0.01%       dd  [kernel]                 [k] vm_normal_page
     0.01%       dd  [kernel]                 [k] init_request_from_bio
     0.01%       dd  [kernel]                 [k] get_request_wait
     0.01%     perf  [kernel]                 [k] dbAdjTree
     0.01%     perf  [kernel]                 [k] yenta_interrupt
     0.01%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.01%       dd  [kernel]                 [k] io_schedule
     0.01%     perf  [kernel]                 [k] rcu_sched_qs
     0.01%     perf  [kernel]                 [k] do_read_cache_page
     0.01%     perf  [kernel]                 [k] zone_watermark_ok
     0.01%     perf  [kernel]                 [k] alloc_buffer_head
     0.01%     perf  [kernel]                 [k] do_page_fault
     0.01%       dd  [kernel]                 [k] down_read_trylock
     0.01%       dd  [kernel]                 [k] __delayacct_blkio_end
     0.01%       dd  [kernel]                 [k] page_cache_async_readahead
     0.01%       dd  [kernel]                 [k] blk_unplug
     0.01%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.01%     perf  [kernel]                 [k] __do_fault
     0.01%     perf  [kernel]                 [k] blk_end_bidi_request
     0.01%       dd  [kernel]                 [k] finish_wait
     0.01%       dd  [kernel]                 [k] __pagevec_release
     0.01%       dd  [kernel]                 [k] req_bio_endio
     0.01%     perf  [kernel]                 [k] __lru_cache_add
     0.01%     perf  [kernel]                 [k] free_buffer_head
     0.01%     perf  [kernel]                 [k] alloc_page_buffers
     0.01%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.01%     perf  [kernel]                 [k] ata_sff_interrupt
     0.01%       dd  [kernel]                 [k] up_read
     0.01%       dd  [kernel]                 [k] sync_page
     0.01%       dd  [kernel]                 [k] ondemand_readahead
     0.01%       dd  [kernel]                 [k] block_sync_page
     0.01%       dd  [kernel]                 [k] elv_set_request
     0.01%       dd  [kernel]                 [k] blk_backing_dev_unplug
     0.01%     perf  [kernel]                 [k] read_cache_page_async
     0.01%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.01%     perf  [kernel]                 [k] next_zones_zonelist
     0.01%     perf  [kernel]                 [k] radix_tree_preload
     0.01%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.01%       dd  [kernel]                 [k] free_hot_page
     0.01%       dd  [kernel]                 [k] pagevec_lookup
     0.01%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.01%       dd  [kernel]                 [k] ioread8
     0.01%       dd  [kernel]                 [k] scsi_finish_command
     0.01%       dd  [kernel]                 [k] irq_handler
     0.00%     perf  [kernel]                 [k] ktime_add_safe
     0.00%     perf  [kernel]                 [k] irq_handler
     0.00%       dd  [kernel]                 [k] page_remove_rmap
     0.00%       dd  [kernel]                 [k] mpage_end_io_read
     0.00%       dd  [kernel]                 [k] blk_recount_segments
     0.00%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.00%     perf  [kernel]                 [k] ktime_get_ts
     0.00%     perf  [kernel]                 [k] do_wp_page
     0.00%     perf  [kernel]                 [k] mangle_path
     0.00%     perf  [kernel]                 [k] strchr
     0.00%     perf  /home/cloos/bin/perf     [.] 0x000000000023ec
     0.00%       dd  [kernel]                 [k] __delayacct_blkio_start
     0.00%       dd  [kernel]                 [k] ra_submit
     0.00%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.00%     perf  [kernel]                 [k] read_cache_page
     0.00%     perf  [kernel]                 [k] get_dirty_limits
     0.00%     perf  [kernel]                 [k] estimate_accuracy
     0.00%     perf  [kernel]                 [k] seq_printf
     0.00%     perf  [kernel]                 [k] blk_update_request
     0.00%     perf  [kernel]                 [k] preempt_schedule_irq
     0.00%       dd  [kernel]                 [k] sync_page_killable
     0.00%       dd  [kernel]                 [k] page_evictable
     0.00%       dd  [kernel]                 [k] copy_strings
     0.00%       dd  [kernel]                 [k] bio_phys_segments
     0.00%       dd  [kernel]                 [k] generic_unplug_device
     0.00%       dd  [kernel]                 [k] current_io_context
     0.00%       dd  [kernel]                 [k] _copy_from_user
     0.00%       dd  [kernel]                 [k] scsi_sg_free
     0.00%       dd  [kernel]                 [k] sd_done
     0.00%     perf  [kernel]                 [k] set_normalized_timespec
     0.00%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.00%     perf  [kernel]                 [k] poll_select_set_timeout
     0.00%     perf  [kernel]                 [k] __pollwait
     0.00%       dd  [kernel]                 [k] test_clear_page_writeback
     0.00%       dd  [kernel]                 [k] link_path_walk
     0.00%       dd  [kernel]                 [k] blk_update_bidi_request
     0.00%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.00%       dd  [kernel]                 [k] kref_get
     0.00%       dd  [kernel]                 [k] scsi_pool_free_command
     0.00%       dd  [kernel]                 [k] scsi_run_queue
     0.00%       dd  [kernel]                 [k] scsi_io_completion
     0.00%       dd  [kernel]                 [k] i8042_interrupt
     0.00%       dd  [kernel]                 [k] __alloc_skb
     0.00%       dd  [kernel]                 [k] tcp_transmit_skb
     0.00%     perf  [kernel]                 [k] remove_wait_queue
     0.00%     perf  [kernel]                 [k] add_wait_queue
     0.00%     perf  [kernel]                 [k] hrtimer_start_range_ns
     0.00%     perf  [kernel]                 [k] wait_on_page_read
     0.00%     perf  [kernel]                 [k] find_vma
     0.00%     perf  [kernel]                 [k] poll_freewait
     0.00%     perf  [kernel]                 [k] sys_poll
     0.00%     perf  [kernel]                 [k] prepend
     0.00%     perf  [kernel]                 [k] mpage_end_io_read
     0.00%     perf  [kernel]                 [k] spin_unlock
     0.00%     perf  [kernel]                 [k] _atomic_dec_and_lock
     0.00%     perf  [kernel]                 [k] put_dec
     0.00%       dd  [kernel]                 [k] __wake_up_sync_key
     0.00%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] lru_cache_add_lru
     0.00%       dd  [kernel]                 [k] do_wp_page
     0.00%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.00%       dd  [kernel]                 [k] __d_lookup
     0.00%       dd  [kernel]                 [k] bio_endio
     0.00%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.00%       dd  [kernel]                 [k] strnlen_user
     0.00%       dd  [kernel]                 [k] credit_entropy_bits
     0.00%       dd  [kernel]                 [k] scsi_decide_disposition
     0.00%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.00%       dd  [kernel]                 [k] scsi_softirq_done
     0.00%       dd  [kernel]                 [k] ip_queue_xmit
     0.00%       dd  [kernel]                 [k] __release_kernel_lock
     0.00%     perf  [kernel]                 [k] dup_mm
     0.00%     perf  [kernel]                 [k] hrtimer_init
     0.00%     perf  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.00%     perf  [kernel]                 [k] __rcu_process_callbacks
     0.00%     perf  [kernel]                 [k] perf_mmap_fault
     0.00%     perf  [kernel]                 [k] filemap_fault
     0.00%     perf  [kernel]                 [k] vm_normal_page
     0.00%     perf  [kernel]                 [k] handle_mm_fault
     0.00%     perf  [kernel]                 [k] mmap_region
     0.00%     perf  [kernel]                 [k] __slab_alloc
     0.00%     perf  [kernel]                 [k] __d_path
     0.00%     perf  [kernel]                 [k] seq_read
     0.00%     perf  [kernel]                 [k] set_bh_page
     0.00%     perf  [kernel]                 [k] end_bio_bh_io_sync
     0.00%     perf  [kernel]                 [k] bio_endio
     0.00%     perf  [kernel]                 [k] bio_fs_destructor
     0.00%     perf  [kernel]                 [k] show_map_vma
     0.00%     perf  [kernel]                 [k] req_bio_endio
     0.00%     perf  [kernel]                 [k] ioread8
     0.00%     perf  [kernel]                 [k] scsi_init_cmd_errh
     0.00%     perf  [kernel]                 [k] preempt_schedule
     0.00%     perf  /lib/libc-2.11.so        [.] __libc_poll
     0.00%     perf  /lib/ld-2.11.so          [.] _dl_name_match_p
     0.00%       dd  [kernel]                 [k] __cond_resched
     0.00%       dd  [kernel]                 [k] prepare_creds
     0.00%       dd  [kernel]                 [k] perf_output_copy
     0.00%       dd  [kernel]                 [k] unmap_region
     0.00%       dd  [kernel]                 [k] mmap_region
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] sys_mmap_pgoff
     0.00%       dd  [kernel]                 [k] page_add_file_rmap
     0.00%       dd  [kernel]                 [k] anon_vma_unlink
     0.00%       dd  [kernel]                 [k] vfs_read
     0.00%       dd  [kernel]                 [k] mntput_no_expire
     0.00%       dd  [kernel]                 [k] bio_fs_destructor
     0.00%       dd  [kernel]                 [k] load_elf_binary
     0.00%       dd  [kernel]                 [k] blk_run_queue
     0.00%       dd  [kernel]                 [k] __scsi_put_command
     0.00%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.00%       dd  [kernel]                 [k] __scsi_release_buffers
     0.00%       dd  [kernel]                 [k] scsi_device_unbusy
     0.00%       dd  [kernel]                 [k] e100_enable_irq
     0.00%       dd  [kernel]                 [k] __kfree_skb
     0.00%       dd  [kernel]                 [k] ip_route_input
     0.00%       dd  [kernel]                 [k] ip_local_deliver
     0.00%       dd  [kernel]                 [k] __inet_lookup_established
     0.00%       dd  [kernel]                 [k] tcp_v4_rcv
     0.00%       dd  [kernel]                 [k] preempt_schedule_irq
     0.00%       dd  [kernel]                 [k] system_call
     0.00%       dd  [kernel]                 [k] error_code
     0.00%       dd  /lib/libc-2.11.so        [.] 0x00000000021f6c
     0.00%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_lookup_symbol_x
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_relocate_object
     0.00%     perf  [kernel]                 [k] __wake_up_sync_key
     0.00%     perf  [kernel]                 [k] copy_process
     0.00%     perf  [kernel]                 [k] run_timer_softirq
     0.00%     perf  [kernel]                 [k] rcu_bh_qs
     0.00%     perf  [kernel]                 [k] call_rcu
     0.00%     perf  [kernel]                 [k] perf_poll
     0.00%     perf  [kernel]                 [k] perf_event_init_task
     0.00%     perf  [kernel]                 [k] mempool_free
     0.00%     perf  [kernel]                 [k] determine_dirtyable_memory
     0.00%     perf  [kernel]                 [k] release_pages
     0.00%     perf  [kernel]                 [k] lru_cache_add_lru
     0.00%     perf  [kernel]                 [k] lock_page
     0.00%     perf  [kernel]                 [k] expand_downwards
     0.00%     perf  [kernel]                 [k] __vm_enough_memory
     0.00%     perf  [kernel]                 [k] vma_adjust
     0.00%     perf  [kernel]                 [k] page_add_new_anon_rmap
     0.00%     perf  [kernel]                 [k] anon_vma_link
     0.00%     perf  [kernel]                 [k] setup_object
     0.00%     perf  [kernel]                 [k] __kmalloc_track_caller
     0.00%     perf  [kernel]                 [k] vfs_read
     0.00%     perf  [kernel]                 [k] may_open
     0.00%     perf  [kernel]                 [k] link_path_walk
     0.00%     perf  [kernel]                 [k] poll_schedule_timeout
     0.00%     perf  [kernel]                 [k] free_poll_entry
     0.00%     perf  [kernel]                 [k] __d_lookup
     0.00%     perf  [kernel]                 [k] dput
     0.00%     perf  [kernel]                 [k] dup_fd
     0.00%     perf  [kernel]                 [k] expand_files
     0.00%     perf  [kernel]                 [k] end_buffer_async_read
     0.00%     perf  [kernel]                 [k] end_buffer_async_write
     0.00%     perf  [kernel]                 [k] m_start
     0.00%     perf  [kernel]                 [k] render_sigset_t
     0.00%     perf  [kernel]                 [k] proc_pid_status
     0.00%     perf  [kernel]                 [k] cap_vm_enough_memory
     0.00%     perf  [kernel]                 [k] cap_task_to_inode
     0.00%     perf  [kernel]                 [k] blk_update_bidi_request
     0.00%     perf  [kernel]                 [k] cfq_idle_slice_timer
     0.00%     perf  [kernel]                 [k] kref_put
     0.00%     perf  [kernel]                 [k] prop_get_global
     0.00%     perf  [kernel]                 [k] prop_norm_single
     0.00%     perf  [kernel]                 [k] __rb_rotate_left
     0.00%     perf  [kernel]                 [k] string
     0.00%     perf  [kernel]                 [k] credit_entropy_bits
     0.00%     perf  [kernel]                 [k] get_device
     0.00%     perf  [kernel]                 [k] scsi_finish_command
     0.00%     perf  [kernel]                 [k] scsi_decide_disposition
     0.00%     perf  [kernel]                 [k] scsi_run_queue
     0.00%     perf  [kernel]                 [k] e100_exec_cmd
     0.00%     perf  [kernel]                 [k] e100_disable_irq
     0.00%     perf  [kernel]                 [k] e100_intr
     0.00%     perf  [kernel]                 [k] __alloc_skb
     0.00%     perf  [kernel]                 [k] rt6_fill_node
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_getline_info
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_file_fopen_internal
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_link_in_internal
     0.00%     perf  /lib/libc-2.11.so        [.] _int_free
     0.00%     perf  /home/cloos/bin/perf     [.] exit@plt
     0.00%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.00%       dd  [kernel]                 [k] pgd_alloc
     0.00%       dd  [kernel]                 [k] test_ti_thread_flag
     0.00%       dd  [kernel]                 [k] try_to_wake_up
     0.00%       dd  [kernel]                 [k] mm_alloc
     0.00%       dd  [kernel]                 [k] current_fs_time
     0.00%       dd  [kernel]                 [k] run_timer_softirq
     0.00%       dd  [kernel]                 [k] free_uid
     0.00%       dd  [kernel]                 [k] __queue_work
     0.00%       dd  [kernel]                 [k] do_futex
     0.00%       dd  [kernel]                 [k] rcu_process_gp_end
     0.00%       dd  [kernel]                 [k] check_for_new_grace_period
     0.00%       dd  [kernel]                 [k] rcu_start_gp
     0.00%       dd  [kernel]                 [k] __call_rcu
     0.00%       dd  [kernel]                 [k] rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] perf_event_pid
     0.00%       dd  [kernel]                 [k] perf_output_begin
     0.00%       dd  [kernel]                 [k] perf_event_comm
     0.00%       dd  [kernel]                 [k] find_get_pages_tag
     0.00%       dd  [kernel]                 [k] wakeup_kswapd
     0.00%       dd  [kernel]                 [k] follow_page
     0.00%       dd  [kernel]                 [k] find_mergeable_anon_vma
     0.00%       dd  [kernel]                 [k] remove_vma
     0.00%       dd  [kernel]                 [k] expand_downwards
     0.00%       dd  [kernel]                 [k] __remove_shared_vm_struct
     0.00%       dd  [kernel]                 [k] unlink_file_vma
     0.00%       dd  [kernel]                 [k] vma_link
     0.00%       dd  [kernel]                 [k] vma_adjust
     0.00%       dd  [kernel]                 [k] __split_vma
     0.00%       dd  [kernel]                 [k] do_munmap
     0.00%       dd  [kernel]                 [k] vma_merge
     0.00%       dd  [kernel]                 [k] filp_close
     0.00%       dd  [kernel]                 [k] vfs_write
     0.00%       dd  [kernel]                 [k] __fput
     0.00%       dd  [kernel]                 [k] test_ti_thread_flag
     0.00%       dd  [kernel]                 [k] do_execve
     0.00%       dd  [kernel]                 [k] exec_permission
     0.00%       dd  [kernel]                 [k] getname
     0.00%       dd  [kernel]                 [k] __follow_mount
     0.00%       dd  [kernel]                 [k] path_put
     0.00%       dd  [kernel]                 [k] path_to_nameidata
     0.00%       dd  [kernel]                 [k] do_lookup
     0.00%       dd  [kernel]                 [k] path_walk
     0.00%       dd  [kernel]                 [k] do_filp_open
     0.00%       dd  [kernel]                 [k] prepend
     0.00%       dd  [kernel]                 [k] dput
     0.00%       dd  [kernel]                 [k] touch_atime
     0.00%       dd  [kernel]                 [k] alloc_fd
     0.00%       dd  [kernel]                 [k] end_bio_bh_io_sync
     0.00%       dd  [kernel]                 [k] mpage_end_io_write
     0.00%       dd  [kernel]                 [k] do_mpage_readpage
     0.00%       dd  [kernel]                 [k] fsnotify
     0.00%       dd  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.00%       dd  [kernel]                 [k] jfs_open
     0.00%       dd  [kernel]                 [k] security_inode_permission
     0.00%       dd  [kernel]                 [k] blk_end_request
     0.00%       dd  [kernel]                 [k] blk_done_softirq
     0.00%       dd  [kernel]                 [k] cfq_get_queue
     0.00%       dd  [kernel]                 [k] prio_tree_replace
     0.00%       dd  [kernel]                 [k] prio_tree_insert
     0.00%       dd  [kernel]                 [k] rb_insert_color
     0.00%       dd  [kernel]                 [k] csum_partial
     0.00%       dd  [kernel]                 [k] __get_user_4
     0.00%       dd  [kernel]                 [k] __put_user_4
     0.00%       dd  [kernel]                 [k] strlen
     0.00%       dd  [kernel]                 [k] strncpy_from_user
     0.00%       dd  [kernel]                 [k] __sg_free_table
     0.00%       dd  [kernel]                 [k] add_timer_randomness
     0.00%       dd  [kernel]                 [k] extract_buf
     0.00%       dd  [kernel]                 [k] extract_entropy
     0.00%       dd  [kernel]                 [k] n_tty_write
     0.00%       dd  [kernel]                 [k] tty_insert_flip_string
     0.00%       dd  [kernel]                 [k] tty_flip_buffer_push
     0.00%       dd  [kernel]                 [k] get_device
     0.00%       dd  [kernel]                 [k] put_device
     0.00%       dd  [kernel]                 [k] scsi_log_send
     0.00%       dd  [kernel]                 [k] scsi_log_completion
     0.00%       dd  [kernel]                 [k] scsi_put_command
     0.00%       dd  [kernel]                 [k] e100_exec_cb
     0.00%       dd  [kernel]                 [k] e100_intr
     0.00%       dd  [kernel]                 [k] e100_poll
     0.00%       dd  [kernel]                 [k] dma_issue_pending_all
     0.00%       dd  [kernel]                 [k] skb_release_data
     0.00%       dd  [kernel]                 [k] skb_checksum
     0.00%       dd  [kernel]                 [k] dev_hard_start_xmit
     0.00%       dd  [kernel]                 [k] netif_receive_skb
     0.00%       dd  [kernel]                 [k] net_rx_action
     0.00%       dd  [kernel]                 [k] ip_local_out
     0.00%       dd  [kernel]                 [k] tcp_parse_aligned_timestamp
     0.00%       dd  [kernel]                 [k] tcp_data_snd_check
     0.00%       dd  [kernel]                 [k] tcp_valid_rtt_meas
     0.00%       dd  [kernel]                 [k] tcp_rcv_established
     0.00%       dd  [kernel]                 [k] tcp_send_ack
     0.00%       dd  [kernel]                 [k] raw_local_deliver
     0.00%       dd  /lib/librt-2.11.so       [.] __GI_clock_gettime
     0.00%       dd  /lib/libpthread-2.11.so  [.] __pthread_initialize_minimal_internal
     0.00%       dd  /lib/libc-2.11.so        [.] _init
     0.00%       dd  /lib/libc-2.11.so        [.] new_composite_name
     0.00%       dd  /lib/libc-2.11.so        [.] _nl_load_locale_from_archive
     0.00%       dd  /lib/libc-2.11.so        [.] _IO_file_overflow_internal
     0.00%       dd  /lib/libc-2.11.so        [.] index
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_close
     0.00%       dd  /lib/ld-2.11.so          [.] _start
     0.00%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_check_map_versions
     0.00%       dd  /lib/ld-2.11.so          [.] __open
     0.00%       dd  /lib/ld-2.11.so          [.] strcmp
     0.00%       dd  /lib/ld-2.11.so          [.] memmove
     0.00%       dd                 b7713b6c  [.] 0x000000b7713b6c
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #5: perf4.report --]
[-- Type: text/plain, Size: 29539 bytes --]

# Samples: 71344
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
     6.73%       dd  [kernel]                 [k] __copy_from_user_ll_nozero
     5.94%       dd  [kernel]                 [k] xtSearch
     5.22%       dd  [kernel]                 [k] dbAlloc
     4.85%       dd  [kernel]                 [k] __clear_user
     4.69%       dd  [kernel]                 [k] extAlloc
     4.03%       dd  [kernel]                 [k] get_page_from_freelist
     3.49%       dd  [kernel]                 [k] lmWriteRecord
     3.30%       dd  [kernel]                 [k] __set_page_dirty
     2.97%       dd  [kernel]                 [k] __wake_up
     2.76%       dd  [kernel]                 [k] xtLookup
     2.69%       dd  [kernel]                 [k] nobh_write_begin
     2.64%       dd  [kernel]                 [k] find_get_page
     2.32%       dd  [kernel]                 [k] __get_metapage
     1.90%       dd  [kernel]                 [k] xtExtend
     1.87%       dd  [kernel]                 [k] dbAllocBits
     1.61%       dd  [kernel]                 [k] generic_file_buffered_write
     1.58%       dd  [kernel]                 [k] dbAllocNext
     1.39%       dd  [kernel]                 [k] release_metapage
     1.39%       dd  [kernel]                 [k] jfs_get_block
     1.33%       dd  [kernel]                 [k] txCommit
     1.32%       dd  [kernel]                 [k] __mark_inode_dirty
     1.30%       dd  [kernel]                 [k] extHint
     1.29%       dd  [kernel]                 [k] txLock
     1.20%       dd  [kernel]                 [k] add_to_page_cache_locked
     1.10%       dd  [kernel]                 [k] nobh_write_end
     1.08%       dd  [kernel]                 [k] __rcu_read_unlock
     0.98%       dd  [kernel]                 [k] __alloc_pages_nodemask
     0.96%       dd  [kernel]                 [k] unlock_page
     0.95%       dd  [kernel]                 [k] lmLog
     0.90%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.86%       dd  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.84%       dd  [kernel]                 [k] do_read_cache_page
     0.82%       dd  [kernel]                 [k] dbAllocDmap
     0.78%       dd  [kernel]                 [k] put_page
     0.73%       dd  [kernel]                 [k] __wake_up_bit
     0.67%       dd  [kernel]                 [k] read_cache_page
     0.67%       dd  [kernel]                 [k] grab_cache_page_write_begin
     0.64%       dd  [kernel]                 [k] __find_get_block_slow
     0.62%       dd  [kernel]                 [k] kmem_cache_alloc
     0.61%       dd  [kernel]                 [k] jfs_write_begin
     0.55%       dd  [kernel]                 [k] zone_watermark_ok
     0.54%       dd  [kernel]                 [k] kmem_cache_free
     0.53%       dd  [kernel]                 [k] iov_iter_fault_in_readable
     0.53%       dd  [kernel]                 [k] read_cache_page_async
     0.52%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.52%       dd  [kernel]                 [k] dbSplit
     0.50%       dd  [kernel]                 [k] txBeginAnon
     0.50%       dd  [kernel]                 [k] iov_iter_advance
     0.50%       dd  [kernel]                 [k] mark_page_accessed
     0.48%       dd  [kernel]                 [k] diWrite
     0.48%       dd  [kernel]                 [k] scsi_dispatch_cmd
     0.48%       dd  [kernel]                 [k] __rcu_read_lock
     0.48%       dd  [kernel]                 [k] scsi_request_fn
     0.46%       dd  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.46%       dd  [kernel]                 [k] inode_add_bytes
     0.45%       dd  [kernel]                 [k] dbAdjTree
     0.42%       dd  [kernel]                 [k] __set_page_dirty_buffers
     0.40%       dd  [kernel]                 [k] alloc_page_buffers
     0.38%       dd  [kernel]                 [k] alloc_buffer_head
     0.38%       dd  [kernel]                 [k] __lru_cache_add
     0.36%       dd  [kernel]                 [k] jfs_dirty_inode
     0.35%       dd  [kernel]                 [k] next_zones_zonelist
     0.34%       dd  [kernel]                 [k] copy_to_dinode
     0.33%       dd  [kernel]                 [k] wait_on_page_read
     0.33%       dd  [kernel]                 [k] mutex_lock
     0.32%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.32%       dd  [kernel]                 [k] read_zero
     0.31%       dd  [kernel]                 [k] free_buffer_head
     0.31%       dd  [kernel]                 [k] mutex_unlock
     0.27%       dd  [kernel]                 [k] _cond_resched
     0.26%       dd  [kernel]                 [k] find_lock_page
     0.23%       dd  [kernel]                 [k] txBegin
     0.22%       dd  [kernel]                 [k] radix_tree_preload
     0.22%       dd  [kernel]                 [k] __set_page_dirty_nobuffers
     0.20%       dd  [kernel]                 [k] set_page_dirty
     0.20%       dd  [kernel]                 [k] spin_unlock
     0.17%       dd  [kernel]                 [k] do_page_fault
     0.17%       dd  [kernel]                 [k] txEnd
     0.16%       dd  [kernel]                 [k] jfs_commit_inode
     0.16%       dd  [kernel]                 [k] schedule
     0.16%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.14%       dd  [kernel]                 [k] handle_mm_fault
     0.14%       dd  [kernel]                 [k] unmap_underlying_metadata
     0.13%       dd  [kernel]                 [k] up_write
     0.13%       dd  [kernel]                 [k] txLockAlloc
     0.13%       dd  [kernel]                 [k] lmGroupCommit
     0.12%       dd  [kernel]                 [k] unmap_vmas
     0.12%       dd  [kernel]                 [k] down_write
     0.11%       dd  [kernel]                 [k] down_read
     0.11%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.10%       dd  [kernel]                 [k] __page_cache_alloc
     0.09%       dd  [kernel]                 [k] up_read
     0.09%       dd  [kernel]                 [k] get_request
     0.09%       dd  [kernel]                 [k] lbmIODone
     0.09%       dd  [kernel]                 [k] blk_end_bidi_request
     0.08%       dd  [kernel]                 [k] T.1339
     0.08%       dd  [kernel]                 [k] free_hot_cold_page
     0.07%       dd  [kernel]                 [k] set_bh_page
     0.07%       dd  [kernel]                 [k] xtInsert
     0.06%       dd  [kernel]                 [k] __do_softirq
     0.06%       dd  [kernel]                 [k] sha_transform
     0.06%       dd  [kernel]                 [k] handle_IRQ_event
     0.05%       dd  [kernel]                 [k] generic_make_request
     0.05%       dd  [kernel]                 [k] rcu_sched_qs
     0.05%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     0.04%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.04%       dd  [kernel]                 [k] ata_sff_interrupt
     0.04%       dd  [kernel]                 [k] __make_request
     0.04%       dd  [kernel]                 [k] serio_interrupt
     0.04%       dd  [kernel]                 [k] release_pages
     0.03%       dd  [kernel]                 [k] test_clear_page_writeback
     0.03%       dd  [kernel]                 [k] lbmWrite
     0.03%       dd  [kernel]                 [k] __do_fault
     0.03%       dd  [kernel]                 [k] jfs_get_inode_flags
     0.03%     perf  [kernel]                 [k] T.1339
     0.03%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.03%       dd  [kernel]                 [k] lmNextPage
     0.02%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.02%       dd  [kernel]                 [k] down_read_trylock
     0.02%       dd  [kernel]                 [k] filemap_fault
     0.02%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.02%       dd  [kernel]                 [k] check_for_new_grace_period
     0.02%       dd  [kernel]                 [k] preempt_schedule
     0.02%     perf  [vdso]                   [.] 0x000000ffffe425
     0.02%     perf  [kernel]                 [k] nobh_write_begin
     0.02%       dd  [kernel]                 [k] anon_vma_prepare
     0.02%       dd  [kernel]                 [k] blk_rq_init
     0.02%       dd  [kernel]                 [k] blk_queue_bounce
     0.02%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.02%       dd  [kernel]                 [k] cfq_set_request
     0.02%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.02%       dd  [kernel]                 [k] find_vma
     0.02%     perf  [kernel]                 [k] dbAlloc
     0.02%       dd  [kernel]                 [k] mempool_alloc
     0.02%       dd  [kernel]                 [k] mpage_end_io_write
     0.02%       dd  [kernel]                 [k] lbmAllocate
     0.01%     perf  [kernel]                 [k] __get_metapage
     0.01%       dd  [kernel]                 [k] __lookup_tag
     0.01%     perf  [kernel]                 [k] release_metapage
     0.01%     perf  /lib/libc-2.11.so        [.] __libc_poll
     0.01%       dd  [kernel]                 [k] vm_normal_page
     0.01%       dd  [kernel]                 [k] init_request_from_bio
     0.01%     perf  [kernel]                 [k] get_page_from_freelist
     0.01%     perf  [kernel]                 [k] __set_page_dirty
     0.01%       dd  [kernel]                 [k] bio_alloc_bioset
     0.01%       dd  [kernel]                 [k] blk_update_request
     0.01%       dd  [kernel]                 [k] kref_put
     0.01%       dd  [kernel]                 [k] strnlen_user
     0.01%       dd  [kernel]                 [k] e100_poll
     0.01%     perf  [kernel]                 [k] extAlloc
     0.01%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.01%     perf  [kernel]                 [k] acpi_pm_read
     0.01%       dd  [kernel]                 [k] mempool_alloc_slab
     0.01%       dd  [kernel]                 [k] test_set_page_writeback
     0.01%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.01%       dd  [kernel]                 [k] dbAdjCtl
     0.01%       dd  [kernel]                 [k] i8042_interrupt
     0.01%     perf  [kernel]                 [k] do_sys_poll
     0.01%     perf  [kernel]                 [k] nobh_write_end
     0.01%       dd  [kernel]                 [k] rcu_process_gp_end
     0.01%       dd  [kernel]                 [k] free_hot_page
     0.01%       dd  [kernel]                 [k] page_remove_rmap
     0.01%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.01%       dd  [kernel]                 [k] get_request_wait
     0.01%       dd  [kernel]                 [k] get_io_context
     0.01%       dd  [kernel]                 [k] __scsi_put_command
     0.01%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.01%       dd                 b71a045f  [.] 0x000000b71a045f
     0.01%     perf  [kernel]                 [k] __wake_up
     0.01%     perf  [kernel]                 [k] __wake_up_bit
     0.01%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.01%       dd  [kernel]                 [k] page_evictable
     0.01%       dd  [kernel]                 [k] copy_strings
     0.01%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.01%     perf  [kernel]                 [k] __generic_file_aio_write
     0.01%     perf  [kernel]                 [k] seq_printf
     0.01%     perf  [kernel]                 [k] jfs_get_block
     0.01%     perf  [kernel]                 [k] xtSearch
     0.01%     perf  [kernel]                 [k] txLock
     0.01%       dd  [kernel]                 [k] sysenter_past_esp
     0.01%       dd  [kernel]                 [k] lru_cache_add_lru
     0.01%       dd  [kernel]                 [k] bio_phys_segments
     0.01%       dd  [kernel]                 [k] lbmStartIO
     0.01%       dd  [kernel]                 [k] blk_run_queue
     0.01%       dd  [kernel]                 [k] cfq_cic_lookup
     0.01%       dd  [kernel]                 [k] rwsem_wake
     0.01%       dd  [kernel]                 [k] scsi_finish_command
     0.01%       dd  [kernel]                 [k] scsi_decide_disposition
     0.01%       dd  [kernel]                 [k] sd_done
     0.01%       dd  [kernel]                 [k] ip_rcv
     0.01%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%     perf  [kernel]                 [k] timespec_add_safe
     0.00%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.00%     perf  [kernel]                 [k] generic_file_buffered_write
     0.00%     perf  [kernel]                 [k] generic_file_aio_write
     0.00%     perf  [kernel]                 [k] __do_fault
     0.00%     perf  [kernel]                 [k] copy_page_range
     0.00%     perf  [kernel]                 [k] do_wp_page
     0.00%     perf  [kernel]                 [k] vfs_write
     0.00%     perf  [kernel]                 [k] xtExtend
     0.00%     perf  [kernel]                 [k] dbAdjTree
     0.00%     perf  [kernel]                 [k] dbAllocNext
     0.00%     perf  [kernel]                 [k] strlen
     0.00%     perf  [kernel]                 [k] scsi_request_fn
     0.00%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.00%       dd  [kernel]                 [k] try_to_wake_up
     0.00%       dd  [kernel]                 [k] vma_prio_tree_remove
     0.00%       dd  [kernel]                 [k] find_mergeable_anon_vma
     0.00%       dd  [kernel]                 [k] do_lookup
     0.00%       dd  [kernel]                 [k] link_path_walk
     0.00%       dd  [kernel]                 [k] __d_lookup
     0.00%       dd  [kernel]                 [k] dbFindLeaf
     0.00%       dd  [kernel]                 [k] metapage_writepage
     0.00%       dd  [kernel]                 [k] __jfs_getxattr
     0.00%       dd  [kernel]                 [k] blk_recount_segments
     0.00%       dd  [kernel]                 [k] scsi_run_queue
     0.00%       dd  [kernel]                 [k] scsi_io_completion
     0.00%       dd  [kernel]                 [k] ip_route_input
     0.00%     perf  [kernel]                 [k] hrtimer_init
     0.00%     perf  [kernel]                 [k] up_read
     0.00%     perf  [kernel]                 [k] getnstimeofday
     0.00%     perf  [kernel]                 [k] __rcu_read_lock
     0.00%     perf  [kernel]                 [k] find_get_page
     0.00%     perf  [kernel]                 [k] unlock_page
     0.00%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.00%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.00%     perf  [kernel]                 [k] find_vma
     0.00%     perf  [kernel]                 [k] rw_verify_area
     0.00%     perf  [kernel]                 [k] fput
     0.00%     perf  [kernel]                 [k] sys_poll
     0.00%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.00%     perf  [kernel]                 [k] xtLookup
     0.00%     perf  [kernel]                 [k] dbAllocBits
     0.00%     perf  [kernel]                 [k] ea_get
     0.00%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.00%     perf  [kernel]                 [k] format_decode
     0.00%     perf  [kernel]                 [k] copy_to_user
     0.00%     perf  [kernel]                 [k] mutex_lock
     0.00%     perf  [kernel]                 [k] do_page_fault
     0.00%       dd  [kernel]                 [k] test_ti_thread_flag
     0.00%       dd  [kernel]                 [k] prepare_to_wait_exclusive
     0.00%       dd  [kernel]                 [k] delayacct_end
     0.00%       dd  [kernel]                 [k] perf_output_begin
     0.00%       dd  [kernel]                 [k] end_page_writeback
     0.00%       dd  [kernel]                 [k] __filemap_fdatawrite_range
     0.00%       dd  [kernel]                 [k] free_pgd_range
     0.00%       dd  [kernel]                 [k] __vma_link_rb
     0.00%       dd  [kernel]                 [k] unlink_file_vma
     0.00%       dd  [kernel]                 [k] mmap_region
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] rw_verify_area
     0.00%       dd  [kernel]                 [k] vfs_write
     0.00%       dd  [kernel]                 [k] vfs_read
     0.00%       dd  [kernel]                 [k] file_move
     0.00%       dd  [kernel]                 [k] put_filp
     0.00%       dd  [kernel]                 [k] path_walk
     0.00%       dd  [kernel]                 [k] bio_endio
     0.00%       dd  [kernel]                 [k] bio_init
     0.00%       dd  [kernel]                 [k] bio_free
     0.00%       dd  [kernel]                 [k] do_mpage_readpage
     0.00%       dd  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.00%       dd  [kernel]                 [k] dbAllocDmapLev
     0.00%       dd  [kernel]                 [k] dbAllocAG
     0.00%       dd  [kernel]                 [k] req_bio_endio
     0.00%       dd  [kernel]                 [k] blk_update_bidi_request
     0.00%       dd  [kernel]                 [k] current_io_context
     0.00%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.00%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.00%       dd  [kernel]                 [k] kobject_put
     0.00%       dd  [kernel]                 [k] __get_user_4
     0.00%       dd  [kernel]                 [k] __sg_free_table
     0.00%       dd  [kernel]                 [k] add_timer_randomness
     0.00%       dd  [kernel]                 [k] scsi_put_command
     0.00%       dd  [kernel]                 [k] scsi_softirq_done
     0.00%       dd  [kernel]                 [k] e100_enable_irq
     0.00%       dd  [kernel]                 [k] yenta_interrupt
     0.00%       dd  [kernel]                 [k] __inet_lookup_established
     0.00%       dd  [kernel]                 [k] tcp_v4_rcv
     0.00%       dd  [kernel]                 [k] error_code
     0.00%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.00%     perf  [kernel]                 [k] restore_i387_xstate
     0.00%     perf  [kernel]                 [k] task_nice
     0.00%     perf  [kernel]                 [k] copy_process
     0.00%     perf  [kernel]                 [k] set_normalized_timespec
     0.00%     perf  [kernel]                 [k] __do_softirq
     0.00%     perf  [kernel]                 [k] remove_wait_queue
     0.00%     perf  [kernel]                 [k] add_wait_queue
     0.00%     perf  [kernel]                 [k] do_gettimeofday
     0.00%     perf  [kernel]                 [k] handle_IRQ_event
     0.00%     perf  [kernel]                 [k] __rcu_read_unlock
     0.00%     perf  [kernel]                 [k] perf_mmap_fault
     0.00%     perf  [kernel]                 [k] generic_segment_checks
     0.00%     perf  [kernel]                 [k] __page_cache_alloc
     0.00%     perf  [kernel]                 [k] should_remove_suid
     0.00%     perf  [kernel]                 [k] file_remove_suid
     0.00%     perf  [kernel]                 [k] wait_on_page_read
     0.00%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.00%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.00%     perf  [kernel]                 [k] set_page_dirty
     0.00%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.00%     perf  [kernel]                 [k] activate_page
     0.00%     perf  [kernel]                 [k] release_pages
     0.00%     perf  [kernel]                 [k] __lru_cache_add
     0.00%     perf  [kernel]                 [k] __pte_alloc
     0.00%     perf  [kernel]                 [k] handle_mm_fault
     0.00%     perf  [kernel]                 [k] page_add_file_rmap
     0.00%     perf  [kernel]                 [k] kmem_cache_alloc
     0.00%     perf  [kernel]                 [k] inode_add_bytes
     0.00%     perf  [kernel]                 [k] get_dumpable
     0.00%     perf  [kernel]                 [k] path_walk
     0.00%     perf  [kernel]                 [k] __pollwait
     0.00%     perf  [kernel]                 [k] file_update_time
     0.00%     perf  [kernel]                 [k] mnt_drop_write
     0.00%     perf  [kernel]                 [k] seq_read
     0.00%     perf  [kernel]                 [k] alloc_page_buffers
     0.00%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.00%     perf  [kernel]                 [k] __find_get_block_slow
     0.00%     perf  [kernel]                 [k] jfs_dirty_inode
     0.00%     perf  [kernel]                 [k] dbAllocDmap
     0.00%     perf  [kernel]                 [k] extHint
     0.00%     perf  [kernel]                 [k] txBeginAnon
     0.00%     perf  [kernel]                 [k] ea_release
     0.00%     perf  [kernel]                 [k] jfs_getxattr
     0.00%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.00%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.00%     perf  [kernel]                 [k] blk_update_request
     0.00%     perf  [kernel]                 [k] __make_request
     0.00%     perf  [kernel]                 [k] number
     0.00%     perf  [kernel]                 [k] strncmp
     0.00%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.00%     perf  [kernel]                 [k] down_read
     0.00%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.00%     perf  /lib/libpthread-2.11.so  [.] __fork
     0.00%     perf  /lib/libc-2.11.so        [.] __GI___libc_read
     0.00%     perf  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%     perf  /lib/ld-2.11.so          [.] strcmp
     0.00%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.00%     perf  /home/cloos/bin/perf     [.] hex2u64
     0.00%       dd  [kernel]                 [k] ptep_set_access_flags
     0.00%       dd  [kernel]                 [k] set_normalized_timespec
     0.00%       dd  [kernel]                 [k] mod_timer
     0.00%       dd  [kernel]                 [k] __queue_work
     0.00%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.00%       dd  [kernel]                 [k] ktime_get_real
     0.00%       dd  [kernel]                 [k] ktime_get_ts
     0.00%       dd  [kernel]                 [k] rcu_bh_qs
     0.00%       dd  [kernel]                 [k] __call_rcu
     0.00%       dd  [kernel]                 [k] call_rcu
     0.00%       dd  [kernel]                 [k] rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] perf_output_copy
     0.00%       dd  [kernel]                 [k] generic_segment_checks
     0.00%       dd  [kernel]                 [k] should_remove_suid
     0.00%       dd  [kernel]                 [k] file_read_actor
     0.00%       dd  [kernel]                 [k] find_get_pages_tag
     0.00%       dd  [kernel]                 [k] __generic_file_aio_write
     0.00%       dd  [kernel]                 [k] __writepage
     0.00%       dd  [kernel]                 [k] get_dirty_limits
     0.00%       dd  [kernel]                 [k] write_cache_pages
     0.00%       dd  [kernel]                 [k] __do_page_cache_readahead
     0.00%       dd  [kernel]                 [k] lru_add_drain
     0.00%       dd  [kernel]                 [k] vma_prio_tree_insert
     0.00%       dd  [kernel]                 [k] follow_page
     0.00%       dd  [kernel]                 [k] do_wp_page
     0.00%       dd  [kernel]                 [k] vm_get_page_prot
     0.00%       dd  [kernel]                 [k] vm_stat_account
     0.00%       dd  [kernel]                 [k] find_extend_vma
     0.00%       dd  [kernel]                 [k] vma_link
     0.00%       dd  [kernel]                 [k] page_add_file_rmap
     0.00%       dd  [kernel]                 [k] anon_vma_free
     0.00%       dd  [kernel]                 [k] kfree
     0.00%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.00%       dd  [kernel]                 [k] __kmalloc
     0.00%       dd  [kernel]                 [k] sys_close
     0.00%       dd  [kernel]                 [k] sys_read
     0.00%       dd  [kernel]                 [k] sys_write
     0.00%       dd  [kernel]                 [k] get_empty_filp
     0.00%       dd  [kernel]                 [k] fput
     0.00%       dd  [kernel]                 [k] cp_new_stat64
     0.00%       dd  [kernel]                 [k] flush_old_exec
     0.00%       dd  [kernel]                 [k] inode_permission
     0.00%       dd  [kernel]                 [k] path_init
     0.00%       dd  [kernel]                 [k] may_open
     0.00%       dd  [kernel]                 [k] do_filp_open
     0.00%       dd  [kernel]                 [k] dput
     0.00%       dd  [kernel]                 [k] generic_write_sync
     0.00%       dd  [kernel]                 [k] end_buffer_async_write
     0.00%       dd  [kernel]                 [k] __bio_add_page
     0.00%       dd  [kernel]                 [k] bio_alloc
     0.00%       dd  [kernel]                 [k] bio_fs_destructor
     0.00%       dd  [kernel]                 [k] fsnotify
     0.00%       dd  [kernel]                 [k] locks_remove_flock
     0.00%       dd  [kernel]                 [k] load_elf_binary
     0.00%       dd  [kernel]                 [k] metapage_get_blocks
     0.00%       dd  [kernel]                 [k] ea_get
     0.00%       dd  [kernel]                 [k] cap_d_instantiate
     0.00%       dd  [kernel]                 [k] elv_set_request
     0.00%       dd  [kernel]                 [k] submit_bio
     0.00%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.00%       dd  [kernel]                 [k] kobject_get
     0.00%       dd  [kernel]                 [k] kref_get
     0.00%       dd  [kernel]                 [k] prio_tree_replace
     0.00%       dd  [kernel]                 [k] prio_tree_remove
     0.00%       dd  [kernel]                 [k] prio_tree_insert
     0.00%       dd  [kernel]                 [k] prop_fraction_percpu
     0.00%       dd  [kernel]                 [k] radix_tree_gang_lookup_tag_slot
     0.00%       dd  [kernel]                 [k] __rb_rotate_right
     0.00%       dd  [kernel]                 [k] rb_erase
     0.00%       dd  [kernel]                 [k] strlcpy
     0.00%       dd  [kernel]                 [k] strlen
     0.00%       dd  [kernel]                 [k] _copy_from_user
     0.00%       dd  [kernel]                 [k] half_md4_transform
     0.00%       dd  [kernel]                 [k] get_random_int
     0.00%       dd  [kernel]                 [k] credit_entropy_bits
     0.00%       dd  [kernel]                 [k] add_disk_randomness
     0.00%       dd  [kernel]                 [k] extract_buf
     0.00%       dd  [kernel]                 [k] extract_entropy
     0.00%       dd  [kernel]                 [k] put_device
     0.00%       dd  [kernel]                 [k] scsi_log_send
     0.00%       dd  [kernel]                 [k] scsi_pool_free_command
     0.00%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.00%       dd  [kernel]                 [k] __scsi_release_buffers
     0.00%       dd  [kernel]                 [k] scsi_device_unbusy
     0.00%       dd  [kernel]                 [k] e100_tx_clean
     0.00%       dd  [kernel]                 [k] e100_watchdog
     0.00%       dd  [kernel]                 [k] irq_handler
     0.00%       dd  [kernel]                 [k] acpi_pm_read
     0.00%       dd  [kernel]                 [k] __alloc_skb
     0.00%       dd  [kernel]                 [k] __netdev_alloc_skb
     0.00%       dd  [kernel]                 [k] sk_run_filter
     0.00%       dd  [kernel]                 [k] ip_local_deliver
     0.00%       dd  [kernel]                 [k] tcp_send_ack
     0.00%       dd  [kernel]                 [k] preempt_schedule_irq
     0.00%       dd  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.00%       dd  /lib/librt-2.11.so       [.] 0x00000000006230
     0.00%       dd  /lib/libpthread-2.11.so  [.] abort@plt
     0.00%       dd  /lib/libpthread-2.11.so  [.] __libc_sigaction
     0.00%       dd  /lib/libc-2.11.so        [.] 0x00000000021f6c
     0.00%       dd  /lib/libc-2.11.so        [.] _IO_vfprintf_internal
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_strstr
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___libc_read
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___register_atfork
     0.00%       dd  /lib/libc-2.11.so        [.] 0x00000000115e76
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_cache_libcmp
     0.00%       dd  /lib/ld-2.11.so          [.] check_match.8476
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_new_object
     0.00%       dd  /lib/ld-2.11.so          [.] __open
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #6: perf5.report --]
[-- Type: text/plain, Size: 26764 bytes --]

# Samples: 15873
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    40.32%       dd  [kernel]                 [k] __copy_from_user_ll_nozero
    22.49%       dd  [kernel]                 [k] __clear_user
     3.06%       dd  [kernel]                 [k] get_page_from_freelist
     1.18%       dd  [kernel]                 [k] ext4_mark_iloc_dirty
     1.13%       dd  [kernel]                 [k] __set_page_dirty
     1.13%       dd  [kernel]                 [k] scsi_dispatch_cmd
     0.93%       dd  [kernel]                 [k] __block_prepare_write
     0.93%       dd  [kernel]                 [k] scsi_request_fn
     0.81%       dd  [kernel]                 [k] kmem_cache_alloc
     0.80%       dd  [kernel]                 [k] __ext4_get_inode_loc
     0.64%       dd  [kernel]                 [k] ext4_da_get_block_prep
     0.62%       dd  [kernel]                 [k] ext4_da_write_end
     0.62%       dd  [kernel]                 [k] jbd2_journal_stop
     0.57%       dd  [kernel]                 [k] add_to_page_cache_locked
     0.49%       dd  [kernel]                 [k] create_empty_buffers
     0.47%       dd  [kernel]                 [k] ext4_ext_calc_metadata_amount
     0.46%       dd  [kernel]                 [k] ext4_da_write_begin
     0.45%       dd  [kernel]                 [k] iov_iter_fault_in_readable
     0.45%       dd  [kernel]                 [k] start_this_handle
     0.43%     perf  [kernel]                 [k] generic_file_buffered_write
     0.41%       dd  [kernel]                 [k] __find_get_block
     0.36%       dd  [kernel]                 [k] __wake_up_bit
     0.34%       dd  [kernel]                 [k] __alloc_pages_nodemask
     0.34%       dd  [kernel]                 [k] __block_commit_write
     0.32%       dd  [kernel]                 [k] generic_file_buffered_write
     0.32%       dd  [kernel]                 [k] jbd2_journal_start
     0.32%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.30%       dd  [kernel]                 [k] ext4_get_blocks
     0.29%     perf  [kernel]                 [k] number
     0.28%     perf  [kernel]                 [k] vsnprintf
     0.28%       dd  [kernel]                 [k] do_get_write_access
     0.28%       dd  [kernel]                 [k] blk_end_bidi_request
     0.27%       dd  [kernel]                 [k] __mark_inode_dirty
     0.26%     perf  [kernel]                 [k] __generic_file_aio_write
     0.24%       dd  [kernel]                 [k] ext4_ext_get_blocks
     0.23%     perf  [kernel]                 [k] nobh_write_begin
     0.23%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.23%       dd  [kernel]                 [k] generic_write_end
     0.23%       dd  [kernel]                 [k] ext4_journal_start_sb
     0.23%       dd  [kernel]                 [k] jbd2_journal_add_journal_head
     0.22%       dd  [kernel]                 [k] find_get_page
     0.22%       dd  [kernel]                 [k] kmem_cache_free
     0.22%       dd  [kernel]                 [k] ata_sff_interrupt
     0.21%       dd  [kernel]                 [k] mark_page_accessed
     0.21%       dd  [kernel]                 [k] __ext4_journal_stop
     0.21%       dd  [kernel]                 [k] T.1339
     0.21%       dd  [kernel]                 [k] handle_mm_fault
     0.20%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.20%     perf  [kernel]                 [k] extAlloc
     0.20%     perf  [kernel]                 [k] format_decode
     0.20%       dd  [kernel]                 [k] alloc_page_buffers
     0.19%       dd  [kernel]                 [k] unlock_page
     0.18%     perf  [kernel]                 [k] T.1339
     0.17%     perf  [kernel]                 [k] nobh_write_end
     0.17%       dd  [kernel]                 [k] block_write_end
     0.17%       dd  [kernel]                 [k] read_zero
     0.16%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.16%       dd  [kernel]                 [k] __wake_up
     0.16%       dd  [kernel]                 [k] wake_up_bit
     0.16%       dd  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.16%       dd  [kernel]                 [k] __lru_cache_add
     0.16%     perf  [kernel]                 [k] dbAlloc
     0.16%       dd  [kernel]                 [k] mark_buffer_dirty
     0.16%       dd  [kernel]                 [k] __find_get_block_slow
     0.15%     perf  [kernel]                 [k] do_page_fault
     0.15%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.15%       dd  [kernel]                 [k] grab_cache_page_write_begin
     0.15%       dd  [kernel]                 [k] ext4_dirty_inode
     0.14%     perf  [kernel]                 [k] __jfs_getxattr
     0.14%     perf  [kernel]                 [k] find_get_page
     0.14%     perf  [kernel]                 [k] generic_file_aio_write
     0.14%     perf  [kernel]                 [k] get_page_from_freelist
     0.13%       dd  [kernel]                 [k] __rcu_read_unlock
     0.13%       dd  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.13%       dd  [kernel]                 [k] ext4_has_free_blocks
     0.13%       dd  [kernel]                 [k] ext4_inode_table
     0.13%     perf  [kernel]                 [k] vfs_write
     0.13%     perf  [kernel]                 [k] __mark_inode_dirty
     0.12%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.12%     perf  [kernel]                 [k] copy_page_range
     0.12%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.11%     perf  [kernel]                 [k] do_sync_write
     0.11%       dd  [kernel]                 [k] put_page
     0.11%       dd  [kernel]                 [k] ext4_claim_free_blocks
     0.11%       dd  [kernel]                 [k] ext4_mark_inode_dirty
     0.11%     perf  [kernel]                 [k] sysenter_past_esp
     0.11%     perf  [kernel]                 [k] iov_iter_advance
     0.11%       dd  [kernel]                 [k] up_read
     0.11%       dd  [kernel]                 [k] rcu_process_gp_end
     0.11%       dd  [kernel]                 [k] ext4_get_group_desc
     0.11%       dd  [kernel]                 [k] ext4_get_inode_flags
     0.11%       dd  [kernel]                 [k] __ext4_journal_get_write_access
     0.11%       dd  [kernel]                 [k] jbd2_journal_dirty_metadata
     0.10%     perf  [kernel]                 [k] getnstimeofday
     0.10%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.10%     perf  [kernel]                 [k] rw_verify_area
     0.10%     perf  [kernel]                 [k] jfs_write_begin
     0.10%     perf  [kernel]                 [k] acpi_pm_read
     0.10%       dd  [kernel]                 [k] check_for_new_grace_period
     0.09%     perf  [kernel]                 [k] __set_page_dirty
     0.09%     perf  [kernel]                 [k] xtSearch
     0.09%       dd  [kernel]                 [k] ext4_reserve_inode_write
     0.09%       dd  [kernel]                 [k] blk_update_request
     0.09%       dd  [kernel]                 [k] down_read
     0.09%       dd  [kernel]                 [k] do_page_fault
     0.09%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.09%       dd  [kernel]                 [k] __do_softirq
     0.09%       dd  [kernel]                 [k] __rcu_read_lock
     0.09%       dd  [kernel]                 [k] iov_iter_advance
     0.09%       dd  [kernel]                 [k] find_lock_page
     0.09%       dd  [kernel]                 [k] zone_watermark_ok
     0.09%       dd  [kernel]                 [k] next_zones_zonelist
     0.08%     perf  [vdso]                   [.] 0x000000ffffe424
     0.08%     perf  [kernel]                 [k] __do_fault
     0.08%     perf  [kernel]                 [k] scsi_request_fn
     0.08%       dd  [kernel]                 [k] test_clear_page_writeback
     0.08%       dd  [kernel]                 [k] block_write_begin
     0.08%       dd  [kernel]                 [k] jbd2_journal_get_write_access
     0.08%     perf  [kernel]                 [k] generic_segment_checks
     0.08%     perf  [kernel]                 [k] mark_page_accessed
     0.08%     perf  [kernel]                 [k] ea_get
     0.07%     perf  [kernel]                 [k] sys_gettimeofday
     0.07%     perf  [kernel]                 [k] __rcu_read_unlock
     0.07%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.07%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.07%     perf  [kernel]                 [k] xtLookup
     0.07%     perf  [kernel]                 [k] dbAllocBits
     0.07%     perf  [kernel]                 [k] dbAllocNext
     0.07%       dd  [kernel]                 [k] handle_IRQ_event
     0.07%       dd  [kernel]                 [k] unmap_vmas
     0.07%       dd  [kernel]                 [k] set_bh_page
     0.07%       dd  [kernel]                 [k] end_buffer_async_write
     0.07%       dd  [kernel]                 [k] radix_tree_preload
     0.06%     perf  [kernel]                 [k] file_update_time
     0.06%     perf  [kernel]                 [k] __get_metapage
     0.06%       dd  [kernel]                 [k] release_pages
     0.06%       dd  [kernel]                 [k] __getblk
     0.06%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.06%     perf  [kernel]                 [k] sys_write
     0.06%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.06%       dd  [kernel]                 [k] down_read_trylock
     0.06%       dd  [kernel]                 [k] alloc_buffer_head
     0.06%       dd  [kernel]                 [k] jbd2_journal_cancel_revoke
     0.06%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.05%     perf  [kernel]                 [k] file_remove_suid
     0.05%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.05%     perf  [kernel]                 [k] put_page
     0.05%     perf  [kernel]                 [k] kfree
     0.05%     perf  [kernel]                 [k] jfs_get_block
     0.05%     perf  [kernel]                 [k] xtExtend
     0.05%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.05%     perf  [kernel]                 [k] copy_to_user
     0.05%       dd  [kernel]                 [k] __page_cache_alloc
     0.05%       dd  [kernel]                 [k] __jbd2_log_space_left
     0.05%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.04%     perf  [kernel]                 [k] find_lock_page
     0.04%     perf  [kernel]                 [k] do_wp_page
     0.04%     perf  [kernel]                 [k] fget_light
     0.04%     perf  [kernel]                 [k] fsnotify
     0.04%     perf  [kernel]                 [k] extHint
     0.04%     perf  [kernel]                 [k] strchr
     0.04%     perf  [kernel]                 [k] schedule
     0.04%       dd  [kernel]                 [k] __ext4_handle_dirty_metadata
     0.04%       dd  [kernel]                 [k] spin_unlock
     0.04%       dd  [kernel]                 [k] jbd2_journal_put_journal_head
     0.04%       dd  [kernel]                 [k] scsi_finish_command
     0.04%       dd  [kernel]                 [k] _cond_resched
     0.04%     perf  [kernel]                 [k] set_page_dirty
     0.04%     perf  [kernel]                 [k] handle_mm_fault
     0.04%     perf  [kernel]                 [k] kmem_cache_alloc
     0.04%     perf  [kernel]                 [k] seq_printf
     0.04%     perf  [kernel]                 [k] generic_write_sync
     0.04%     perf  [kernel]                 [k] jfs_dirty_inode
     0.04%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.04%       dd  [kernel]                 [k] free_hot_cold_page
     0.04%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.03%     perf  [kernel]                 [k] should_remove_suid
     0.03%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.03%     perf  [kernel]                 [k] test_clear_page_writeback
     0.03%     perf  [kernel]                 [k] kmem_cache_free
     0.03%     perf  [kernel]                 [k] prepend
     0.03%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.03%     perf  [kernel]                 [k] dbAllocDmap
     0.03%     perf  [kernel]                 [k] ea_release
     0.03%     perf  [kernel]                 [k] mutex_lock
     0.03%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.03%       dd  [kernel]                 [k] add_timer_randomness
     0.03%       dd  [kernel]                 [k] scsi_put_command
     0.03%       dd  [kernel]                 [k] scsi_run_queue
     0.03%     perf  [kernel]                 [k] __do_softirq
     0.03%     perf  [kernel]                 [k] __wake_up_bit
     0.03%     perf  [kernel]                 [k] current_kernel_time
     0.03%     perf  [kernel]                 [k] do_gettimeofday
     0.03%     perf  [kernel]                 [k] __rcu_read_lock
     0.03%     perf  [kernel]                 [k] zone_watermark_ok
     0.03%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.03%     perf  [kernel]                 [k] __lru_cache_add
     0.03%     perf  [kernel]                 [k] find_vma
     0.03%     perf  [kernel]                 [k] inode_add_bytes
     0.03%     perf  [kernel]                 [k] __fsnotify_parent
     0.03%     perf  [kernel]                 [k] dbAdjTree
     0.03%     perf  [kernel]                 [k] spin_unlock
     0.03%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.03%     perf  [kernel]                 [k] radix_tree_preload
     0.03%     perf  [kernel]                 [k] down_read
     0.03%       dd  [kernel]                 [k] find_vma
     0.03%       dd  [kernel]                 [k] unmap_underlying_metadata
     0.03%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.03%       dd  [kernel]                 [k] scsi_softirq_done
     0.02%     perf  [kernel]                 [k] __wake_up
     0.02%     perf  [kernel]                 [k] up_read
     0.02%     perf  [kernel]                 [k] perf_mmap_fault
     0.02%     perf  [kernel]                 [k] unlock_page
     0.02%     perf  [kernel]                 [k] vm_normal_page
     0.02%     perf  [kernel]                 [k] mnt_clone_write
     0.02%     perf  [kernel]                 [k] alloc_page_buffers
     0.02%     perf  [kernel]                 [k] jfs_getxattr
     0.02%     perf  [kernel]                 [k] cap_file_permission
     0.02%     perf  [kernel]                 [k] blk_end_bidi_request
     0.02%     perf  [kernel]                 [k] __make_request
     0.02%     perf  [kernel]                 [k] _atomic_dec_and_lock
     0.02%     perf  [kernel]                 [k] put_dec
     0.02%     perf  [kernel]                 [k] strncmp
     0.02%     perf  [kernel]                 [k] strlen
     0.02%     perf  [kernel]                 [k] mutex_unlock
     0.02%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.02%       dd  [kernel]                 [k] __do_fault
     0.02%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.02%       dd  [kernel]                 [k] end_bio_bh_io_sync
     0.02%       dd  [kernel]                 [k] ext4_get_inode_loc
     0.02%       dd  [kernel]                 [k] req_bio_endio
     0.02%       dd  [kernel]                 [k] blk_update_bidi_request
     0.02%       dd  [kernel]                 [k] __make_request
     0.02%       dd  [kernel]                 [k] blk_done_softirq
     0.02%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.02%       dd  [kernel]                 [k] kobject_get
     0.02%       dd  [kernel]                 [k] scsi_io_completion
     0.01%     perf  [kernel]                 [k] timespec_trunc
     0.01%     perf  [kernel]                 [k] current_fs_time
     0.01%     perf  [kernel]                 [k] hrtimer_try_to_cancel
     0.01%     perf  [kernel]                 [k] rcu_sched_qs
     0.01%     perf  [kernel]                 [k] __page_cache_alloc
     0.01%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.01%     perf  [kernel]                 [k] read_cache_page_async
     0.01%     perf  [kernel]                 [k] filemap_fault
     0.01%     perf  [kernel]                 [k] poll_freewait
     0.01%     perf  [kernel]                 [k] __d_path
     0.01%     perf  [kernel]                 [k] mangle_path
     0.01%     perf  [kernel]                 [k] __find_get_block_slow
     0.01%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.01%     perf  [kernel]                 [k] show_map_vma
     0.01%     perf  [kernel]                 [k] dbSplit
     0.01%     perf  [kernel]                 [k] txLock
     0.01%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.01%     perf  [kernel]                 [k] put_dec_trunc
     0.01%     perf  [kernel]                 [k] scsi_pool_free_command
     0.01%     perf  [kernel]                 [k] ata_sff_interrupt
     0.01%     perf  [kernel]                 [k] _cond_resched
     0.01%     perf  [kernel]                 [k] preempt_schedule
     0.01%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.01%       dd  [kernel]                 [k] try_to_wake_up
     0.01%       dd  [kernel]                 [k] mempool_free
     0.01%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.01%       dd  [kernel]                 [k] page_evictable
     0.01%       dd  [kernel]                 [k] vm_normal_page
     0.01%       dd  [kernel]                 [k] do_mmap_pgoff
     0.01%       dd  [kernel]                 [k] fget_light
     0.01%       dd  [kernel]                 [k] file_update_time
     0.01%       dd  [kernel]                 [k] mnt_want_write_file
     0.01%       dd  [kernel]                 [k] __brelse
     0.01%       dd  [kernel]                 [k] unlock_buffer
     0.01%       dd  [kernel]                 [k] bio_free
     0.01%       dd  [kernel]                 [k] kobject_put
     0.01%       dd  [kernel]                 [k] kref_get
     0.01%       dd  [kernel]                 [k] credit_entropy_bits
     0.01%       dd  [kernel]                 [k] __scsi_release_buffers
     0.01%       dd  [kernel]                 [k] scsi_device_unbusy
     0.01%       dd  [kernel]                 [k] yenta_interrupt
     0.01%       dd  [kernel]                 [k] netif_receive_skb
     0.01%       dd  /lib/libc-2.11.so        [.] __GI___printf_fp
     0.01%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_lookup_symbol_x
     0.01%       dd  /bin/dd                  [.] dd_copy
     0.01%       dd                  804fa9d  [.] 0x0000000804fa9d
     0.01%     perf  [kernel]                 [k] sysenter_do_call
     0.01%     perf  [kernel]                 [k] dup_mm
     0.01%     perf  [kernel]                 [k] get_pid_task
     0.01%     perf  [kernel]                 [k] down_read_trylock
     0.01%     perf  [kernel]                 [k] module_put
     0.01%     perf  [kernel]                 [k] do_read_cache_page
     0.01%     perf  [kernel]                 [k] page_evictable
     0.01%     perf  [kernel]                 [k] next_zones_zonelist
     0.01%     perf  [kernel]                 [k] lock_page
     0.01%     perf  [kernel]                 [k] __vma_link_rb
     0.01%     perf  [kernel]                 [k] __vm_enough_memory
     0.01%     perf  [kernel]                 [k] __anon_vma_link
     0.01%     perf  [kernel]                 [k] page_add_file_rmap
     0.01%     perf  [kernel]                 [k] page_remove_rmap
     0.01%     perf  [kernel]                 [k] anon_vma_unlink
     0.01%     perf  [kernel]                 [k] kmem_cache_alloc_notrace
     0.01%     perf  [kernel]                 [k] do_sys_open
     0.01%     perf  [kernel]                 [k] sys_read
     0.01%     perf  [kernel]                 [k] generic_fillattr
     0.01%     perf  [kernel]                 [k] acl_permission_check
     0.01%     perf  [kernel]                 [k] path_walk
     0.01%     perf  [kernel]                 [k] do_sys_poll
     0.01%     perf  [kernel]                 [k] d_path
     0.01%     perf  [kernel]                 [k] __d_lookup
     0.01%     perf  [kernel]                 [k] dput
     0.01%     perf  [kernel]                 [k] dup_fd
     0.01%     perf  [kernel]                 [k] mnt_drop_write
     0.01%     perf  [kernel]                 [k] mnt_want_write_file
     0.01%     perf  [kernel]                 [k] free_buffer_head
     0.01%     perf  [kernel]                 [k] bio_put
     0.01%     perf  [kernel]                 [k] m_next
     0.01%     perf  [kernel]                 [k] show_map
     0.01%     perf  [kernel]                 [k] m_start
     0.01%     perf  [kernel]                 [k] proc_lookup_de
     0.01%     perf  [kernel]                 [k] proc_pid_status
     0.01%     perf  [kernel]                 [k] release_metapage
     0.01%     perf  [kernel]                 [k] txBeginAnon
     0.01%     perf  [kernel]                 [k] security_file_permission
     0.01%     perf  [kernel]                 [k] security_inode_getattr
     0.01%     perf  [kernel]                 [k] cap_task_to_inode
     0.01%     perf  [kernel]                 [k] blk_update_request
     0.01%     perf  [kernel]                 [k] get_request_wait
     0.01%     perf  [kernel]                 [k] rb_insert_color
     0.01%     perf  [kernel]                 [k] string
     0.01%     perf  [kernel]                 [k] strnlen
     0.01%     perf  [kernel]                 [k] mix_pool_bytes_extract
     0.01%     perf  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.01%     perf  [kernel]                 [k] scsi_decide_disposition
     0.01%     perf  [kernel]                 [k] scsi_init_cmd_errh
     0.01%     perf  [kernel]                 [k] scsi_io_completion
     0.01%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.01%     perf  /lib/libc-2.11.so        [.] _IO_getline_internal
     0.01%     perf  /lib/libc-2.11.so        [.] _IO_default_xsputn_internal
     0.01%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.01%       dd  [kernel]                 [k] do_group_exit
     0.01%       dd  [kernel]                 [k] __queue_work
     0.01%       dd  [kernel]                 [k] prepare_creds
     0.01%       dd  [kernel]                 [k] commit_creds
     0.01%       dd  [kernel]                 [k] rcu_bh_qs
     0.01%       dd  [kernel]                 [k] __generic_file_aio_write
     0.01%       dd  [kernel]                 [k] do_wp_page
     0.01%       dd  [kernel]                 [k] __vm_enough_memory
     0.01%       dd  [kernel]                 [k] vma_adjust
     0.01%       dd  [kernel]                 [k] spin_unlock
     0.01%       dd  [kernel]                 [k] anon_vma_prepare
     0.01%       dd  [kernel]                 [k] rw_verify_area
     0.01%       dd  [kernel]                 [k] file_free_rcu
     0.01%       dd  [kernel]                 [k] __fput
     0.01%       dd  [kernel]                 [k] path_get
     0.01%       dd  [kernel]                 [k] __d_lookup
     0.01%       dd  [kernel]                 [k] submit_bh
     0.01%       dd  [kernel]                 [k] bio_put
     0.01%       dd  [kernel]                 [k] bio_fs_destructor
     0.01%       dd  [kernel]                 [k] fsnotify
     0.01%       dd  [kernel]                 [k] inotify_inode_queue_event
     0.01%       dd  [kernel]                 [k] ext4_check_dir_entry
     0.01%       dd  [kernel]                 [k] ext4_find_entry
     0.01%       dd  [kernel]                 [k] dtSearch
     0.01%       dd  [kernel]                 [k] cap_inode_need_killpriv
     0.01%       dd  [kernel]                 [k] get_vfs_caps_from_disk
     0.01%       dd  [kernel]                 [k] blk_end_request
     0.01%       dd  [kernel]                 [k] blk_run_queue
     0.01%       dd  [kernel]                 [k] kref_put
     0.01%       dd  [kernel]                 [k] rb_insert_color
     0.01%       dd  [kernel]                 [k] sha_transform
     0.01%       dd  [kernel]                 [k] csum_partial
     0.01%       dd  [kernel]                 [k] strnlen_user
     0.01%       dd  [kernel]                 [k] _copy_from_user
     0.01%       dd  [kernel]                 [k] __sg_free_table
     0.01%       dd  [kernel]                 [k] add_disk_randomness
     0.01%       dd  [kernel]                 [k] n_tty_write
     0.01%       dd  [kernel]                 [k] get_device
     0.01%       dd  [kernel]                 [k] scsi_log_completion
     0.01%       dd  [kernel]                 [k] scsi_pool_free_command
     0.01%       dd  [kernel]                 [k] scsi_decide_disposition
     0.01%       dd  [kernel]                 [k] scsi_sg_free
     0.01%       dd  [kernel]                 [k] __alloc_skb
     0.01%       dd  [kernel]                 [k] tcp_v4_rcv
     0.01%       dd  [kernel]                 [k] schedule
     0.01%       dd  [kernel]                 [k] preempt_schedule
     0.01%       dd  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.01%       dd  /lib/libc-2.11.so        [.] __GI_getenv
     0.01%       dd  /lib/libc-2.11.so        [.] _IO_new_do_write
     0.01%       dd  /lib/libc-2.11.so        [.] __GI_strcmp
     0.01%       dd  /lib/libc-2.11.so        [.] __GI___libc_read
     0.01%       dd  /lib/ld-2.11.so          [.] security_init
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_map_object
     0.01%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_relocate_object
     0.01%       dd  /lib/ld-2.11.so          [.] 0x0000000000baa4
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_fixup
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_name_match_p
     0.01%       dd  /bin/dd                  [.] print_stats
     0.01%       dd  /bin/dd                  [.] iread
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #7: perf6.report --]
[-- Type: text/plain, Size: 32231 bytes --]

# Samples: 26633
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    19.21%       dd  [kernel]                 [k] __copy_from_user_ll
    11.34%       dd  [kernel]                 [k] __clear_user
    10.06%       dd  [kernel]                 [k] get_page_from_freelist
     7.95%       dd  [kernel]                 [k] __set_page_dirty_nobuffers
     3.15%       dd  [kernel]                 [k] add_to_page_cache_locked
     2.47%       dd  [kernel]                 [k] __alloc_pages_nodemask
     1.67%     perf  [kernel]                 [k] generic_file_buffered_write
     1.33%       dd  [kernel]                 [k] ____pagevec_lru_add
     1.32%       dd  [kernel]                 [k] prepare_pages
     1.30%       dd  [kernel]                 [k] zone_watermark_ok
     1.26%       dd  [kernel]                 [k] find_get_page
     1.22%       dd  [kernel]                 [k] find_or_create_page
     1.19%       dd  [kernel]                 [k] btrfs_copy_from_user
     1.05%       dd  [kernel]                 [k] __lru_cache_add
     0.95%     perf  [kernel]                 [k] __generic_file_aio_write
     0.87%       dd  [kernel]                 [k] read_zero
     0.86%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.82%       dd  [kernel]                 [k] next_zones_zonelist
     0.79%       dd  [kernel]                 [k] page_mkclean
     0.75%       dd  [kernel]                 [k] put_page
     0.67%       dd  [kernel]                 [k] __rcu_read_unlock
     0.66%     perf  [kernel]                 [k] T.1339
     0.65%       dd  [kernel]                 [k] T.1339
     0.63%       dd  [kernel]                 [k] btrfs_drop_pages
     0.62%     perf  [kernel]                 [k] iov_iter_advance
     0.61%       dd  [kernel]                 [k] find_lock_page
     0.59%     perf  [kernel]                 [k] do_sync_write
     0.58%     perf  [kernel]                 [k] nobh_write_end
     0.57%     perf  [kernel]                 [k] generic_file_aio_write
     0.57%     perf  [kernel]                 [k] vfs_write
     0.56%       dd  [kernel]                 [k] clear_page_dirty_for_io
     0.55%     perf  [kernel]                 [k] __jfs_getxattr
     0.54%       dd  [kernel]                 [k] __mark_inode_dirty
     0.53%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.53%     perf  [kernel]                 [k] nobh_write_begin
     0.50%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.50%       dd  [kernel]                 [k] radix_tree_preload
     0.48%     perf  [kernel]                 [k] sysenter_past_esp
     0.47%     perf  [kernel]                 [k] acpi_pm_read
     0.45%       dd  [kernel]                 [k] unlock_page
     0.44%     perf  [kernel]                 [k] find_get_page
     0.44%       dd  [kernel]                 [k] do_page_fault
     0.41%     perf  [kernel]                 [k] sys_write
     0.39%     perf  [kernel]                 [k] getnstimeofday
     0.38%     perf  [kernel]                 [k] generic_segment_checks
     0.38%       dd  [kernel]                 [k] _cond_resched
     0.37%     perf  [kernel]                 [k] rw_verify_area
     0.35%     perf  [kernel]                 [k] jfs_write_begin
     0.33%     perf  [kernel]                 [k] copy_to_user
     0.31%     perf  [kernel]                 [k] ea_get
     0.31%       dd  [kernel]                 [k] mark_page_accessed
     0.30%       dd  [kernel]                 [k] btrfs_set_page_dirty
     0.30%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.29%       dd  [kernel]                 [k] set_page_extent_mapped
     0.28%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.27%       dd  [kernel]                 [k] __rcu_read_lock
     0.27%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.25%       dd  [kernel]                 [k] __wake_up_bit
     0.25%       dd  [kernel]                 [k] schedule
     0.24%       dd  [kernel]                 [k] handle_mm_fault
     0.24%       dd  [kernel]                 [k] set_page_dirty
     0.24%       dd  [kernel]                 [k] unmap_vmas
     0.23%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.23%     perf  [vdso]                   [.] 0x000000ffffe424
     0.21%     perf  [kernel]                 [k] sys_gettimeofday
     0.21%     perf  [kernel]                 [k] __mark_inode_dirty
     0.21%       dd  [kernel]                 [k] dirty_and_release_pages
     0.19%       dd  [kernel]                 [k] free_hot_cold_page
     0.18%     perf  [kernel]                 [k] __rcu_read_unlock
     0.18%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.17%     perf  [kernel]                 [k] kfree
     0.17%     perf  [kernel]                 [k] number
     0.17%       dd  [kernel]                 [k] map_private_extent_buffer
     0.15%     perf  [kernel]                 [k] fsnotify
     0.15%     perf  [kernel]                 [k] extAlloc
     0.15%     perf  [kernel]                 [k] jfs_getxattr
     0.15%     perf  [kernel]                 [k] put_page
     0.15%     perf  [kernel]                 [k] fget_light
     0.15%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.15%       dd  [kernel]                 [k] sha_transform
     0.15%       dd  [kernel]                 [k] rcu_sched_qs
     0.14%     perf  [kernel]                 [k] current_fs_time
     0.14%     perf  [kernel]                 [k] do_gettimeofday
     0.14%     perf  [kernel]                 [k] file_update_time
     0.14%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.14%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.14%     perf  [kernel]                 [k] vsnprintf
     0.13%       dd  [kernel]                 [k] read_extent_buffer
     0.12%     perf  [kernel]                 [k] __set_page_dirty
     0.12%       dd  [kernel]                 [k] generic_bin_search
     0.12%     perf  [kernel]                 [k] find_lock_page
     0.12%     perf  [kernel]                 [k] strlen
     0.11%     perf  [kernel]                 [k] timespec_trunc
     0.11%     perf  [kernel]                 [k] file_remove_suid
     0.11%     perf  [kernel]                 [k] __fsnotify_parent
     0.11%     perf  [kernel]                 [k] schedule
     0.11%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.11%     perf  [kernel]                 [k] generic_write_sync
     0.11%     perf  [kernel]                 [k] strncmp
     0.11%     perf  [kernel]                 [k] xtSearch
     0.11%     perf  [kernel]                 [k] dbAlloc
     0.10%     perf  [kernel]                 [k] should_remove_suid
     0.10%     perf  [kernel]                 [k] ea_release
     0.09%     perf  [kernel]                 [k] xtLookup
     0.09%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.09%     perf  [kernel]                 [k] _cond_resched
     0.09%     perf  [kernel]                 [k] get_page_from_freelist
     0.09%       dd  [kernel]                 [k] __do_softirq
     0.09%     perf  [kernel]                 [k] __wake_up_bit
     0.09%     perf  [kernel]                 [k] copy_page_range
     0.09%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.08%       dd  [kernel]                 [k] scsi_dispatch_cmd
     0.08%     perf  [kernel]                 [k] jfs_dirty_inode
     0.08%     perf  [kernel]                 [k] dbAllocBits
     0.08%     perf  [kernel]                 [k] format_decode
     0.08%       dd  [kernel]                 [k] map_extent_buffer
     0.08%     perf  [kernel]                 [k] current_kernel_time
     0.08%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.08%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.08%       dd  [kernel]                 [k] btrfs_search_slot
     0.08%       dd  [kernel]                 [k] blk_end_bidi_request
     0.07%     perf  [kernel]                 [k] __rcu_read_lock
     0.07%     perf  [kernel]                 [k] set_page_dirty
     0.07%     perf  [kernel]                 [k] down_read
     0.07%     perf  [kernel]                 [k] unlock_page
     0.07%     perf  [kernel]                 [k] mark_page_accessed
     0.07%     perf  [kernel]                 [k] mutex_lock
     0.07%       dd  [kernel]                 [k] __do_fault
     0.07%       dd  [kernel]                 [k] kmem_cache_alloc
     0.06%       dd  [kernel]                 [k] rcu_process_gp_end
     0.06%       dd  [kernel]                 [k] check_for_new_grace_period
     0.06%     perf  [kernel]                 [k] __do_fault
     0.06%     perf  [kernel]                 [k] cap_file_permission
     0.06%       dd  [kernel]                 [k] filemap_fault
     0.06%       dd  [kernel]                 [k] find_vma
     0.06%       dd  [kernel]                 [k] btrfs_key_generation
     0.05%     perf  [kernel]                 [k] sysenter_do_call
     0.05%     perf  [kernel]                 [k] extHint
     0.05%     perf  [kernel]                 [k] do_page_fault
     0.05%       dd  [kernel]                 [k] ata_sff_interrupt
     0.05%     perf  [kernel]                 [k] up_read
     0.05%     perf  [kernel]                 [k] mnt_clone_write
     0.05%     perf  [kernel]                 [k] jfs_get_block
     0.05%     perf  [kernel]                 [k] xtExtend
     0.05%       dd  [kernel]                 [k] release_pages
     0.05%       dd  [kernel]                 [k] vm_normal_page
     0.05%       dd  [kernel]                 [k] btrfs_file_write
     0.05%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.05%     perf  [kernel]                 [k] security_file_permission
     0.05%       dd  [kernel]                 [k] anon_vma_prepare
     0.04%     perf  [kernel]                 [k] mutex_unlock
     0.04%       dd  [kernel]                 [k] scsi_request_fn
     0.03%     perf  [kernel]                 [k] seq_printf
     0.03%     perf  [kernel]                 [k] dbAllocNext
     0.03%     perf  [kernel]                 [k] __get_metapage
     0.03%       dd  [kernel]                 [k] free_hot_page
     0.03%       dd  [kernel]                 [k] page_remove_rmap
     0.03%       dd  [kernel]                 [k] verify_parent_transid
     0.03%       dd  [kernel]                 [k] btrfs_key_blockptr
     0.03%       dd  [kernel]                 [k] btrfs_queue_worker
     0.03%     perf  [kernel]                 [k] __wake_up
     0.03%     perf  [kernel]                 [k] mangle_path
     0.03%     perf  [kernel]                 [k] dbAllocDmap
     0.03%       dd  [kernel]                 [k] up_read
     0.03%       dd  [kernel]                 [k] handle_IRQ_event
     0.03%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.03%       dd  [kernel]                 [k] copy_strings
     0.03%       dd  [kernel]                 [k] extent_buffer_uptodate
     0.03%     perf  [kernel]                 [k] __do_softirq
     0.03%     perf  [kernel]                 [k] rcu_sched_qs
     0.03%     perf  [kernel]                 [k] do_sys_poll
     0.03%     perf  [kernel]                 [k] mnt_drop_write
     0.03%     perf  [kernel]                 [k] dbSplit
     0.03%     perf  [kernel]                 [k] strchr
     0.03%       dd  [kernel]                 [k] btrfs_clear_path_blocking
     0.03%       dd  [kernel]                 [k] test_range_bit
     0.02%     perf  [kernel]                 [k] perf_mmap_fault
     0.02%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.02%     perf  [kernel]                 [k] read_cache_page_async
     0.02%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.02%     perf  [kernel]                 [k] kmem_cache_alloc
     0.02%     perf  [kernel]                 [k] __d_path
     0.02%     perf  [kernel]                 [k] dbAdjTree
     0.02%     perf  [kernel]                 [k] release_metapage
     0.02%       dd  [kernel]                 [k] __wake_up
     0.02%       dd  [kernel]                 [k] down_read_trylock
     0.02%       dd  [kernel]                 [k] lru_cache_add_lru
     0.02%       dd  [kernel]                 [k] page_evictable
     0.02%       dd  [kernel]                 [k] link_path_walk
     0.02%       dd  [kernel]                 [k] read_block_for_search
     0.02%       dd  [kernel]                 [k] find_extent_buffer
     0.02%       dd  [kernel]                 [k] blk_update_request
     0.02%       dd  [kernel]                 [k] preempt_schedule
     0.02%     perf  [kernel]                 [k] read_cache_page
     0.02%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.02%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.02%       dd  [kernel]                 [k] try_to_wake_up
     0.02%       dd  [kernel]                 [k] btrfs_header_nritems
     0.02%       dd  [kernel]                 [k] merge_state
     0.02%       dd  [kernel]                 [k] unmap_extent_buffer
     0.02%       dd  [kernel]                 [k] __make_request
     0.02%     perf  [kernel]                 [k] __lru_cache_add
     0.02%     perf  [kernel]                 [k] handle_mm_fault
     0.02%     perf  [kernel]                 [k] kmem_cache_free
     0.02%     perf  [kernel]                 [k] inode_add_bytes
     0.02%     perf  [kernel]                 [k] alloc_buffer_head
     0.02%     perf  [kernel]                 [k] alloc_page_buffers
     0.02%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.02%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.02%       dd  [kernel]                 [k] __call_rcu
     0.02%       dd  [kernel]                 [k] unlink_file_vma
     0.02%       dd  [kernel]                 [k] mmap_region
     0.02%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.02%       dd  [kernel]                 [k] load_elf_binary
     0.02%       dd  [kernel]                 [k] __btrfs_cow_block
     0.02%       dd  [kernel]                 [k] btrfs_buffer_uptodate
     0.02%       dd  [kernel]                 [k] insert_state
     0.02%       dd  [kernel]                 [k] btrfs_tree_unlock
     0.02%       dd  [kernel]                 [k] blk_update_bidi_request
     0.02%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.02%       dd  [kernel]                 [k] scsi_run_queue
     0.02%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.02%       dd  /bin/dd                  [.] dd_copy
     0.02%       dd                 b70ef45f  [.] 0x000000b70ef45f
     0.01%     perf  [kernel]                 [k] do_read_cache_page
     0.01%     perf  [kernel]                 [k] filemap_fault
     0.01%     perf  [kernel]                 [k] next_zones_zonelist
     0.01%     perf  [kernel]                 [k] anon_vma_link
     0.01%     perf  [kernel]                 [k] prepend
     0.01%     perf  [kernel]                 [k] free_buffer_head
     0.01%     perf  [kernel]                 [k] __find_get_block_slow
     0.01%     perf  [kernel]                 [k] txBeginAnon
     0.01%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.01%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.01%       dd  [kernel]                 [k] mprotect_fixup
     0.01%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.01%       dd  [kernel]                 [k] kmem_cache_free
     0.01%       dd  [kernel]                 [k] inotify_inode_queue_event
     0.01%       dd  [kernel]                 [k] btrfs_unlock_up_safe
     0.01%       dd  [kernel]                 [k] bin_search
     0.01%       dd  [kernel]                 [k] unlock_up
     0.01%       dd  [kernel]                 [k] check_force_delalloc
     0.01%       dd  [kernel]                 [k] block_group_cache_tree_search
     0.01%       dd  [kernel]                 [k] btrfs_reserve_metadata_for_delalloc
     0.01%       dd  [kernel]                 [k] btrfs_set_timespec_nsec
     0.01%       dd  [kernel]                 [k] set_extent_bit
     0.01%       dd  [kernel]                 [k] __btrfs_map_block
     0.01%       dd  [kernel]                 [k] btrfs_spin_on_block
     0.01%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.01%       dd  [kernel]                 [k] prio_tree_insert
     0.01%       dd  [kernel]                 [k] strnlen_user
     0.01%       dd  [kernel]                 [k] e100_enable_irq
     0.01%     perf  [kernel]                 [k] dup_mm
     0.01%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.01%     perf  [kernel]                 [k] __page_cache_alloc
     0.01%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.01%     perf  [kernel]                 [k] zone_watermark_ok
     0.01%     perf  [kernel]                 [k] do_wp_page
     0.01%     perf  [kernel]                 [k] path_put
     0.01%     perf  [kernel]                 [k] __pollwait
     0.01%     perf  [kernel]                 [k] d_path
     0.01%     perf  [kernel]                 [k] mnt_want_write_file
     0.01%     perf  [kernel]                 [k] show_map_vma
     0.01%     perf  [kernel]                 [k] txLock
     0.01%     perf  [kernel]                 [k] blk_end_bidi_request
     0.01%     perf  [kernel]                 [k] _atomic_dec_and_lock
     0.01%     perf  [kernel]                 [k] put_dec
     0.01%     perf  [kernel]                 [k] ata_sff_interrupt
     0.01%     perf  [kernel]                 [k] preempt_schedule
     0.01%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.01%       dd  [kernel]                 [k] current_kernel_time
     0.01%       dd  [kernel]                 [k] perf_output_copy
     0.01%       dd  [kernel]                 [k] generic_file_aio_read
     0.01%       dd  [kernel]                 [k] generic_write_checks
     0.01%       dd  [kernel]                 [k] vfs_read
     0.01%       dd  [kernel]                 [k] exec_permission
     0.01%       dd  [kernel]                 [k] fsnotify
     0.01%       dd  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.01%       dd  [kernel]                 [k] setup_items_for_insert
     0.01%       dd  [kernel]                 [k] btrfs_alloc_free_block
     0.01%       dd  [kernel]                 [k] btrfs_check_data_free_space
     0.01%       dd  [kernel]                 [k] end_workqueue_bio
     0.01%       dd  [kernel]                 [k] btrfs_set_extent_delalloc
     0.01%       dd  [kernel]                 [k] lookup_extent_mapping
     0.01%       dd  [kernel]                 [k] btrfs_set_timespec_sec
     0.01%       dd  [kernel]                 [k] btrfs_set_inode_transid
     0.01%       dd  [kernel]                 [k] free_extent_buffer
     0.01%       dd  [kernel]                 [k] lock_extent
     0.01%       dd  [kernel]                 [k] set_extent_buffer_dirty
     0.01%       dd  [kernel]                 [k] btrfs_clear_lock_blocking
     0.01%       dd  [kernel]                 [k] btrfs_try_spin_lock
     0.01%       dd  [kernel]                 [k] btrfs_tree_lock
     0.01%       dd  [kernel]                 [k] btrfs_set_lock_blocking
     0.01%       dd  [kernel]                 [k] search_bitmap
     0.01%       dd  [kernel]                 [k] tree_insert
     0.01%       dd  [kernel]                 [k] add_delayed_tree_ref
     0.01%       dd  [kernel]                 [k] blk_done_softirq
     0.01%       dd  [kernel]                 [k] strncpy_from_user
     0.01%       dd  [kernel]                 [k] half_md4_transform
     0.01%       dd  [kernel]                 [k] add_timer_randomness
     0.01%       dd  [kernel]                 [k] scsi_finish_command
     0.01%       dd  [kernel]                 [k] scsi_next_command
     0.01%       dd  [kernel]                 [k] yenta_interrupt
     0.01%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.01%       dd  /lib/ld-2.11.so          [.] check_match.8476
     0.00%     perf  [kernel]                 [k] arch_dup_task_struct
     0.00%     perf  [kernel]                 [k] test_ti_thread_flag
     0.00%     perf  [kernel]                 [k] task_fork_fair
     0.00%     perf  [kernel]                 [k] copy_process
     0.00%     perf  [kernel]                 [k] hrtimer_start_range_ns
     0.00%     perf  [kernel]                 [k] hrtimer_try_to_cancel
     0.00%     perf  [kernel]                 [k] down_read_trylock
     0.00%     perf  [kernel]                 [k] ktime_get_ts
     0.00%     perf  [kernel]                 [k] perf_poll
     0.00%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.00%     perf  [kernel]                 [k] activate_page
     0.00%     perf  [kernel]                 [k] vma_prio_tree_add
     0.00%     perf  [kernel]                 [k] vm_normal_page
     0.00%     perf  [kernel]                 [k] find_vma
     0.00%     perf  [kernel]                 [k] unmap_region
     0.00%     perf  [kernel]                 [k] mmap_region
     0.00%     perf  [kernel]                 [k] do_mmap_pgoff
     0.00%     perf  [kernel]                 [k] page_add_file_rmap
     0.00%     perf  [kernel]                 [k] reuse_swap_page
     0.00%     perf  [kernel]                 [k] sys_read
     0.00%     perf  [kernel]                 [k] __fput
     0.00%     perf  [kernel]                 [k] may_open
     0.00%     perf  [kernel]                 [k] link_path_walk
     0.00%     perf  [kernel]                 [k] estimate_accuracy
     0.00%     perf  [kernel]                 [k] poll_schedule_timeout
     0.00%     perf  [kernel]                 [k] sys_poll
     0.00%     perf  [kernel]                 [k] spin_unlock
     0.00%     perf  [kernel]                 [k] dup_fd
     0.00%     perf  [kernel]                 [k] alloc_fd
     0.00%     perf  [kernel]                 [k] seq_putc
     0.00%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.00%     perf  [kernel]                 [k] show_map
     0.00%     perf  [kernel]                 [k] proc_lookup_de
     0.00%     perf  [kernel]                 [k] render_sigset_t
     0.00%     perf  [kernel]                 [k] dbFindCtl
     0.00%     perf  [kernel]                 [k] spin_unlock
     0.00%     perf  [kernel]                 [k] rb_insert_color
     0.00%     perf  [kernel]                 [k] down_write
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_default_xsputn_internal
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_str_init_static_internal
     0.00%     perf  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%     perf  /lib/ld-2.11.so          [.] _dl_fixup
     0.00%       dd  [kernel]                 [k] sysenter_past_esp
     0.00%       dd  [kernel]                 [k] nommu_sync_single_for_device
     0.00%       dd  [kernel]                 [k] sched_move_task
     0.00%       dd  [kernel]                 [k] timespec_trunc
     0.00%       dd  [kernel]                 [k] spin_unlock_irq
     0.00%       dd  [kernel]                 [k] __queue_work
     0.00%       dd  [kernel]                 [k] __task_pid_nr_ns
     0.00%       dd  [kernel]                 [k] __mutex_init
     0.00%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.00%       dd  [kernel]                 [k] release_tgcred
     0.00%       dd  [kernel]                 [k] ktime_get_ts
     0.00%       dd  [kernel]                 [k] perf_event_comm
     0.00%       dd  [kernel]                 [k] sync_page
     0.00%       dd  [kernel]                 [k] get_dirty_limits
     0.00%       dd  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.00%       dd  [kernel]                 [k] vma_prio_tree_insert
     0.00%       dd  [kernel]                 [k] get_unmapped_area
     0.00%       dd  [kernel]                 [k] remove_vma
     0.00%       dd  [kernel]                 [k] __vm_enough_memory
     0.00%       dd  [kernel]                 [k] unmap_region
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] __anon_vma_link
     0.00%       dd  [kernel]                 [k] __kmalloc
     0.00%       dd  [kernel]                 [k] __dentry_open
     0.00%       dd  [kernel]                 [k] sys_write
     0.00%       dd  [kernel]                 [k] fget_light
     0.00%       dd  [kernel]                 [k] generic_fillattr
     0.00%       dd  [kernel]                 [k] cp_new_stat64
     0.00%       dd  [kernel]                 [k] get_arg_page
     0.00%       dd  [kernel]                 [k] acl_permission_check
     0.00%       dd  [kernel]                 [k] inode_permission
     0.00%       dd  [kernel]                 [k] getname
     0.00%       dd  [kernel]                 [k] do_lookup
     0.00%       dd  [kernel]                 [k] path_walk
     0.00%       dd  [kernel]                 [k] sys_dup3
     0.00%       dd  [kernel]                 [k] d_instantiate
     0.00%       dd  [kernel]                 [k] d_path
     0.00%       dd  [kernel]                 [k] __d_lookup
     0.00%       dd  [kernel]                 [k] expand_files
     0.00%       dd  [kernel]                 [k] mntput_no_expire
     0.00%       dd  [kernel]                 [k] bio_put
     0.00%       dd  [kernel]                 [k] bvec_alloc_bs
     0.00%       dd  [kernel]                 [k] bio_alloc_bioset
     0.00%       dd  [kernel]                 [k] locks_remove_posix
     0.00%       dd  [kernel]                 [k] removed_exe_file_vma
     0.00%       dd  [kernel]                 [k] check_block
     0.00%       dd  [kernel]                 [k] btrfs_alloc_path
     0.00%       dd  [kernel]                 [k] btrfs_lock_root_node
     0.00%       dd  [kernel]                 [k] btrfs_get_alloc_profile
     0.00%       dd  [kernel]                 [k] btrfs_init_new_buffer
     0.00%       dd  [kernel]                 [k] btrfs_lookup_block_group
     0.00%       dd  [kernel]                 [k] check_ref_cleanup
     0.00%       dd  [kernel]                 [k] btrfs_unreserve_metadata_for_delalloc
     0.00%       dd  [kernel]                 [k] btrfs_reserve_extent
     0.00%       dd  [kernel]                 [k] btrfs_lookup_xattr
     0.00%       dd  [kernel]                 [k] btrfs_set_buffer_uptodate
     0.00%       dd  [kernel]                 [k] btrfs_find_create_tree_block
     0.00%       dd  [kernel]                 [k] btrfs_mark_buffer_dirty
     0.00%       dd  [kernel]                 [k] clean_tree_block
     0.00%       dd  [kernel]                 [k] btree_submit_bio_hook
     0.00%       dd  [kernel]                 [k] wait_current_trans
     0.00%       dd  [kernel]                 [k] btrfs_throttle
     0.00%       dd  [kernel]                 [k] start_transaction
     0.00%       dd  [kernel]                 [k] fill_inode_item
     0.00%       dd  [kernel]                 [k] btrfs_update_inode
     0.00%       dd  [kernel]                 [k] btrfs_dirty_inode
     0.00%       dd  [kernel]                 [k] btrfs_create
     0.00%       dd  [kernel]                 [k] btrfs_set_bit_hook
     0.00%       dd  [kernel]                 [k] __tree_search
     0.00%       dd  [kernel]                 [k] btrfs_item_offset
     0.00%       dd  [kernel]                 [k] btrfs_set_key_blockptr
     0.00%       dd  [kernel]                 [k] btrfs_set_inode_block_group
     0.00%       dd  [kernel]                 [k] btrfs_set_inode_nbytes
     0.00%       dd  [kernel]                 [k] btrfs_set_inode_sequence
     0.00%       dd  [kernel]                 [k] free_extent_state
     0.00%       dd  [kernel]                 [k] alloc_extent_state
     0.00%       dd  [kernel]                 [k] spin_unlock
     0.00%       dd  [kernel]                 [k] set_extent_dirty
     0.00%       dd  [kernel]                 [k] memmove_extent_buffer
     0.00%       dd  [kernel]                 [k] write_extent_buffer
     0.00%       dd  [kernel]                 [k] alloc_extent_buffer
     0.00%       dd  [kernel]                 [k] btrfs_map_bio
     0.00%       dd  [kernel]                 [k] bitmap_clear_bits
     0.00%       dd  [kernel]                 [k] find_ref_head
     0.00%       dd  [kernel]                 [k] btrfs_find_delayed_ref_head
     0.00%       dd  [kernel]                 [k] crypto_shash_update
     0.00%       dd  [kernel]                 [k] chksum_update
     0.00%       dd  [kernel]                 [k] req_bio_endio
     0.00%       dd  [kernel]                 [k] blk_rq_init
     0.00%       dd  [kernel]                 [k] blk_run_queue
     0.00%       dd  [kernel]                 [k] get_request
     0.00%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.00%       dd  [kernel]                 [k] kref_put
     0.00%       dd  [kernel]                 [k] prop_norm_single
     0.00%       dd  [kernel]                 [k] prop_fraction_single
     0.00%       dd  [kernel]                 [k] __rb_rotate_right
     0.00%       dd  [kernel]                 [k] rb_next
     0.00%       dd  [kernel]                 [k] rb_prev
     0.00%       dd  [kernel]                 [k] __get_user_4
     0.00%       dd  [kernel]                 [k] strncmp
     0.00%       dd  [kernel]                 [k] _copy_from_user
     0.00%       dd  [kernel]                 [k] crc32c
     0.00%       dd  [kernel]                 [k] extract_buf
     0.00%       dd  [kernel]                 [k] __scsi_put_command
     0.00%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.00%       dd  [kernel]                 [k] scsi_decide_disposition
     0.00%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.00%       dd  [kernel]                 [k] scsi_sg_free
     0.00%       dd  [kernel]                 [k] __scsi_release_buffers
     0.00%       dd  [kernel]                 [k] scsi_softirq_done
     0.00%       dd  [kernel]                 [k] sd_done
     0.00%       dd  [kernel]                 [k] e100_disable_irq
     0.00%       dd  [kernel]                 [k] acpi_pm_read
     0.00%       dd  [kernel]                 [k] skb_pull
     0.00%       dd  [kernel]                 [k] kfree_skb
     0.00%       dd  [kernel]                 [k] eth_type_trans
     0.00%       dd  [kernel]                 [k] tcp_v4_rcv
     0.00%       dd  [kernel]                 [k] tpacket_rcv
     0.00%       dd  [kernel]                 [k] mutex_unlock
     0.00%       dd  [kernel]                 [k] error_code
     0.00%       dd  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_bsearch
     0.00%       dd  /lib/libc-2.11.so        [.] ptmalloc_init
     0.00%       dd  /lib/libc-2.11.so        [.] _getopt_internal_r
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_start
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_cache_libcmp
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_unload_cache
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_setup_hash
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_relocate_object
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_map_object_deps
     0.00%       dd  /lib/ld-2.11.so          [.] strcmp
     0.00%       dd  /lib/ld-2.11.so          [.] 0x0000000001943e
     0.00%       dd  /bin/dd                  [.] main
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #8: perf7.report --]
[-- Type: text/plain, Size: 33242 bytes --]

# Samples: 66514
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
     7.26%       dd  [kernel]                 [k] __copy_from_user_ll_nozero
     5.20%       dd  [kernel]                 [k] __clear_user
     4.47%       dd  [kernel]                 [k] get_page_from_freelist
     4.40%       dd  [kernel]                 [k] __set_page_dirty
     2.85%       dd  [kernel]                 [k] __block_prepare_write
     2.61%       dd  [kernel]                 [k] ext4_da_write_begin
     2.26%       dd  [kernel]                 [k] ext4_da_get_block_prep
     1.94%       dd  [kernel]                 [k] ext4_da_write_end
     1.93%       dd  [kernel]                 [k] ext4_mark_iloc_dirty
     1.82%       dd  [kernel]                 [k] generic_file_buffered_write
     1.54%       dd  [kernel]                 [k] kmem_cache_alloc
     1.49%     perf  [kernel]                 [k] generic_file_buffered_write
     1.45%       dd  [kernel]                 [k] __ext4_get_inode_loc
     1.40%       dd  [kernel]                 [k] add_to_page_cache_locked
     1.35%       dd  [kernel]                 [k] ext4_get_blocks
     1.33%       dd  [kernel]                 [k] jbd2_journal_stop
     1.22%       dd  [kernel]                 [k] T.1339
     1.19%       dd  [kernel]                 [k] start_this_handle
     1.13%       dd  [kernel]                 [k] generic_write_end
     1.08%       dd  [kernel]                 [k] ext4_ext_get_blocks
     1.07%       dd  [kernel]                 [k] __alloc_pages_nodemask
     1.07%       dd  [kernel]                 [k] iov_iter_copy_from_user_atomic
     1.06%       dd  [kernel]                 [k] ext4_has_free_blocks
     1.04%       dd  [kernel]                 [k] ext4_ext_calc_metadata_amount
     0.99%       dd  [kernel]                 [k] find_get_page
     0.99%       dd  [kernel]                 [k] __rcu_read_unlock
     0.94%     perf  [kernel]                 [k] __generic_file_aio_write
     0.91%       dd  [kernel]                 [k] jbd2_journal_start
     0.90%       dd  [kernel]                 [k] do_get_write_access
     0.84%       dd  [kernel]                 [k] __wake_up
     0.77%       dd  [kernel]                 [k] grab_cache_page_write_begin
     0.76%       dd  [kernel]                 [k] __find_get_block
     0.74%       dd  [kernel]                 [k] __block_commit_write
     0.69%     perf  [kernel]                 [k] T.1339
     0.69%       dd  [kernel]                 [k] __find_get_block_slow
     0.68%       dd  [kernel]                 [k] __mark_inode_dirty
     0.66%       dd  [kernel]                 [k] zone_watermark_ok
     0.62%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.61%       dd  [kernel]                 [k] block_write_begin
     0.60%       dd  [kernel]                 [k] iov_iter_fault_in_readable
     0.60%       dd  [kernel]                 [k] ext4_mark_inode_dirty
     0.59%       dd  [kernel]                 [k] alloc_page_buffers
     0.57%       dd  [kernel]                 [k] block_write_end
     0.55%       dd  [kernel]                 [k] iov_iter_advance
     0.55%     perf  [kernel]                 [k] vfs_write
     0.55%       dd  [kernel]                 [k] kmem_cache_free
     0.54%       dd  [kernel]                 [k] jbd2_journal_dirty_metadata
     0.51%       dd  [kernel]                 [k] __ext4_journal_stop
     0.49%     perf  [kernel]                 [k] generic_file_aio_write
     0.49%     perf  [kernel]                 [k] iov_iter_advance
     0.49%       dd  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.49%       dd  [kernel]                 [k] ext4_reserve_inode_write
     0.49%       dd  [kernel]                 [k] spin_unlock
     0.48%       dd  [kernel]                 [k] ext4_claim_free_blocks
     0.48%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.48%     perf  [kernel]                 [k] nobh_write_end
     0.47%       dd  [kernel]                 [k] __getblk
     0.46%       dd  [kernel]                 [k] jbd2_journal_add_journal_head
     0.46%     perf  [kernel]                 [k] nobh_write_begin
     0.46%       dd  [kernel]                 [k] alloc_buffer_head
     0.45%     perf  [kernel]                 [k] __jfs_getxattr
     0.45%     perf  [kernel]                 [k] sysenter_past_esp
     0.45%       dd  [kernel]                 [k] create_empty_buffers
     0.45%       dd  [kernel]                 [k] __lru_cache_add
     0.45%       dd  [kernel]                 [k] read_zero
     0.44%       dd  [kernel]                 [k] next_zones_zonelist
     0.43%     perf  [kernel]                 [k] do_sync_write
     0.43%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.43%       dd  [kernel]                 [k] __ext4_handle_dirty_metadata
     0.42%       dd  [kernel]                 [k] __rcu_read_lock
     0.42%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.40%       dd  [kernel]                 [k] ext4_get_group_desc
     0.39%     perf  [kernel]                 [k] find_get_page
     0.39%       dd  [kernel]                 [k] __wake_up_bit
     0.38%       dd  [kernel]                 [k] jbd2_journal_put_journal_head
     0.38%       dd  [kernel]                 [k] __ext4_journal_get_write_access
     0.37%     perf  [kernel]                 [k] generic_segment_checks
     0.37%       dd  [kernel]                 [k] ext4_dirty_inode
     0.36%     perf  [kernel]                 [k] rw_verify_area
     0.35%     perf  [kernel]                 [k] getnstimeofday
     0.35%       dd  [kernel]                 [k] put_page
     0.35%       dd  [kernel]                 [k] mark_page_accessed
     0.33%       dd  [kernel]                 [k] jbd2_journal_cancel_revoke
     0.33%       dd  [kernel]                 [k] jbd2_journal_get_write_access
     0.30%       dd  [kernel]                 [k] find_lock_page
     0.30%       dd  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.30%     perf  [kernel]                 [k] sys_write
     0.30%     perf  [kernel]                 [k] jfs_write_begin
     0.30%     perf  [kernel]                 [k] acpi_pm_read
     0.29%       dd  [kernel]                 [k] wake_up_bit
     0.28%       dd  [kernel]                 [k] _cond_resched
     0.27%       dd  [kernel]                 [k] ext4_journal_start_sb
     0.27%       dd  [kernel]                 [k] radix_tree_preload
     0.26%     perf  [kernel]                 [k] ea_get
     0.26%     perf  [kernel]                 [k] copy_to_user
     0.25%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.25%     perf  [kernel]                 [k] __rcu_read_unlock
     0.24%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.23%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.23%       dd  [kernel]                 [k] mark_buffer_dirty
     0.23%       dd  [kernel]                 [k] ext4_inode_table
     0.22%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.21%       dd  [kernel]                 [k] schedule
     0.21%     perf  [kernel]                 [k] fget_light
     0.21%     perf  [kernel]                 [k] __mark_inode_dirty
     0.21%     perf  [vdso]                   [.] 0x000000ffffe424
     0.20%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.19%       dd  [kernel]                 [k] do_page_fault
     0.18%     perf  [kernel]                 [k] sys_gettimeofday
     0.18%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.18%     perf  [kernel]                 [k] xtSearch
     0.17%       dd  [kernel]                 [k] unlock_page
     0.17%     perf  [kernel]                 [k] put_page
     0.16%     perf  [kernel]                 [k] fsnotify
     0.16%     perf  [kernel]                 [k] schedule
     0.15%       dd  [kernel]                 [k] down_read
     0.15%       dd  [kernel]                 [k] unmap_underlying_metadata
     0.14%       dd  [kernel]                 [k] rcu_sched_qs
     0.14%     perf  [kernel]                 [k] extAlloc
     0.14%     perf  [kernel]                 [k] kfree
     0.14%     perf  [kernel]                 [k] generic_write_sync
     0.14%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.14%     perf  [kernel]                 [k] dbAlloc
     0.14%     perf  [kernel]                 [k] file_update_time
     0.14%       dd  [kernel]                 [k] handle_IRQ_event
     0.14%       dd  [kernel]                 [k] __page_cache_alloc
     0.14%       dd  [kernel]                 [k] unmap_vmas
     0.13%       dd  [kernel]                 [k] ext4_get_inode_loc
     0.13%     perf  [kernel]                 [k] jfs_getxattr
     0.13%       dd  [kernel]                 [k] handle_mm_fault
     0.13%       dd  [kernel]                 [k] ext4_get_inode_flags
     0.12%     perf  [kernel]                 [k] find_lock_page
     0.12%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.12%     perf  [kernel]                 [k] __rcu_read_lock
     0.12%     perf  [kernel]                 [k] should_remove_suid
     0.12%     perf  [kernel]                 [k] get_page_from_freelist
     0.11%     perf  [kernel]                 [k] current_fs_time
     0.11%     perf  [kernel]                 [k] __fsnotify_parent
     0.11%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.11%     perf  [kernel]                 [k] ea_release
     0.11%     perf  [kernel]                 [k] unlock_page
     0.11%     perf  [kernel]                 [k] do_gettimeofday
     0.10%     perf  [kernel]                 [k] file_remove_suid
     0.10%       dd  [kernel]                 [k] set_bh_page
     0.10%       dd  [kernel]                 [k] __jbd2_log_space_left
     0.10%     perf  [kernel]                 [k] mark_page_accessed
     0.10%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.10%       dd  [kernel]                 [k] up_read
     0.10%       dd  [kernel]                 [k] free_hot_cold_page
     0.09%     perf  [kernel]                 [k] __set_page_dirty
     0.09%       dd  [kernel]                 [k] __brelse
     0.09%     perf  [kernel]                 [k] strncmp
     0.09%     perf  [kernel]                 [k] __wake_up_bit
     0.09%       dd  [kernel]                 [k] unlock_buffer
     0.08%     perf  [kernel]                 [k] timespec_trunc
     0.08%     perf  [kernel]                 [k] xtLookup
     0.08%     perf  [kernel]                 [k] dbAllocBits
     0.08%     perf  [kernel]                 [k] mutex_lock
     0.08%     perf  [kernel]                 [k] current_kernel_time
     0.08%     perf  [kernel]                 [k] jfs_dirty_inode
     0.08%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.07%     perf  [kernel]                 [k] _cond_resched
     0.07%     perf  [kernel]                 [k] __wake_up
     0.07%     perf  [kernel]                 [k] cap_file_permission
     0.07%     perf  [kernel]                 [k] strlen
     0.07%       dd  [kernel]                 [k] sha_transform
     0.07%     perf  [kernel]                 [k] __get_metapage
     0.07%       dd  [kernel]                 [k] yenta_interrupt
     0.06%     perf  [kernel]                 [k] dbAllocNext
     0.06%     perf  [kernel]                 [k] set_page_dirty
     0.06%       dd  [kernel]                 [k] __do_softirq
     0.06%     perf  [kernel]                 [k] number
     0.05%     perf  [kernel]                 [k] sysenter_do_call
     0.05%     perf  [kernel]                 [k] jfs_get_block
     0.05%     perf  [kernel]                 [k] xtExtend
     0.05%     perf  [kernel]                 [k] vsnprintf
     0.05%     perf  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.05%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.05%     perf  [kernel]                 [k] mutex_unlock
     0.05%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.05%     perf  [kernel]                 [k] down_read
     0.04%     perf  [kernel]                 [k] rcu_sched_qs
     0.04%     perf  [kernel]                 [k] security_file_permission
     0.04%     perf  [kernel]                 [k] up_read
     0.04%     perf  [kernel]                 [k] release_metapage
     0.04%       dd  [kernel]                 [k] release_pages
     0.04%     perf  [kernel]                 [k] format_decode
     0.04%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.04%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.03%     perf  [kernel]                 [k] copy_page_range
     0.03%     perf  [kernel]                 [k] extHint
     0.03%       dd  [kernel]                 [k] serio_interrupt
     0.03%     perf  [kernel]                 [k] __do_fault
     0.03%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.03%     perf  [kernel]                 [k] mnt_clone_write
     0.03%     perf  [kernel]                 [k] __do_softirq
     0.03%     perf  [kernel]                 [k] handle_IRQ_event
     0.03%       dd  [kernel]                 [k] find_vma
     0.03%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.03%     perf  [kernel]                 [k] mangle_path
     0.03%     perf  [kernel]                 [k] dbAllocDmap
     0.03%     perf  [kernel]                 [k] do_page_fault
     0.03%       dd  [kernel]                 [k] __do_fault
     0.02%     perf  [kernel]                 [k] txLock
     0.02%       dd  [kernel]                 [k] preempt_schedule
     0.02%     perf  [kernel]                 [k] mnt_drop_write
     0.02%     perf  [kernel]                 [k] read_cache_page
     0.02%     perf  [kernel]                 [k] txBeginAnon
     0.02%       dd  [kernel]                 [k] check_for_new_grace_period
     0.02%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.02%       dd  [kernel]                 [k] anon_vma_prepare
     0.02%       dd  [kernel]                 [k] scsi_dispatch_cmd
     0.02%       dd  [kernel]                 [k] irq_handler
     0.02%     perf  [kernel]                 [k] __find_get_block_slow
     0.02%       dd  [kernel]                 [k] page_evictable
     0.02%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.02%     perf  [kernel]                 [k] dbSplit
     0.02%       dd  [kernel]                 [k] try_to_wake_up
     0.02%       dd  [kernel]                 [k] down_read_trylock
     0.02%       dd  [kernel]                 [k] page_remove_rmap
     0.02%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.02%     perf  [kernel]                 [k] alloc_buffer_head
     0.02%       dd  [kernel]                 [k] rcu_process_gp_end
     0.02%     perf  [kernel]                 [k] do_read_cache_page
     0.02%     perf  [kernel]                 [k] read_cache_page_async
     0.02%     perf  [kernel]                 [k] free_buffer_head
     0.01%     perf  [kernel]                 [k] inode_add_bytes
     0.01%     perf  [kernel]                 [k] mnt_want_write_file
     0.01%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.01%     perf  [kernel]                 [k] yenta_interrupt
     0.01%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.01%       dd  [kernel]                 [k] filemap_fault
     0.01%     perf  [kernel]                 [k] zone_watermark_ok
     0.01%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.01%     perf  [kernel]                 [k] next_zones_zonelist
     0.01%     perf  [kernel]                 [k] kmem_cache_free
     0.01%     perf  [kernel]                 [k] kmem_cache_alloc
     0.01%       dd  [kernel]                 [k] vm_normal_page
     0.01%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.01%     perf  [kernel]                 [k] find_vma
     0.01%     perf  [kernel]                 [k] seq_printf
     0.01%     perf  [kernel]                 [k] dbAdjTree
     0.01%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.01%       dd  [kernel]                 [k] copy_strings
     0.01%       dd  [kernel]                 [k] blk_end_bidi_request
     0.01%       dd  [kernel]                 [k] strnlen_user
     0.01%     perf  [kernel]                 [k] handle_mm_fault
     0.01%     perf  [kernel]                 [k] alloc_page_buffers
     0.01%     perf  [kernel]                 [k] radix_tree_preload
     0.01%       dd  [kernel]                 [k] scsi_request_fn
     0.01%     perf  [kernel]                 [k] perf_mmap_fault
     0.01%     perf  [kernel]                 [k] activate_page
     0.01%     perf  [kernel]                 [k] __lru_cache_add
     0.01%     perf  [kernel]                 [k] do_sys_poll
     0.01%     perf  [kernel]                 [k] scsi_request_fn
     0.01%     perf  [kernel]                 [k] irq_handler
     0.01%       dd  [kernel]                 [k] __generic_file_aio_write
     0.01%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.01%       dd  [kernel]                 [k] i8042_interrupt
     0.01%     perf  [kernel]                 [k] up_write
     0.01%     perf  [kernel]                 [k] set_bh_page
     0.01%     perf  [kernel]                 [k] spin_unlock
     0.01%     perf  [kernel]                 [k] rb_insert_color
     0.01%     perf  [kernel]                 [k] down_write
     0.01%       dd  [kernel]                 [k] run_timer_softirq
     0.01%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.01%       dd  [kernel]                 [k] free_hot_page
     0.01%       dd  [kernel]                 [k] do_wp_page
     0.01%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.01%       dd  [kernel]                 [k] ioread8
     0.01%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.01%       dd                 b6ffe590  [.] 0x000000b6ffe590
     0.00%     perf  [kernel]                 [k] __queue_work
     0.00%     perf  [kernel]                 [k] wait_on_page_read
     0.00%     perf  [kernel]                 [k] do_wp_page
     0.00%     perf  [kernel]                 [k] strchr
     0.00%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.00%       dd  [kernel]                 [k] __queue_work
     0.00%       dd  [kernel]                 [k] rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] get_dirty_limits
     0.00%       dd  [kernel]                 [k] __vma_link
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.00%       dd  [kernel]                 [k] do_sync_write
     0.00%       dd  [kernel]                 [k] fget_light
     0.00%       dd  [kernel]                 [k] get_empty_filp
     0.00%       dd  [kernel]                 [k] do_filp_open
     0.00%       dd  [kernel]                 [k] inotify_inode_queue_event
     0.00%       dd  [kernel]                 [k] load_elf_binary
     0.00%       dd  [kernel]                 [k] scsi_softirq_done
     0.00%     perf  [kernel]                 [k] arch_dup_task_struct
     0.00%     perf  [kernel]                 [k] try_to_wake_up
     0.00%     perf  [kernel]                 [k] copy_process
     0.00%     perf  [kernel]                 [k] remove_wait_queue
     0.00%     perf  [kernel]                 [k] down_read_trylock
     0.00%     perf  [kernel]                 [k] ktime_get_ts
     0.00%     perf  [kernel]                 [k] perf_poll
     0.00%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.00%     perf  [kernel]                 [k] filemap_fault
     0.00%     perf  [kernel]                 [k] get_dirty_limits
     0.00%     perf  [kernel]                 [k] test_clear_page_writeback
     0.00%     perf  [kernel]                 [k] mmap_region
     0.00%     perf  [kernel]                 [k] prepend
     0.00%     perf  [kernel]                 [k] spin_unlock
     0.00%     perf  [kernel]                 [k] seq_read
     0.00%     perf  [kernel]                 [k] proc_pid_status
     0.00%     perf  [kernel]                 [k] xtInsert
     0.00%     perf  [kernel]                 [k] put_dec_trunc
     0.00%     perf  [kernel]                 [k] ip_local_deliver
     0.00%     perf  /lib/libpthread-2.11.so  [.] __write_nocancel
     0.00%       dd  [kernel]                 [k] sysenter_past_esp
     0.00%       dd  [kernel]                 [k] __call_rcu
     0.00%       dd  [kernel]                 [k] lru_cache_add_lru
     0.00%       dd  [kernel]                 [k] free_pgd_range
     0.00%       dd  [kernel]                 [k] free_pgtables
     0.00%       dd  [kernel]                 [k] unlink_file_vma
     0.00%       dd  [kernel]                 [k] mmap_region
     0.00%       dd  [kernel]                 [k] vfs_read
     0.00%       dd  [kernel]                 [k] path_put
     0.00%       dd  [kernel]                 [k] do_lookup
     0.00%       dd  [kernel]                 [k] link_path_walk
     0.00%       dd  [kernel]                 [k] __d_lookup
     0.00%       dd  [kernel]                 [k] security_file_permission
     0.00%       dd  [kernel]                 [k] prio_tree_replace
     0.00%       dd  [kernel]                 [k] rb_insert_color
     0.00%       dd  [kernel]                 [k] _copy_from_user
     0.00%       dd  [kernel]                 [k] error_code
     0.00%       dd  /lib/libc-2.11.so        [.] _nl_load_locale_from_archive
     0.00%       dd  /lib/libc-2.11.so        [.] _IO_fclose@@GLIBC_2.1
     0.00%       dd  /lib/libc-2.11.so        [.] 0x000000000a1877
     0.00%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.00%       dd  /bin/dd                  [.] T.275
     0.00%     perf  [kernel]                 [k] pgd_alloc
     0.00%     perf  [kernel]                 [k] wake_up_new_task
     0.00%     perf  [kernel]                 [k] alloc_pid
     0.00%     perf  [kernel]                 [k] __mutex_init
     0.00%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.00%     perf  [kernel]                 [k] mempool_free_slab
     0.00%     perf  [kernel]                 [k] release_pages
     0.00%     perf  [kernel]                 [k] vma_prio_tree_add
     0.00%     perf  [kernel]                 [k] unmap_vmas
     0.00%     perf  [kernel]                 [k] __vma_link_rb
     0.00%     perf  [kernel]                 [k] anon_vma_unlink
     0.00%     perf  [kernel]                 [k] fd_install
     0.00%     perf  [kernel]                 [k] path_get
     0.00%     perf  [kernel]                 [k] generic_permission
     0.00%     perf  [kernel]                 [k] do_lookup
     0.00%     perf  [kernel]                 [k] link_path_walk
     0.00%     perf  [kernel]                 [k] estimate_accuracy
     0.00%     perf  [kernel]                 [k] poll_select_set_timeout
     0.00%     perf  [kernel]                 [k] __pollwait
     0.00%     perf  [kernel]                 [k] __d_path
     0.00%     perf  [kernel]                 [k] d_instantiate
     0.00%     perf  [kernel]                 [k] __d_lookup
     0.00%     perf  [kernel]                 [k] dput
     0.00%     perf  [kernel]                 [k] seq_path
     0.00%     perf  [kernel]                 [k] end_bio_bh_io_sync
     0.00%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.00%     perf  [kernel]                 [k] bio_fs_destructor
     0.00%     perf  [kernel]                 [k] show_map_vma
     0.00%     perf  [kernel]                 [k] m_start
     0.00%     perf  [kernel]                 [k] proc_pid_make_inode
     0.00%     perf  [kernel]                 [k] render_sigset_t
     0.00%     perf  [kernel]                 [k] dbFindCtl
     0.00%     perf  [kernel]                 [k] dbAllocAG
     0.00%     perf  [kernel]                 [k] security_vm_enough_memory
     0.00%     perf  [kernel]                 [k] req_bio_endio
     0.00%     perf  [kernel]                 [k] blk_end_bidi_request
     0.00%     perf  [kernel]                 [k] get_request
     0.00%     perf  [kernel]                 [k] __make_request
     0.00%     perf  [kernel]                 [k] _atomic_dec_and_lock
     0.00%     perf  [kernel]                 [k] prop_fraction_percpu
     0.00%     perf  [kernel]                 [k] __rb_rotate_left
     0.00%     perf  [kernel]                 [k] put_dec
     0.00%     perf  [kernel]                 [k] ioread8
     0.00%     perf  [kernel]                 [k] scsi_finish_command
     0.00%     perf  [kernel]                 [k] scsi_put_command
     0.00%     perf  [kernel]                 [k] __scsi_release_buffers
     0.00%     perf  [kernel]                 [k] ata_sff_interrupt
     0.00%     perf  [kernel]                 [k] serio_interrupt
     0.00%     perf  [kernel]                 [k] __skb_clone
     0.00%     perf  [kernel]                 [k] preempt_schedule
     0.00%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.00%     perf  [kernel]                 [k] error_code
     0.00%     perf  /lib/libc-2.11.so        [.] fgets
     0.00%     perf  /lib/libc-2.11.so        [.] __GI___open
     0.00%     perf  /lib/libc-2.11.so        [.] __libc_poll
     0.00%     perf  /home/cloos/bin/perf     [.] geteuid@plt
     0.00%       dd  [kernel]                 [k] nommu_sync_single_for_device
     0.00%       dd  [kernel]                 [k] sys_execve
     0.00%       dd  [kernel]                 [k] sched_move_task
     0.00%       dd  [kernel]                 [k] unshare_fd
     0.00%       dd  [kernel]                 [k] mm_init
     0.00%       dd  [kernel]                 [k] flush_signal_handlers
     0.00%       dd  [kernel]                 [k] spin_unlock_irq
     0.00%       dd  [kernel]                 [k] flush_itimer_signals
     0.00%       dd  [kernel]                 [k] sigprocmask
     0.00%       dd  [kernel]                 [k] task_tgid_nr_ns
     0.00%       dd  [kernel]                 [k] __task_pid_nr_ns
     0.00%       dd  [kernel]                 [k] __blocking_notifier_call_chain
     0.00%       dd  [kernel]                 [k] acct_collect
     0.00%       dd  [kernel]                 [k] perf_output_begin
     0.00%       dd  [kernel]                 [k] perf_event_comm
     0.00%       dd  [kernel]                 [k] generic_segment_checks
     0.00%       dd  [kernel]                 [k] file_read_actor
     0.00%       dd  [kernel]                 [k] generic_file_aio_read
     0.00%       dd  [kernel]                 [k] generic_file_aio_write
     0.00%       dd  [kernel]                 [k] mempool_free_slab
     0.00%       dd  [kernel]                 [k] file_ra_state_init
     0.00%       dd  [kernel]                 [k] global_reclaimable_pages
     0.00%       dd  [kernel]                 [k] vma_prio_tree_remove
     0.00%       dd  [kernel]                 [k] arch_get_unmapped_area_topdown
     0.00%       dd  [kernel]                 [k] __vma_link_rb
     0.00%       dd  [kernel]                 [k] vma_link
     0.00%       dd  [kernel]                 [k] vma_adjust
     0.00%       dd  [kernel]                 [k] sys_brk
     0.00%       dd  [kernel]                 [k] exit_mmap
     0.00%       dd  [kernel]                 [k] mprotect_fixup
     0.00%       dd  [kernel]                 [k] __anon_vma_link
     0.00%       dd  [kernel]                 [k] anon_vma_unlink
     0.00%       dd  [kernel]                 [k] __dentry_open
     0.00%       dd  [kernel]                 [k] nameidata_to_filp
     0.00%       dd  [kernel]                 [k] rw_verify_area
     0.00%       dd  [kernel]                 [k] vfs_write
     0.00%       dd  [kernel]                 [k] fget
     0.00%       dd  [kernel]                 [k] vfs_fstat
     0.00%       dd  [kernel]                 [k] do_execve
     0.00%       dd  [kernel]                 [k] acl_permission_check
     0.00%       dd  [kernel]                 [k] exec_permission
     0.00%       dd  [kernel]                 [k] generic_permission
     0.00%       dd  [kernel]                 [k] inode_permission
     0.00%       dd  [kernel]                 [k] path_walk
     0.00%       dd  [kernel]                 [k] __d_path
     0.00%       dd  [kernel]                 [k] file_update_time
     0.00%       dd  [kernel]                 [k] expand_files
     0.00%       dd  [kernel]                 [k] mnt_drop_write
     0.00%       dd  [kernel]                 [k] mntput_no_expire
     0.00%       dd  [kernel]                 [k] mpage_end_io_read
     0.00%       dd  [kernel]                 [k] dnotify_flush
     0.00%       dd  [kernel]                 [k] added_exe_file_vma
     0.00%       dd  [kernel]                 [k] ext4_check_dir_entry
     0.00%       dd  [kernel]                 [k] ext4_new_inode
     0.00%       dd  [kernel]                 [k] ext4_ind_get_blocks
     0.00%       dd  [kernel]                 [k] is_file_shm_hugepages
     0.00%       dd  [kernel]                 [k] security_vm_enough_memory_mm
     0.00%       dd  [kernel]                 [k] cap_inode_permission
     0.00%       dd  [kernel]                 [k] cap_file_permission
     0.00%       dd  [kernel]                 [k] blk_update_request
     0.00%       dd  [kernel]                 [k] blk_update_bidi_request
     0.00%       dd  [kernel]                 [k] blk_done_softirq
     0.00%       dd  [kernel]                 [k] find_next_zero_bit
     0.00%       dd  [kernel]                 [k] kobject_get
     0.00%       dd  [kernel]                 [k] prio_tree_insert
     0.00%       dd  [kernel]                 [k] prop_norm_percpu
     0.00%       dd  [kernel]                 [k] prop_fraction_single
     0.00%       dd  [kernel]                 [k] __rb_rotate_left
     0.00%       dd  [kernel]                 [k] __rb_rotate_right
     0.00%       dd  [kernel]                 [k] rb_erase
     0.00%       dd  [kernel]                 [k] rwsem_wake
     0.00%       dd  [kernel]                 [k] csum_partial
     0.00%       dd  [kernel]                 [k] __get_user_4
     0.00%       dd  [kernel]                 [k] copy_to_user
     0.00%       dd  [kernel]                 [k] half_md4_transform
     0.00%       dd  [kernel]                 [k] tty_wakeup
     0.00%       dd  [kernel]                 [k] tty_write
     0.00%       dd  [kernel]                 [k] tty_insert_flip_string
     0.00%       dd  [kernel]                 [k] scsi_finish_command
     0.00%       dd  [kernel]                 [k] __scsi_release_buffers
     0.00%       dd  [kernel]                 [k] scsi_io_completion
     0.00%       dd  [kernel]                 [k] ata_sff_interrupt
     0.00%       dd  [kernel]                 [k] e100_exec_cb
     0.00%       dd  [kernel]                 [k] e100_disable_irq
     0.00%       dd  [kernel]                 [k] e100_intr
     0.00%       dd  [kernel]                 [k] e100_poll
     0.00%       dd  [kernel]                 [k] acpi_pm_read
     0.00%       dd  [kernel]                 [k] __copy_skb_header
     0.00%       dd  [kernel]                 [k] __kfree_skb
     0.00%       dd  [kernel]                 [k] napi_gro_flush
     0.00%       dd  [kernel]                 [k] net_rx_action
     0.00%       dd  [kernel]                 [k] sk_run_filter
     0.00%       dd  [kernel]                 [k] tcp_ack
     0.00%       dd  [kernel]                 [k] tcp_rcv_established
     0.00%       dd  [kernel]                 [k] tcp_established_options
     0.00%       dd  [kernel]                 [k] ret_from_exception
     0.00%       dd  [kernel]                 [k] system_call
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_getenv
     0.00%       dd  /lib/libc-2.11.so        [.] _int_malloc
     0.00%       dd  /lib/libc-2.11.so        [.] index
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___stpcpy
     0.00%       dd  /lib/ld-2.11.so          [.] dl_main
     0.00%       dd  /lib/ld-2.11.so          [.] check_match.8476
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_fixup
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_debug_initialize
     0.00%       dd  /lib/ld-2.11.so          [.] __mmap
     0.00%       dd  /lib/ld-2.11.so          [.] 0x0000000001943e
     0.00%       dd  /bin/dd                  [.] iread
     0.00%       dd  /bin/dd                  [.] set_program_name
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #9: perf8.report --]
[-- Type: text/plain, Size: 24418 bytes --]

# Samples: 24816
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    15.90%       dd  [kernel]                 [k] __copy_to_user_ll
     7.04%     perf  [kernel]                 [k] generic_file_buffered_write
     4.47%     perf  [kernel]                 [k] __generic_file_aio_write
     2.28%     perf  [kernel]                 [k] sysenter_past_esp
     2.28%     perf  [kernel]                 [k] iov_iter_advance
     2.26%     perf  [kernel]                 [k] generic_file_aio_write
     2.19%     perf  [kernel]                 [k] __jfs_getxattr
     2.15%     perf  [kernel]                 [k] nobh_write_end
     2.15%     perf  [kernel]                 [k] nobh_write_begin
     2.10%     perf  [kernel]                 [k] do_sync_write
     2.07%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     2.04%     perf  [kernel]                 [k] vfs_write
     1.91%     perf  [kernel]                 [k] getnstimeofday
     1.82%     perf  [kernel]                 [k] find_get_page
     1.66%     perf  [kernel]                 [k] generic_segment_checks
     1.61%     perf  [kernel]                 [k] ea_get
     1.58%     perf  [kernel]                 [k] rw_verify_area
     1.37%     perf  [kernel]                 [k] jfs_write_begin
     1.34%       dd  [kernel]                 [k] __bio_add_page
     1.31%     perf  [kernel]                 [k] copy_to_user
     1.18%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     1.17%     perf  [kernel]                 [k] sys_write
     1.16%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     1.10%     perf  [vdso]                   [.] 0x000000ffffe424
     1.06%       dd  [kernel]                 [k] get_page_from_freelist
     1.02%     perf  [kernel]                 [k] fget_light
     0.96%     perf  [kernel]                 [k] acpi_pm_read
     0.93%     perf  [kernel]                 [k] sys_gettimeofday
     0.93%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.85%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.84%     perf  [kernel]                 [k] __mark_inode_dirty
     0.82%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     0.80%     perf  [kernel]                 [k] fsnotify
     0.77%     perf  [kernel]                 [k] kfree
     0.73%     perf  [kernel]                 [k] file_update_time
     0.69%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.67%     perf  [kernel]                 [k] generic_write_sync
     0.67%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.66%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.65%     perf  [kernel]                 [k] put_page
     0.64%       dd  [kernel]                 [k] __make_request
     0.63%     perf  [kernel]                 [k] jfs_getxattr
     0.62%     perf  [kernel]                 [k] do_gettimeofday
     0.60%     perf  [kernel]                 [k] find_lock_page
     0.59%     perf  [kernel]                 [k] __rcu_read_unlock
     0.56%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.54%     perf  [kernel]                 [k] __fsnotify_parent
     0.54%     perf  [kernel]                 [k] ea_release
     0.53%     perf  [kernel]                 [k] strncmp
     0.52%     perf  [kernel]                 [k] file_remove_suid
     0.51%     perf  [kernel]                 [k] unlock_page
     0.50%     perf  [kernel]                 [k] timespec_trunc
     0.47%     perf  [kernel]                 [k] T.1339
     0.47%     perf  [kernel]                 [k] current_fs_time
     0.46%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.45%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.45%     perf  [kernel]                 [k] should_remove_suid
     0.42%     perf  [kernel]                 [k] schedule
     0.41%     perf  [kernel]                 [k] strlen
     0.38%       dd  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.36%     perf  [kernel]                 [k] current_kernel_time
     0.36%     perf  [kernel]                 [k] _cond_resched
     0.35%       dd  [kernel]                 [k] do_mpage_readpage
     0.34%     perf  [kernel]                 [k] jfs_dirty_inode
     0.33%     perf  [kernel]                 [k] mark_page_accessed
     0.33%       dd  [kernel]                 [k] get_request
     0.31%     perf  [kernel]                 [k] __rcu_read_lock
     0.30%     perf  [kernel]                 [k] cap_file_permission
     0.29%     perf  [kernel]                 [k] set_page_dirty
     0.27%     perf  [kernel]                 [k] mutex_lock
     0.27%     perf  [kernel]                 [k] __wake_up_bit
     0.27%     perf  [kernel]                 [k] down_read
     0.22%     perf  [kernel]                 [k] up_read
     0.22%     perf  [kernel]                 [k] security_file_permission
     0.22%     perf  [kernel]                 [k] mutex_unlock
     0.21%       dd  [kernel]                 [k] blk_rq_init
     0.20%       dd  [kernel]                 [k] yenta_interrupt
     0.20%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.19%     perf  [kernel]                 [k] sysenter_do_call
     0.19%       dd  [kernel]                 [k] acpi_pm_read
     0.17%     perf  [kernel]                 [k] vsnprintf
     0.17%       dd  [kernel]                 [k] add_to_page_cache_locked
     0.16%       dd  [kernel]                 [k] generic_make_request
     0.15%       dd  [kernel]                 [k] generic_file_aio_read
     0.15%     perf  [kernel]                 [k] extAlloc
     0.15%       dd  [kernel]                 [k] T.1339
     0.12%     perf  [kernel]                 [k] number
     0.12%     perf  [kernel]                 [k] dbAlloc
     0.12%     perf  [kernel]                 [k] usb_hcd_irq	[usbcore]
     0.12%       dd  [kernel]                 [k] bio_add_page
     0.12%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.11%     perf  [kernel]                 [k] format_decode
     0.11%       dd  [kernel]                 [k] handle_IRQ_event
     0.11%       dd  [kernel]                 [k] irq_handler
     0.10%       dd  [kernel]                 [k] file_read_actor
     0.10%       dd  [kernel]                 [k] kmem_cache_alloc
     0.10%       dd  [kernel]                 [k] init_request_from_bio
     0.10%     perf  [kernel]                 [k] xtSearch
     0.10%       dd  [kernel]                 [k] __alloc_pages_nodemask
     0.09%     perf  [kernel]                 [k] mnt_drop_write
     0.09%     perf  [kernel]                 [k] mnt_clone_write
     0.09%       dd  [kernel]                 [k] find_get_page
     0.08%     perf  [kernel]                 [k] copy_page_range
     0.08%     perf  [kernel]                 [k] __set_page_dirty
     0.08%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.07%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.07%       dd  [kernel]                 [k] radix_tree_lookup
     0.07%     perf  [kernel]                 [k] do_sys_poll
     0.07%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.06%     perf  [kernel]                 [k] handle_IRQ_event
     0.06%     perf  [kernel]                 [k] get_page_from_freelist
     0.06%     perf  [kernel]                 [k] dbAllocBits
     0.06%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.06%       dd  [kernel]                 [k] ext4_get_blocks
     0.06%     perf  [kernel]                 [k] hrtimer_try_to_cancel
     0.06%     perf  [kernel]                 [k] xtLookup
     0.06%       dd  [kernel]                 [k] put_page
     0.06%       dd  [kernel]                 [k] ext4_ext_get_blocks
     0.06%       dd  [kernel]                 [k] cfq_set_request
     0.06%     perf  [kernel]                 [k] __wake_up
     0.06%       dd  [kernel]                 [k] ondemand_readahead
     0.05%     perf  [kernel]                 [k] do_page_fault
     0.05%       dd  [kernel]                 [k] __rcu_read_unlock
     0.05%       dd  [kernel]                 [k] __lru_cache_add
     0.05%       dd  [kernel]                 [k] mpage_readpages
     0.05%       dd  [kernel]                 [k] do_page_fault
     0.05%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.05%       dd  [kernel]                 [k] handle_mm_fault
     0.05%       dd  [kernel]                 [k] radix_tree_preload
     0.04%     perf  [kernel]                 [k] extHint
     0.04%     perf  [kernel]                 [k] yenta_interrupt
     0.04%       dd  [kernel]                 [k] __lock_page_killable
     0.04%       dd  [kernel]                 [k] __do_page_cache_readahead
     0.04%       dd  [kernel]                 [k] submit_bio
     0.04%       dd  [kernel]                 [k] zone_watermark_ok
     0.04%       dd  [kernel]                 [k] bio_init
     0.04%       dd  [kernel]                 [k] scsi_dispatch_cmd
     0.04%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.04%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.04%     perf  [kernel]                 [k] __do_fault
     0.04%       dd  [kernel]                 [k] prepare_to_wait_exclusive
     0.04%       dd  [kernel]                 [k] __rcu_read_lock
     0.04%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.04%       dd  [kernel]                 [k] page_cache_async_readahead
     0.04%       dd  [kernel]                 [k] unmap_vmas
     0.04%       dd  [kernel]                 [k] scsi_request_fn
     0.03%     perf  [kernel]                 [k] kmem_cache_alloc
     0.03%     perf  [kernel]                 [k] mnt_want_write_file
     0.03%     perf  /lib/libpthread-2.11.so  [.] __write_nocancel
     0.03%       dd  [kernel]                 [k] free_hot_cold_page
     0.03%       dd  [kernel]                 [k] next_zones_zonelist
     0.03%       dd  [kernel]                 [k] ext4_data_block_valid
     0.03%       dd  [kernel]                 [k] ioread8
     0.03%     perf  [kernel]                 [k] __find_get_block_slow
     0.03%     perf  [kernel]                 [k] jfs_get_block
     0.03%     perf  [kernel]                 [k] dbAllocDmap
     0.03%     perf  [kernel]                 [k] dbAllocNext
     0.03%     perf  [kernel]                 [k] __get_metapage
     0.03%     perf  /lib/libc-2.11.so        [.] __libc_poll
     0.03%       dd  [kernel]                 [k] sync_page
     0.03%       dd  [kernel]                 [k] mark_page_accessed
     0.03%       dd  [kernel]                 [k] bio_alloc_bioset
     0.03%       dd  [kernel]                 [k] blk_backing_dev_unplug
     0.02%     perf  [kernel]                 [k] do_wp_page
     0.02%     perf  [kernel]                 [k] show_map_vma
     0.02%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.02%       dd  [kernel]                 [k] mempool_alloc_slab
     0.02%       dd  [kernel]                 [k] block_sync_page
     0.02%       dd  [kernel]                 [k] mpage_alloc
     0.02%       dd  [kernel]                 [k] get_io_context
     0.02%     perf  [kernel]                 [k] __do_softirq
     0.02%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.02%     perf  [kernel]                 [k] __lru_cache_add
     0.02%     perf  [kernel]                 [k] poll_schedule_timeout
     0.02%     perf  [kernel]                 [k] prepend
     0.02%     perf  [kernel]                 [k] xtExtend
     0.02%     perf  [kernel]                 [k] release_metapage
     0.02%       dd  [kernel]                 [k] __do_softirq
     0.02%       dd  [kernel]                 [k] ktime_get_ts
     0.02%       dd  [kernel]                 [k] ra_submit
     0.02%       dd  [kernel]                 [k] __do_fault
     0.02%       dd  [kernel]                 [k] blk_queue_bounce
     0.02%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.02%       dd  [kernel]                 [k] __wait_on_bit_lock
     0.02%       dd  [kernel]                 [k] down_read
     0.02%     perf  [kernel]                 [k] ktime_get_ts
     0.02%     perf  [kernel]                 [k] perf_poll
     0.02%     perf  [kernel]                 [k] handle_mm_fault
     0.02%     perf  [kernel]                 [k] poll_freewait
     0.02%     perf  [kernel]                 [k] d_path
     0.02%       dd  [kernel]                 [k] __delayacct_blkio_end
     0.02%       dd  [kernel]                 [k] mempool_alloc
     0.02%       dd  [kernel]                 [k] release_pages
     0.02%       dd  [kernel]                 [k] bio_get_nr_vecs
     0.02%       dd  [kernel]                 [k] bio_phys_segments
     0.02%       dd  [kernel]                 [k] current_io_context
     0.02%       dd  [kernel]                 [k] blk_recount_segments
     0.02%       dd  [kernel]                 [k] io_schedule
     0.01%     perf  [kernel]                 [k] remove_wait_queue
     0.01%     perf  [kernel]                 [k] rcu_sched_qs
     0.01%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.01%     perf  [kernel]                 [k] find_vma
     0.01%     perf  [kernel]                 [k] __pollwait
     0.01%     perf  [kernel]                 [k] sys_poll
     0.01%     perf  [kernel]                 [k] dbAdjTree
     0.01%     perf  [kernel]                 [k] spin_unlock
     0.01%     perf  [kernel]                 [k] txBeginAnon
     0.01%     perf  [kernel]                 [k] strchr
     0.01%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.01%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.01%       dd  [kernel]                 [k] set_normalized_timespec
     0.01%       dd  [kernel]                 [k] __delayacct_blkio_start
     0.01%       dd  [kernel]                 [k] bio_alloc
     0.01%       dd  [kernel]                 [k] elv_set_request
     0.01%       dd  [kernel]                 [k] blk_unplug
     0.01%       dd  [kernel]                 [k] generic_unplug_device
     0.01%       dd  [kernel]                 [k] ata_sff_interrupt
     0.01%       dd  [kernel]                 [k] _cond_resched
     0.01%     perf  [kernel]                 [k] add_wait_queue
     0.01%     perf  [kernel]                 [k] hrtimer_init
     0.01%     perf  [kernel]                 [k] hrtimer_start_range_ns
     0.01%     perf  [kernel]                 [k] rcu_process_gp_end
     0.01%     perf  [kernel]                 [k] zone_watermark_ok
     0.01%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.01%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.01%     perf  [kernel]                 [k] poll_select_set_timeout
     0.01%     perf  [kernel]                 [k] free_poll_entry
     0.01%     perf  [kernel]                 [k] __d_path
     0.01%     perf  [kernel]                 [k] dbSplit
     0.01%     perf  [kernel]                 [k] txLock
     0.01%     perf  [kernel]                 [k] scsi_request_fn
     0.01%     perf  [kernel]                 [k] ata_sff_interrupt
     0.01%     perf  [kernel]                 [k] down_write
     0.01%       dd  [kernel]                 [k] __wake_up
     0.01%       dd  [kernel]                 [k] sync_page_killable
     0.01%       dd  [kernel]                 [k] lru_cache_add_lru
     0.01%       dd  [kernel]                 [k] bvec_alloc_bs
     0.01%       dd  [kernel]                 [k] ext4_get_block
     0.01%       dd  [kernel]                 [k] blk_update_request
     0.01%       dd  [kernel]                 [k] blk_end_bidi_request
     0.01%       dd  [kernel]                 [k] cfq_cic_lookup
     0.01%       dd  [kernel]                 [k] schedule
     0.01%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.01%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_lookup_symbol_x
     0.00%     perf  [kernel]                 [k] get_gate_vma
     0.00%     perf  [kernel]                 [k] wake_up_new_task
     0.00%     perf  [kernel]                 [k] __wake_up_sync_key
     0.00%     perf  [kernel]                 [k] dup_mm
     0.00%     perf  [kernel]                 [k] copy_process
     0.00%     perf  [kernel]                 [k] set_normalized_timespec
     0.00%     perf  [kernel]                 [k] hrtimer_cancel
     0.00%     perf  [kernel]                 [k] down_read_trylock
     0.00%     perf  [kernel]                 [k] spin_unlock_irqrestore
     0.00%     perf  [kernel]                 [k] __rcu_process_callbacks
     0.00%     perf  [kernel]                 [k] perf_mmap_fault
     0.00%     perf  [kernel]                 [k] perf_event_init_task
     0.00%     perf  [kernel]                 [k] do_read_cache_page
     0.00%     perf  [kernel]                 [k] read_cache_page_async
     0.00%     perf  [kernel]                 [k] read_cache_page
     0.00%     perf  [kernel]                 [k] determine_dirtyable_memory
     0.00%     perf  [kernel]                 [k] get_dirty_limits
     0.00%     perf  [kernel]                 [k] lru_cache_add_lru
     0.00%     perf  [kernel]                 [k] vma_prio_tree_add
     0.00%     perf  [kernel]                 [k] next_zones_zonelist
     0.00%     perf  [kernel]                 [k] unmap_vmas
     0.00%     perf  [kernel]                 [k] __vm_enough_memory
     0.00%     perf  [kernel]                 [k] unlink_file_vma
     0.00%     perf  [kernel]                 [k] page_add_file_rmap
     0.00%     perf  [kernel]                 [k] reuse_swap_page
     0.00%     perf  [kernel]                 [k] kmem_cache_free
     0.00%     perf  [kernel]                 [k] kmem_cache_alloc_notrace
     0.00%     perf  [kernel]                 [k] sys_lseek
     0.00%     perf  [kernel]                 [k] fput
     0.00%     perf  [kernel]                 [k] generic_fillattr
     0.00%     perf  [kernel]                 [k] cp_new_stat64
     0.00%     perf  [kernel]                 [k] path_get
     0.00%     perf  [kernel]                 [k] d_alloc
     0.00%     perf  [kernel]                 [k] __d_instantiate
     0.00%     perf  [kernel]                 [k] __d_lookup
     0.00%     perf  [kernel]                 [k] dput
     0.00%     perf  [kernel]                 [k] expand_files
     0.00%     perf  [kernel]                 [k] seq_release_private
     0.00%     perf  [kernel]                 [k] mangle_path
     0.00%     perf  [kernel]                 [k] seq_path
     0.00%     perf  [kernel]                 [k] seq_printf
     0.00%     perf  [kernel]                 [k] writeback_in_progress
     0.00%     perf  [kernel]                 [k] free_buffer_head
     0.00%     perf  [kernel]                 [k] alloc_buffer_head
     0.00%     perf  [kernel]                 [k] alloc_page_buffers
     0.00%     perf  [kernel]                 [k] dnotify_flush
     0.00%     perf  [kernel]                 [k] proc_lookup_de
     0.00%     perf  [kernel]                 [k] proc_pid_status
     0.00%     perf  [kernel]                 [k] metapage_get_blocks
     0.00%     perf  [kernel]                 [k] blk_end_bidi_request
     0.00%     perf  [kernel]                 [k] get_io_context
     0.00%     perf  [kernel]                 [k] prop_norm_percpu
     0.00%     perf  [kernel]                 [k] prop_fraction_percpu
     0.00%     perf  [kernel]                 [k] __rb_rotate_left
     0.00%     perf  [kernel]                 [k] rb_insert_color
     0.00%     perf  [kernel]                 [k] put_dec_trunc
     0.00%     perf  [kernel]                 [k] put_dec
     0.00%     perf  [kernel]                 [k] csum_partial
     0.00%     perf  [kernel]                 [k] credit_entropy_bits
     0.00%     perf  [kernel]                 [k] scsi_decide_disposition
     0.00%     perf  [kernel]                 [k] scsi_init_cmd_errh
     0.00%     perf  [kernel]                 [k] e100_intr
     0.00%     perf  [kernel]                 [k] e100_enable_irq
     0.00%     perf  [kernel]                 [k] e100_poll
     0.00%     perf  [kernel]                 [k] irq_handler
     0.00%     perf  [kernel]                 [k] i8042_interrupt
     0.00%     perf  [kernel]                 [k] ip_queue_xmit
     0.00%     perf  [kernel]                 [k] tcp_sync_mss
     0.00%     perf  [kernel]                 [k] fib6_age
     0.00%     perf  [kernel]                 [k] io_schedule
     0.00%     perf  [kernel]                 [k] preempt_schedule_irq
     0.00%     perf  /lib/libc-2.11.so        [.] __fopen_internal
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_getline_info
     0.00%     perf  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%     perf  /lib/ld-2.11.so          [.] _dl_fixup
     0.00%     perf  /home/cloos/bin/perf     [.] exit@plt
     0.00%       dd  [kernel]                 [k] sysenter_past_esp
     0.00%       dd  [kernel]                 [k] do_exit
     0.00%       dd  [kernel]                 [k] put_cred_rcu
     0.00%       dd  [kernel]                 [k] prepare_creds
     0.00%       dd  [kernel]                 [k] current_kernel_time
     0.00%       dd  [kernel]                 [k] check_for_new_grace_period
     0.00%       dd  [kernel]                 [k] rcu_sched_qs
     0.00%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] filemap_fault
     0.00%       dd  [kernel]                 [k] vma_prio_tree_add
     0.00%       dd  [kernel]                 [k] find_vma
     0.00%       dd  [kernel]                 [k] unmap_region
     0.00%       dd  [kernel]                 [k] mmap_region
     0.00%       dd  [kernel]                 [k] page_remove_rmap
     0.00%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.00%       dd  [kernel]                 [k] anon_vma_prepare
     0.00%       dd  [kernel]                 [k] sys_write
     0.00%       dd  [kernel]                 [k] fget_light
     0.00%       dd  [kernel]                 [k] fget
     0.00%       dd  [kernel]                 [k] fput
     0.00%       dd  [kernel]                 [k] copy_strings
     0.00%       dd  [kernel]                 [k] kernel_read
     0.00%       dd  [kernel]                 [k] acl_permission_check
     0.00%       dd  [kernel]                 [k] d_alloc
     0.00%       dd  [kernel]                 [k] __d_lookup
     0.00%       dd  [kernel]                 [k] bio_endio
     0.00%       dd  [kernel]                 [k] ext4_readpages
     0.00%       dd  [kernel]                 [k] jbd2_journal_stop
     0.00%       dd  [kernel]                 [k] dtSearch
     0.00%       dd  [kernel]                 [k] security_inode_permission
     0.00%       dd  [kernel]                 [k] req_bio_endio
     0.00%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.00%       dd  [kernel]                 [k] __rb_rotate_right
     0.00%       dd  [kernel]                 [k] sha_transform
     0.00%       dd  [kernel]                 [k] strnlen_user
     0.00%       dd  [kernel]                 [k] tty_insert_flip_string
     0.00%       dd  [kernel]                 [k] scsi_next_command
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_bsearch
     0.00%       dd  /lib/libc-2.11.so        [.] msort_with_tmp
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_getenv
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___printf_fp
     0.00%       dd  /lib/libc-2.11.so        [.] __libc_open64
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_sort_fini
     0.00%       dd  /lib/ld-2.11.so          [.] match_symbol
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_next_ld_env_entry
     0.00%       dd  /lib/ld-2.11.so          [.] strcmp
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #10: perf9.report --]
[-- Type: text/plain, Size: 33507 bytes --]

# Samples: 41396
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    11.98%       dd  [kernel]                 [k] __copy_to_user_ll
     8.74%       dd  [kernel]                 [k] __extent_read_full_page
     5.37%       dd  [kernel]                 [k] get_page_from_freelist
     4.09%       dd  [kernel]                 [k] __btrfs_map_block
     3.87%       dd  [kernel]                 [k] insert_state
     2.49%       dd  [kernel]                 [k] btrfs_lookup_bio_sums
     2.13%       dd  [kernel]                 [k] generic_file_aio_read
     2.04%       dd  [kernel]                 [k] lookup_extent_mapping
     1.99%       dd  [kernel]                 [k] __tree_search
     1.92%       dd  [kernel]                 [k] set_extent_bit
     1.85%       dd  [kernel]                 [k] btrfs_get_extent
     1.80%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     1.77%       dd  [kernel]                 [k] submit_extent_page
     1.73%       dd  [kernel]                 [k] file_read_actor
     1.73%       dd  [kernel]                 [k] read_extent_buffer
     1.69%       dd  [kernel]                 [k] add_to_page_cache_locked
     1.60%       dd  [kernel]                 [k] __bio_add_page
     1.51%       dd  [kernel]                 [k] test_range_bit
     1.47%       dd  [kernel]                 [k] __rcu_read_unlock
     1.39%       dd  [kernel]                 [k] __alloc_pages_nodemask
     1.17%       dd  [kernel]                 [k] kmem_cache_alloc
     1.13%       dd  [kernel]                 [k] find_get_page
     1.12%       dd  [kernel]                 [k] btrfs_merge_bio_hook
     1.10%       dd  [kernel]                 [k] generic_bin_search
     1.06%       dd  [kernel]                 [k] lock_extent_bits
     1.02%       dd  [kernel]                 [k] __do_page_cache_readahead
     0.93%       dd  [kernel]                 [k] put_page
     0.86%       dd  [kernel]                 [k] set_state_private
     0.84%       dd  [kernel]                 [k] btrfs_lookup_ordered_extent
     0.81%       dd  [kernel]                 [k] ____pagevec_lru_add
     0.74%       dd  [kernel]                 [k] zone_watermark_ok
     0.73%       dd  [kernel]                 [k] btrfs_find_ordered_sum
     0.73%       dd  [kernel]                 [k] extent_readpages
     0.72%       dd  [kernel]                 [k] __rcu_read_lock
     0.71%       dd  [kernel]                 [k] btrfs_map_block
     0.71%       dd  [kernel]                 [k] rb_insert_color
     0.66%       dd  [kernel]                 [k] map_private_extent_buffer
     0.66%       dd  [kernel]                 [k] scsi_dispatch_cmd
     0.62%       dd  [kernel]                 [k] lock_extent
     0.57%       dd  [kernel]                 [k] scsi_request_fn
     0.56%     perf  [kernel]                 [k] generic_file_buffered_write
     0.54%       dd  [kernel]                 [k] T.1339
     0.53%       dd  [kernel]                 [k] btrfs_set_bit_hook
     0.52%       dd  [kernel]                 [k] __make_request
     0.50%       dd  [kernel]                 [k] next_zones_zonelist
     0.48%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.43%       dd  [kernel]                 [k] btrfs_search_slot
     0.43%       dd  [kernel]                 [k] bio_add_page
     0.38%       dd  [kernel]                 [k] free_extent_map
     0.37%     perf  [kernel]                 [k] __generic_file_aio_write
     0.37%       dd  [kernel]                 [k] merge_state
     0.36%       dd  [kernel]                 [k] map_extent_buffer
     0.35%       dd  [kernel]                 [k] radix_tree_lookup
     0.33%       dd  [kernel]                 [k] blk_end_bidi_request
     0.32%       dd  [kernel]                 [k] __rb_rotate_left
     0.30%       dd  [kernel]                 [k] radix_tree_preload
     0.30%       dd  [kernel]                 [k] schedule
     0.29%       dd  [kernel]                 [k] mark_page_accessed
     0.25%       dd  [kernel]                 [k] _cond_resched
     0.25%       dd  [kernel]                 [k] do_page_fault
     0.24%       dd  [kernel]                 [k] alloc_extent_state
     0.23%       dd  [kernel]                 [k] get_request
     0.22%       dd  [kernel]                 [k] ata_sff_interrupt
     0.21%       dd  [kernel]                 [k] mutex_lock
     0.21%       dd  [kernel]                 [k] rb_next
     0.20%       dd  [kernel]                 [k] __init_waitqueue_head
     0.20%       dd  [kernel]                 [k] unmap_vmas
     0.19%     perf  [kernel]                 [k] T.1339
     0.19%     perf  [kernel]                 [k] sysenter_past_esp
     0.19%     perf  [kernel]                 [k] iov_iter_advance
     0.19%     perf  [kernel]                 [k] nobh_write_begin
     0.18%     perf  [kernel]                 [k] nobh_write_end
     0.18%       dd  [kernel]                 [k] rcu_sched_qs
     0.17%     perf  [kernel]                 [k] vfs_write
     0.17%     perf  [kernel]                 [k] generic_file_aio_write
     0.17%       dd  [kernel]                 [k] __wake_up
     0.16%     perf  [kernel]                 [k] __jfs_getxattr
     0.16%       dd  [kernel]                 [k] handle_mm_fault
     0.16%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.15%       dd  [kernel]                 [k] handle_IRQ_event
     0.15%       dd  [kernel]                 [k] read_block_for_search
     0.15%       dd  [kernel]                 [k] generic_make_request
     0.15%       dd  [kernel]                 [k] unmap_extent_buffer
     0.14%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.14%       dd  [kernel]                 [k] verify_parent_transid
     0.14%       dd  [kernel]                 [k] free_hot_cold_page
     0.14%     perf  [kernel]                 [k] do_sync_write
     0.14%     perf  [kernel]                 [k] generic_segment_checks
     0.14%     perf  [kernel]                 [k] getnstimeofday
     0.13%     perf  [kernel]                 [k] find_get_page
     0.13%       dd  [kernel]                 [k] __do_softirq
     0.13%     perf  [kernel]                 [k] rw_verify_area
     0.13%     perf  [kernel]                 [k] sys_write
     0.13%       dd  [kernel]                 [k] btrfs_key_generation
     0.12%     perf  [kernel]                 [k] jfs_write_begin
     0.12%     perf  [kernel]                 [k] ea_get
     0.12%       dd  [kernel]                 [k] mutex_unlock
     0.12%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.11%     perf  [kernel]                 [k] xtSearch
     0.11%       dd  [kernel]                 [k] extent_buffer_uptodate
     0.11%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.11%       dd  [kernel]                 [k] __queue_work
     0.11%       dd  [kernel]                 [k] btrfs_queue_worker
     0.10%       dd  [kernel]                 [k] btrfs_map_bio
     0.10%     perf  [kernel]                 [k] dbAlloc
     0.10%       dd  [kernel]                 [k] btrfs_key_blockptr
     0.10%       dd  [kernel]                 [k] btrfs_clear_path_blocking
     0.10%       dd  [kernel]                 [k] set_page_extent_mapped
     0.10%       dd  [kernel]                 [k] find_extent_buffer
     0.10%       dd  [kernel]                 [k] release_pages
     0.10%       dd  [kernel]                 [k] preempt_schedule
     0.09%     perf  [kernel]                 [k] fsnotify
     0.09%       dd  [kernel]                 [k] btrfs_lookup_csum
     0.09%     perf  [kernel]                 [k] acpi_pm_read
     0.09%       dd  [kernel]                 [k] bin_search
     0.09%     perf  [kernel]                 [k] get_page_from_freelist
     0.09%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.09%       dd  [kernel]                 [k] btrfs_tree_unlock
     0.08%     perf  [kernel]                 [k] schedule
     0.08%       dd  [kernel]                 [k] unlock_up
     0.08%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.08%     perf  [kernel]                 [k] generic_write_sync
     0.07%     perf  [vdso]                   [.] 0x000000ffffe424
     0.07%     perf  [kernel]                 [k] copy_to_user
     0.07%       dd  [kernel]                 [k] btrfs_item_size
     0.07%     perf  [kernel]                 [k] extAlloc
     0.07%       dd  [kernel]                 [k] btrfs_header_nritems
     0.07%       dd  [kernel]                 [k] cfq_set_request
     0.07%     perf  [kernel]                 [k] sys_gettimeofday
     0.07%     perf  [kernel]                 [k] find_lock_page
     0.07%     perf  [kernel]                 [k] __mark_inode_dirty
     0.07%     perf  [kernel]                 [k] __rcu_read_unlock
     0.07%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.06%     perf  [kernel]                 [k] mark_page_accessed
     0.06%     perf  [kernel]                 [k] put_page
     0.06%       dd  [kernel]                 [k] btrfs_try_spin_lock
     0.06%     perf  [kernel]                 [k] unlock_page
     0.06%     perf  [kernel]                 [k] fget_light
     0.06%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.06%     perf  [kernel]                 [k] jfs_getxattr
     0.06%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.06%       dd  [kernel]                 [k] btrfs_item_offset
     0.05%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.05%       dd  [kernel]                 [k] __kmalloc
     0.05%     perf  [kernel]                 [k] file_update_time
     0.05%     perf  [kernel]                 [k] __set_page_dirty
     0.05%     perf  [kernel]                 [k] vsnprintf
     0.05%     perf  [kernel]                 [k] file_remove_suid
     0.05%       dd  [kernel]                 [k] btrfs_clear_lock_blocking
     0.05%     perf  [kernel]                 [k] number
     0.05%     perf  [kernel]                 [k] _cond_resched
     0.04%       dd  [kernel]                 [k] btrfs_buffer_uptodate
     0.04%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.04%     perf  [kernel]                 [k] dbAllocBits
     0.04%     perf  [kernel]                 [k] __get_metapage
     0.04%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.04%       dd  [kernel]                 [k] find_vma
     0.04%       dd  [kernel]                 [k] kmem_cache_alloc_notrace
     0.04%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.04%     perf  [kernel]                 [k] __wake_up
     0.04%     perf  [kernel]                 [k] should_remove_suid
     0.04%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.04%       dd  [kernel]                 [k] get_io_context
     0.04%     perf  [kernel]                 [k] timespec_trunc
     0.04%     perf  [kernel]                 [k] format_decode
     0.04%     perf  [kernel]                 [k] strncmp
     0.04%     perf  [kernel]                 [k] strlen
     0.04%       dd  [kernel]                 [k] filemap_fault
     0.04%       dd  [kernel]                 [k] __lru_cache_add
     0.04%       dd  [kernel]                 [k] kmem_cache_free
     0.04%       dd  [kernel]                 [k] btrfs_unlock_up_safe
     0.04%       dd  [kernel]                 [k] btrfs_find_tree_block
     0.03%     perf  [kernel]                 [k] copy_page_range
     0.03%     perf  [kernel]                 [k] kfree
     0.03%     perf  [kernel]                 [k] xtLookup
     0.03%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.03%       dd  [kernel]                 [k] anon_vma_prepare
     0.03%       dd  [kernel]                 [k] btrfs_release_path
     0.03%       dd  [kernel]                 [k] btrfs_submit_bio_hook
     0.03%       dd  [kernel]                 [k] submit_bio
     0.03%     perf  [kernel]                 [k] __wake_up_bit
     0.03%     perf  [kernel]                 [k] __fsnotify_parent
     0.03%     perf  [kernel]                 [k] jfs_dirty_inode
     0.03%     perf  [kernel]                 [k] mutex_lock
     0.03%       dd  [kernel]                 [k] vm_normal_page
     0.03%       dd  [kernel]                 [k] __do_fault
     0.03%       dd  [kernel]                 [k] submit_one_bio
     0.03%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.03%     perf  [kernel]                 [k] current_fs_time
     0.03%     perf  [kernel]                 [k] do_gettimeofday
     0.03%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.03%     perf  [kernel]                 [k] ea_release
     0.03%       dd  [kernel]                 [k] free_hot_page
     0.03%       dd  [kernel]                 [k] btrfs_bio_wq_end_io
     0.03%       dd  [kernel]                 [k] btrfs_spin_on_block
     0.03%       dd  [kernel]                 [k] btrfs_tree_lock
     0.03%     perf  [kernel]                 [k] current_kernel_time
     0.03%     perf  [kernel]                 [k] __rcu_read_lock
     0.03%     perf  [kernel]                 [k] dbAllocNext
     0.03%     perf  [kernel]                 [k] extHint
     0.03%     perf  [kernel]                 [k] cap_file_permission
     0.02%     perf  [kernel]                 [k] set_page_dirty
     0.02%     perf  [kernel]                 [k] jfs_get_block
     0.02%     perf  [kernel]                 [k] xtExtend
     0.02%     perf  [kernel]                 [k] release_metapage
     0.02%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.02%       dd  [kernel]                 [k] kfree
     0.02%       dd  [kernel]                 [k] bio_alloc_bioset
     0.02%       dd  [kernel]                 [k] btrfs_root_node
     0.02%       dd  [kernel]                 [k] scsi_run_queue
     0.02%       dd  [kernel]                 [k] bio_put
     0.02%       dd  [kernel]                 [k] btrfs_set_lock_blocking
     0.02%       dd  [kernel]                 [k] cfq_cic_lookup
     0.02%       dd  [kernel]                 [k] sha_transform
     0.02%     perf  [kernel]                 [k] sysenter_do_call
     0.02%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.02%     perf  [kernel]                 [k] read_cache_page
     0.02%     perf  [kernel]                 [k] security_file_permission
     0.02%     perf  [kernel]                 [k] ata_sff_interrupt
     0.02%     perf  [kernel]                 [k] mutex_unlock
     0.02%     perf  [kernel]                 [k] down_read
     0.02%     perf  [kernel]                 [k] do_page_fault
     0.02%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.02%       dd  [kernel]                 [k] page_evictable
     0.02%       dd  [kernel]                 [k] blk_queue_bounce
     0.02%       dd  [kernel]                 [k] bvec_alloc_bs
     0.02%       dd  [kernel]                 [k] blk_rq_init
     0.02%       dd  [kernel]                 [k] blk_update_request
     0.02%       dd  [kernel]                 [k] current_io_context
     0.02%       dd  [kernel]                 [k] kref_put
     0.02%       dd  [kernel]                 [k] credit_entropy_bits
     0.02%       dd  [kernel]                 [k] add_timer_randomness
     0.02%     perf  [kernel]                 [k] kmem_cache_free
     0.02%     perf  [kernel]                 [k] __find_get_block_slow
     0.02%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.02%       dd  [kernel]                 [k] mempool_alloc_slab
     0.02%       dd  [kernel]                 [k] free_extent_buffer
     0.02%       dd  [kernel]                 [k] get_request_wait
     0.01%     perf  [kernel]                 [k] rcu_sched_qs
     0.01%     perf  [kernel]                 [k] zone_watermark_ok
     0.01%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.01%     perf  [kernel]                 [k] inode_add_bytes
     0.01%     perf  [kernel]                 [k] __d_path
     0.01%     perf  [kernel]                 [k] txLock
     0.01%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.01%       dd  [kernel]                 [k] copy_strings
     0.01%       dd  [kernel]                 [k] check_block
     0.01%       dd  [kernel]                 [k] btrfs_free_path
     0.01%       dd  [kernel]                 [k] init_request_from_bio
     0.01%       dd  [kernel]                 [k] blk_run_queue
     0.01%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.01%       dd  [kernel]                 [k] scsi_finish_command
     0.01%       dd  [kernel]                 [k] scsi_decide_disposition
     0.01%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.01%     perf  [kernel]                 [k] handle_mm_fault
     0.01%     perf  [kernel]                 [k] dbAdjTree
     0.01%     perf  [kernel]                 [k] dbAllocDmap
     0.01%     perf  [kernel]                 [k] blk_end_bidi_request
     0.01%       dd  [kernel]                 [k] down_read_trylock
     0.01%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.01%       dd  [kernel]                 [k] mempool_alloc
     0.01%       dd  [kernel]                 [k] __d_lookup
     0.01%       dd  [kernel]                 [k] bio_init
     0.01%       dd  [kernel]                 [k] bio_alloc
     0.01%       dd  [kernel]                 [k] mpage_end_io_read
     0.01%       dd  [kernel]                 [k] btrfs_set_path_blocking
     0.01%       dd  [kernel]                 [k] btrfs_lock_root_node
     0.01%       dd  [kernel]                 [k] end_workqueue_bio
     0.01%       dd  [kernel]                 [k] strnlen_user
     0.01%       dd  [kernel]                 [k] __scsi_put_command
     0.01%       dd  [kernel]                 [k] scsi_io_completion
     0.01%       dd  [kernel]                 [k] sd_done
     0.01%     perf  [kernel]                 [k] up_read
     0.01%     perf  [kernel]                 [k] perf_mmap_fault
     0.01%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.01%     perf  [kernel]                 [k] __do_fault
     0.01%     perf  [kernel]                 [k] kmem_cache_alloc
     0.01%     perf  [kernel]                 [k] mnt_clone_write
     0.01%     perf  [kernel]                 [k] mnt_want_write_file
     0.01%     perf  [kernel]                 [k] alloc_buffer_head
     0.01%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.01%     perf  [kernel]                 [k] strchr
     0.01%       dd  [kernel]                 [k] req_bio_endio
     0.01%       dd  [kernel]                 [k] blk_recount_segments
     0.01%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.01%       dd  [kernel]                 [k] radix_tree_next_hole
     0.01%       dd  [kernel]                 [k] scsi_device_unbusy
     0.01%       dd                 b70d746f  [.] 0x000000b70d746f
     0.01%     perf  [kernel]                 [k] __do_softirq
     0.01%     perf  [kernel]                 [k] handle_IRQ_event
     0.01%     perf  [kernel]                 [k] do_read_cache_page
     0.01%     perf  [kernel]                 [k] do_sys_poll
     0.01%     perf  [kernel]                 [k] seq_printf
     0.01%     perf  [kernel]                 [k] dbSplit
     0.01%     perf  [kernel]                 [k] txBeginAnon
     0.01%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.01%       dd  [kernel]                 [k] run_timer_softirq
     0.01%       dd  [kernel]                 [k] queue_work
     0.01%       dd  [kernel]                 [k] up_read
     0.01%       dd  [kernel]                 [k] check_for_new_grace_period
     0.01%       dd  [kernel]                 [k] unlock_page
     0.01%       dd  [kernel]                 [k] test_clear_page_writeback
     0.01%       dd  [kernel]                 [k] __vm_enough_memory
     0.01%       dd  [kernel]                 [k] mmap_region
     0.01%       dd  [kernel]                 [k] page_remove_rmap
     0.01%       dd  [kernel]                 [k] setup_object
     0.01%       dd  [kernel]                 [k] touch_atime
     0.01%       dd  [kernel]                 [k] end_buffer_async_write
     0.01%       dd  [kernel]                 [k] fsnotify
     0.01%       dd  [kernel]                 [k] elv_set_request
     0.01%       dd  [kernel]                 [k] blk_done_softirq
     0.01%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.01%       dd  [kernel]                 [k] get_device
     0.01%       dd  [kernel]                 [k] put_device
     0.01%       dd  [kernel]                 [k] scsi_put_command
     0.01%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_relocate_object
     0.00%     perf  [kernel]                 [k] dup_mm
     0.00%     perf  [kernel]                 [k] wait_on_page_read
     0.00%     perf  [kernel]                 [k] read_cache_page_async
     0.00%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.00%     perf  [kernel]                 [k] activate_page
     0.00%     perf  [kernel]                 [k] do_wp_page
     0.00%     perf  [kernel]                 [k] mnt_drop_write
     0.00%     perf  [kernel]                 [k] mangle_path
     0.00%     perf  [kernel]                 [k] alloc_page_buffers
     0.00%     perf  [kernel]                 [k] show_map_vma
     0.00%     perf  [kernel]                 [k] show_map
     0.00%     perf  [kernel]                 [k] btrfs_queue_worker
     0.00%     perf  [kernel]                 [k] _copy_from_user
     0.00%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.00%       dd  [kernel]                 [k] sysenter_past_esp
     0.00%       dd  [kernel]                 [k] __wake_up_bit
     0.00%       dd  [kernel]                 [k] rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] do_wp_page
     0.00%       dd  [kernel]                 [k] __get_user_pages
     0.00%       dd  [kernel]                 [k] __remove_shared_vm_struct
     0.00%       dd  [kernel]                 [k] anon_vma_unlink
     0.00%       dd  [kernel]                 [k] __slab_alloc
     0.00%       dd  [kernel]                 [k] get_slab
     0.00%       dd  [kernel]                 [k] nameidata_to_filp
     0.00%       dd  [kernel]                 [k] bio_endio
     0.00%       dd  [kernel]                 [k] bio_phys_segments
     0.00%       dd  [kernel]                 [k] dnotify_flush
     0.00%       dd  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.00%       dd  [kernel]                 [k] btrfs_item_size_nr
     0.00%       dd  [kernel]                 [k] cap_inode_permission
     0.00%       dd  [kernel]                 [k] kobject_put
     0.00%       dd  [kernel]                 [k] prio_tree_replace
     0.00%       dd  [kernel]                 [k] prio_tree_insert
     0.00%       dd  [kernel]                 [k] radix_tree_prev_hole
     0.00%       dd  [kernel]                 [k] _copy_from_user
     0.00%       dd  [kernel]                 [k] scsi_pool_free_command
     0.00%       dd  [kernel]                 [k] scsi_sg_free
     0.00%       dd  [kernel]                 [k] scsi_softirq_done
     0.00%       dd  [kernel]                 [k] error_code
     0.00%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.00%       dd  /lib/ld-2.11.so          [.] 0x0000000000c08a
     0.00%     perf  [kernel]                 [k] arch_dup_task_struct
     0.00%     perf  [kernel]                 [k] try_to_wake_up
     0.00%     perf  [kernel]                 [k] wake_up_new_task
     0.00%     perf  [kernel]                 [k] get_signal_to_deliver
     0.00%     perf  [kernel]                 [k] hrtimer_try_to_cancel
     0.00%     perf  [kernel]                 [k] hrtimer_cancel
     0.00%     perf  [kernel]                 [k] up_write
     0.00%     perf  [kernel]                 [k] ktime_get_ts
     0.00%     perf  [kernel]                 [k] check_for_new_grace_period
     0.00%     perf  [kernel]                 [k] perf_poll
     0.00%     perf  [kernel]                 [k] perf_event_init_task
     0.00%     perf  [kernel]                 [k] __page_cache_alloc
     0.00%     perf  [kernel]                 [k] release_pages
     0.00%     perf  [kernel]                 [k] __lru_cache_add
     0.00%     perf  [kernel]                 [k] page_evictable
     0.00%     perf  [kernel]                 [k] next_zones_zonelist
     0.00%     perf  [kernel]                 [k] vm_normal_page
     0.00%     perf  [kernel]                 [k] find_vma
     0.00%     perf  [kernel]                 [k] __vma_link_rb
     0.00%     perf  [kernel]                 [k] unmap_region
     0.00%     perf  [kernel]                 [k] do_mmap_pgoff
     0.00%     perf  [kernel]                 [k] anon_vma_link
     0.00%     perf  [kernel]                 [k] sys_lseek
     0.00%     perf  [kernel]                 [k] __fput
     0.00%     perf  [kernel]                 [k] acl_permission_check
     0.00%     perf  [kernel]                 [k] exec_permission
     0.00%     perf  [kernel]                 [k] new_inode
     0.00%     perf  [kernel]                 [k] mntput_no_expire
     0.00%     perf  [kernel]                 [k] seq_putc
     0.00%     perf  [kernel]                 [k] set_bh_page
     0.00%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.00%     perf  [kernel]                 [k] bio_fs_destructor
     0.00%     perf  [kernel]                 [k] mpage_end_io_read
     0.00%     perf  [kernel]                 [k] locks_remove_flock
     0.00%     perf  [kernel]                 [k] m_next
     0.00%     perf  [kernel]                 [k] proc_alloc_inode
     0.00%     perf  [kernel]                 [k] render_sigset_t
     0.00%     perf  [kernel]                 [k] proc_pid_status
     0.00%     perf  [kernel]                 [k] xtSplitRoot
     0.00%     perf  [kernel]                 [k] xtInsert
     0.00%     perf  [kernel]                 [k] req_bio_endio
     0.00%     perf  [kernel]                 [k] blk_update_request
     0.00%     perf  [kernel]                 [k] blk_update_bidi_request
     0.00%     perf  [kernel]                 [k] disk_map_sector_rcu
     0.00%     perf  [kernel]                 [k] kobject_put
     0.00%     perf  [kernel]                 [k] prop_fraction_single
     0.00%     perf  [kernel]                 [k] __rb_rotate_left
     0.00%     perf  [kernel]                 [k] put_dec_trunc
     0.00%     perf  [kernel]                 [k] string
     0.00%     perf  [kernel]                 [k] __sg_free_table
     0.00%     perf  [kernel]                 [k] add_timer_randomness
     0.00%     perf  [kernel]                 [k] __scsi_put_command
     0.00%     perf  [kernel]                 [k] scsi_put_command
     0.00%     perf  [kernel]                 [k] __scsi_release_buffers
     0.00%     perf  [kernel]                 [k] scsi_request_fn
     0.00%     perf  [kernel]                 [k] scsi_next_command
     0.00%     perf  [kernel]                 [k] preempt_schedule_irq
     0.00%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.00%     perf  /lib/libpthread-2.11.so  [.] __fork
     0.00%     perf  /lib/libc-2.11.so        [.] fgets
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_old_init
     0.00%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.00%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.00%     perf  /home/cloos/bin/perf     [.] hex2u64
     0.00%       dd  [kernel]                 [k] init_new_context
     0.00%       dd  [kernel]                 [k] do_set_thread_area
     0.00%       dd  [kernel]                 [k] sched_move_task
     0.00%       dd  [kernel]                 [k] run_hrtimer_softirq
     0.00%       dd  [kernel]                 [k] hrtimer_try_to_cancel
     0.00%       dd  [kernel]                 [k] rcu_bh_qs
     0.00%       dd  [kernel]                 [k] rcu_process_gp_end
     0.00%       dd  [kernel]                 [k] __call_rcu
     0.00%       dd  [kernel]                 [k] perf_output_unlock
     0.00%       dd  [kernel]                 [k] perf_event_mmap_ctx
     0.00%       dd  [kernel]                 [k] __perf_event_mmap
     0.00%       dd  [kernel]                 [k] ondemand_readahead
     0.00%       dd  [kernel]                 [k] lru_cache_add_lru
     0.00%       dd  [kernel]                 [k] vma_prio_tree_remove
     0.00%       dd  [kernel]                 [k] vm_stat_account
     0.00%       dd  [kernel]                 [k] expand_downwards
     0.00%       dd  [kernel]                 [k] unlink_file_vma
     0.00%       dd  [kernel]                 [k] mprotect_fixup
     0.00%       dd  [kernel]                 [k] sys_mprotect
     0.00%       dd  [kernel]                 [k] filp_close
     0.00%       dd  [kernel]                 [k] rw_verify_area
     0.00%       dd  [kernel]                 [k] vfs_read
     0.00%       dd  [kernel]                 [k] file_free_rcu
     0.00%       dd  [kernel]                 [k] fget_light
     0.00%       dd  [kernel]                 [k] __fput
     0.00%       dd  [kernel]                 [k] check_unsafe_exec
     0.00%       dd  [kernel]                 [k] path_get
     0.00%       dd  [kernel]                 [k] __follow_mount
     0.00%       dd  [kernel]                 [k] path_put
     0.00%       dd  [kernel]                 [k] link_path_walk
     0.00%       dd  [kernel]                 [k] do_filp_open
     0.00%       dd  [kernel]                 [k] spin_unlock
     0.00%       dd  [kernel]                 [k] spin_unlock
     0.00%       dd  [kernel]                 [k] alloc_fd
     0.00%       dd  [kernel]                 [k] bio_get_nr_vecs
     0.00%       dd  [kernel]                 [k] bio_fs_destructor
     0.00%       dd  [kernel]                 [k] __fsnotify_parent
     0.00%       dd  [kernel]                 [k] load_elf_binary
     0.00%       dd  [kernel]                 [k] jfs_release
     0.00%       dd  [kernel]                 [k] btrfs_congested_fn
     0.00%       dd  [kernel]                 [k] is_file_shm_hugepages
     0.00%       dd  [kernel]                 [k] get_vfs_caps_from_disk
     0.00%       dd  [kernel]                 [k] security_inode_getattr
     0.00%       dd  [kernel]                 [k] cap_file_permission
     0.00%       dd  [kernel]                 [k] blk_get_backing_dev_info
     0.00%       dd  [kernel]                 [k] blk_backing_dev_unplug
     0.00%       dd  [kernel]                 [k] blk_unplug_timeout
     0.00%       dd  [kernel]                 [k] blk_update_bidi_request
     0.00%       dd  [kernel]                 [k] blk_end_request
     0.00%       dd  [kernel]                 [k] cfq_get_queue
     0.00%       dd  [kernel]                 [k] kref_get
     0.00%       dd  [kernel]                 [k] prio_tree_remove
     0.00%       dd  [kernel]                 [k] copy_to_user
     0.00%       dd  [kernel]                 [k] __sg_free_table
     0.00%       dd  [kernel]                 [k] add_disk_randomness
     0.00%       dd  [kernel]                 [k] n_tty_write
     0.00%       dd  [kernel]                 [k] tty_ldisc_deref
     0.00%       dd  [kernel]                 [k] tty_insert_flip_string
     0.00%       dd  [kernel]                 [k] tty_flip_buffer_push
     0.00%       dd  [kernel]                 [k] __scsi_release_buffers
     0.00%       dd  [kernel]                 [k] pfifo_fast_dequeue
     0.00%       dd  [kernel]                 [k] netlink_broadcast
     0.00%       dd  /lib/libc-2.11.so        [.] _nl_find_locale
     0.00%       dd  /lib/libc-2.11.so        [.] _IO_vfprintf_internal
     0.00%       dd  /lib/libc-2.11.so        [.] _IO_new_file_xsputn
     0.00%       dd  /lib/libc-2.11.so        [.] __strchrnul
     0.00%       dd  /lib/libc-2.11.so        [.] intel_check_word
     0.00%       dd  /lib/libc-2.11.so        [.] ___fprintf_chk
     0.00%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_map_object_from_fd
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_init_paths
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_cache_libcmp
     0.00%       dd  /lib/ld-2.11.so          [.] check_match.8476
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_lookup_symbol_x
     0.00%       dd  /lib/ld-2.11.so          [.] __libc_close
     0.00%       dd  /lib/ld-2.11.so          [.] __mmap
     0.00%       dd  /lib/ld-2.11.so          [.] 0x0000000001943e
     0.00%       dd  /bin/dd                  [.] 0x000000000013b6
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #11: perf10.report --]
[-- Type: text/plain, Size: 28650 bytes --]

# Samples: 25313
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    17.81%       dd  [kernel]                 [k] __copy_to_user_ll
    12.23%       dd  [kernel]                 [k] do_mpage_readpage
     9.82%       dd  [kernel]                 [k] get_page_from_freelist
     3.43%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     3.37%       dd  [kernel]                 [k] generic_file_aio_read
     3.37%       dd  [kernel]                 [k] __bio_add_page
     3.34%       dd  [kernel]                 [k] add_to_page_cache_locked
     2.82%       dd  [kernel]                 [k] file_read_actor
     2.68%       dd  [kernel]                 [k] __alloc_pages_nodemask
     1.96%       dd  [kernel]                 [k] scsi_request_fn
     1.89%       dd  [kernel]                 [k] scsi_dispatch_cmd
     1.83%       dd  [kernel]                 [k] __do_page_cache_readahead
     1.76%       dd  [kernel]                 [k] find_get_page
     1.59%       dd  [kernel]                 [k] ____pagevec_lru_add
     1.55%       dd  [kernel]                 [k] put_page
     1.46%       dd  [kernel]                 [k] __rcu_read_unlock
     1.33%       dd  [kernel]                 [k] zone_watermark_ok
     1.29%       dd  [kernel]                 [k] mpage_readpages
     1.16%       dd  [kernel]                 [k] __lru_cache_add
     1.01%       dd  [kernel]                 [k] bio_add_page
     0.98%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.83%       dd  [kernel]                 [k] next_zones_zonelist
     0.66%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.66%       dd  [kernel]                 [k] radix_tree_preload
     0.60%     perf  [kernel]                 [k] generic_file_buffered_write
     0.56%       dd  [kernel]                 [k] __rcu_read_lock
     0.56%       dd  [kernel]                 [k] blk_end_bidi_request
     0.56%       dd  [kernel]                 [k] __make_request
     0.51%       dd  [kernel]                 [k] radix_tree_lookup
     0.49%       dd  [kernel]                 [k] get_request
     0.42%     perf  [kernel]                 [k] __generic_file_aio_write
     0.40%       dd  [kernel]                 [k] _cond_resched
     0.39%       dd  [kernel]                 [k] unmap_vmas
     0.39%       dd  [kernel]                 [k] do_page_fault
     0.38%       dd  [kernel]                 [k] kmem_cache_alloc
     0.36%       dd  [kernel]                 [k] schedule
     0.32%       dd  [kernel]                 [k] handle_mm_fault
     0.31%       dd  [kernel]                 [k] mark_page_accessed
     0.29%     perf  [kernel]                 [k] T.1339
     0.27%       dd  [kernel]                 [k] generic_make_request
     0.26%       dd  [kernel]                 [k] mpage_end_io_read
     0.25%     perf  [kernel]                 [k] find_get_page
     0.25%       dd  [kernel]                 [k] T.1339
     0.24%     perf  [kernel]                 [k] nobh_write_begin
     0.22%     perf  [kernel]                 [k] xtSearch
     0.22%       dd  [kernel]                 [k] free_hot_cold_page
     0.22%       dd  [kernel]                 [k] ata_sff_interrupt
     0.22%     perf  [kernel]                 [k] sysenter_past_esp
     0.22%     perf  [kernel]                 [k] __jfs_getxattr
     0.21%     perf  [kernel]                 [k] iov_iter_advance
     0.21%     perf  [kernel]                 [k] vfs_write
     0.20%     perf  [kernel]                 [k] nobh_write_end
     0.20%       dd  [kernel]                 [k] ext4_get_blocks
     0.19%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.18%     perf  [kernel]                 [k] do_sync_write
     0.18%     perf  [kernel]                 [k] generic_file_aio_write
     0.17%       dd  [kernel]                 [k] ext4_ext_get_blocks
     0.17%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.16%     perf  [kernel]                 [k] extAlloc
     0.15%     perf  [kernel]                 [k] dbAlloc
     0.15%     perf  [kernel]                 [k] number
     0.15%       dd  [kernel]                 [k] sha_transform
     0.15%     perf  [kernel]                 [k] getnstimeofday
     0.15%     perf  [kernel]                 [k] rw_verify_area
     0.14%       dd  [kernel]                 [k] handle_IRQ_event
     0.14%       dd  [kernel]                 [k] __do_softirq
     0.13%     perf  [kernel]                 [k] sys_write
     0.13%     perf  [kernel]                 [k] vsnprintf
     0.13%       dd  [kernel]                 [k] ext4_data_block_valid
     0.13%       dd  [kernel]                 [k] blk_rq_init
     0.13%       dd  [kernel]                 [k] cfq_set_request
     0.13%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.12%     perf  [kernel]                 [k] get_page_from_freelist
     0.12%     perf  [kernel]                 [k] generic_segment_checks
     0.12%       dd  [kernel]                 [k] __wake_up_bit
     0.12%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.11%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.11%     perf  [kernel]                 [k] jfs_write_begin
     0.11%     perf  [kernel]                 [k] acpi_pm_read
     0.11%       dd  [kernel]                 [k] bio_alloc_bioset
     0.11%       dd  [kernel]                 [k] rcu_sched_qs
     0.11%     perf  [kernel]                 [k] ea_get
     0.10%     perf  [kernel]                 [k] __rcu_read_unlock
     0.10%     perf  [kernel]                 [k] __set_page_dirty
     0.10%       dd  [kernel]                 [k] __wake_up
     0.09%     perf  [vdso]                   [.] 0x000000ffffe424
     0.09%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.09%     perf  [kernel]                 [k] fsnotify
     0.09%     perf  [kernel]                 [k] __mark_inode_dirty
     0.09%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.09%       dd  [kernel]                 [k] ext4_get_block
     0.09%       dd  [kernel]                 [k] blk_update_request
     0.08%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.08%     perf  [kernel]                 [k] put_page
     0.08%     perf  [kernel]                 [k] file_remove_suid
     0.08%     perf  [kernel]                 [k] unlock_page
     0.08%     perf  [kernel]                 [k] copy_page_range
     0.08%     perf  [kernel]                 [k] schedule
     0.08%     perf  [kernel]                 [k] copy_to_user
     0.08%       dd  [kernel]                 [k] ondemand_readahead
     0.08%       dd  [kernel]                 [k] release_pages
     0.08%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.07%     perf  [kernel]                 [k] mark_page_accessed
     0.07%     perf  [kernel]                 [k] __get_metapage
     0.07%     perf  [kernel]                 [k] format_decode
     0.07%       dd  [kernel]                 [k] kref_put
     0.07%     perf  [kernel]                 [k] generic_write_sync
     0.07%     perf  [kernel]                 [k] xtLookup
     0.07%     perf  [kernel]                 [k] ea_release
     0.06%     perf  [kernel]                 [k] should_remove_suid
     0.06%       dd  [kernel]                 [k] filemap_fault
     0.06%       dd  [kernel]                 [k] anon_vma_prepare
     0.06%       dd  [kernel]                 [k] mpage_alloc
     0.06%     perf  [kernel]                 [k] sys_gettimeofday
     0.06%     perf  [kernel]                 [k] __wake_up_bit
     0.06%     perf  [kernel]                 [k] jfs_getxattr
     0.06%     perf  [kernel]                 [k] find_lock_page
     0.06%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.06%       dd  [kernel]                 [k] up_read
     0.06%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.05%     perf  [kernel]                 [k] __wake_up
     0.05%     perf  [kernel]                 [k] current_fs_time
     0.05%     perf  [kernel]                 [k] file_update_time
     0.05%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.05%       dd  [kernel]                 [k] down_read_trylock
     0.05%       dd  [kernel]                 [k] mempool_alloc
     0.05%       dd  [kernel]                 [k] find_vma
     0.05%       dd  [kernel]                 [k] kmem_cache_free
     0.05%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.05%       dd  [kernel]                 [k] submit_bio
     0.05%     perf  [kernel]                 [k] timespec_trunc
     0.05%     perf  [kernel]                 [k] kfree
     0.05%     perf  [kernel]                 [k] jfs_dirty_inode
     0.05%     perf  [kernel]                 [k] strncmp
     0.05%       dd  [kernel]                 [k] scsi_finish_command
     0.05%       dd  [kernel]                 [k] preempt_schedule
     0.04%     perf  [kernel]                 [k] do_gettimeofday
     0.04%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.04%     perf  [kernel]                 [k] release_metapage
     0.04%       dd  [kernel]                 [k] __do_fault
     0.04%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.04%       dd  [kernel]                 [k] blk_queue_bounce
     0.04%       dd  [kernel]                 [k] init_request_from_bio
     0.04%       dd  [kernel]                 [k] get_io_context
     0.04%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.04%     perf  [kernel]                 [k] fget_light
     0.04%     perf  [kernel]                 [k] dbAllocBits
     0.04%       dd  [kernel]                 [k] free_hot_page
     0.04%       dd  [kernel]                 [k] ra_submit
     0.04%       dd  [kernel]                 [k] vm_normal_page
     0.04%       dd  [kernel]                 [k] bio_get_nr_vecs
     0.04%       dd  [kernel]                 [k] bio_init
     0.04%       dd  [kernel]                 [k] blk_done_softirq
     0.04%     perf  [kernel]                 [k] current_kernel_time
     0.04%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.04%     perf  [kernel]                 [k] xtExtend
     0.04%     perf  [kernel]                 [k] extHint
     0.04%     perf  [kernel]                 [k] strlen
     0.04%     perf  [kernel]                 [k] do_page_fault
     0.04%       dd  [kernel]                 [k] unlock_page
     0.04%       dd  [kernel]                 [k] page_cache_async_readahead
     0.04%       dd  [kernel]                 [k] down_read
     0.03%     perf  [kernel]                 [k] perf_mmap_fault
     0.03%     perf  [kernel]                 [k] mutex_unlock
     0.03%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.03%       dd  [kernel]                 [k] generic_unplug_device
     0.03%     perf  [kernel]                 [k] __rcu_read_lock
     0.03%     perf  [kernel]                 [k] __do_fault
     0.03%     perf  [kernel]                 [k] handle_mm_fault
     0.03%     perf  [kernel]                 [k] kmem_cache_free
     0.03%     perf  [kernel]                 [k] jfs_get_block
     0.03%     perf  [kernel]                 [k] dbAllocDmap
     0.03%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.03%     perf  [kernel]                 [k] cap_file_permission
     0.03%     perf  [kernel]                 [k] strchr
     0.03%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.03%     perf  [kernel]                 [k] mutex_lock
     0.03%       dd  [kernel]                 [k] page_evictable
     0.03%       dd  [kernel]                 [k] bio_fs_destructor
     0.03%       dd  [kernel]                 [k] blk_unplug
     0.03%       dd  [kernel]                 [k] get_request_wait
     0.03%       dd  [kernel]                 [k] blk_recount_segments
     0.03%       dd  [kernel]                 [k] __scsi_put_command
     0.03%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.03%       dd  [kernel]                 [k] scsi_run_queue
     0.02%     perf  [kernel]                 [k] do_read_cache_page
     0.02%     perf  [kernel]                 [k] read_cache_page
     0.02%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.02%     perf  [kernel]                 [k] dbAdjTree
     0.02%     perf  [kernel]                 [k] dbAllocNext
     0.02%     perf  [kernel]                 [k] _cond_resched
     0.02%       dd  [kernel]                 [k] mempool_free
     0.02%       dd  [kernel]                 [k] kobject_put
     0.02%       dd  [kernel]                 [k] strnlen_user
     0.02%       dd  [kernel]                 [k] sd_done
     0.02%     perf  [kernel]                 [k] inode_add_bytes
     0.02%     perf  [kernel]                 [k] mpage_end_io_read
     0.02%     perf  [kernel]                 [k] __fsnotify_parent
     0.02%     perf  [kernel]                 [k] txLock
     0.02%     perf  [kernel]                 [k] down_read
     0.02%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.02%       dd  [kernel]                 [k] put_pages_list
     0.02%       dd  [kernel]                 [k] do_wp_page
     0.02%       dd  [kernel]                 [k] bvec_alloc_bs
     0.02%       dd  [kernel]                 [k] ext4_readpages
     0.02%       dd  [kernel]                 [k] req_bio_endio
     0.02%       dd  [kernel]                 [k] credit_entropy_bits
     0.02%       dd  [kernel]                 [k] scsi_decide_disposition
     0.02%       dd  [kernel]                 [k] scsi_softirq_done
     0.02%       dd  /lib/ld-2.11.so          [.] strcmp
     0.02%     perf  [kernel]                 [k] up_read
     0.02%     perf  [kernel]                 [k] up_write
     0.02%     perf  [kernel]                 [k] rcu_sched_qs
     0.02%     perf  [kernel]                 [k] perf_poll
     0.02%     perf  [kernel]                 [k] zone_watermark_ok
     0.02%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.02%     perf  [kernel]                 [k] prepend
     0.02%     perf  [kernel]                 [k] mnt_drop_write
     0.02%     perf  [kernel]                 [k] mnt_clone_write
     0.02%     perf  [kernel]                 [k] alloc_buffer_head
     0.02%     perf  [kernel]                 [k] dbSplit
     0.02%     perf  [kernel]                 [k] security_file_permission
     0.02%       dd  [kernel]                 [k] __remove_shared_vm_struct
     0.02%       dd  [kernel]                 [k] copy_strings
     0.02%       dd  [kernel]                 [k] do_lookup
     0.02%       dd  [kernel]                 [k] bio_phys_segments
     0.02%       dd  [kernel]                 [k] mpage_bio_submit
     0.02%       dd  [kernel]                 [k] blk_update_bidi_request
     0.02%       dd  [kernel]                 [k] current_io_context
     0.02%       dd  [kernel]                 [k] cfq_cic_lookup
     0.02%       dd  [kernel]                 [k] add_timer_randomness
     0.02%       dd  [kernel]                 [k] scsi_put_command
     0.02%       dd  [kernel]                 [k] scsi_sg_free
     0.02%       dd  [kernel]                 [k] scsi_device_unbusy
     0.01%     perf  [kernel]                 [k] dup_mm
     0.01%     perf  [kernel]                 [k] __do_softirq
     0.01%     perf  [kernel]                 [k] down_read_trylock
     0.01%     perf  [kernel]                 [k] read_cache_page_async
     0.01%     perf  [kernel]                 [k] set_page_dirty
     0.01%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.01%     perf  [kernel]                 [k] do_wp_page
     0.01%     perf  [kernel]                 [k] mangle_path
     0.01%     perf  [kernel]                 [k] __find_get_block_slow
     0.01%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.01%     perf  [kernel]                 [k] show_map_vma
     0.01%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.01%     perf  [kernel]                 [k] blk_end_bidi_request
     0.01%     perf  [kernel]                 [k] ata_sff_interrupt
     0.01%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.01%       dd  [kernel]                 [k] run_timer_softirq
     0.01%       dd  [kernel]                 [k] mempool_alloc_slab
     0.01%       dd  [kernel]                 [k] mmap_region
     0.01%       dd  [kernel]                 [k] page_remove_rmap
     0.01%       dd  [kernel]                 [k] blk_run_queue
     0.01%       dd  [kernel]                 [k] scsi_log_send
     0.01%       dd  [kernel]                 [k] __scsi_release_buffers
     0.01%       dd  [kernel]                 [k] scsi_io_completion
     0.01%       dd                 b7771cc4  [.] 0x000000b7771cc4
     0.01%     perf  [kernel]                 [k] sysenter_do_call
     0.01%     perf  [kernel]                 [k] timespec_add_safe
     0.01%     perf  [kernel]                 [k] rcu_process_gp_end
     0.01%     perf  [kernel]                 [k] wait_on_page_read
     0.01%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.01%     perf  [kernel]                 [k] find_vma
     0.01%     perf  [kernel]                 [k] __vma_link_rb
     0.01%     perf  [kernel]                 [k] kmem_cache_alloc
     0.01%     perf  [kernel]                 [k] __d_path
     0.01%     perf  [kernel]                 [k] mnt_want_write_file
     0.01%     perf  [kernel]                 [k] seq_printf
     0.01%     perf  [kernel]                 [k] seq_read
     0.01%     perf  [kernel]                 [k] bio_free
     0.01%     perf  [kernel]                 [k] xtInsert
     0.01%     perf  [kernel]                 [k] dbAllocCtl
     0.01%     perf  [kernel]                 [k] cfq_idle_slice_timer
     0.01%     perf  [kernel]                 [k] radix_tree_preload
     0.01%     perf  [kernel]                 [k] credit_entropy_bits
     0.01%     perf  [kernel]                 [k] mix_pool_bytes_extract
     0.01%     perf  [kernel]                 [k] scsi_request_fn
     0.01%     perf  /lib/libc-2.11.so        [.] __libc_poll
     0.01%       dd  [kernel]                 [k] sysenter_past_esp
     0.01%       dd  [kernel]                 [k] test_ti_thread_flag
     0.01%       dd  [kernel]                 [k] unshare_fd
     0.01%       dd  [kernel]                 [k] up_write
     0.01%       dd  [kernel]                 [k] rcu_process_gp_end
     0.01%       dd  [kernel]                 [k] perf_output_copy
     0.01%       dd  [kernel]                 [k] lru_cache_add_lru
     0.01%       dd  [kernel]                 [k] vma_adjust
     0.01%       dd  [kernel]                 [k] mprotect_fixup
     0.01%       dd  [kernel]                 [k] path_get
     0.01%       dd  [kernel]                 [k] __d_lookup
     0.01%       dd  [kernel]                 [k] mntput_no_expire
     0.01%       dd  [kernel]                 [k] bio_put
     0.01%       dd  [kernel]                 [k] bio_free
     0.01%       dd  [kernel]                 [k] load_elf_binary
     0.01%       dd  [kernel]                 [k] elv_set_request
     0.01%       dd  [kernel]                 [k] blk_end_request
     0.01%       dd  [kernel]                 [k] kobject_get
     0.01%       dd  [kernel]                 [k] kref_get
     0.01%       dd  [kernel]                 [k] prio_tree_insert
     0.01%       dd  [kernel]                 [k] rb_insert_color
     0.01%       dd  [kernel]                 [k] _copy_from_user
     0.01%       dd  [kernel]                 [k] __sg_free_table
     0.01%       dd  [kernel]                 [k] do_output_char
     0.01%       dd  [kernel]                 [k] put_device
     0.01%       dd  [kernel]                 [k] scsi_log_completion
     0.01%       dd  [kernel]                 [k] acpi_pm_read
     0.01%       dd  [kernel]                 [k] error_code
     0.01%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_map_object_from_fd
     0.01%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.01%       dd  /lib/ld-2.11.so          [.] ___fxstat64
     0.01%       dd  /bin/dd                  [.] dd_copy
     0.00%     perf  [kernel]                 [k] pte_alloc_one
     0.00%     perf  [kernel]                 [k] get_gate_vma
     0.00%     perf  [kernel]                 [k] lock_task_sighand
     0.00%     perf  [kernel]                 [k] __queue_work
     0.00%     perf  [kernel]                 [k] remove_wait_queue
     0.00%     perf  [kernel]                 [k] __hrtimer_start_range_ns
     0.00%     perf  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.00%     perf  [kernel]                 [k] hrtimer_cancel
     0.00%     perf  [kernel]                 [k] filemap_fault
     0.00%     perf  [kernel]                 [k] mempool_free
     0.00%     perf  [kernel]                 [k] test_clear_page_writeback
     0.00%     perf  [kernel]                 [k] __lru_cache_add
     0.00%     perf  [kernel]                 [k] next_zones_zonelist
     0.00%     perf  [kernel]                 [k] __vm_enough_memory
     0.00%     perf  [kernel]                 [k] page_add_file_rmap
     0.00%     perf  [kernel]                 [k] nameidata_to_filp
     0.00%     perf  [kernel]                 [k] do_sys_open
     0.00%     perf  [kernel]                 [k] generic_file_llseek
     0.00%     perf  [kernel]                 [k] __d_instantiate
     0.00%     perf  [kernel]                 [k] d_instantiate
     0.00%     perf  [kernel]                 [k] d_path
     0.00%     perf  [kernel]                 [k] free_buffer_head
     0.00%     perf  [kernel]                 [k] alloc_page_buffers
     0.00%     perf  [kernel]                 [k] xtSplitRoot
     0.00%     perf  [kernel]                 [k] dbAllocDmapLev
     0.00%     perf  [kernel]                 [k] txBeginAnon
     0.00%     perf  [kernel]                 [k] security_vm_enough_memory_mm
     0.00%     perf  [kernel]                 [k] blk_update_request
     0.00%     perf  [kernel]                 [k] blk_run_queue
     0.00%     perf  [kernel]                 [k] disk_map_sector_rcu
     0.00%     perf  [kernel]                 [k] __rb_rotate_left
     0.00%     perf  [kernel]                 [k] put_dec_trunc
     0.00%     perf  [kernel]                 [k] bitmap_scnlistprintf
     0.00%     perf  [kernel]                 [k] scsi_log_completion
     0.00%     perf  [kernel]                 [k] __scsi_put_command
     0.00%     perf  [kernel]                 [k] scsi_device_unbusy
     0.00%     perf  [kernel]                 [k] scsi_softirq_done
     0.00%     perf  [kernel]                 [k] down_write
     0.00%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.00%     perf  /lib/libpthread-2.11.so  [.] __write_nocancel
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_vfprintf_internal
     0.00%     perf  /lib/ld-2.11.so          [.] _dl_name_match_p
     0.00%       dd  [kernel]                 [k] sysenter_do_call
     0.00%       dd  [kernel]                 [k] sched_move_task
     0.00%       dd  [kernel]                 [k] set_normalized_timespec
     0.00%       dd  [kernel]                 [k] __queue_work
     0.00%       dd  [kernel]                 [k] sys_clock_gettime
     0.00%       dd  [kernel]                 [k] add_wait_queue
     0.00%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.00%       dd  [kernel]                 [k] prepare_creds
     0.00%       dd  [kernel]                 [k] module_put
     0.00%       dd  [kernel]                 [k] __call_rcu
     0.00%       dd  [kernel]                 [k] __rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] rcu_process_callbacks
     0.00%       dd  [kernel]                 [k] perf_output_begin
     0.00%       dd  [kernel]                 [k] __perf_event_mmap
     0.00%       dd  [kernel]                 [k] mempool_free_slab
     0.00%       dd  [kernel]                 [k] __free_pages
     0.00%       dd  [kernel]                 [k] vma_prio_tree_remove
     0.00%       dd  [kernel]                 [k] follow_page
     0.00%       dd  [kernel]                 [k] vma_wants_writenotify
     0.00%       dd  [kernel]                 [k] expand_downwards
     0.00%       dd  [kernel]                 [k] __vm_enough_memory
     0.00%       dd  [kernel]                 [k] unlink_file_vma
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] page_add_file_rmap
     0.00%       dd  [kernel]                 [k] anon_vma_unlink
     0.00%       dd  [kernel]                 [k] do_sync_read
     0.00%       dd  [kernel]                 [k] rw_verify_area
     0.00%       dd  [kernel]                 [k] vfs_write
     0.00%       dd  [kernel]                 [k] vfs_read
     0.00%       dd  [kernel]                 [k] sys_write
     0.00%       dd  [kernel]                 [k] fput
     0.00%       dd  [kernel]                 [k] kernel_read
     0.00%       dd  [kernel]                 [k] release_open_intent
     0.00%       dd  [kernel]                 [k] path_put
     0.00%       dd  [kernel]                 [k] link_path_walk
     0.00%       dd  [kernel]                 [k] do_path_lookup
     0.00%       dd  [kernel]                 [k] do_filp_open
     0.00%       dd  [kernel]                 [k] __d_free
     0.00%       dd  [kernel]                 [k] spin_unlock
     0.00%       dd  [kernel]                 [k] touch_atime
     0.00%       dd  [kernel]                 [k] block_sync_page
     0.00%       dd  [kernel]                 [k] bio_endio
     0.00%       dd  [kernel]                 [k] bio_alloc
     0.00%       dd  [kernel]                 [k] fsnotify
     0.00%       dd  [kernel]                 [k] inotify_inode_queue_event
     0.00%       dd  [kernel]                 [k] locks_remove_flock
     0.00%       dd  [kernel]                 [k] ext4_ext_drop_refs
     0.00%       dd  [kernel]                 [k] ext4_ext_find_extent
     0.00%       dd  [kernel]                 [k] security_file_permission
     0.00%       dd  [kernel]                 [k] security_file_mmap
     0.00%       dd  [kernel]                 [k] blk_backing_dev_unplug
     0.00%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.00%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.00%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.00%       dd  [kernel]                 [k] radix_tree_next_hole
     0.00%       dd  [kernel]                 [k] __rb_rotate_left
     0.00%       dd  [kernel]                 [k] rb_erase
     0.00%       dd  [kernel]                 [k] strrchr
     0.00%       dd  [kernel]                 [k] half_md4_transform
     0.00%       dd  [kernel]                 [k] write_null
     0.00%       dd  [kernel]                 [k] add_disk_randomness
     0.00%       dd  [kernel]                 [k] tty_flip_buffer_push
     0.00%       dd  [kernel]                 [k] get_device
     0.00%       dd  [kernel]                 [k] scsi_pool_free_command
     0.00%       dd  [kernel]                 [k] scsi_next_command
     0.00%       dd  /lib/libpthread-2.11.so  [.] __pthread_rwlock_wrlock_internal
     0.00%       dd  /lib/libc-2.11.so        [.] __init_misc
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_start
     0.00%       dd  /lib/ld-2.11.so          [.] version_check_doit
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_cache_libcmp
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_load_cache_lookup
     0.00%       dd  /lib/ld-2.11.so          [.] 0x0000000000bc59
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_map_object_deps
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_sysdep_start
     0.00%       dd  /lib/ld-2.11.so          [.] __mmap
     0.00%       dd  /lib/ld-2.11.so          [.] strlen
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #12: perf11.report --]
[-- Type: text/plain, Size: 33524 bytes --]

# Samples: 39236
#
# Overhead  Command            Shared Object  Symbol
# ........  .......  .......................  ......
#
    13.23%       dd  [kernel]                 [k] __copy_to_user_ll
     8.96%       dd  [kernel]                 [k] do_mpage_readpage
     7.42%       dd  [kernel]                 [k] get_page_from_freelist
     2.78%       dd  [kernel]                 [k] generic_file_aio_read
     2.76%       dd  [kernel]                 [k] __bio_add_page
     2.50%       dd  [kernel]                 [k] add_to_page_cache_locked
     2.48%       dd  [kernel]                 [k] __blk_recalc_rq_segments
     2.19%       dd  [kernel]                 [k] file_read_actor
     2.00%       dd  [kernel]                 [k] __alloc_pages_nodemask
     1.92%     perf  [kernel]                 [k] generic_file_buffered_write
     1.88%       dd  [kernel]                 [k] scsi_dispatch_cmd
     1.79%       dd  [kernel]                 [k] scsi_request_fn
     1.62%       dd  [kernel]                 [k] find_get_page
     1.60%       dd  [kernel]                 [k] __make_request
     1.38%       dd  [kernel]                 [k] __do_page_cache_readahead
     1.23%       dd  [kernel]                 [k] put_page
     1.15%       dd  [kernel]                 [k] zone_watermark_ok
     1.14%     perf  [kernel]                 [k] __generic_file_aio_write
     1.11%       dd  [kernel]                 [k] ____pagevec_lru_add
     1.11%       dd  [kernel]                 [k] __rcu_read_unlock
     1.03%       dd  [kernel]                 [k] mpage_readpages
     0.96%       dd  [kernel]                 [k] get_request
     0.93%       dd  [kernel]                 [k] xtSearch
     0.83%       dd  [kernel]                 [k] __lru_cache_add
     0.83%       dd  [kernel]                 [k] add_to_page_cache_lru
     0.76%       dd  [kernel]                 [k] T.1339
     0.75%       dd  [kernel]                 [k] next_zones_zonelist
     0.71%     perf  [kernel]                 [k] iov_iter_advance
     0.70%       dd  [kernel]                 [k] bio_add_page
     0.68%     perf  [kernel]                 [k] nobh_write_end
     0.65%     perf  [kernel]                 [k] generic_file_aio_write
     0.64%     perf  [kernel]                 [k] vfs_write
     0.64%     perf  [kernel]                 [k] iov_iter_copy_from_user_atomic
     0.62%     perf  [kernel]                 [k] sysenter_past_esp
     0.60%     perf  [kernel]                 [k] __jfs_getxattr
     0.60%     perf  [kernel]                 [k] do_sync_write
     0.59%     perf  [kernel]                 [k] nobh_write_begin
     0.58%       dd  [kernel]                 [k] radix_tree_lookup_slot
     0.56%     perf  [kernel]                 [k] T.1339
     0.55%       dd  [kernel]                 [k] kmem_cache_alloc
     0.53%     perf  [kernel]                 [k] find_get_page
     0.53%       dd  [kernel]                 [k] radix_tree_preload
     0.51%     perf  [kernel]                 [k] getnstimeofday
     0.50%       dd  [kernel]                 [k] xtLookup
     0.48%       dd  [kernel]                 [k] generic_make_request
     0.47%     perf  [kernel]                 [k] rw_verify_area
     0.47%       dd  [kernel]                 [k] __rcu_read_lock
     0.44%       dd  [kernel]                 [k] blk_end_bidi_request
     0.43%     perf  [kernel]                 [k] generic_segment_checks
     0.42%     perf  [kernel]                 [k] acpi_pm_read
     0.39%     perf  [kernel]                 [k] ea_get
     0.38%       dd  [kernel]                 [k] radix_tree_lookup
     0.37%     perf  [kernel]                 [k] jfs_write_begin
     0.35%     perf  [kernel]                 [k] copy_to_user
     0.33%     perf  [kernel]                 [k] scsi_dispatch_cmd
     0.32%     perf  [kernel]                 [k] sys_write
     0.32%     perf  [kernel]                 [k] iov_iter_fault_in_readable
     0.32%       dd  [kernel]                 [k] do_page_fault
     0.31%       dd  [kernel]                 [k] mark_page_accessed
     0.31%       dd  [kernel]                 [k] _cond_resched
     0.30%     perf  [kernel]                 [k] balance_dirty_pages_ratelimited_nr
     0.30%     perf  [kernel]                 [k] __copy_from_user_ll_nozero
     0.29%     perf  [kernel]                 [k] sys_gettimeofday
     0.28%       dd  [kernel]                 [k] handle_mm_fault
     0.27%       dd  [kernel]                 [k] unmap_vmas
     0.27%       dd  [kernel]                 [k] jfs_get_block
     0.26%     perf  [kernel]                 [k] __mark_inode_dirty
     0.26%       dd  [kernel]                 [k] ata_sff_interrupt
     0.26%       dd  [kernel]                 [k] __get_metapage
     0.24%     perf  [vdso]                   [.] 0x000000ffffe424
     0.24%     perf  [kernel]                 [k] grab_cache_page_write_begin
     0.23%     perf  [kernel]                 [k] __set_page_dirty_buffers
     0.23%     perf  [kernel]                 [k] file_update_time
     0.23%     perf  [kernel]                 [k] fsnotify
     0.22%     perf  [kernel]                 [k] inotify_inode_queue_event
     0.21%       dd  [kernel]                 [k] __wake_up
     0.20%       dd  [kernel]                 [k] free_hot_cold_page
     0.20%     perf  [kernel]                 [k] put_page
     0.19%       dd  [kernel]                 [k] mpage_end_io_read
     0.19%     perf  [kernel]                 [k] jfs_getxattr
     0.18%     perf  [kernel]                 [k] generic_write_sync
     0.18%     perf  [kernel]                 [k] radix_tree_lookup_slot
     0.18%       dd  [kernel]                 [k] cfq_set_request
     0.18%     perf  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.17%     perf  [kernel]                 [k] fget_light
     0.17%     perf  [kernel]                 [k] ea_release
     0.17%     perf  [kernel]                 [k] __rcu_read_unlock
     0.17%     perf  [kernel]                 [k] file_remove_suid
     0.17%       dd  [kernel]                 [k] schedule
     0.17%     perf  [kernel]                 [k] kfree
     0.17%       dd  [kernel]                 [k] bio_alloc_bioset
     0.16%     perf  [kernel]                 [k] timespec_trunc
     0.16%     perf  [kernel]                 [k] extAlloc
     0.16%     perf  [kernel]                 [k] dbAlloc
     0.16%     perf  [kernel]                 [k] vsnprintf
     0.16%       dd  [kernel]                 [k] blk_rq_init
     0.15%     perf  [kernel]                 [k] do_gettimeofday
     0.15%     perf  [kernel]                 [k] strncmp
     0.15%     perf  [kernel]                 [k] find_lock_page
     0.15%       dd  [kernel]                 [k] mempool_alloc
     0.14%     perf  [kernel]                 [k] scsi_request_fn
     0.14%       dd  [kernel]                 [k] release_metapage
     0.14%       dd  [kernel]                 [k] handle_IRQ_event
     0.14%     perf  [kernel]                 [k] xtSearch
     0.13%     perf  [kernel]                 [k] should_remove_suid
     0.13%     perf  [kernel]                 [k] unlock_page
     0.13%     perf  [kernel]                 [k] __wake_up_bit
     0.13%     perf  [kernel]                 [k] __fsnotify_parent
     0.13%     perf  [kernel]                 [k] cap_inode_need_killpriv
     0.13%     perf  [kernel]                 [k] strlen
     0.12%       dd  [kernel]                 [k] __do_softirq
     0.12%     perf  [kernel]                 [k] jfs_dirty_inode
     0.12%       dd  [kernel]                 [k] kmem_cache_free
     0.12%       dd  [kernel]                 [k] __wake_up_bit
     0.11%     perf  [kernel]                 [k] current_fs_time
     0.11%       dd  [kernel]                 [k] unlock_page
     0.11%     perf  [kernel]                 [k] current_kernel_time
     0.11%     perf  [kernel]                 [k] __set_page_dirty
     0.10%     perf  [kernel]                 [k] number
     0.10%     perf  [kernel]                 [k] mark_page_accessed
     0.10%     perf  [kernel]                 [k] get_page_from_freelist
     0.10%     perf  [kernel]                 [k] _cond_resched
     0.10%       dd  [kernel]                 [k] sha_transform
     0.10%       dd  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.09%     perf  [kernel]                 [k] __rcu_read_lock
     0.09%       dd  [kernel]                 [k] mpage_alloc
     0.09%     perf  /home/cloos/bin/perf     [.] __cmd_record
     0.09%     perf  [kernel]                 [k] set_page_dirty
     0.09%       dd  [kernel]                 [k] blk_rq_bio_prep
     0.09%       dd  [kernel]                 [k] init_request_from_bio
     0.08%       dd  [kernel]                 [k] blk_queue_bounce
     0.08%       dd  [kernel]                 [k] submit_bio
     0.08%     perf  [kernel]                 [k] xtLookup
     0.08%     perf  [kernel]                 [k] security_file_permission
     0.08%     perf  [kernel]                 [k] cap_file_permission
     0.08%       dd  [kernel]                 [k] __do_fault
     0.07%     perf  [kernel]                 [k] sysenter_do_call
     0.07%     perf  [kernel]                 [k] mutex_lock
     0.07%       dd  [kernel]                 [k] do_read_cache_page
     0.07%       dd  [kernel]                 [k] get_io_context
     0.07%       dd  [kernel]                 [k] blk_recount_segments
     0.07%       dd  [kernel]                 [k] blk_update_request
     0.07%       dd  [kernel]                 [k] rcu_sched_qs
     0.06%       dd  [kernel]                 [k] ondemand_readahead
     0.06%     perf  [kernel]                 [k] copy_page_range
     0.06%       dd  [kernel]                 [k] cfq_cic_lookup
     0.06%     perf  [kernel]                 [k] jfs_get_block
     0.06%     perf  [kernel]                 [k] blk_end_bidi_request
     0.06%       dd  [kernel]                 [k] read_cache_page_async
     0.06%       dd  [kernel]                 [k] bio_init
     0.06%     perf  [kernel]                 [k] up_read
     0.06%     perf  [kernel]                 [k] dbAllocBits
     0.06%     perf  [kernel]                 [k] format_decode
     0.06%     perf  [kernel]                 [k] down_read
     0.06%       dd  [kernel]                 [k] mempool_free
     0.05%     perf  [kernel]                 [k] __get_metapage
     0.05%     perf  [kernel]                 [k] security_inode_need_killpriv
     0.05%       dd  [kernel]                 [k] get_pageblock_flags_group
     0.05%     perf  [kernel]                 [k] __wake_up
     0.05%     perf  [kernel]                 [k] schedule
     0.05%     perf  [kernel]                 [k] mutex_unlock
     0.05%       dd  [kernel]                 [k] mempool_alloc_slab
     0.05%       dd  [kernel]                 [k] scsi_init_cmd_errh
     0.05%     perf  [kernel]                 [k] xtExtend
     0.05%       dd  [kernel]                 [k] ktime_get_ts
     0.05%       dd  [kernel]                 [k] read_cache_page
     0.05%       dd  [kernel]                 [k] find_vma
     0.05%       dd  [kernel]                 [k] get_request_wait
     0.05%       dd  [kernel]                 [k] mdio_ctrl_hw
     0.05%     perf  [kernel]                 [k] handle_IRQ_event
     0.05%       dd  [kernel]                 [k] down_read_trylock
     0.05%       dd  [kernel]                 [k] page_cache_async_readahead
     0.05%       dd  [kernel]                 [k] release_pages
     0.05%       dd  [kernel]                 [k] vm_normal_page
     0.05%       dd  [kernel]                 [k] scsi_finish_command
     0.04%     perf  [kernel]                 [k] mnt_drop_write
     0.04%     perf  [kernel]                 [k] ata_sff_interrupt
     0.04%       dd  [kernel]                 [k] anon_vma_prepare
     0.04%       dd  [kernel]                 [k] bio_phys_segments
     0.04%       dd  [kernel]                 [k] bvec_alloc_bs
     0.04%     perf  [kernel]                 [k] __do_fault
     0.04%       dd  [kernel]                 [k] mix_pool_bytes_extract
     0.04%     perf  [kernel]                 [k] do_page_fault
     0.04%       dd  [kernel]                 [k] page_add_new_anon_rmap
     0.04%       dd  [kernel]                 [k] credit_entropy_bits
     0.04%     perf  [kernel]                 [k] handle_mm_fault
     0.04%     perf  [kernel]                 [k] mnt_clone_write
     0.04%     perf  [kernel]                 [k] dbAllocNext
     0.04%     perf  [kernel]                 [k] extHint
     0.04%       dd  [kernel]                 [k] prepare_to_wait_exclusive
     0.04%       dd  [kernel]                 [k] bio_get_nr_vecs
     0.04%       dd  [kernel]                 [k] down_read
     0.03%       dd  [kernel]                 [k] free_page_and_swap_cache
     0.03%       dd  [kernel]                 [k] blk_unplug
     0.03%       dd  [kernel]                 [k] preempt_schedule
     0.03%     perf  [kernel]                 [k] __alloc_pages_nodemask
     0.03%     perf  [kernel]                 [k] mpage_end_io_read
     0.03%       dd  [kernel]                 [k] up_read
     0.03%       dd  [kernel]                 [k] blk_run_queue
     0.03%     perf  [kernel]                 [k] rcu_sched_qs
     0.03%     perf  [kernel]                 [k] add_to_page_cache_locked
     0.03%     perf  [kernel]                 [k] do_read_cache_page
     0.03%     perf  [kernel]                 [k] kmem_cache_alloc
     0.03%       dd  [kernel]                 [k] page_evictable
     0.03%       dd  [kernel]                 [k] mpage_bio_submit
     0.03%       dd  [kernel]                 [k] current_io_context
     0.03%       dd  [kernel]                 [k] scsi_device_unbusy
     0.03%       dd  [kernel]                 [k] scsi_io_completion
     0.03%       dd  [kernel]                 [k] acpi_pm_read
     0.02%     perf  [kernel]                 [k] __do_softirq
     0.02%     perf  [kernel]                 [k] perf_mmap_fault
     0.02%     perf  [kernel]                 [k] inode_add_bytes
     0.02%     perf  [kernel]                 [k] __find_get_block_slow
     0.02%     perf  [kernel]                 [k] dbSplit
     0.02%       dd  [kernel]                 [k] filemap_fault
     0.02%       dd  [kernel]                 [k] blk_done_softirq
     0.02%       dd  [kernel]                 [k] add_timer_randomness
     0.02%       dd  [kernel]                 [k] sd_done
     0.02%     perf  [kernel]                 [k] zone_watermark_ok
     0.02%     perf  [kernel]                 [k] mnt_want_write_file
     0.02%     perf  [kernel]                 [k] mangle_path
     0.02%     perf  [kernel]                 [k] dbAllocDmap
     0.02%     perf  [kernel]                 [k] txBeginAnon
     0.02%     perf  [kernel]                 [k] strchr
     0.02%     perf  [kernel]                 [k] ehci_watchdog	[ehci_hcd]
     0.02%       dd  [kernel]                 [k] set_normalized_timespec
     0.02%       dd  [kernel]                 [k] copy_strings
     0.02%       dd  [kernel]                 [k] elv_set_request
     0.02%       dd  [kernel]                 [k] blk_update_bidi_request
     0.02%       dd  [kernel]                 [k] kref_put
     0.02%     perf  [kernel]                 [k] add_to_page_cache_lru
     0.02%     perf  [kernel]                 [k] ____pagevec_lru_add
     0.02%     perf  [kernel]                 [k] __lru_cache_add
     0.02%     perf  [kernel]                 [k] alloc_page_buffers
     0.02%     perf  [kernel]                 [k] release_metapage
     0.02%     perf  /lib/libpthread-2.11.so  [.] __libc_write
     0.02%       dd  [kernel]                 [k] wait_on_page_read
     0.02%       dd  [kernel]                 [k] page_remove_rmap
     0.02%       dd  [kernel]                 [k] block_sync_page
     0.02%       dd  [kernel]                 [k] jfs_readpages
     0.02%       dd  [kernel]                 [k] scsi_softirq_done
     0.02%       dd  [kernel]                 [k] __wait_on_bit_lock
     0.02%     perf  [kernel]                 [k] read_cache_page
     0.02%     perf  [kernel]                 [k] kmem_cache_free
     0.02%     perf  [kernel]                 [k] dbAdjTree
     0.02%       dd  [kernel]                 [k] __lock_page_killable
     0.02%       dd  [kernel]                 [k] ra_submit
     0.02%       dd  [kernel]                 [k] link_path_walk
     0.02%       dd  [kernel]                 [k] bio_fs_destructor
     0.02%       dd  [kernel]                 [k] generic_unplug_device
     0.02%       dd  [kernel]                 [k] strnlen_user
     0.02%       dd  [kernel]                 [k] put_device
     0.02%       dd  [kernel]                 [k] __scsi_release_buffers
     0.02%       dd  [kernel]                 [k] scsi_run_queue
     0.01%     perf  [kernel]                 [k] activate_page
     0.01%     perf  [kernel]                 [k] do_wp_page
     0.01%     perf  [kernel]                 [k] prepend
     0.01%     perf  [kernel]                 [k] alloc_buffer_head
     0.01%     perf  [kernel]                 [k] blk_update_request
     0.01%     perf  [kernel]                 [k] scsi_init_cmd_errh
     0.01%     perf  /lib/libc-2.11.so        [.] __GI___gettimeofday_internal
     0.01%       dd  [kernel]                 [k] put_pages_list
     0.01%       dd  [kernel]                 [k] anon_vma_unlink
     0.01%       dd  [kernel]                 [k] prio_tree_insert
     0.01%       dd  [kernel]                 [k] scsi_log_send
     0.01%       dd  [kernel]                 [k] scsi_put_command
     0.01%     perf  [kernel]                 [k] read_cache_page_async
     0.01%     perf  [kernel]                 [k] free_buffer_head
     0.01%     perf  [kernel]                 [k] req_bio_endio
     0.01%     perf  [kernel]                 [k] mix_pool_bytes_extract
     0.01%       dd  [kernel]                 [k] hrtimer_peek_ahead_timers
     0.01%       dd  [kernel]                 [k] delayacct_end
     0.01%       dd  [kernel]                 [k] sync_page_killable
     0.01%       dd  [kernel]                 [k] __d_lookup
     0.01%       dd  [kernel]                 [k] bio_put
     0.01%       dd  [kernel]                 [k] bio_alloc
     0.01%       dd  [kernel]                 [k] req_bio_endio
     0.01%       dd  [kernel]                 [k] blk_backing_dev_unplug
     0.01%       dd  [kernel]                 [k] kobject_put
     0.01%       dd  [kernel]                 [k] copy_to_user
     0.01%       dd  [kernel]                 [k] get_device
     0.01%       dd  [kernel]                 [k] io_schedule
     0.01%     perf  [kernel]                 [k] dup_mm
     0.01%     perf  [kernel]                 [k] find_vma
     0.01%     perf  [kernel]                 [k] seq_printf
     0.01%     perf  [kernel]                 [k] unmap_underlying_metadata
     0.01%     perf  [kernel]                 [k] blk_done_softirq
     0.01%     perf  [kernel]                 [k] scsi_io_completion
     0.01%     perf  [kernel]                 [k] schedule_hrtimeout_range
     0.01%       dd  [kernel]                 [k] finish_wait
     0.01%       dd  [kernel]                 [k] generic_segment_checks
     0.01%       dd  [kernel]                 [k] mempool_free_slab
     0.01%       dd  [kernel]                 [k] free_hot_page
     0.01%       dd  [kernel]                 [k] do_wp_page
     0.01%       dd  [kernel]                 [k] mprotect_fixup
     0.01%       dd  [kernel]                 [k] sys_close
     0.01%       dd  [kernel]                 [k] blk_end_request
     0.01%       dd  [kernel]                 [k] disk_map_sector_rcu
     0.01%       dd  [kernel]                 [k] _atomic_dec_and_lock
     0.01%       dd  [kernel]                 [k] kobject_get
     0.01%       dd  [kernel]                 [k] scsi_pool_free_command
     0.01%       dd  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.01%       dd  [kernel]                 [k] scsi_decide_disposition
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_map_object_from_fd
     0.01%       dd  /lib/ld-2.11.so          [.] do_lookup_x
     0.01%     perf  [kernel]                 [k] set_normalized_timespec
     0.01%     perf  [kernel]                 [k] down_read_trylock
     0.01%     perf  [kernel]                 [k] ktime_get_ts
     0.01%     perf  [kernel]                 [k] filemap_fault
     0.01%     perf  [kernel]                 [k] __set_page_dirty_nobuffers
     0.01%     perf  [kernel]                 [k] next_zones_zonelist
     0.01%     perf  [kernel]                 [k] page_add_file_rmap
     0.01%     perf  [kernel]                 [k] do_sys_poll
     0.01%     perf  [kernel]                 [k] d_path
     0.01%     perf  [kernel]                 [k] seq_path
     0.01%     perf  [kernel]                 [k] set_bh_page
     0.01%     perf  [kernel]                 [k] bio_put
     0.01%     perf  [kernel]                 [k] bio_fs_destructor
     0.01%     perf  [kernel]                 [k] show_map_vma
     0.01%     perf  [kernel]                 [k] xtInsert
     0.01%     perf  [kernel]                 [k] spin_unlock
     0.01%     perf  [kernel]                 [k] txLock
     0.01%     perf  [kernel]                 [k] blk_update_bidi_request
     0.01%     perf  [kernel]                 [k] _atomic_dec_and_lock
     0.01%     perf  [kernel]                 [k] kref_put
     0.01%     perf  [kernel]                 [k] radix_tree_preload
     0.01%     perf  [kernel]                 [k] rb_insert_color
     0.01%     perf  [kernel]                 [k] put_dec
     0.01%     perf  [kernel]                 [k] add_timer_randomness
     0.01%     perf  [kernel]                 [k] scsi_finish_command
     0.01%     perf  [kernel]                 [k] __scsi_put_command
     0.01%     perf  [kernel]                 [k] scsi_put_command
     0.01%     perf  [kernel]                 [k] scsi_handle_queue_ramp_up
     0.01%     perf  [kernel]                 [k] scsi_decide_disposition
     0.01%     perf  [kernel]                 [k] sd_done
     0.01%     perf  [kernel]                 [k] preempt_schedule
     0.01%     perf  /home/cloos/bin/perf     [.] pid_synthesize_mmap_samples
     0.01%       dd  [kernel]                 [k] sysenter_past_esp
     0.01%       dd  [kernel]                 [k] try_to_wake_up
     0.01%       dd  [kernel]                 [k] __call_rcu
     0.01%       dd  [kernel]                 [k] sync_page
     0.01%       dd  [kernel]                 [k] lru_cache_add_lru
     0.01%       dd  [kernel]                 [k] find_mergeable_anon_vma
     0.01%       dd  [kernel]                 [k] remove_vma
     0.01%       dd  [kernel]                 [k] mmap_region
     0.01%       dd  [kernel]                 [k] do_sync_read
     0.01%       dd  [kernel]                 [k] get_empty_filp
     0.01%       dd  [kernel]                 [k] path_put
     0.01%       dd  [kernel]                 [k] dput
     0.01%       dd  [kernel]                 [k] bio_endio
     0.01%       dd  [kernel]                 [k] _copy_from_user
     0.01%       dd  [kernel]                 [k] __sg_free_table
     0.01%       dd  [kernel]                 [k] scsi_log_completion
     0.01%       dd  [kernel]                 [k] system_call
     0.01%       dd  /lib/libc-2.11.so        [.] _dl_addr
     0.01%       dd  /lib/ld-2.11.so          [.] check_match.8476
     0.01%       dd  /lib/ld-2.11.so          [.] _dl_relocate_object
     0.00%     perf  [kernel]                 [k] remove_wait_queue
     0.00%     perf  [kernel]                 [k] hrtimer_init
     0.00%     perf  [kernel]                 [k] up_write
     0.00%     perf  [kernel]                 [k] rcu_process_gp_end
     0.00%     perf  [kernel]                 [k] check_for_new_grace_period
     0.00%     perf  [kernel]                 [k] __call_rcu
     0.00%     perf  [kernel]                 [k] perf_event_task
     0.00%     perf  [kernel]                 [k] wait_on_page_read
     0.00%     perf  [kernel]                 [k] mempool_free_slab
     0.00%     perf  [kernel]                 [k] mempool_free
     0.00%     perf  [kernel]                 [k] __get_free_pages
     0.00%     perf  [kernel]                 [k] test_clear_page_writeback
     0.00%     perf  [kernel]                 [k] release_pages
     0.00%     perf  [kernel]                 [k] vm_stat_account
     0.00%     perf  [kernel]                 [k] unmap_region
     0.00%     perf  [kernel]                 [k] page_remove_rmap
     0.00%     perf  [kernel]                 [k] page_add_new_anon_rmap
     0.00%     perf  [kernel]                 [k] anon_vma_prepare
     0.00%     perf  [kernel]                 [k] anon_vma_link
     0.00%     perf  [kernel]                 [k] kmem_cache_alloc_notrace
     0.00%     perf  [kernel]                 [k] link_path_walk
     0.00%     perf  [kernel]                 [k] do_path_lookup
     0.00%     perf  [kernel]                 [k] free_poll_entry
     0.00%     perf  [kernel]                 [k] poll_freewait
     0.00%     perf  [kernel]                 [k] sys_poll
     0.00%     perf  [kernel]                 [k] __d_path
     0.00%     perf  [kernel]                 [k] dput
     0.00%     perf  [kernel]                 [k] dup_fd
     0.00%     perf  [kernel]                 [k] alloc_fd
     0.00%     perf  [kernel]                 [k] seq_read
     0.00%     perf  [kernel]                 [k] bio_endio
     0.00%     perf  [kernel]                 [k] bio_free
     0.00%     perf  [kernel]                 [k] proc_pid_make_inode
     0.00%     perf  [kernel]                 [k] proc_pid_status
     0.00%     perf  [kernel]                 [k] dbAllocAG
     0.00%     perf  [kernel]                 [k] security_vm_enough_memory
     0.00%     perf  [kernel]                 [k] string
     0.00%     perf  [kernel]                 [k] strnlen
     0.00%     perf  [kernel]                 [k] cmpxchg8b
     0.00%     perf  [kernel]                 [k] credit_entropy_bits
     0.00%     perf  [kernel]                 [k] get_device
     0.00%     perf  [kernel]                 [k] scsi_log_completion
     0.00%     perf  [kernel]                 [k] scsi_sg_free
     0.00%     perf  [kernel]                 [k] __scsi_release_buffers
     0.00%     perf  [kernel]                 [k] scsi_device_unbusy
     0.00%     perf  [kernel]                 [k] scsi_softirq_done
     0.00%     perf  [kernel]                 [k] down_write
     0.00%     perf  /lib/libpthread-2.11.so  [.] __write_nocancel
     0.00%     perf  /lib/libc-2.11.so        [.] _IO_vfprintf_internal
     0.00%     perf  /lib/libc-2.11.so        [.] ptmalloc_unlock_all
     0.00%     perf  /home/cloos/bin/perf     [.] hex2u64
     0.00%       dd  [kernel]                 [k] nommu_map_page
     0.00%       dd  [kernel]                 [k] init_fpu
     0.00%       dd  [kernel]                 [k] pgd_alloc
     0.00%       dd  [kernel]                 [k] mm_release
     0.00%       dd  [kernel]                 [k] timespec_trunc
     0.00%       dd  [kernel]                 [k] run_timer_softirq
     0.00%       dd  [kernel]                 [k] __queue_work
     0.00%       dd  [kernel]                 [k] do_futex
     0.00%       dd  [kernel]                 [k] acct_collect
     0.00%       dd  [kernel]                 [k] rcu_bh_qs
     0.00%       dd  [kernel]                 [k] check_for_new_grace_period
     0.00%       dd  [kernel]                 [k] __delayacct_blkio_end
     0.00%       dd  [kernel]                 [k] perf_event_tid
     0.00%       dd  [kernel]                 [k] perf_output_copy
     0.00%       dd  [kernel]                 [k] perf_output_begin
     0.00%       dd  [kernel]                 [k] perf_event_mmap_ctx
     0.00%       dd  [kernel]                 [k] __lock_page
     0.00%       dd  [kernel]                 [k] vma_prio_tree_add
     0.00%       dd  [kernel]                 [k] follow_page
     0.00%       dd  [kernel]                 [k] __get_user_pages
     0.00%       dd  [kernel]                 [k] get_user_pages
     0.00%       dd  [kernel]                 [k] vm_stat_account
     0.00%       dd  [kernel]                 [k] arch_get_unmapped_area_topdown
     0.00%       dd  [kernel]                 [k] find_vma_prev
     0.00%       dd  [kernel]                 [k] __vma_link_file
     0.00%       dd  [kernel]                 [k] __vma_link
     0.00%       dd  [kernel]                 [k] unmap_region
     0.00%       dd  [kernel]                 [k] vma_adjust
     0.00%       dd  [kernel]                 [k] __split_vma
     0.00%       dd  [kernel]                 [k] do_munmap
     0.00%       dd  [kernel]                 [k] vma_merge
     0.00%       dd  [kernel]                 [k] do_mmap_pgoff
     0.00%       dd  [kernel]                 [k] sys_mmap_pgoff
     0.00%       dd  [kernel]                 [k] exit_mmap
     0.00%       dd  [kernel]                 [k] __anon_vma_link
     0.00%       dd  [kernel]                 [k] page_add_file_rmap
     0.00%       dd  [kernel]                 [k] kfree
     0.00%       dd  [kernel]                 [k] filp_close
     0.00%       dd  [kernel]                 [k] __dentry_open
     0.00%       dd  [kernel]                 [k] nameidata_to_filp
     0.00%       dd  [kernel]                 [k] do_sys_open
     0.00%       dd  [kernel]                 [k] rw_verify_area
     0.00%       dd  [kernel]                 [k] vfs_write
     0.00%       dd  [kernel]                 [k] sys_read
     0.00%       dd  [kernel]                 [k] sys_write
     0.00%       dd  [kernel]                 [k] fget_light
     0.00%       dd  [kernel]                 [k] __fput
     0.00%       dd  [kernel]                 [k] sys_fstat64
     0.00%       dd  [kernel]                 [k] open_exec
     0.00%       dd  [kernel]                 [k] do_execve
     0.00%       dd  [kernel]                 [k] acl_permission_check
     0.00%       dd  [kernel]                 [k] exec_permission
     0.00%       dd  [kernel]                 [k] generic_permission
     0.00%       dd  [kernel]                 [k] path_init
     0.00%       dd  [kernel]                 [k] release_open_intent
     0.00%       dd  [kernel]                 [k] getname
     0.00%       dd  [kernel]                 [k] __d_path
     0.00%       dd  [kernel]                 [k] touch_atime
     0.00%       dd  [kernel]                 [k] alloc_fd
     0.00%       dd  [kernel]                 [k] bio_free
     0.00%       dd  [kernel]                 [k] inotify_dentry_parent_queue_event
     0.00%       dd  [kernel]                 [k] locks_remove_flock
     0.00%       dd  [kernel]                 [k] load_script
     0.00%       dd  [kernel]                 [k] load_elf_binary
     0.00%       dd  [kernel]                 [k] cfq_idle_slice_timer
     0.00%       dd  [kernel]                 [k] prio_tree_replace
     0.00%       dd  [kernel]                 [k] __rb_rotate_left
     0.00%       dd  [kernel]                 [k] __rb_rotate_right
     0.00%       dd  [kernel]                 [k] half_md4_transform
     0.00%       dd  [kernel]                 [k] n_tty_write
     0.00%       dd  [kernel]                 [k] tty_insert_flip_string
     0.00%       dd  [kernel]                 [k] __scsi_put_command
     0.00%       dd  [kernel]                 [k] scsi_sg_free
     0.00%       dd  [kernel]                 [k] scsi_next_command
     0.00%       dd  [kernel]                 [k] e100_exec_cmd
     0.00%       dd  [kernel]                 [k] e100_enable_irq
     0.00%       dd  [kernel]                 [k] net_rx_action
     0.00%       dd  [kernel]                 [k] down_write
     0.00%       dd  /lib/libpthread-2.11.so  [.] 0x00000000010806
     0.00%       dd  /lib/libc-2.11.so        [.] _nl_intern_locale_data
     0.00%       dd  /lib/libc-2.11.so        [.] 0x00000000021f1a
     0.00%       dd  /lib/libc-2.11.so        [.] _nl_normalize_codeset
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_getenv
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___printf_fp
     0.00%       dd  /lib/libc-2.11.so        [.] __GI_strstr
     0.00%       dd  /lib/libc-2.11.so        [.] __strchrnul
     0.00%       dd  /lib/libc-2.11.so        [.] intel_02_known_compare
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___libc_write
     0.00%       dd  /lib/libc-2.11.so        [.] __GI___register_atfork
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_cache_libcmp
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_load_cache_lookup
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_lookup_symbol_x
     0.00%       dd  /lib/ld-2.11.so          [.] _dl_fixup
     0.00%       dd  /lib/ld-2.11.so          [.] match_symbol
     0.00%       dd  /lib/ld-2.11.so          [.] munmap
     0.00%       dd  /lib/ld-2.11.so          [.] strcmp
     0.00%       dd  /lib/ld-2.11.so          [.] strlen
     0.00%       dd                 ffffe415  [.] 0x000000ffffe415
#
# (For a higher level overview, try: perf report --sort comm,dso)
#


[-- Attachment #13: Type: text/plain, Size: 75 bytes --]


-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-15 15:45 ` Chris Mason
@ 2010-02-17 20:04     ` James Cloos
  2010-02-17 20:04     ` James Cloos
  1 sibling, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-17 20:04 UTC (permalink / raw)
  To: linux-kernel
  Cc: linux-fsdevel, jfs-discussion, linux-ext4, linux-btrfs, Chris Mason

While getting re-aquainted with perf, and after upgrading to the current
version, I see that do_page_fault, unmap_vmas, get_page_from_freelist,
handle_mm_fault, __do_fault and T.1339 tend to get the most time when the
kernel is spinning.

T.1339 is in this part of kallsyms:

,----< grep -E -C9 ^c1027750 /proc/kallsyms >
| c10266a0 T sched_setscheduler_nocheck
| c10266c0 T sched_setscheduler
| c10266e0 t do_sched_setscheduler
| c1026770 T sys_sched_setparam
| c1026790 T sys_sched_setscheduler
| c10267c0 T sched_fork
| c1026900 T task_oncpu_function_call
| c1026960 T print_cfs_rq
| c1027640 T scheduler_tick
| c1027750 t T.1339
| c10277f0 T schedule_tail
| c1027830 t ftrace_raw_output_sched_kthread_stop
| c1027930 t ftrace_raw_output_sched_kthread_stop_ret
| c1027a30 t ftrace_raw_output_sched_wait_task
| c1027b40 t ftrace_raw_output_sched_switch
| c1027c90 t ftrace_raw_output_sched_migrate_task
| c1027da0 t ftrace_raw_output_sched_process_wait
| c1027eb0 t ftrace_raw_output_sched_process_fork
| c1027fc0 t ftrace_raw_output_sched_stat_runtime
`----

suggesting that T.1339 is part of sched.

This leaves the impression that the issue is related to either
scheduling or paging.

I don't see any commits in the rc6..rc7 timeframe which stand out
as likely suspects.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

------------------------------------------------------------------------------
SOLARIS 10 is the OS for Data Centers - provides features such as DTrace,
Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW
http://p.sf.net/sfu/solaris-dev2dev

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-17 20:04     ` James Cloos
  0 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-17 20:04 UTC (permalink / raw)
  To: linux-kernel
  Cc: Chris Mason, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

While getting re-aquainted with perf, and after upgrading to the current
version, I see that do_page_fault, unmap_vmas, get_page_from_freelist,
handle_mm_fault, __do_fault and T.1339 tend to get the most time when the
kernel is spinning.

T.1339 is in this part of kallsyms:

,----< grep -E -C9 ^c1027750 /proc/kallsyms >
| c10266a0 T sched_setscheduler_nocheck
| c10266c0 T sched_setscheduler
| c10266e0 t do_sched_setscheduler
| c1026770 T sys_sched_setparam
| c1026790 T sys_sched_setscheduler
| c10267c0 T sched_fork
| c1026900 T task_oncpu_function_call
| c1026960 T print_cfs_rq
| c1027640 T scheduler_tick
| c1027750 t T.1339
| c10277f0 T schedule_tail
| c1027830 t ftrace_raw_output_sched_kthread_stop
| c1027930 t ftrace_raw_output_sched_kthread_stop_ret
| c1027a30 t ftrace_raw_output_sched_wait_task
| c1027b40 t ftrace_raw_output_sched_switch
| c1027c90 t ftrace_raw_output_sched_migrate_task
| c1027da0 t ftrace_raw_output_sched_process_wait
| c1027eb0 t ftrace_raw_output_sched_process_fork
| c1027fc0 t ftrace_raw_output_sched_stat_runtime
`----

suggesting that T.1339 is part of sched.

This leaves the impression that the issue is related to either
scheduling or paging.

I don't see any commits in the rc6..rc7 timeframe which stand out
as likely suspects.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-17 20:04     ` James Cloos
  (?)
@ 2010-02-19 21:02     ` Dave Airlie
  2010-02-19 21:37       ` James Cloos
  -1 siblings, 1 reply; 21+ messages in thread
From: Dave Airlie @ 2010-02-19 21:02 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

On Thu, Feb 18, 2010 at 6:04 AM, James Cloos <cloos@jhcloos.com> wrote:
> While getting re-aquainted with perf, and after upgrading to the current
> version, I see that do_page_fault, unmap_vmas, get_page_from_freelist,
> handle_mm_fault, __do_fault and T.1339 tend to get the most time when the
> kernel is spinning.

dmesg? lspci -vv?

We have an AGP related issue with WC/UC paging that might be related,
but I'd need more info.

Dave.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-14  3:11 [Regression] Filesystem I/O is CPU-bound in rc7 and rc8 James Cloos
  2010-02-14 16:40 ` James Cloos
@ 2010-02-19 21:22   ` Mike Fedyk
  2010-02-15 15:45 ` Chris Mason
  2010-02-19 21:22   ` Mike Fedyk
  3 siblings, 0 replies; 21+ messages in thread
From: Mike Fedyk @ 2010-02-19 21:22 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

On Sat, Feb 13, 2010 at 7:11 PM, James Cloos <cloos@jhcloos.com> wrote:
> Sometime between rc6 and rc7 all filesystem I/O started using 100% CP=
U,
> usually on the order of 60% sys, 40% user.
>
> I've tried this with each of ext4, jfs and btrfs filesystems. =C2=A0A=
ll show
> the same issue.
>

Are you sure you're not running with any of the debugging options
enabled?  I see the same, but I have debugging enabled (rawhide
kernel).

> Using dd(1) to read from the block specials directly works as well an=
d
> as fast as it always has; only reading or writing to mounted filesyst=
ems
> is affected.
>
> Box is 32-bit x86, PentiumIII-M; drives are ide using libata.
>
> If the btrfs fs is mounted, the slowdown is enought to trigger the
> hung_task call trace (120s) on the btrfs-transac process.
>
> But the regression is just as apparent when only jfs and ext4 are mou=
nted.
>
> The only filesystems I've found which avoid the regression are tmpfs =
and
> devtmpfs.
>
> I didn't have time to write up a report when I noticed this in rc7 bu=
t
> had to boot back into rc6 for work.
>
> Some of the commits since rc7 looked like they might have addressed t=
his
> regression, but it persists in rc8.
>
> -JimC
> --
> James Cloos <cloos@jhcloos.com> =C2=A0 =C2=A0 =C2=A0 =C2=A0 OpenPGP: =
1024D/ED7DAEA6
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =C2=A0http://vger.kernel.org/majordomo-info.ht=
ml
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-19 21:22   ` Mike Fedyk
  0 siblings, 0 replies; 21+ messages in thread
From: Mike Fedyk @ 2010-02-19 21:22 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

On Sat, Feb 13, 2010 at 7:11 PM, James Cloos <cloos@jhcloos.com> wrote:
> Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
> usually on the order of 60% sys, 40% user.
>
> I've tried this with each of ext4, jfs and btrfs filesystems.  All show
> the same issue.
>

Are you sure you're not running with any of the debugging options
enabled?  I see the same, but I have debugging enabled (rawhide
kernel).

> Using dd(1) to read from the block specials directly works as well and
> as fast as it always has; only reading or writing to mounted filesystems
> is affected.
>
> Box is 32-bit x86, PentiumIII-M; drives are ide using libata.
>
> If the btrfs fs is mounted, the slowdown is enought to trigger the
> hung_task call trace (120s) on the btrfs-transac process.
>
> But the regression is just as apparent when only jfs and ext4 are mounted.
>
> The only filesystems I've found which avoid the regression are tmpfs and
> devtmpfs.
>
> I didn't have time to write up a report when I noticed this in rc7 but
> had to boot back into rc6 for work.
>
> Some of the commits since rc7 looked like they might have addressed this
> regression, but it persists in rc8.
>
> -JimC
> --
> James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-19 21:22   ` Mike Fedyk
  0 siblings, 0 replies; 21+ messages in thread
From: Mike Fedyk @ 2010-02-19 21:22 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

On Sat, Feb 13, 2010 at 7:11 PM, James Cloos <cloos@jhcloos.com> wrote:
> Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
> usually on the order of 60% sys, 40% user.
>
> I've tried this with each of ext4, jfs and btrfs filesystems.  All show
> the same issue.
>

Are you sure you're not running with any of the debugging options
enabled?  I see the same, but I have debugging enabled (rawhide
kernel).

> Using dd(1) to read from the block specials directly works as well and
> as fast as it always has; only reading or writing to mounted filesystems
> is affected.
>
> Box is 32-bit x86, PentiumIII-M; drives are ide using libata.
>
> If the btrfs fs is mounted, the slowdown is enought to trigger the
> hung_task call trace (120s) on the btrfs-transac process.
>
> But the regression is just as apparent when only jfs and ext4 are mounted.
>
> The only filesystems I've found which avoid the regression are tmpfs and
> devtmpfs.
>
> I didn't have time to write up a report when I noticed this in rc7 but
> had to boot back into rc6 for work.
>
> Some of the commits since rc7 looked like they might have addressed this
> regression, but it persists in rc8.
>
> -JimC
> --
> James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-19 21:02     ` Dave Airlie
@ 2010-02-19 21:37       ` James Cloos
  2010-02-19 22:03           ` Dave Airlie
  0 siblings, 1 reply; 21+ messages in thread
From: James Cloos @ 2010-02-19 21:37 UTC (permalink / raw)
  To: Dave Airlie
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

>>>>> "Dave" == Dave Airlie <airlied@gmail.com> writes:

Dave> On Thu, Feb 18, 2010 at 6:04 AM, James Cloos <cloos@jhcloos.com> wrote:
>> While getting re-aquainted with perf, and after upgrading to the current
>> version, I see that do_page_fault, unmap_vmas, get_page_from_freelist,
>> handle_mm_fault, __do_fault and T.1339 tend to get the most time when the
>> kernel is spinning.

Dave> We have an AGP related issue with WC/UC paging that might be related,
Dave> but I'd need more info.

OK.  I do use radeon kms (r100).

On a lark, I tried out a non-kms boot this morning, but X never started;
I was going to update all of the user-mode code (dri2, mesa, x, the ati
driver, etc) before tyring again to check whether kms was required to
hit the issue.  That takes a few hours to do on this box.

Dave> dmesg?

The only reelvant part of dmesg are of the form:

Feb 14 01:37:27 [kernel] [ 3720.582130] INFO: task btrfs-transacti:2527 blocked for more than 120 seconds.
Feb 14 01:37:27 [kernel] [ 3720.582140] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 14 01:37:27 [kernel] [ 3720.582149] btrfs-transac D da2ba5b0     0  2527      2 0x00000000
Feb 14 01:37:27 [kernel] [ 3720.582163]  da33af60 00000046 c170f240 da2ba5b0 037a730a 00000343 da2ba854 da2ba5b0
Feb 14 01:37:27 [kernel] [ 3720.582181]  df9557f4 da2ba5b0 df9557f8 da33af84 c152c406 df9557f8 df9557f8 da2ba5b0
Feb 14 01:37:27 [kernel] [ 3720.582199]  ffffffff da212000 da33afa0 da2ba5b0 da33af8c c152bf81 da33afc0 c11b4ecc
Feb 14 01:37:27 [kernel] [ 3720.582216] Call Trace:
Feb 14 01:37:27 [kernel] [ 3720.582241]  [<c152c406>] __mutex_lock_slowpath+0x96/0x120
Feb 14 01:37:27 [kernel] [ 3720.582253]  [<c152bf81>] mutex_lock+0x11/0x20
Feb 14 01:37:27 [kernel] [ 3720.582269]  [<c11b4ecc>] transaction_kthread+0xcc/0x1e0
Feb 14 01:37:27 [kernel] [ 3720.582281]  [<c152b643>] ? preempt_schedule+0x33/0x50
Feb 14 01:37:27 [kernel] [ 3720.582292]  [<c11b4e00>] ? transaction_kthread+0x0/0x1e0
Feb 14 01:37:27 [kernel] [ 3720.582306]  [<c104c684>] kthread+0x74/0x80
Feb 14 01:37:27 [kernel] [ 3720.582316]  [<c104c610>] ? kthread+0x0/0x80
Feb 14 01:37:27 [kernel] [ 3720.582328]  [<c1003356>] kernel_thread_helper+0x6/0x10

Dave>  lspci -vv?

00:00.0 Host bridge: Intel Corporation 82815 815 Chipset Host Bridge and Memory Controller Hub (rev 04)
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 0
	Region 0: Memory at e8000000 (32-bit, prefetchable) [size=64M]
	Capabilities: [88] Vendor Specific Information <?>
	Capabilities: [a0] AGP version 2.0
		Status: RQ=32 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW- AGP3- Rate=x1,x2,x4
		Command: RQ=1 ArqSz=0 Cal=0 SBA+ AGP+ GART64- 64bit- FW- Rate=x4
	Kernel driver in use: agpgart-intel

00:01.0 PCI bridge: Intel Corporation 82815 815 Chipset AGP Bridge (rev 04) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fc000000-fdffffff
	Prefetchable memory behind bridge: e0000000-e7ffffff
	Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR+
	BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 03) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=00, secondary=02, subordinate=10, sec-latency=32
	I/O behind bridge: 0000d000-0000ffff
	Memory behind bridge: f2000000-fbffffff
	Prefetchable memory behind bridge: 20000000-27ffffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

00:1f.0 ISA bridge: Intel Corporation 82801BAM ISA Bridge (LPC) (rev 03)
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Kernel modules: intel-rng

00:1f.1 IDE interface: Intel Corporation 82801BAM IDE U100 Controller (rev 03) (prog-if 80 [Master])
	Subsystem: Intel Corporation Device 4541
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
	Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
	Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
	Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
	Region 4: I/O ports at bfa0 [size=16]
	Kernel driver in use: ata_piix

00:1f.2 USB Controller: Intel Corporation 82801BA/BAM USB Controller #1 (rev 03) (prog-if 00 [UHCI])
	Subsystem: Intel Corporation Device 4541
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin D routed to IRQ 10
	Region 4: I/O ports at bce0 [size=32]
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

01:00.0 VGA compatible controller: ATI Technologies Inc Radeon Mobility M7 LW [Radeon Mobility 7500] (prog-if 00 [VGA controller])
	Subsystem: Dell Radeon Mobility M7 LW (Dell Inspiron 8100)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping+ SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (2000ns min), Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at e0000000 (32-bit, prefetchable) [size=128M]
	Region 1: I/O ports at cc00 [size=256]
	Region 2: Memory at fcff0000 (32-bit, non-prefetchable) [size=64K]
	Expansion ROM at fc000000 [disabled] [size=128K]
	Capabilities: [58] AGP version 2.0
		Status: RQ=48 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW- AGP3- Rate=x1,x2,x4
		Command: RQ=32 ArqSz=0 Cal=0 SBA+ AGP+ GART64- 64bit- FW- Rate=x4
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: radeon
	Kernel modules: radeonfb

02:03.0 Multimedia audio controller: ESS Technology ES1983S Maestro-3i PCI Audio Accelerator (rev 10)
	Subsystem: Dell ES1983S Maestro-3i (Dell Inspiron 8100)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (500ns min, 6000ns max)
	Interrupt: pin A routed to IRQ 5
	Region 0: I/O ports at dc00 [size=256]
	Region 1: Memory at f6ffe000 (32-bit, non-prefetchable) [size=8K]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: Maestro3

02:06.0 PCI bridge: Actiontec Electronics Inc Mini-PCI bridge (rev 11) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 32 bytes
	Bus: primary=02, secondary=08, subordinate=08, sec-latency=32
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: f8000000-f9ffffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [80] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Capabilities: [90] CompactPCI hot-swap <?>

02:0f.0 CardBus bridge: Texas Instruments PCI4451 PC card Cardbus Controller
	Subsystem: Dell PCI4451 PC card CardBus Controller (Inspiron 8100)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 168, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=4K]
	Bus: primary=02, secondary=09, subordinate=0c, sec-latency=176
	Memory window 0: 20000000-23fff000 (prefetchable)
	Memory window 1: 28000000-2bfff000
	I/O window 0: 0000d000-0000d0ff
	I/O window 1: 0000d400-0000d4ff
	BridgeCtl: Parity- SERR- ISA- VGA- MAbort- >Reset+ 16bInt+ PostWrite+
	16-bit legacy interface ports at 0001
	Kernel driver in use: yenta_cardbus

02:0f.1 CardBus bridge: Texas Instruments PCI4451 PC card Cardbus Controller
	Subsystem: Dell PCI4451 PC card CardBus Controller (Inspiron 8100)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 168, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f2001000 (32-bit, non-prefetchable) [size=4K]
	Bus: primary=02, secondary=0d, subordinate=10, sec-latency=176
	Memory window 0: 24000000-27fff000 (prefetchable)
	Memory window 1: 2c000000-2ffff000
	I/O window 0: 0000d800-0000d8ff
	I/O window 1: 0000f000-0000f0ff
	BridgeCtl: Parity- SERR- ISA- VGA- MAbort- >Reset- 16bInt- PostWrite+
	16-bit legacy interface ports at 0001
	Kernel driver in use: yenta_cardbus

02:0f.2 FireWire (IEEE 1394): Texas Instruments PCI4451 IEEE-1394 Controller (prog-if 10 [OHCI])
	Subsystem: Dell PCI4451 IEEE-1394 Controller (Dell Inspiron 8100)
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (500ns min, 1000ns max), Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f6ffd800 (32-bit, non-prefetchable) [size=2K]
	Region 1: Memory at f6ff8000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: firewire_ohci

08:04.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 08)
	Subsystem: Actiontec Electronics Inc EtherExpress PRO/100B (TX) (MiniPCI Ethernet+Modem)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (2000ns min, 14000ns max), Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f8fff000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at ecc0 [size=64]
	Region 2: Memory at f8e00000 (32-bit, non-prefetchable) [size=1M]
	Expansion ROM at f9000000 [disabled] [size=1M]
	Capabilities: [dc] Power Management version 2
		Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=2 PME-
	Kernel driver in use: e100

08:08.0 Communication controller: Agere Systems WinModem 56k (rev 01)
	Subsystem: Actiontec Electronics Inc LT WinModem 56k (MiniPCI Ethernet+Modem)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0 (63000ns min, 3500ns max)
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at f8ffec00 (32-bit, non-prefetchable) [size=256]
	Region 1: I/O ports at ecb8 [size=8]
	Region 2: I/O ports at e800 [size=256]
	Capabilities: [f8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0d:00.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 62) (prog-if 00 [UHCI])
	Subsystem: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 22, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 1: Memory at 2c000800 (32-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at d880 [size=32]
	Capabilities: [80] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

0d:00.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 62) (prog-if 00 [UHCI])
	Subsystem: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 22, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 1: Memory at 2c000900 (32-bit, non-prefetchable) [size=256]
	Region 4: I/O ports at d8a0 [size=32]
	Capabilities: [80] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: uhci_hcd
	Kernel modules: uhci-hcd

0d:00.2 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 65) (prog-if 20 [EHCI])
	Subsystem: VIA Technologies, Inc. USB 2.0
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 22, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at 2c000a00 (32-bit, non-prefetchable) [size=256]
	Region 1: Memory at 2c000b00 (32-bit, non-prefetchable) [size=256]
	Capabilities: [80] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ehci_hcd
	Kernel modules: ehci-hcd

0d:00.3 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev 80) (prog-if 10 [OHCI])
	Subsystem: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64 (8000ns max), Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 10
	Region 0: Memory at 2c000000 (32-bit, non-prefetchable) [size=2K]
	Region 1: I/O ports at d800 [size=128]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: firewire_ohci

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-19 21:22   ` Mike Fedyk
  (?)
@ 2010-02-19 21:44     ` James Cloos
  -1 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-19 21:44 UTC (permalink / raw)
  To: Mike Fedyk
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

>>>>> "Mike" =3D=3D Mike Fedyk <mfedyk@mikefedyk.com> writes:

Mike> On Sat, Feb 13, 2010 at 7:11 PM, James Cloos <cloos@jhcloos.com> =
wrote:
>> Sometime between rc6 and rc7 all filesystem I/O started using 100% C=
PU,
>> usually on the order of 60% sys, 40% user.
>>=20
>> I've tried this with each of ext4, jfs and btrfs filesystems. =A0All=
 show
>> the same issue.
>>=20

Mike> Are you sure you're not running with any of the debugging options
Mike> enabled?  I see the same, but I have debugging enabled (rawhide
Mike> kernel).

There are some:

CONFIG_HAVE_DMA_API_DEBUG=3Dy
CONFIG_X86_DEBUGCTLMSR=3Dy
CONFIG_CFG80211_DEBUGFS=3Dy
CONFIG_MAC80211_DEBUGFS=3Dy
CONFIG_PNP_DEBUG_MESSAGES=3Dy
CONFIG_FIREWIRE_OHCI_DEBUG=3Dy
CONFIG_USB_SERIAL_DEBUG=3Dm
CONFIG_DEBUG_FS=3Dy
CONFIG_DEBUG_KERNEL=3Dy
CONFIG_SCHED_DEBUG=3Dy
CONFIG_DEBUG_BUGVERBOSE=3Dy
CONFIG_DEBUG_MEMORY_INIT=3Dy
CONFIG_DEBUG_STACKOVERFLOW=3Dy
CONFIG_KEYS_DEBUG_PROC_KEYS=3Dy

I forgot about /proc/sched_debug, but am back at -rc6 right now; I'll
post an example of /proc/sched_debug output during the issue when I'm
next in rc8 or later.

-JimC
--=20
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-19 21:44     ` James Cloos
  0 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-19 21:44 UTC (permalink / raw)
  To: Mike Fedyk
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

>>>>> "Mike" == Mike Fedyk <mfedyk@mikefedyk.com> writes:

Mike> On Sat, Feb 13, 2010 at 7:11 PM, James Cloos <cloos@jhcloos.com> wrote:
>> Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
>> usually on the order of 60% sys, 40% user.
>> 
>> I've tried this with each of ext4, jfs and btrfs filesystems.  All show
>> the same issue.
>> 

Mike> Are you sure you're not running with any of the debugging options
Mike> enabled?  I see the same, but I have debugging enabled (rawhide
Mike> kernel).

There are some:

CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_PNP_DEBUG_MESSAGES=y
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_USB_SERIAL_DEBUG=m
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y

I forgot about /proc/sched_debug, but am back at -rc6 right now; I'll
post an example of /proc/sched_debug output during the issue when I'm
next in rc8 or later.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-19 21:44     ` James Cloos
  0 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-19 21:44 UTC (permalink / raw)
  To: Mike Fedyk
  Cc: linux-kernel, linux-fsdevel, linux-ext4, linux-btrfs, jfs-discussion

>>>>> "Mike" == Mike Fedyk <mfedyk@mikefedyk.com> writes:

Mike> On Sat, Feb 13, 2010 at 7:11 PM, James Cloos <cloos@jhcloos.com> wrote:
>> Sometime between rc6 and rc7 all filesystem I/O started using 100% CPU,
>> usually on the order of 60% sys, 40% user.
>> 
>> I've tried this with each of ext4, jfs and btrfs filesystems.  All show
>> the same issue.
>> 

Mike> Are you sure you're not running with any of the debugging options
Mike> enabled?  I see the same, but I have debugging enabled (rawhide
Mike> kernel).

There are some:

CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_PNP_DEBUG_MESSAGES=y
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_USB_SERIAL_DEBUG=m
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y

I forgot about /proc/sched_debug, but am back at -rc6 right now; I'll
post an example of /proc/sched_debug output during the issue when I'm
next in rc8 or later.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-19 21:37       ` James Cloos
@ 2010-02-19 22:03           ` Dave Airlie
  0 siblings, 0 replies; 21+ messages in thread
From: Dave Airlie @ 2010-02-19 22:03 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

>
> OK. =A0I do use radeon kms (r100).

I've just sent a pull req to Linus
http://bugzilla.kernel.org/show_bug.cgi?id=3D15328
this

http://bugzilla.kernel.org/attachment.cgi?id=3D25107
patch.

The relevant piece of dmesg is at the top ;-), where
it states if you have PAT support or not, but I
suspect if you try this patch it'll fix it.

Dave.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-19 22:03           ` Dave Airlie
  0 siblings, 0 replies; 21+ messages in thread
From: Dave Airlie @ 2010-02-19 22:03 UTC (permalink / raw)
  To: James Cloos
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

>
> OK.  I do use radeon kms (r100).

I've just sent a pull req to Linus
http://bugzilla.kernel.org/show_bug.cgi?id=15328
this

http://bugzilla.kernel.org/attachment.cgi?id=25107
patch.

The relevant piece of dmesg is at the top ;-), where
it states if you have PAT support or not, but I
suspect if you try this patch it'll fix it.

Dave.

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-19 22:03           ` Dave Airlie
  (?)
@ 2010-02-20 15:59             ` James Cloos
  -1 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-20 15:59 UTC (permalink / raw)
  To: Dave Airlie
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

>>>>> "Dave" =3D=3D Dave Airlie <airlied@gmail.com> writes:

>>=20
>> OK. =A0I do use radeon kms (r100).

Dave> I've just sent a pull req to Linus
Dave> http://bugzilla.kernel.org/show_bug.cgi?id=3D15328
Dave> this

Dave> http://bugzilla.kernel.org/attachment.cgi?id=3D25107
Dave> patch.

I'll give that a try as soon as I can schedule a reboot.

Dave> The relevant piece of dmesg is at the top ;-), where
Dave> it states if you have PAT support or not, but I
Dave> suspect if you try this patch it'll fix it.

Oh.  I thought you were referring to errors....

Anyway, no, the p3m does not support PAT:

[    0.000000] PAT not supported by CPU.

-JimC
--=20
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-20 15:59             ` James Cloos
  0 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-20 15:59 UTC (permalink / raw)
  To: Dave Airlie
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

>>>>> "Dave" == Dave Airlie <airlied@gmail.com> writes:

>> 
>> OK.  I do use radeon kms (r100).

Dave> I've just sent a pull req to Linus
Dave> http://bugzilla.kernel.org/show_bug.cgi?id=15328
Dave> this

Dave> http://bugzilla.kernel.org/attachment.cgi?id=25107
Dave> patch.

I'll give that a try as soon as I can schedule a reboot.

Dave> The relevant piece of dmesg is at the top ;-), where
Dave> it states if you have PAT support or not, but I
Dave> suspect if you try this patch it'll fix it.

Oh.  I thought you were referring to errors....

Anyway, no, the p3m does not support PAT:

[    0.000000] PAT not supported by CPU.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
@ 2010-02-20 15:59             ` James Cloos
  0 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-20 15:59 UTC (permalink / raw)
  To: Dave Airlie
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

>>>>> "Dave" == Dave Airlie <airlied@gmail.com> writes:

>> 
>> OK.  I do use radeon kms (r100).

Dave> I've just sent a pull req to Linus
Dave> http://bugzilla.kernel.org/show_bug.cgi?id=15328
Dave> this

Dave> http://bugzilla.kernel.org/attachment.cgi?id=25107
Dave> patch.

I'll give that a try as soon as I can schedule a reboot.

Dave> The relevant piece of dmesg is at the top ;-), where
Dave> it states if you have PAT support or not, but I
Dave> suspect if you try this patch it'll fix it.

Oh.  I thought you were referring to errors....

Anyway, no, the p3m does not support PAT:

[    0.000000] PAT not supported by CPU.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [Regression] Filesystem I/O is CPU-bound in rc7 and rc8
  2010-02-19 22:03           ` Dave Airlie
  (?)
  (?)
@ 2010-02-21  2:12           ` James Cloos
  -1 siblings, 0 replies; 21+ messages in thread
From: James Cloos @ 2010-02-21  2:12 UTC (permalink / raw)
  To: Dave Airlie
  Cc: linux-kernel, Chris Mason, linux-fsdevel, linux-ext4,
	linux-btrfs, jfs-discussion

>>>>> "Dave" == Dave Airlie <airlied@gmail.com> writes:

Dave> I've just sent a pull req to Linus
Dave> http://bugzilla.kernel.org/show_bug.cgi?id=15328
Dave> this
Dave> http://bugzilla.kernel.org/attachment.cgi?id=25107
Dave> patch.

That patch does indeed fix the regression here.

Thanks!

It looks like my report is a dup of bug 15328.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2010-02-21  2:12 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-02-14  3:11 [Regression] Filesystem I/O is CPU-bound in rc7 and rc8 James Cloos
2010-02-14 16:40 ` James Cloos
2010-02-14 23:12 ` Rafael J. Wysocki
2010-02-15 15:45 ` Chris Mason
2010-02-15 19:27   ` James Cloos
2010-02-17 20:04   ` James Cloos
2010-02-17 20:04     ` James Cloos
2010-02-19 21:02     ` Dave Airlie
2010-02-19 21:37       ` James Cloos
2010-02-19 22:03         ` Dave Airlie
2010-02-19 22:03           ` Dave Airlie
2010-02-20 15:59           ` James Cloos
2010-02-20 15:59             ` James Cloos
2010-02-20 15:59             ` James Cloos
2010-02-21  2:12           ` James Cloos
2010-02-19 21:22 ` Mike Fedyk
2010-02-19 21:22   ` Mike Fedyk
2010-02-19 21:22   ` Mike Fedyk
2010-02-19 21:44   ` James Cloos
2010-02-19 21:44     ` James Cloos
2010-02-19 21:44     ` James Cloos

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.