xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
From: Paulina Szubarczyk <paulinaszubarczyk@gmail.com>
To: xen-devel@lists.xenproject.org, roger.pau@citrix.com
Cc: anthony.perard@citrix.com, sstabellini@kernel.org,
	Paulina Szubarczyk <paulinaszubarczyk@gmail.com>,
	P.Gawkowski@ii.pw.edu.pl
Subject: [PATCH 0/1] qemu-qdisk: indirect descriptors
Date: Wed, 15 Jun 2016 19:01:01 +0200	[thread overview]
Message-ID: <1466010062-16270-1-git-send-email-paulinaszubarczyk@gmail.com> (raw)

In the meantime I tried an implementation for indirect descriptors for qemu.
Described further in the next mail. It is based on current staging branch of qemu. 

From tests I did not observed an improvement. A decrease of bandwith starts 
earlier when the block size increase then for staging branch, especially for 
higher values of iodepth[1]. 
I run it under gprof and all the results are available on my github[2] 
but below is a part of flat profile for staging and indirect descriptors when 
fio is run with iodepth=256 and bs=256 for 300 sec. 

In the indirect descriptors implementation more time is spent in ioreq_unmap
function with smaller number of calls. I tried to check if it cooperate better
with grant copy running in the same time vmstat but then rapidly memory is 
exhausted and swap-out/in, the part of the listings are below, and that is not
a case for poor grant copy implementation. 
I tried also different values of MAX_INDIRECT_SEGMENTS in the range 
{256, 128, 64, 32, 16} without bigger difference.

I would appreciate any suggestions how to approach the problem.  

flat profiles:
indirect descriptors
 Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls   s/call   s/call  name    
13.19      1.12     1.12   653798     0.00     0.00  get_clock_realtime
 10.13      1.98     0.86    83570     0.00     0.00  ioreq_unmap
  4.77      2.38     0.41 31245461     0.00     0.00  rcu_read_unlock
  4.12      2.73     0.35    83423     0.00     0.00  ioreq_map
  3.65      3.04     0.31 20900170     0.00     0.00  phys_page_find
  3.12      3.31     0.27 20886790     0.00     0.00  address_space_rw
  2.24      3.50     0.19 20886790     0.00     0.00  address_space_translate
  2.00      3.67     0.17 10849312     0.00     0.00  test_and_clear_bit
  1.88      3.83     0.16 31245456     0.00     0.00  rcu_read_lock
  1.71      3.98     0.14 41773586     0.00     0.00  memory_access_is_direct
  1.65      4.12     0.14 10330994     0.00     0.00  xen_map_cache_unlocked
  1.59      4.25     0.14 20886785     0.00     0.00  address_space_translate_internal
  1.53      4.38     0.13 10339152     0.00     0.00  cpu_inw
  1.41      4.50     0.12 10458730     0.00     0.00  find_portio
  1.30      4.61     0.11 10389053     0.00     0.00  cpu_physical_memory_rw
  1.12      4.71     0.10 10358655     0.00     0.00  qemu_get_ram_block
  1.06      4.79     0.09 31245450     0.00     0.00  xen_enabled
  1.06      4.88     0.09 10447242     0.00     0.00  portio_read
  1.06      4.97     0.09   237496     0.00     0.00  cpu_ioreq_pio
  1.06      5.07     0.09     1557     0.00     0.00  vnc_refresh_server_sur

 staging 
 Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls   s/call   s/call  name    
 11.51      1.61     1.61   970388     0.00     0.00  get_clock_realtime
  9.58      2.95     1.34  1186036     0.00     0.00  ioreq_unmap
  5.50      3.72     0.77  1187881     0.00     0.00  ioreq_map
  4.15      4.30     0.58 31195245     0.00     0.00  rcu_read_unlock
  2.50      4.65     0.35 31195243     0.00     0.00  rcu_read_lock
  2.50      5.00     0.35 20866261     0.00     0.00  phys_page_find
  1.79      5.25     0.25 20852888     0.00     0.00  address_space_rw
  1.36      5.44     0.19  4912499     0.00     0.00  qemu_coroutine_switch
  1.22      5.61     0.17 20852881     0.00     0.00  address_space_translate
  1.22      5.78     0.17  6141137     0.00     0.00  bdrv_is_inserted
  1.07      5.93     0.15  2455277     0.00     0.00  tracked_request_end
  1.07      6.08     0.15  1187877     0.00     0.00  ioreq_parse
  1.00      6.22     0.14 20852887     0.00     0.00  address_space_translate_internal
  1.00      6.36     0.14  2456463     0.00     0.00  qemu_aio_unref
  1.00      6.50     0.14  2456156     0.00     0.00  qemu_coroutine_enter
  0.93      6.63     0.13 41705784     0.00     0.00  memory_access_is_direct

vmstat listings:
grant map
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  1     11     62   2775    638    0    0  4052   244 16250 14124  6 20 71  2  1
 1  0     11     58   2779    638    0    0  4308     0 16227 14254  7 18 74  1  0
 1  0     11     56   2781    638    0    0  2320  1456 16310 14124  6 19 74  0  1
 1  0     13     67   2776    631    0    1  3924  1372 14720 14019  6 20 74  0  1
 1  0     13     66   2779    631    0    0  2768     0 16105 14038  6 19 74  0  0
 1  0     13     63   2782    631    0    0  3000     0 14471 14002  6 19 74  0  0
 1  0     13     58   2786    632    0    0  3988    36 12383 13135  7 19 73  1  0
 1  0     13     56   2789    632    0    0  2488   116 12417 13853  6 20 74  0  0
 1  0     13     61   2788    627    0    0  2556   296 12402 13382  7 20 73  0  0
 2  0     13     59   2791    627    0    0  2552     0 16114 14085  7 18 74  0  1
 1  0     13     56   2793    627    0    0  2320     0 16155 14092  5 20 75  0  0
 1  0     14     69   2787    621    0    1  2848  1248 16766 14480  7 19 73  1  1
 1  0     14     65   2792    620    0    0  4356     6 16369 14136  6 20 74  0  0
 1  0     14     62   2795    621    0    0  3020     0 16079 14079  7 19 74  0  1
 1  0     14     59   2798    621    0    0  2964     0 16229 14084  5 19 75  0  1
 1  0     14     57   2800    621    0    0  2172     0 16454 14257  6 18 75  0  0
 2  0     15     69   2794    614    0    0  3024   712 16416 14241  7 18 73  1  1
 1  0     15     67   2797    615    0    0  2936    32 16168 14084  6 19 74  0  0

grant map with indirect desriptors
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1      0     89   1900   1477    0    0  5760    24 9876 11438  5 19 76  0  0
 2  0      0     83   1906   1478    0    0  5568     0 8829 12670  5 19 76  1  0
 1  0      0     78   1911   1477    0    0  4736     0 8649 11291  5 19 76  0  1
 1  0      0     73   1916   1478    0    0  5120   984 8746 11946  5 20 75  0  0
 1  0      0     66   1922   1478    0    0  6016     0 8959 11785  6 18 76  0  1
 1  0      0     61   1927   1478    0    0  5312    32 9031 11559  5 18 76  1  0
 2  0      0     56   1932   1477    0    0  4608     0 9170 12156  5 19 75  0  1
 2  0      0     63   1937   1466    0    0  4992    28 8205 11871  5 21 74  0  0
 2  0      0     57   1942   1466    0    0  4928     0 8249 12198  5 18 76  0  0
 1  0      0     67   1948   1450    0    0  5376     8 10813 11381  6 20 74  0  0
 2  0      0     63   1952   1450    0    0  4288   192 9651 11814  5 20 70  4  1
 1  0      0     59   1956   1450    0    0  4096     0 8960 12058  4 19 76  0  0
 1  0      0     68   1962   1434    0    0  5184    12 9207 12089  5 20 75  0  1
 1  0      0     64   1966   1434    0    0  4096     0 8433 12016  5 20 75  0  0
 1  0      0     60   1970   1434    0    0  4224   140 10919 10750  5 18 76  0  0
 1  0      0     55   1976   1434    0    0  5440     0 8362 12207  5 19 76  0  0
 1  0      0     60   1980   1425    0    0  3776    64 8437 12020  6 19 74  1  1
 1  0      0     55   1984   1425    0    0  4416     0 8902 11962  6 17 76  0  0

grant copy
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  1      0     63   2789    760    0    0  2268     8 36651 32671  6 19 74  0  1
 0  0      0     62   2791    760    0    0  1240     4 36465 33543  5 18 75  1  1
 1  0      0     61   2792    760    0    0  1584     0 36237 32312  4 21 74  0  1
 3  0      0     59   2794    760    0    0  1628     0 36475 32888  4 20 75  0  1
 2  0      0     57   2796    760    0    0  1968     0 34898 33329  5 19 75  0  1
 0  0      0     55   2798    759    0    0  1948     0 31510 31938  4 20 75  0  1
 1  0      0     66   2794    753    0    0  2244    12 36692 34147  5 18 75  1  1
 1  1      0     64   2796    753    0    0  1792    20 29159 32907  5 18 76  0  1
 1  0      0     62   2798    753    0    0  2416     0 37445 35323  2 19 77  1  1
 2  0      0     59   2800    753    0    0  2188     0 35741 32670  4 20 76  0  1
 1  0      0     58   2802    753    0    0  1772     0 36770 34468  4 17 78  0  1
 0  0      0     56   2803    752    0    0  1260     0 36317 33152  4 19 76  0  1
 1  0      0     55   2805    753    0    0  1216     0 36364 32263  4 19 76  0  1
 4  0      0     67   2802    743    0    0  1068    16 35886 32045  5 19 75  1  1
 2  0      0     65   2805    743    0    0  2928     0 28347 33364  5 20 74  0  1
 2  0      0     62   2807    743    0    0  1944     0 36737 35010  4 17 78  0  1
 1  0      0     61   2808    743    0    0  1540     0 35855 31968  3 20 76  0  1
 1  0      0     58   2810    743    0    0  2268     0 36047 31639  4 20 75  0  1

grant copy with indirect descriptors
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0 29    296     55      9    150    0  153 124752 167260 11989 13103  2  7  6 84  0
 0 28    296     53      9    156    0    0  2692   620  596  852  0  0  0 100  0
 0 27    296     72      1    159    0    0  7896  2072  675  882  0  2  0 98  0
 2 11    313     53     13    146    0   17 35392 17032 2920 3640  1  5 10 83  0
 0 13    324     58      1    139    0   10 25240 10688 2270 2584  0  3 22 74  0
 0 27    447     72      0    117    0  122 126116 120204 10926 12598  1  5  4 90  0
 1 24    450     67      0    130    0    3 11968  3608  772 1283  0  2  5 93  0
 0 21    486     71      0    133    0   35  5176 34968  596  952  0  2 46 53  0
 0 18    486     62      0    141    0    0  9352     0  652 1029  0  1 35 64  0
 0 14    488     80      0    146    0    2 12068  2124  584  706  1  2 26 71  0
 0 27    619     59      8    104    0  131 126264 128104 10722 12416  1  7  5 87  0
 0 22    619     78      0    111    0    0 16652    28 1267 1865  0  2  0 98  0
 0 25    800     68      5     81    0  180 166844 176752 13875 16661  1  6  0 92  0
 0 19    800     55      1    100    0    0 14436   200  763  909  0  1  0 99  0
 2 15    801     57      6    105    0    1 16308  1080 1103 1461  0  4 17 79  0
 0 28    832     68      0     82    0   30 179036 30080 14029 16746  2  8  6 83  0
 0 17    831     57      0     94    0    0 14140     0 1082 1316  0  2 13 86  0
 1 18    849     68      1    101    0   17  9908 17444  691  873  0  2  6 92  0

[1] https://docs.google.com/spreadsheets/d/1E6AMiB8ceJpExL6jWpH9u2yy6DZxzhmDUyFf-eUuJ0c/edit#gid=1390267663
[2] https://github.com/paulina-szubarczyk/xen-benchmark/tree/master/gprof

Thanks and regards, 
Paulina

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

             reply	other threads:[~2016-06-15 17:02 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-06-15 17:01 Paulina Szubarczyk [this message]
2016-06-15 17:01 ` [PATCH 1/1] " Paulina Szubarczyk

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1466010062-16270-1-git-send-email-paulinaszubarczyk@gmail.com \
    --to=paulinaszubarczyk@gmail.com \
    --cc=P.Gawkowski@ii.pw.edu.pl \
    --cc=anthony.perard@citrix.com \
    --cc=roger.pau@citrix.com \
    --cc=sstabellini@kernel.org \
    --cc=xen-devel@lists.xenproject.org \
    --subject='Re: [PATCH 0/1] qemu-qdisk: indirect descriptors' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).