From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2001883698390801544==" MIME-Version: 1.0 From: huangqingxin at ruijie.com.cn Subject: [SPDK] ioat performance questions Date: Mon, 04 Dec 2017 13:38:13 +0000 Message-ID: <6DA4AE090ED3494A8051AF4C22FDA0911D1402F0@FZEX4.ruijie.com.cn> List-ID: To: spdk@lists.01.org --===============2001883698390801544== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable hi, When I run the ioat_perf provided by spdk , I get this result. [root(a)localhost kperf]# ./ioat_kperf -n 8 Total 8 Channels, Queue_Depth 256, Transfer Size 4096 Bytes, Total Transfer= Size 4 GB Running I/O . . . . . . . . Channel 0 Bandwidth 661 MiB/s Channel 1 Bandwidth 660 MiB/s Channel 2 Bandwidth 661 MiB/s Channel 3 Bandwidth 661 MiB/s Channel 4 Bandwidth 661 MiB/s Channel 5 Bandwidth 661 MiB/s Channel 6 Bandwidth 661 MiB/s Channel 7 Bandwidth 661 MiB/s Total Channel Bandwidth: 5544 MiB/s Average Bandwidth Per Channel: 660 MiB/s [root(a)localhost kperf]# ./ioat_kperf -n 4 Total 4 Channels, Queue_Depth 256, Transfer Size 4096 Bytes, Total Transfer= Size 4 GB Running I/O . . . . . Channel 0 Bandwidth 1319 MiB/s Channel 1 Bandwidth 1322 MiB/s Channel 2 Bandwidth 1319 MiB/s Channel 3 Bandwidth 1318 MiB/s Total Channel Bandwidth: 5530 MiB/s Average Bandwidth Per Channel: 1318 MiB/s [root(a)localhost kperf]# [root(a)localhost kperf]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz Stepping: 2 CPU MHz: 1200.000 CPU max MHz: 2400.0000 CPU min MHz: 1200.0000 BogoMIPS: 4799.90 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-5,12-17 NUMA node1 CPU(s): 6-11,18-23 I found the `Total Channel Bandwidth` can not increase with more channels. = What's the limitation? Does the performance of ioat dma on E5 V3 can only a= ccess around 5GB/s ? Any helps will be appreciated! --===============2001883698390801544== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWw+CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRl eHQvaHRtbDsgY2hhcnNldD1nYjIzMTIiPgo8c3R5bGU+Ym9keSB7IGxpbmUtaGVpZ2h0OiAxLjU7 IH1ib2R5IHsgZm9udC1zaXplOiAxMC41cHQ7IGZvbnQtZmFtaWx5OiDLzszlOyBjb2xvcjogcmdi KDAsIDAsIDApOyBsaW5lLWhlaWdodDogMS41OyB9PC9zdHlsZT4KPC9oZWFkPgo8Ym9keT4KPGRp dj48c3Bhbj48L3NwYW4+CjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiDLzszlLCBUYWhvbWE7Ij4K PGRpdj5oaSwmbmJzcDs8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PldoZW4gSSBydW4gdGhl IGlvYXRfcGVyZiBwcm92aWRlZCBieSBzcGRrICwgSSBnZXQgdGhpcyByZXN1bHQuPC9kaXY+Cjxk aXY+PGJyPgo8L2Rpdj4KPGRpdj5bcm9vdEBsb2NhbGhvc3QmbmJzcDtrcGVyZl0jJm5ic3A7Li9p b2F0X2twZXJmJm5ic3A7LW4mbmJzcDs4PGJyPgpUb3RhbCZuYnNwOzgmbmJzcDtDaGFubmVscywm bmJzcDtRdWV1ZV9EZXB0aCZuYnNwOzI1NiwmbmJzcDtUcmFuc2ZlciZuYnNwO1NpemUmbmJzcDs0 MDk2Jm5ic3A7Qnl0ZXMsJm5ic3A7VG90YWwmbmJzcDtUcmFuc2ZlciZuYnNwO1NpemUmbmJzcDs0 Jm5ic3A7R0I8YnI+ClJ1bm5pbmcmbmJzcDtJL08mbmJzcDsuJm5ic3A7LiZuYnNwOy4mbmJzcDsu Jm5ic3A7LiZuYnNwOy4mbmJzcDsuJm5ic3A7Ljxicj4KQ2hhbm5lbCZuYnNwOzAmbmJzcDtCYW5k d2lkdGgmbmJzcDs2NjEmbmJzcDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzEmbmJzcDtCYW5kd2lk dGgmbmJzcDs2NjAmbmJzcDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzImbmJzcDtCYW5kd2lkdGgm bmJzcDs2NjEmbmJzcDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzMmbmJzcDtCYW5kd2lkdGgmbmJz cDs2NjEmbmJzcDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzQmbmJzcDtCYW5kd2lkdGgmbmJzcDs2 NjEmbmJzcDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzUmbmJzcDtCYW5kd2lkdGgmbmJzcDs2NjEm bmJzcDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzYmbmJzcDtCYW5kd2lkdGgmbmJzcDs2NjEmbmJz cDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzcmbmJzcDtCYW5kd2lkdGgmbmJzcDs2NjEmbmJzcDtN aUIvczxicj4KVG90YWwmbmJzcDtDaGFubmVsJm5ic3A7QmFuZHdpZHRoOiZuYnNwOzU1NDQmbmJz cDtNaUIvczxicj4KQXZlcmFnZSZuYnNwO0JhbmR3aWR0aCZuYnNwO1BlciZuYnNwO0NoYW5uZWw6 Jm5ic3A7NjYwJm5ic3A7TWlCL3M8YnI+Cltyb290QGxvY2FsaG9zdCZuYnNwO2twZXJmXSMmbmJz cDsuL2lvYXRfa3BlcmYmbmJzcDstbiZuYnNwOzQ8YnI+ClRvdGFsJm5ic3A7NCZuYnNwO0NoYW5u ZWxzLCZuYnNwO1F1ZXVlX0RlcHRoJm5ic3A7MjU2LCZuYnNwO1RyYW5zZmVyJm5ic3A7U2l6ZSZu YnNwOzQwOTYmbmJzcDtCeXRlcywmbmJzcDtUb3RhbCZuYnNwO1RyYW5zZmVyJm5ic3A7U2l6ZSZu YnNwOzQmbmJzcDtHQjxicj4KUnVubmluZyZuYnNwO0kvTyZuYnNwOy4mbmJzcDsuJm5ic3A7LiZu YnNwOy4mbmJzcDsuPGJyPgpDaGFubmVsJm5ic3A7MCZuYnNwO0JhbmR3aWR0aCZuYnNwOzEzMTkm bmJzcDtNaUIvczxicj4KQ2hhbm5lbCZuYnNwOzEmbmJzcDtCYW5kd2lkdGgmbmJzcDsxMzIyJm5i c3A7TWlCL3M8YnI+CkNoYW5uZWwmbmJzcDsyJm5ic3A7QmFuZHdpZHRoJm5ic3A7MTMxOSZuYnNw O01pQi9zPGJyPgpDaGFubmVsJm5ic3A7MyZuYnNwO0JhbmR3aWR0aCZuYnNwOzEzMTgmbmJzcDtN aUIvczxicj4KVG90YWwmbmJzcDtDaGFubmVsJm5ic3A7QmFuZHdpZHRoOiZuYnNwOzU1MzAmbmJz cDtNaUIvczxicj4KQXZlcmFnZSZuYnNwO0JhbmR3aWR0aCZuYnNwO1BlciZuYnNwO0NoYW5uZWw6 Jm5ic3A7MTMxOCZuYnNwO01pQi9zPGJyPgpbcm9vdEBsb2NhbGhvc3QmbmJzcDtrcGVyZl0jPGJy Pgo8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2Pltyb290QGxvY2FsaG9zdCZuYnNwO2twZXJm XSMmbmJzcDtsc2NwdTxicj4KQXJjaGl0ZWN0dXJlOiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwO3g4Nl82NDxicj4KQ1BVJm5ic3A7b3At bW9kZShzKTombmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsz Mi1iaXQsJm5ic3A7NjQtYml0PGJyPgpCeXRlJm5ic3A7T3JkZXI6Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7TGl0 dGxlJm5ic3A7RW5kaWFuPGJyPgpDUFUocyk6Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7MjQ8YnI+Ck9uLWxpbmUmbmJzcDtDUFUocykmbmJzcDtsaXN0OiZuYnNwOyZuYnNw OyZuYnNwOzAtMjM8YnI+ClRocmVhZChzKSZuYnNwO3BlciZuYnNwO2NvcmU6Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Mjxicj4KQ29yZShzKSZuYnNwO3BlciZuYnNwO3NvY2tldDombmJzcDsmbmJz cDsmbmJzcDsmbmJzcDs2PGJyPgpTb2NrZXQocyk6Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Mjxicj4K TlVNQSZuYnNwO25vZGUocyk6Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Mjxicj4KVmVuZG9yJm5ic3A7SUQ6Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7R2VudWluZUludGVsPGJyPgpDUFUmbmJzcDtmYW1pbHk6Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Njxi cj4KTW9kZWw6Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7NjM8 YnI+Ck1vZGVsJm5ic3A7bmFtZTombmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtJbnRlbChSKSZuYnNwO1hlb24oUikm bmJzcDtDUFUmbmJzcDtFNS0yNjIwJm5ic3A7djMmbmJzcDtAJm5ic3A7Mi40MEdIejxicj4KU3Rl cHBpbmc6Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Mjxicj4KQ1BVJm5ic3A7TUh6OiZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOzEyMDAuMDAwPGJyPgpDUFUmbmJzcDttYXgmbmJz cDtNSHo6Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7MjQwMC4wMDAwPGJyPgpDUFUmbmJzcDttaW4mbmJzcDtNSHo6Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7MTIwMC4wMDAwPGJyPgpCb2dvTUlQUzombmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDs0Nzk5 LjkwPGJyPgpWaXJ0dWFsaXphdGlvbjombmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDtWVC14PGJyPgpMMWQmbmJzcDtjYWNoZTombmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDszMks8YnI+CkwxaSZuYnNwO2NhY2hlOiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOzMySzxicj4KTDIm bmJzcDtjYWNoZTombmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsyNTZLPGJyPgpMMyZuYnNwO2Nh Y2hlOiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOzE1MzYwSzxicj4KTlVNQSZuYnNwO25vZGUw Jm5ic3A7Q1BVKHMpOiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOzAtNSwxMi0xNzxicj4K TlVNQSZuYnNwO25vZGUxJm5ic3A7Q1BVKHMpOiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OzYtMTEsMTgtMjM8YnI+CjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+SSBmb3VuZCB0aGUg YFRvdGFsIENoYW5uZWwgQmFuZHdpZHRoYCBjYW4gbm90IGluY3JlYXNlIHdpdGggbW9yZSBjaGFu bmVscy4gV2hhdCdzIHRoZSBsaW1pdGF0aW9uPyBEb2VzIHRoZSBwZXJmb3JtYW5jZSBvZiBpb2F0 IGRtYSBvbiBFNSBWMyBjYW4gb25seSBhY2Nlc3MgYXJvdW5kIDVHQi9zID88L2Rpdj4KPC9kaXY+ CjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiDLzszlLCBUYWhvbWE7Ij48YnI+CjwvZGl2Pgo8ZGl2 IHN0eWxlPSJmb250LWZhbWlseTogy87M5SwgVGFob21hOyI+QW55IGhlbHBzIHdpbGwgYmUgYXBw cmVjaWF0ZWQhPC9kaXY+CjwvZGl2Pgo8L2JvZHk+CjwvaHRtbD4K --===============2001883698390801544==--