From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============7291638636156751373==" MIME-Version: 1.0 From: Shahar Salzman Subject: Re: [SPDK] Hugepage allocation, and issue with non contiguous memory Date: Thu, 01 Feb 2018 07:59:15 +0000 Message-ID: In-Reply-To: 1517418025.25907.108.camel@intel.com List-ID: To: spdk@lists.01.org --===============7291638636156751373== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Ben, Thanks for the response, I'll stay tuned for updates on the dpdk side. One small question, are there places where large memory blocks are expected= to be contiguous? Before preallocating memory, we would sometimes fail to = open an RDMA session since there was not enough contiguous memory, but it s= eems that this is not a requirement, since the large allocated block is the= n split into a mempool. Shahar ________________________________ From: SPDK on behalf of Walker, Benjamin Sent: Wednesday, January 31, 2018 7:00:51 PM To: spdk(a)lists.01.org Subject: Re: [SPDK] Hugepage allocation, and issue with non contiguous memo= ry On Mon, 2018-01-29 at 12:11 +0000, Shahar Salzman wrote: > On our system, we make extensive use of hugepages, so only a fraction of = the > hugepages are for spdk usage, and the memory allocated may be fragmented = at > the hugepage level. > Initially we used "--socket-mem=3D2048,0", but init time was very long, p= robably > since dpdk built its hugepage info from all the hugepages on the system. > For the fragmentation I am running a small program that initializes dpdk > before the rest of the hugepage owners start allocating their pages. > > Is there a better way to limit the # of pages that dpdk works on, and to > preallocate a contiguous amount of hugepages? This is a common scenario that many users of SPDK have run into. However, S= PDK is just using DPDK's memory allocation framework, so SPDK can't solve it by itself. We've been working with the DPDK team for nearly a year now to capt= ure all of the challenges that SPDK users have had and to turn them into improvements to the core memory management system in DPDK. There is current= ly a very large patch series that makes memory allocation in DPDK dynamic. See t= his thread: https://dpdk.org/ml/archives/dev/2017-December/084302.html I don't know when that will land, but as soon as it does land SPDK will take advantage of it. That's the "right" way to fix this, in the long term. In the short term, what you're doing is probably the best practice. _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============7291638636156751373== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWw+CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRl eHQvaHRtbDsgY2hhcnNldD11cy1hc2NpaSI+CjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyIgc3R5bGU9 ImRpc3BsYXk6bm9uZTsiPjwhLS0gUCB7bWFyZ2luLXRvcDowO21hcmdpbi1ib3R0b206MDt9IC0t Pjwvc3R5bGU+CjwvaGVhZD4KPGJvZHkgZGlyPSJsdHIiPgo8ZGl2IGlkPSJkaXZ0YWdkZWZhdWx0 d3JhcHBlciIgc3R5bGU9ImZvbnQtc2l6ZToxMnB0O2NvbG9yOiMwMDAwMDA7Zm9udC1mYW1pbHk6 Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZjsiIGRpcj0ibHRyIj4KPHAgc3R5bGU9Im1hcmdp bi10b3A6MDttYXJnaW4tYm90dG9tOjAiPkhpIEJlbiw8L3A+CjxwIHN0eWxlPSJtYXJnaW4tdG9w OjA7bWFyZ2luLWJvdHRvbTowIj48YnI+CjwvcD4KPHAgc3R5bGU9Im1hcmdpbi10b3A6MDttYXJn aW4tYm90dG9tOjAiPlRoYW5rcyBmb3IgdGhlIHJlc3BvbnNlLCBJJ2xsIHN0YXkgdHVuZWQgZm9y IHVwZGF0ZXMgb24gdGhlIGRwZGsgc2lkZS48L3A+CjxwIHN0eWxlPSJtYXJnaW4tdG9wOjA7bWFy Z2luLWJvdHRvbTowIj5PbmUgc21hbGwgcXVlc3Rpb24sIGFyZSB0aGVyZSBwbGFjZXMgd2hlcmUg bGFyZ2UgbWVtb3J5IGJsb2NrcyBhcmUgZXhwZWN0ZWQgdG8gYmUgY29udGlndW91cz8gQmVmb3Jl IHByZWFsbG9jYXRpbmcgbWVtb3J5LCB3ZSB3b3VsZCBzb21ldGltZXMgZmFpbCB0byBvcGVuIGFu IFJETUEgc2Vzc2lvbiBzaW5jZSB0aGVyZSB3YXMgbm90IGVub3VnaCBjb250aWd1b3VzIG1lbW9y eSwKIGJ1dCBpdCBzZWVtcyB0aGF0IHRoaXMgaXMgbm90IGEgcmVxdWlyZW1lbnQsIHNpbmNlIHRo ZSBsYXJnZSBhbGxvY2F0ZWQgYmxvY2sgaXMgdGhlbiBzcGxpdCBpbnRvIGEgbWVtcG9vbC48L3A+ CjxwIHN0eWxlPSJtYXJnaW4tdG9wOjA7bWFyZ2luLWJvdHRvbTowIj48YnI+CjwvcD4KPHAgc3R5 bGU9Im1hcmdpbi10b3A6MDttYXJnaW4tYm90dG9tOjAiPlNoYWhhcjwvcD4KPC9kaXY+CjxociBz dHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7d2lkdGg6OTglIiB0YWJpbmRleD0iLTEiPgo8ZGl2 IGlkPSJkaXZScGx5RndkTXNnIiBkaXI9Imx0ciI+PGZvbnQgZmFjZT0iQ2FsaWJyaSwgc2Fucy1z ZXJpZiIgc3R5bGU9ImZvbnQtc2l6ZToxMXB0IiBjb2xvcj0iIzAwMDAwMCI+PGI+RnJvbTo8L2I+ IFNQREsgJmx0O3NwZGstYm91bmNlc0BsaXN0cy4wMS5vcmcmZ3Q7IG9uIGJlaGFsZiBvZiBXYWxr ZXIsIEJlbmphbWluICZsdDtiZW5qYW1pbi53YWxrZXJAaW50ZWwuY29tJmd0Ozxicj4KPGI+U2Vu dDo8L2I+IFdlZG5lc2RheSwgSmFudWFyeSAzMSwgMjAxOCA3OjAwOjUxIFBNPGJyPgo8Yj5Ubzo8 L2I+IHNwZGtAbGlzdHMuMDEub3JnPGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtTUERLXSBIdWdl cGFnZSBhbGxvY2F0aW9uLCBhbmQgaXNzdWUgd2l0aCBub24gY29udGlndW91cyBtZW1vcnk8L2Zv bnQ+CjxkaXY+Jm5ic3A7PC9kaXY+CjwvZGl2Pgo8ZGl2IGNsYXNzPSJCb2R5RnJhZ21lbnQiPjxm b250IHNpemU9IjIiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDsiPgo8ZGl2IGNsYXNzPSJQ bGFpblRleHQiPk9uIE1vbiwgMjAxOC0wMS0yOSBhdCAxMjoxMSAmIzQzOzAwMDAsIFNoYWhhciBT YWx6bWFuIHdyb3RlOjxicj4KJmd0OyBPbiBvdXIgc3lzdGVtLCB3ZSBtYWtlIGV4dGVuc2l2ZSB1 c2Ugb2YgaHVnZXBhZ2VzLCBzbyBvbmx5IGEgZnJhY3Rpb24gb2YgdGhlPGJyPgomZ3Q7IGh1Z2Vw YWdlcyBhcmUgZm9yIHNwZGsgdXNhZ2UsIGFuZCB0aGUgbWVtb3J5IGFsbG9jYXRlZCBtYXkgYmUg ZnJhZ21lbnRlZCBhdDxicj4KJmd0OyB0aGUgaHVnZXBhZ2UgbGV2ZWwuPGJyPgomZ3Q7IEluaXRp YWxseSB3ZSB1c2VkICZxdW90Oy0tc29ja2V0LW1lbT0yMDQ4LDAmcXVvdDssIGJ1dCBpbml0IHRp bWUgd2FzIHZlcnkgbG9uZywgcHJvYmFibHk8YnI+CiZndDsgc2luY2UgZHBkayBidWlsdCBpdHMg aHVnZXBhZ2UgaW5mbyBmcm9tIGFsbCB0aGUgaHVnZXBhZ2VzIG9uIHRoZSBzeXN0ZW0uPGJyPgom Z3Q7Jm5ic3A7IEZvciB0aGUgZnJhZ21lbnRhdGlvbiBJIGFtIHJ1bm5pbmcgYSBzbWFsbCBwcm9n cmFtIHRoYXQgaW5pdGlhbGl6ZXMgZHBkazxicj4KJmd0OyBiZWZvcmUgdGhlIHJlc3Qgb2YgdGhl IGh1Z2VwYWdlIG93bmVycyBzdGFydCBhbGxvY2F0aW5nIHRoZWlyIHBhZ2VzLjxicj4KJmd0OyA8 YnI+CiZndDsgSXMgdGhlcmUgYSBiZXR0ZXIgd2F5IHRvIGxpbWl0IHRoZSAjIG9mIHBhZ2VzIHRo YXQgZHBkayB3b3JrcyBvbiwgYW5kIHRvPGJyPgomZ3Q7IHByZWFsbG9jYXRlIGEgY29udGlndW91 cyBhbW91bnQgb2YgaHVnZXBhZ2VzPzxicj4KPGJyPgpUaGlzIGlzIGEgY29tbW9uIHNjZW5hcmlv IHRoYXQgbWFueSB1c2VycyBvZiBTUERLIGhhdmUgcnVuIGludG8uIEhvd2V2ZXIsIFNQREs8YnI+ CmlzIGp1c3QgdXNpbmcgRFBESydzIG1lbW9yeSBhbGxvY2F0aW9uIGZyYW1ld29yaywgc28gU1BE SyBjYW4ndCBzb2x2ZSBpdCBieTxicj4KaXRzZWxmLiBXZSd2ZSBiZWVuIHdvcmtpbmcgd2l0aCB0 aGUgRFBESyB0ZWFtIGZvciBuZWFybHkgYSB5ZWFyIG5vdyB0byBjYXB0dXJlPGJyPgphbGwgb2Yg dGhlIGNoYWxsZW5nZXMgdGhhdCBTUERLIHVzZXJzIGhhdmUgaGFkIGFuZCB0byB0dXJuIHRoZW0g aW50bzxicj4KaW1wcm92ZW1lbnRzIHRvIHRoZSBjb3JlIG1lbW9yeSBtYW5hZ2VtZW50IHN5c3Rl bSBpbiBEUERLLiBUaGVyZSBpcyBjdXJyZW50bHkgYTxicj4KdmVyeSBsYXJnZSBwYXRjaCBzZXJp ZXMgdGhhdCBtYWtlcyBtZW1vcnkgYWxsb2NhdGlvbiBpbiBEUERLIGR5bmFtaWMuIFNlZSB0aGlz PGJyPgp0aHJlYWQ6PGJyPgo8YnI+CjxhIGhyZWY9Imh0dHBzOi8vZHBkay5vcmcvbWwvYXJjaGl2 ZXMvZGV2LzIwMTctRGVjZW1iZXIvMDg0MzAyLmh0bWwiPmh0dHBzOi8vZHBkay5vcmcvbWwvYXJj aGl2ZXMvZGV2LzIwMTctRGVjZW1iZXIvMDg0MzAyLmh0bWw8L2E+PGJyPgo8YnI+CkkgZG9uJ3Qg a25vdyB3aGVuIHRoYXQgd2lsbCBsYW5kLCBidXQgYXMgc29vbiBhcyBpdCBkb2VzIGxhbmQgU1BE SyB3aWxsIHRha2U8YnI+CmFkdmFudGFnZSBvZiBpdC4gVGhhdCdzIHRoZSAmcXVvdDtyaWdodCZx dW90OyB3YXkgdG8gZml4IHRoaXMsIGluIHRoZSBsb25nIHRlcm0uPGJyPgo8YnI+CkluIHRoZSBz aG9ydCB0ZXJtLCB3aGF0IHlvdSdyZSBkb2luZyBpcyBwcm9iYWJseSB0aGUgYmVzdCBwcmFjdGlj ZS48YnI+Cjxicj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X188YnI+ClNQREsgbWFpbGluZyBsaXN0PGJyPgpTUERLQGxpc3RzLjAxLm9yZzxicj4KPGEgaHJl Zj0iaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0aW5mby9zcGRrIj5odHRwczovL2xp c3RzLjAxLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3NwZGs8L2E+PGJyPgo8YnI+CjwvZGl2Pgo8L3Nw YW4+PC9mb250PjwvZGl2Pgo8L2JvZHk+CjwvaHRtbD4K --===============7291638636156751373==--