From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0104118071334188388==" MIME-Version: 1.0 From: Ilan Steinberg Subject: Re: [SPDK] SPDK + user space appliance Date: Tue, 23 Jan 2018 08:20:20 +0000 Message-ID: In-Reply-To: 1516382916.25907.15.camel@intel.com List-ID: To: spdk@lists.01.org --===============0104118071334188388== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable +Shahar Salzman Ilan Steinberg R&D Team Lead Kaminario m: +972-528-337-818 a: 1 Haotsma st. Yokneam, Israel w: kaminario.com e: ilan.steinberg(a)kaminario.com ________________________________ From: Walker, Benjamin Sent: 19 January 2018 7:28 PM To: spdk(a)lists.01.org Cc: Yael Shavit; Amir Sasson; Ilan Steinberg Subject: Re: [SPDK] SPDK + user space appliance On Sun, 2018-01-14 at 09:09 +0000, Shahar Salzman wrote: > We have been integrating spdk into our system using a blockdev module, > currently only a POC version. > Our use case is a user space appliance processing IOs, with an SPDK front= end > to do the NVMeF. > > Currently all of the user bdevs are created via configuration file, but w= e are > working to add functions + rpc's which allow creation/deletion of these > namespaces. Ok - so you have our NVMe-oF target in front and it routes I/Os to your sys= tem via a custom bdev module that you wrote. This is exactly how we intended fo= r you to use it, so it looks good to me. > IO is sent to user space via callback, implementation is up to user space= , but > obviously the longer it lingers there the lower the performance, we use a= set > of rings + threads processing them, so that the time spent in the applian= ce is > minimal. > Going back from user space we use a single ring (multiple producers single > consumer) onto which the completions are inserted, and the ring poll func= tion > is registered with spdk core (spdk_poller_register). What do you mean by I/O is sent to user space via callback? When was the I/O ever in kernel space? Is your storage system running in the kernel? --===============0104118071334188388== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWw+CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRl eHQvaHRtbDsgY2hhcnNldD1pc28tODg1OS0xIj4KPHN0eWxlIHR5cGU9InRleHQvY3NzIiBzdHls ZT0iZGlzcGxheTpub25lOyI+PCEtLSBQIHttYXJnaW4tdG9wOjA7bWFyZ2luLWJvdHRvbTowO30g LS0+PC9zdHlsZT4KPC9oZWFkPgo8Ym9keSBkaXI9Imx0ciI+CjxkaXYgaWQ9ImRpdnRhZ2RlZmF1 bHR3cmFwcGVyIiBzdHlsZT0iZm9udC1zaXplOjEycHQ7Y29sb3I6IzAwMDAwMDtmb250LWZhbWls eTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmOyIgZGlyPSJsdHIiPgo8cCBzdHlsZT0ibWFy Z2luLXRvcDowO21hcmdpbi1ib3R0b206MCI+JiM0MztTaGFoYXIgU2Fsem1hbjwvcD4KPHAgc3R5 bGU9Im1hcmdpbi10b3A6MDttYXJnaW4tYm90dG9tOjAiPjxicj4KPC9wPgo8ZGl2IGlkPSJTaWdu YXR1cmUiPgo8ZGl2IGlkPSJkaXZ0YWdkZWZhdWx0d3JhcHBlciIgZGlyPSJsdHIiIHN0eWxlPSJm b250LXNpemU6MTJwdDsgY29sb3I6cmdiKDAsMCwwKTsgZm9udC1mYW1pbHk6Q2FsaWJyaSxBcmlh bCxIZWx2ZXRpY2Esc2Fucy1zZXJpZixFbW9qaUZvbnQsJnF1b3Q7QXBwbGUgQ29sb3IgRW1vamkm cXVvdDssJnF1b3Q7U2Vnb2UgVUkgRW1vamkmcXVvdDssTm90b0NvbG9yRW1vamksJnF1b3Q7U2Vn b2UgVUkgU3ltYm9sJnF1b3Q7LCZxdW90O0FuZHJvaWQgRW1vamkmcXVvdDssRW1vamlTeW1ib2xz Ij4KPGRpdiBuYW1lPSJkaXZ0YWdkZWZhdWx0d3JhcHBlciIgc3R5bGU9ImZvbnQtZmFtaWx5OkNh bGlicmksQXJpYWwsSGVsdmV0aWNhLHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTo7IG1hcmdpbjowIj4K PHRhYmxlIGNsYXNzPSJ4X01zb05vcm1hbFRhYmxlIiBib3JkZXI9IjAiIGNlbGxzcGFjaW5nPSIw IiBjZWxscGFkZGluZz0iMCIgd2lkdGg9IjU4OCIgc3R5bGU9ImNvbG9yOnJnYigzMywzMywzMyk7 IGZvbnQtc2l6ZToxNXB4OyBib3JkZXItY29sbGFwc2U6Y29sbGFwc2UiPgo8dGJvZHk+Cjx0cj4K PHRkIHdpZHRoPSI1NjkiIGNvbHNwYW49IjIiIHN0eWxlPSJ3aWR0aDo0MjYuNnB0OyBwYWRkaW5n OjBjbSA1LjRwdCI+CjxwIGNsYXNzPSJ4X01zb05vcm1hbCIgc3R5bGU9Im1hcmdpbjowY20gMGNt IDAuMDAwMXB0OyBmb250LXNpemU6MTFwdDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxzYW5zLXNlcmlm Ij4KPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxNHB0OyBmb250LWZhbWlseTpBcmlhbCxzYW5zLXNl cmlmOyBjb2xvcjpyZ2IoMTQsNjgsMTYyKSI+SWxhbiBTdGVpbmJlcmc8L3NwYW4+PC9wPgo8L3Rk Pgo8L3RyPgo8dHI+Cjx0ZCB3aWR0aD0iNTY5IiBjb2xzcGFuPSIyIiBzdHlsZT0id2lkdGg6NDI2 LjZwdDsgcGFkZGluZzowY20gNS40cHQiPgo8cCBjbGFzcz0ieF9Nc29Ob3JtYWwiIHN0eWxlPSJt YXJnaW46MGNtIDBjbSAwLjAwMDFwdDsgZm9udC1zaXplOjExcHQ7IGZvbnQtZmFtaWx5OkNhbGli cmksc2Fucy1zZXJpZiI+CjxpPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNl cmlmOyBjb2xvcjpyZ2IoMjksMjksMjkpIj5SJmFtcDtEIFRlYW0gTGVhZDwvc3Bhbj48L2k+PC9w Pgo8L3RkPgo8L3RyPgo8dHI+Cjx0ZCB3aWR0aD0iNTY5IiBjb2xzcGFuPSIyIiBzdHlsZT0id2lk dGg6NDI2LjZwdDsgcGFkZGluZzowY20gNS40cHQiPgo8cCBjbGFzcz0ieF9Nc29Ob3JtYWwiIHN0 eWxlPSJtYXJnaW46MGNtIDBjbSAwLjAwMDFwdDsgZm9udC1zaXplOjExcHQ7IGZvbnQtZmFtaWx5 OkNhbGlicmksc2Fucy1zZXJpZiI+CjxiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxz YW5zLXNlcmlmOyBjb2xvcjpyZ2IoMjksMjksMjkpIj5LYW1pbmFyaW88L3NwYW4+PC9iPjxzcGFu IHN0eWxlPSJmb250LXNpemU6MTBwdCI+PC9zcGFuPjwvcD4KPC90ZD4KPC90cj4KPHRyPgo8dGQg d2lkdGg9IjM4IiB2YWxpZ249InRvcCIgc3R5bGU9IndpZHRoOjI4LjZwdDsgcGFkZGluZzowY20g NS40cHQiPgo8cCBjbGFzcz0ieF9Nc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW46MGNtIDBjbSAwLjAw MDFwdDsgZm9udC1zaXplOjExcHQ7IGZvbnQtZmFtaWx5OkNhbGlicmksc2Fucy1zZXJpZiI+Cjxz cGFuIHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmOyBjb2xvcjpyZ2IoNzIsMCwx MjMpIj5tPC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmOyBj b2xvcjpyZ2IoMjM2LDcyLDIxKSI+Ojwvc3Bhbj48L3A+CjwvdGQ+Cjx0ZCB3aWR0aD0iNTExIiBz dHlsZT0id2lkdGg6MzgzLjZwdDsgcGFkZGluZzowY20gNS40cHQiPgo8cCBjbGFzcz0ieF9Nc29O b3JtYWwiIHN0eWxlPSJtYXJnaW46MGNtIDBjbSAwLjAwMDFwdDsgZm9udC1zaXplOjExcHQ7IGZv bnQtZmFtaWx5OkNhbGlicmksc2Fucy1zZXJpZiI+CjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTpB cmlhbCxzYW5zLXNlcmlmOyBjb2xvcjpyZ2IoMjksMjksMjkpIj4mIzQzOzk3Mi01MjgtMzM3LTgx ODwvc3Bhbj48L3A+CjwvdGQ+CjwvdHI+Cjx0cj4KPHRkIHdpZHRoPSIzOCIgdmFsaWduPSJ0b3Ai IHN0eWxlPSJ3aWR0aDoyOC42cHQ7IHBhZGRpbmc6MGNtIDUuNHB0Ij4KPHAgY2xhc3M9InhfTXNv Tm9ybWFsIiBzdHlsZT0ibWFyZ2luOjBjbSAwY20gMC4wMDAxcHQ7IGZvbnQtc2l6ZToxMXB0OyBm b250LWZhbWlseTpDYWxpYnJpLHNhbnMtc2VyaWYiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6 QXJpYWwsc2Fucy1zZXJpZjsgY29sb3I6cmdiKDcyLDAsMTIzKSI+YTwvc3Bhbj48c3BhbiBzdHls ZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjsgY29sb3I6cmdiKDIzNiw3MiwyMSkiPjo8 L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMHB0Ij48L3NwYW4+PC9wPgo8L3RkPgo8dGQg d2lkdGg9IjUxMSIgdmFsaWduPSJ0b3AiIHN0eWxlPSJ3aWR0aDozODMuNnB0OyBwYWRkaW5nOjBj bSA1LjRwdCI+MSZuYnNwO0hhb3RzbWEgc3QuIFlva25lYW0sIElzcmFlbDwvdGQ+CjwvdHI+Cjx0 cj4KPHRkIHdpZHRoPSIzOCIgdmFsaWduPSJ0b3AiIHN0eWxlPSJ3aWR0aDoyOC42cHQ7IHBhZGRp bmc6MGNtIDUuNHB0Ij4KPHAgY2xhc3M9InhfTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luOjBjbSAw Y20gMC4wMDAxcHQ7IGZvbnQtc2l6ZToxMXB0OyBmb250LWZhbWlseTpDYWxpYnJpLHNhbnMtc2Vy aWYiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjsgY29sb3I6cmdi KDcyLDAsMTIzKSI+dzwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1z ZXJpZjsgY29sb3I6cmdiKDIzNiw3MiwyMSkiPjo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtc2l6 ZToxMHB0Ij48L3NwYW4+PC9wPgo8L3RkPgo8dGQgd2lkdGg9IjUxMSIgdmFsaWduPSJ0b3AiIHN0 eWxlPSJ3aWR0aDozODMuNnB0OyBwYWRkaW5nOjBjbSA1LjRwdCI+CjxwIGNsYXNzPSJ4X01zb05v cm1hbCIgc3R5bGU9Im1hcmdpbjowY20gMGNtIDAuMDAwMXB0OyBmb250LXNpemU6MTFwdDsgZm9u dC1mYW1pbHk6Q2FsaWJyaSxzYW5zLXNlcmlmIj4KPGEgaHJlZj0iaHR0cDovL2thbWluYXJpby5j b20vIiB0YXJnZXQ9Il9ibGFuayIgcmVsPSJub29wZW5lciBub3JlZmVycmVyIiBpZD0iTFBOb0xQ Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjsgY29sb3I6cmdiKDI3 LDEyMywxOTkpIj5rYW1pbmFyaW8uY29tPC9zcGFuPjwvYT48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjE0cHQ7IGZvbnQtZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7IGNvbG9yOnJnYigyOSwyOSwyOSki PiZuYnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjE0cHQ7IGZvbnQtZmFtaWx5OkFy aWFsLHNhbnMtc2VyaWY7IGNvbG9yOnJnYig3MiwwLDEyMykiPiZuYnNwOzwvc3Bhbj48c3BhbiBz dHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjsgY29sb3I6cmdiKDcyLDAsMTIzKSI+ ZTwvc3Bhbj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6QXJpYWwsc2Fucy1zZXJpZjsgY29sb3I6 cmdiKDIzNiw3MiwyMSkiPjombmJzcDs8L3NwYW4+PHU+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5 OkFyaWFsLHNhbnMtc2VyaWY7IGNvbG9yOnJnYigwLDExMiwxOTIpIj5pbGFuLnN0ZWluYmVyZ0Br YW1pbmFyaW8uY29tPC9zcGFuPjwvdT48L3A+CjwvdGQ+CjwvdHI+Cjx0ciBzdHlsZT0iaGVpZ2h0 OjEzLjk1cHQiPgo8dGQgd2lkdGg9IjU2OSIgY29sc3Bhbj0iMiIgc3R5bGU9IndpZHRoOjQyNi42 cHQ7IHBhZGRpbmc6MGNtIDUuNHB0OyBoZWlnaHQ6MTMuOTVwdCI+CjxwIGNsYXNzPSJ4X01zb05v cm1hbCIgc3R5bGU9Im1hcmdpbjowY20gMGNtIDAuMDAwMXB0OyBmb250LXNpemU6MTFwdDsgZm9u dC1mYW1pbHk6Q2FsaWJyaSxzYW5zLXNlcmlmIj4KPHNwYW4gaWQ9Im1zLXJ0ZXJhbmdlcGFzdGUt c3RhcnQiPjwvc3Bhbj48L3A+CjwvdGQ+CjwvdHI+CjwvdGJvZHk+CjwvdGFibGU+Cjxicj4KPC9k aXY+CjwvZGl2Pgo8L2Rpdj4KPGJyPgo8YnI+CjxkaXYgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwg MCk7Ij4KPGhyIHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jazt3aWR0aDo5OCUiIHRhYmluZGV4 PSItMSI+CjxkaXYgaWQ9ImRpdlJwbHlGd2RNc2ciIGRpcj0ibHRyIj48Zm9udCBmYWNlPSJDYWxp YnJpLCBzYW5zLXNlcmlmIiBzdHlsZT0iZm9udC1zaXplOjExcHQiIGNvbG9yPSIjMDAwMDAwIj48 Yj5Gcm9tOjwvYj4gV2Fsa2VyLCBCZW5qYW1pbiAmbHQ7YmVuamFtaW4ud2Fsa2VyQGludGVsLmNv bSZndDs8YnI+CjxiPlNlbnQ6PC9iPiAxOSBKYW51YXJ5IDIwMTggNzoyOCBQTTxicj4KPGI+VG86 PC9iPiBzcGRrQGxpc3RzLjAxLm9yZzxicj4KPGI+Q2M6PC9iPiBZYWVsIFNoYXZpdDsgQW1pciBT YXNzb247IElsYW4gU3RlaW5iZXJnPGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtTUERLXSBTUERL ICYjNDM7IHVzZXIgc3BhY2UgYXBwbGlhbmNlPC9mb250Pgo8ZGl2PiZuYnNwOzwvZGl2Pgo8L2Rp dj4KPGRpdiBjbGFzcz0iQm9keUZyYWdtZW50Ij48Zm9udCBzaXplPSIyIj48c3BhbiBzdHlsZT0i Zm9udC1zaXplOjExcHQ7Ij4KPGRpdiBjbGFzcz0iUGxhaW5UZXh0Ij5PbiBTdW4sIDIwMTgtMDEt MTQgYXQgMDk6MDkgJiM0MzswMDAwLCBTaGFoYXIgU2Fsem1hbiB3cm90ZTo8YnI+CiZndDsgV2Ug aGF2ZSBiZWVuIGludGVncmF0aW5nIHNwZGsgaW50byBvdXIgc3lzdGVtIHVzaW5nIGEgYmxvY2tk ZXYgbW9kdWxlLDxicj4KJmd0OyBjdXJyZW50bHkgb25seSBhIFBPQyB2ZXJzaW9uLjxicj4KJmd0 OyBPdXIgdXNlIGNhc2UgaXMgYSB1c2VyIHNwYWNlIGFwcGxpYW5jZSBwcm9jZXNzaW5nIElPcywg d2l0aCBhbiBTUERLIGZyb250ZW5kPGJyPgomZ3Q7IHRvIGRvIHRoZSBOVk1lRi48YnI+CiZndDsg PGJyPgomZ3Q7IEN1cnJlbnRseSBhbGwgb2YgdGhlIHVzZXIgYmRldnMgYXJlIGNyZWF0ZWQgdmlh IGNvbmZpZ3VyYXRpb24gZmlsZSwgYnV0IHdlIGFyZTxicj4KJmd0OyB3b3JraW5nIHRvIGFkZCBm dW5jdGlvbnMgJiM0MzsgcnBjJ3Mgd2hpY2ggYWxsb3cgY3JlYXRpb24vZGVsZXRpb24gb2YgdGhl c2U8YnI+CiZndDsgbmFtZXNwYWNlcy48YnI+Cjxicj4KT2sgLSBzbyB5b3UgaGF2ZSBvdXIgTlZN ZS1vRiB0YXJnZXQgaW4gZnJvbnQgYW5kIGl0IHJvdXRlcyBJL09zIHRvIHlvdXIgc3lzdGVtPGJy Pgp2aWEgYSBjdXN0b20gYmRldiBtb2R1bGUgdGhhdCB5b3Ugd3JvdGUuIFRoaXMgaXMgZXhhY3Rs eSBob3cgd2UgaW50ZW5kZWQgZm9yIHlvdTxicj4KdG8gdXNlIGl0LCBzbyBpdCBsb29rcyBnb29k IHRvIG1lLjxicj4KPGJyPgomZ3Q7IElPIGlzIHNlbnQgdG8gdXNlciBzcGFjZSB2aWEgY2FsbGJh Y2ssIGltcGxlbWVudGF0aW9uIGlzIHVwIHRvIHVzZXIgc3BhY2UsIGJ1dDxicj4KJmd0OyBvYnZp b3VzbHkgdGhlIGxvbmdlciBpdCBsaW5nZXJzIHRoZXJlIHRoZSBsb3dlciB0aGUgcGVyZm9ybWFu Y2UsIHdlIHVzZSBhIHNldDxicj4KJmd0OyBvZiByaW5ncyAmIzQzOyB0aHJlYWRzIHByb2Nlc3Np bmcgdGhlbSwgc28gdGhhdCB0aGUgdGltZSBzcGVudCBpbiB0aGUgYXBwbGlhbmNlIGlzPGJyPgom Z3Q7IG1pbmltYWwuPGJyPgomZ3Q7IEdvaW5nIGJhY2sgZnJvbSB1c2VyIHNwYWNlIHdlIHVzZSBh IHNpbmdsZSByaW5nIChtdWx0aXBsZSBwcm9kdWNlcnMgc2luZ2xlPGJyPgomZ3Q7IGNvbnN1bWVy KSBvbnRvIHdoaWNoIHRoZSBjb21wbGV0aW9ucyBhcmUgaW5zZXJ0ZWQsIGFuZCB0aGUgcmluZyBw b2xsIGZ1bmN0aW9uPGJyPgomZ3Q7IGlzIHJlZ2lzdGVyZWQgd2l0aCBzcGRrIGNvcmUgKHNwZGtf cG9sbGVyX3JlZ2lzdGVyKS48YnI+Cjxicj4KV2hhdCBkbyB5b3UgbWVhbiBieSBJL08gaXMgc2Vu dCB0byB1c2VyIHNwYWNlIHZpYSBjYWxsYmFjaz8gV2hlbiB3YXMgdGhlIEkvTzxicj4KZXZlciBp biBrZXJuZWwgc3BhY2U/IElzIHlvdXIgc3RvcmFnZSBzeXN0ZW0gcnVubmluZyBpbiB0aGUga2Vy bmVsPzxicj4KPC9kaXY+Cjwvc3Bhbj48L2ZvbnQ+PC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9ib2R5 Pgo8L2h0bWw+Cg== --===============0104118071334188388==--