memcpy over NTB

memcpy over NTB

[Glimcher, Boris]

[-- Attachment #1: Type: text/plain, Size: 430 bytes --]

Hi,

I've been testing NTB over Microsemi PCIe switch for a few weeks using NTB_PERF
and other tools from the kernel over ntb_hw_switchtec.c (from https://github.com/Microsemi/switchtec-kernel)

I'd like to move to user space now and for starters I would like to do a simple MEMCPY over NTB to the peer's memory.

What the best approach for this? What kernel drivers I must use/modify?

Thanks,
--
Best regards, Boris

[-- Attachment #2: Type: text/html, Size: 2754 bytes --]

Re: memcpy over NTB

[Logan Gunthorpe]

On 5/1/2018 6:09 AM, Glimcher, Boris wrote:
> I’d like to move to user space now and for starters I would like to do a 
> simple MEMCPY over NTB to the peer’s memory.

This isn't supported by the kernel at all. It also introduces 
significant security concerns. I'm afraid you're on your own if you want 
to do something like that.

I'd suggest developing a custom in-kernel NTB client to suit your needs 
instead.

Logan

Re: memcpy over NTB

[Dave Jiang]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]


On 5/1/2018 5:09 AM, Glimcher, Boris wrote:
>
> Hi,
>
> I’ve been testing NTB over Microsemi PCIe switch for a few weeks using 
> NTB_PERF
>
> and other tools from the kernel over ntb_hw_switchtec.c (from 
> https://github.com/Microsemi/switchtec-kernel 
> <https://github.com/Microsemi/switchtec-kernel>)
>
> I’d like to move to user space now and for starters I would like to do 
> a simple MEMCPY over NTB to the peer’s memory.
>
> What the best approach for this? What kernel drivers I must use/modify?
>
There are a lot of people that create VFIO user drivers in place of the 
kernel driver. That's probably an approach. Also, VFIO mediated device 
approach may also be a possibility. Have the kernel driver configure and 
carve out a virtual device for your app to consume.


> Thanks,
>
> --
>
> Best regards, Boris
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to linux-ntb+unsubscribe@googlegroups.com 
> <mailto:linux-ntb+unsubscribe@googlegroups.com>.
> To post to this group, send email to linux-ntb@googlegroups.com 
> <mailto:linux-ntb@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C017448F2%40MX302CL01.corp.emc.com 
> <https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C017448F2%40MX302CL01.corp.emc.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[-- Attachment #2: Type: text/html, Size: 4521 bytes --]

RE: memcpy over NTB

[Glimcher, Boris]

So one approach could be to modify kernel driver and use ioctl from user space to submit buffers for transfer and kernel driver will copy or use dma to copy the buffer to the other side?
What do you think? Valid?

--
Best regards, Boris
Let me google that for you


-----Original Message-----
From: Logan Gunthorpe [mailto:logang@deltatee.com] 
Sent: Tuesday, May 1, 2018 18:41
To: Glimcher, Boris; linux-ntb@googlegroups.com
Subject: Re: memcpy over NTB

On 5/1/2018 6:09 AM, Glimcher, Boris wrote:
> I’d like to move to user space now and for starters I would like to do 
> a simple MEMCPY over NTB to the peer’s memory.

This isn't supported by the kernel at all. It also introduces significant security concerns. I'm afraid you're on your own if you want to do something like that.

I'd suggest developing a custom in-kernel NTB client to suit your needs instead.

Logan

Re: memcpy over NTB

[Logan Gunthorpe]

On 5/1/2018 10:09 AM, Glimcher, Boris wrote:
> So one approach could be to modify kernel driver and use ioctl from user space to submit buffers for transfer and kernel driver will copy or use dma to copy the buffer to the other side?
> What do you think? Valid?

Yes, I think that'd be a good choice. Create an in-kernel NTB client 
with a char device which has an IOCTL or read/write interface for 
submitting buffers. This keeps MW setup and DMA operations inside the 
kernel.

Logan

RE: memcpy over NTB

[Glimcher, Boris]

Is it also possible with this in-kernel NTB client with a char device to MMAP somehow MW to user? 

So the kernel code will be limited just to setup the connection section and will not take part during data transfer at all?
Cause I see that eventually kernel driver just does "memcpy_toio"...

Am I missing anything?

--
Best regards, Boris
Let me google that for you


-----Original Message-----
From: Logan Gunthorpe [mailto:logang@deltatee.com] 
Sent: Tuesday, May 1, 2018 19:15
To: Glimcher, Boris; linux-ntb@googlegroups.com
Subject: Re: memcpy over NTB

On 5/1/2018 10:09 AM, Glimcher, Boris wrote:
> So one approach could be to modify kernel driver and use ioctl from user space to submit buffers for transfer and kernel driver will copy or use dma to copy the buffer to the other side?
> What do you think? Valid?

Yes, I think that'd be a good choice. Create an in-kernel NTB client with a char device which has an IOCTL or read/write interface for submitting buffers. This keeps MW setup and DMA operations inside the kernel.

Logan

Re: memcpy over NTB

[Logan Gunthorpe]

On 5/1/2018 10:20 AM, Glimcher, Boris wrote:
> Is it also possible with this in-kernel NTB client with a char device to MMAP somehow MW to user?

Yes, this is possible but generally discouraged for drivers.

If you are attempting to optimize for performance then it's much better 
to submit large buffers to a DMA engine.

If you are trying to have random access to the MW you'll have to be very 
careful about synchronization with the opposite side.

Logan

RE: memcpy over NTB

[Glimcher, Boris]

The synchronization is not an issue, I believe, as I'm looking just to mirror large memory area from one side to another

--
Best regards, Boris
Let me google that for you


-----Original Message-----
From: Logan Gunthorpe [mailto:logang@deltatee.com] 
Sent: Tuesday, May 1, 2018 19:26
To: Glimcher, Boris; linux-ntb@googlegroups.com
Subject: Re: memcpy over NTB

On 5/1/2018 10:20 AM, Glimcher, Boris wrote:
> Is it also possible with this in-kernel NTB client with a char device to MMAP somehow MW to user?

Yes, this is possible but generally discouraged for drivers.

If you are attempting to optimize for performance then it's much better to submit large buffers to a DMA engine.

If you are trying to have random access to the MW you'll have to be very careful about synchronization with the opposite side.

Logan

Re: memcpy over NTB

[Logan Gunthorpe]

On 5/2/2018 2:06 AM, Glimcher, Boris wrote:
> The synchronization is not an issue, I believe, as I'm looking just to mirror large memory area from one side to another

What happens if one side is reading while the other side is changing the 
memory? They'd read some of the data before the change and some of the 
data after the change resulting in inconsistent data. "just mirroring" 
doesn't solve those issues.

Logan

RE: memcpy over NTB

[Glimcher, Boris]

Invalid situation. One side is always writing. Other side is never reading until first side is down. (High Availability stuff)

--
Best regards, Boris
Let me google that for you


-----Original Message-----
From: linux-ntb@googlegroups.com [mailto:linux-ntb@googlegroups.com] On Behalf Of Logan Gunthorpe
Sent: Wednesday, May 2, 2018 18:43
To: Glimcher, Boris; linux-ntb@googlegroups.com
Subject: Re: memcpy over NTB

On 5/2/2018 2:06 AM, Glimcher, Boris wrote:
> The synchronization is not an issue, I believe, as I'm looking just to 
> mirror large memory area from one side to another

What happens if one side is reading while the other side is changing the memory? They'd read some of the data before the change and some of the data after the change resulting in inconsistent data. "just mirroring" 
doesn't solve those issues.

Logan

--
You received this message because you are subscribed to the Google Groups "linux-ntb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com.
To post to this group, send email to linux-ntb@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/4f05cc88-5ad6-e8f0-070c-0cc62d30d878%40deltatee.com.
For more options, visit https://groups.google.com/d/optout.