PCIe P2P access to GPU memory

PCIe P2P access to GPU memory

[Dirk Eibach]

I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with
mainline drivers.
I can acquire a dmabuf from the GPU and pass it to my PCIe
framegrabber. But I don't see a way to get the PCIe bus address of the
video memory which I need for the P2P transfer. Is there already any
infrastructure in place for doing this? If not, what would be the
right way to implement this? Add another callback to the dmabuf
struct?

Cheers
Dirk
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Koenig, Christian]

[-- Attachment #1.1: Type: text/plain, Size: 812 bytes --]

This is work in progress.

I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this.

I can prepare you a branch based on current upstream kernel next week if you want to test this.

Regards,
Christian.

Am 29.09.2018 09:01 schrieb Dirk Eibach <dirk.eibach@googlemail.com>:
I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with
mainline drivers.
I can acquire a dmabuf from the GPU and pass it to my PCIe
framegrabber. But I don't see a way to get the PCIe bus address of the
video memory which I need for the P2P transfer. Is there already any
infrastructure in place for doing this? If not, what would be the
right way to implement this? Add another callback to the dmabuf
struct?

Cheers
Dirk

[-- Attachment #1.2: Type: text/html, Size: 1586 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Dirk Eibach]

> This is work in progress.
>
> I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this.

Great news! Can you give me a link to this series so I can already have a look?

> I can prepare you a branch based on current upstream kernel next week if you want to test this.

That would be awesome, thanks.

Cheers
Dirk
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Dieter Nützel]

Hello Dirk,

I think Christian is talking about this branch:

https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p

His 'home' is, here:
https://cgit.freedesktop.org/~deathsimple/linux/

Happy hacking! ;-)

Dieter

Am 29.09.2018 10:17, schrieb Dirk Eibach:
>> This is work in progress.
>> 
>> I published patches to enable DMA_buf P2P a few months ago, but now 
>> I'm waiting for the PCI subsystem to pick up core support for this.
> 
> Great news! Can you give me a link to this series so I can already have 
> a look?
> 
>> I can prepare you a branch based on current upstream kernel next week 
>> if you want to test this.
> 
> That would be awesome, thanks.
> 
> Cheers
> Dirk
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Dirk Eibach]

Hi Christian,

I took the repository Dieter suggested and gave it a spin. I am not
sure, if I got the API right.

In userspace I create a bo and get a fd.
bo = gbm_bo_create(gbm, w, h, GBM_FORMAT_ARGB8888, GBM_BO_USE_LINEAR);
dmabuf_fd = gbm_bo_get_fd(bo);

Then I pass the fd to the grabber driver. There I get a sgt, that I
want to pass to the DMA-machine.
struct dma_buf_attach_info dma_buf_attach_info = {
    .dev = &engine->xdev->pdev->dev,
    .peer2peer = true,
};
dma_buf = dma_buf_get(xdma_dma_buf_ioctl.dma_buf_fd);
dma_buf_attach_info.dmabuf = dma_buf;
dma_buf_attachment = dma_buf_attach(&dma_buf_attach_info);
sgt = dma_buf_map_attachment(dma_buf_attachment, DMA_TO_DEVICE);

But the addresses in the sgt do not look like PCIe addresses at at all.

Any ideas?

Cheers
Dirk
Am Sa., 29. Sep. 2018 um 11:53 Uhr schrieb Dieter Nützel <Dieter@nuetzel-hh.de>:
>
> Hello Dirk,
>
> I think Christian is talking about this branch:
>
> https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p
>
> His 'home' is, here:
> https://cgit.freedesktop.org/~deathsimple/linux/
>
> Happy hacking! ;-)
>
> Dieter
>
> Am 29.09.2018 10:17, schrieb Dirk Eibach:
> >> This is work in progress.
> >>
> >> I published patches to enable DMA_buf P2P a few months ago, but now
> >> I'm waiting for the PCI subsystem to pick up core support for this.
> >
> > Great news! Can you give me a link to this series so I can already have
> > a look?
> >
> >> I can prepare you a branch based on current upstream kernel next week
> >> if you want to test this.
> >
> > That would be awesome, thanks.
> >
> > Cheers
> > Dirk
> > _______________________________________________
> > dri-devel mailing list
> > dri-devel@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Christian König]

Hi Dirk,

> Any ideas?
Could be anything. Most likely cause is that some function things the 
two device can't communicate directly with each other.

Please provide a full dmesg and "lspci -t -nn -v" as attachment and note 
which devices should communicate with which.

I'm currently on vacation so it could take a while till I have time to 
look deeper into this.

Christian.

Am 01.10.2018 um 15:07 schrieb Dirk Eibach:
> Hi Christian,
>
> I took the repository Dieter suggested and gave it a spin. I am not
> sure, if I got the API right.
>
> In userspace I create a bo and get a fd.
> bo = gbm_bo_create(gbm, w, h, GBM_FORMAT_ARGB8888, GBM_BO_USE_LINEAR);
> dmabuf_fd = gbm_bo_get_fd(bo);
>
> Then I pass the fd to the grabber driver. There I get a sgt, that I
> want to pass to the DMA-machine.
> struct dma_buf_attach_info dma_buf_attach_info = {
>      .dev = &engine->xdev->pdev->dev,
>      .peer2peer = true,
> };
> dma_buf = dma_buf_get(xdma_dma_buf_ioctl.dma_buf_fd);
> dma_buf_attach_info.dmabuf = dma_buf;
> dma_buf_attachment = dma_buf_attach(&dma_buf_attach_info);
> sgt = dma_buf_map_attachment(dma_buf_attachment, DMA_TO_DEVICE);
>
> But the addresses in the sgt do not look like PCIe addresses at at all.
>
> Any ideas?
>
> Cheers
> Dirk
> Am Sa., 29. Sep. 2018 um 11:53 Uhr schrieb Dieter Nützel <Dieter@nuetzel-hh.de>:
>> Hello Dirk,
>>
>> I think Christian is talking about this branch:
>>
>> https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p
>>
>> His 'home' is, here:
>> https://cgit.freedesktop.org/~deathsimple/linux/
>>
>> Happy hacking! ;-)
>>
>> Dieter
>>
>> Am 29.09.2018 10:17, schrieb Dirk Eibach:
>>>> This is work in progress.
>>>>
>>>> I published patches to enable DMA_buf P2P a few months ago, but now
>>>> I'm waiting for the PCI subsystem to pick up core support for this.
>>> Great news! Can you give me a link to this series so I can already have
>>> a look?
>>>
>>>> I can prepare you a branch based on current upstream kernel next week
>>>> if you want to test this.
>>> That would be awesome, thanks.
>>>
>>> Cheers
>>> Dirk
>>> _______________________________________________
>>> dri-devel mailing list
>>> dri-devel@lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Dirk Eibach]

[-- Attachment #1.1: Type: text/plain, Size: 2584 bytes --]

Hi Christian,

Could I simply force it by overwriting peer2peer after the attach call?

Cheers
Dirk

> Hi Dirk,
>
> > Any ideas?
> Could be anything. Most likely cause is that some function things the
> two device can't communicate directly with each other.
>
> Please provide a full dmesg and "lspci -t -nn -v" as attachment and note
> which devices should communicate with which.
>
> I'm currently on vacation so it could take a while till I have time to
> look deeper into this.
>
> Christian.
>
> Am 01.10.2018 um 15:07 schrieb Dirk Eibach:
> > Hi Christian,
> >
> > I took the repository Dieter suggested and gave it a spin. I am not
> > sure, if I got the API right.
> >
> > In userspace I create a bo and get a fd.
> > bo = gbm_bo_create(gbm, w, h, GBM_FORMAT_ARGB8888, GBM_BO_USE_LINEAR);
> > dmabuf_fd = gbm_bo_get_fd(bo);
> >
> > Then I pass the fd to the grabber driver. There I get a sgt, that I
> > want to pass to the DMA-machine.
> > struct dma_buf_attach_info dma_buf_attach_info = {
> >      .dev = &engine->xdev->pdev->dev,
> >      .peer2peer = true,
> > };
> > dma_buf = dma_buf_get(xdma_dma_buf_ioctl.dma_buf_fd);
> > dma_buf_attach_info.dmabuf = dma_buf;
> > dma_buf_attachment = dma_buf_attach(&dma_buf_attach_info);
> > sgt = dma_buf_map_attachment(dma_buf_attachment, DMA_TO_DEVICE);
> >
> > But the addresses in the sgt do not look like PCIe addresses at at all.
> >
> > Any ideas?
> >
> > Cheers
> > Dirk
> > Am Sa., 29. Sep. 2018 um 11:53 Uhr schrieb Dieter Nützel <
> Dieter@nuetzel-hh.de>:
> >> Hello Dirk,
> >>
> >> I think Christian is talking about this branch:
> >>
> >> https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p
> >>
> >> His 'home' is, here:
> >> https://cgit.freedesktop.org/~deathsimple/linux/
> >>
> >> Happy hacking! ;-)
> >>
> >> Dieter
> >>
> >> Am 29.09.2018 10:17, schrieb Dirk Eibach:
> >>>> This is work in progress.
> >>>>
> >>>> I published patches to enable DMA_buf P2P a few months ago, but now
> >>>> I'm waiting for the PCI subsystem to pick up core support for this.
> >>> Great news! Can you give me a link to this series so I can already have
> >>> a look?
> >>>
> >>>> I can prepare you a branch based on current upstream kernel next week
> >>>> if you want to test this.
> >>> That would be awesome, thanks.
> >>>
> >>> Cheers
> >>> Dirk
> >>> _______________________________________________
> >>> dri-devel mailing list
> >>> dri-devel@lists.freedesktop.org
> >>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>
>

[-- Attachment #1.2: Type: text/html, Size: 4009 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Daniel Vetter]

On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian
<Christian.Koenig@amd.com> wrote:
>
> This is work in progress.
>
> I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this.
>
> I can prepare you a branch based on current upstream kernel next week if you want to test this.

Thread hijack, since I just read the lwn article on p2p for storage folks:

https://lwn.net/Articles/767281/

Totally doesn't match what I remember of your dma-buf p2p work. Do we
need to rework, or is this some kind of higher-level interface on top
of the raw p2p pci stuff? Or do I misremember.

Thanks, Daniel

>
> Regards,
> Christian.
>
> Am 29.09.2018 09:01 schrieb Dirk Eibach <dirk.eibach@googlemail.com>:
> I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with
> mainline drivers.
> I can acquire a dmabuf from the GPU and pass it to my PCIe
> framegrabber. But I don't see a way to get the PCIe bus address of the
> video memory which I need for the P2P transfer. Is there already any
> infrastructure in place for doing this? If not, what would be the
> right way to implement this? Add another callback to the dmabuf
> struct?
>
> Cheers
> Dirk
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Koenig, Christian]

Am 03.10.2018 um 09:14 schrieb Daniel Vetter:
> On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian
> <Christian.Koenig@amd.com> wrote:
>> This is work in progress.
>>
>> I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this.
>>
>> I can prepare you a branch based on current upstream kernel next week if you want to test this.
> Thread hijack, since I just read the lwn article on p2p for storage folks:
>
> https://lwn.net/Articles/767281/
>
> Totally doesn't match what I remember of your dma-buf p2p work. Do we
> need to rework, or is this some kind of higher-level interface on top
> of the raw p2p pci stuff? Or do I misremember.

The P2P from the storage folks are some mix of raw p2p capability 
detection for PCI and higher level interface.

I'm actually waiting for that stuff to land to extract the p2p 
capability detection and use it for this DMA-buf stuff as well.

Alternative would be to convince Logan to extract the capability 
detection in his patch set as well, but so far he wasn't very keen about 
that.

Christian.

>
> Thanks, Daniel
>
>> Regards,
>> Christian.
>>
>> Am 29.09.2018 09:01 schrieb Dirk Eibach <dirk.eibach@googlemail.com>:
>> I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with
>> mainline drivers.
>> I can acquire a dmabuf from the GPU and pass it to my PCIe
>> framegrabber. But I don't see a way to get the PCIe bus address of the
>> video memory which I need for the P2P transfer. Is there already any
>> infrastructure in place for doing this? If not, what would be the
>> right way to implement this? Add another callback to the dmabuf
>> struct?
>>
>> Cheers
>> Dirk
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>
>

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Daniel Vetter]

On Wed, Oct 03, 2018 at 08:54:44AM +0000, Koenig, Christian wrote:
> Am 03.10.2018 um 09:14 schrieb Daniel Vetter:
> > On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian
> > <Christian.Koenig@amd.com> wrote:
> >> This is work in progress.
> >>
> >> I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this.
> >>
> >> I can prepare you a branch based on current upstream kernel next week if you want to test this.
> > Thread hijack, since I just read the lwn article on p2p for storage folks:
> >
> > https://lwn.net/Articles/767281/
> >
> > Totally doesn't match what I remember of your dma-buf p2p work. Do we
> > need to rework, or is this some kind of higher-level interface on top
> > of the raw p2p pci stuff? Or do I misremember.
> 
> The P2P from the storage folks are some mix of raw p2p capability 
> detection for PCI and higher level interface.
> 
> I'm actually waiting for that stuff to land to extract the p2p 
> capability detection and use it for this DMA-buf stuff as well.
> 
> Alternative would be to convince Logan to extract the capability 
> detection in his patch set as well, but so far he wasn't very keen about 
> that.

So no plan to make dma-buf some kind of orchestrator and use the p2p
map_sg stuff? That seems to be the new/additional bits I haven't seen yet,
and I'm wondered how we should fit dma-buf into that picture.
-Daniel

> 
> Christian.
> 
> >
> > Thanks, Daniel
> >
> >> Regards,
> >> Christian.
> >>
> >> Am 29.09.2018 09:01 schrieb Dirk Eibach <dirk.eibach@googlemail.com>:
> >> I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with
> >> mainline drivers.
> >> I can acquire a dmabuf from the GPU and pass it to my PCIe
> >> framegrabber. But I don't see a way to get the PCIe bus address of the
> >> video memory which I need for the P2P transfer. Is there already any
> >> infrastructure in place for doing this? If not, what would be the
> >> right way to implement this? Add another callback to the dmabuf
> >> struct?
> >>
> >> Cheers
> >> Dirk
> >> _______________________________________________
> >> dri-devel mailing list
> >> dri-devel@lists.freedesktop.org
> >> https://lists.freedesktop.org/mailman/listinfo/dri-devel
> >
> >
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: PCIe P2P access to GPU memory

[Christian König]

Am 03.10.2018 um 11:06 schrieb Daniel Vetter:
> On Wed, Oct 03, 2018 at 08:54:44AM +0000, Koenig, Christian wrote:
>> Am 03.10.2018 um 09:14 schrieb Daniel Vetter:
>>> On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian
>>> <Christian.Koenig@amd.com> wrote:
>>>> This is work in progress.
>>>>
>>>> I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this.
>>>>
>>>> I can prepare you a branch based on current upstream kernel next week if you want to test this.
>>> Thread hijack, since I just read the lwn article on p2p for storage folks:
>>>
>>> https://lwn.net/Articles/767281/
>>>
>>> Totally doesn't match what I remember of your dma-buf p2p work. Do we
>>> need to rework, or is this some kind of higher-level interface on top
>>> of the raw p2p pci stuff? Or do I misremember.
>> The P2P from the storage folks are some mix of raw p2p capability
>> detection for PCI and higher level interface.
>>
>> I'm actually waiting for that stuff to land to extract the p2p
>> capability detection and use it for this DMA-buf stuff as well.
>>
>> Alternative would be to convince Logan to extract the capability
>> detection in his patch set as well, but so far he wasn't very keen about
>> that.
> So no plan to make dma-buf some kind of orchestrator and use the p2p
> map_sg stuff?

Not sure about that yet. Didn't had time to take a closer look.

Christian.

>   That seems to be the new/additional bits I haven't seen yet,
> and I'm wondered how we should fit dma-buf into that picture.
> -Daniel
>
>> Christian.
>>
>>> Thanks, Daniel
>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>> Am 29.09.2018 09:01 schrieb Dirk Eibach <dirk.eibach@googlemail.com>:
>>>> I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with
>>>> mainline drivers.
>>>> I can acquire a dmabuf from the GPU and pass it to my PCIe
>>>> framegrabber. But I don't see a way to get the PCIe bus address of the
>>>> video memory which I need for the P2P transfer. Is there already any
>>>> infrastructure in place for doing this? If not, what would be the
>>>> right way to implement this? Add another callback to the dmabuf
>>>> struct?
>>>>
>>>> Cheers
>>>> Dirk
>>>> _______________________________________________
>>>> dri-devel mailing list
>>>> dri-devel@lists.freedesktop.org
>>>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>>>

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel