Hotplug support for VFIO

Hotplug support for VFIO

[Eelco Chaudron]

Hi,

I was wondering if some one is working on VFIO hotplug support?

Cheers,

Eelco

Re: Hotplug support for VFIO

[Burakov, Anatoly]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eelco Chaudron
> Sent: Thursday, February 2, 2017 11:24 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Hotplug support for VFIO
> 
> Hi,
> 
> I was wondering if some one is working on VFIO hotplug support?
> 
> Cheers,
> 
> Eelco

Hi Eelco,

Please forgive me my ignorance on this matter, but doesn't it work at the moment? I would assume that if regular PCI hotplug works (with igb_uio), then so would hotplug with VFIO, as it basically utilizes the same PCI infrastructure igb_uio does. That said, I'm not aware of any patches submitted that had to do with VFIO and hotplug, so I guess the answer is, not at the moment.

Thanks,
Anatoly

Re: Hotplug support for VFIO

[Eelco Chaudron]

On 02/02/17 13:05, Burakov, Anatoly wrote:
> Hi Eelco,
>
> Please forgive me my ignorance on this matter, but doesn't it work at the moment? I would assume that if regular PCI hotplug works (with igb_uio), then so would hotplug with VFIO, as it basically utilizes the same PCI infrastructure igb_uio does. That said, I'm not aware of any patches submitted that had to do with VFIO and hotplug, so I guess the answer is, not at the moment.
>
> Thanks,
> Anatoly
I was asking as the documentation explicitly mentions its not supported.
http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug

- "To detach a port, the port should be backed by a device that igb_uio 
manages. VFIO is not supported."

I could not find any specific reason why it's not supported, so if some 
one can explain this it would help also...

Cheers,

Eelco

Re: Hotplug support for VFIO

[Alejandro Lucero]

It seems none is working on this VFIO support.

I will work on this if there is no reply to this thread saying the opposite
the next days.

On Thu, Feb 2, 2017 at 12:58 PM, Eelco Chaudron <echaudron@redhat.com>
wrote:

> On 02/02/17 13:05, Burakov, Anatoly wrote:
>
>> Hi Eelco,
>>
>> Please forgive me my ignorance on this matter, but doesn't it work at the
>> moment? I would assume that if regular PCI hotplug works (with igb_uio),
>> then so would hotplug with VFIO, as it basically utilizes the same PCI
>> infrastructure igb_uio does. That said, I'm not aware of any patches
>> submitted that had to do with VFIO and hotplug, so I guess the answer is,
>> not at the moment.
>>
>> Thanks,
>> Anatoly
>>
> I was asking as the documentation explicitly mentions its not supported.
> http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug
>
> - "To detach a port, the port should be backed by a device that igb_uio
> manages. VFIO is not supported."
>
> I could not find any specific reason why it's not supported, so if some
> one can explain this it would help also...
>
> Cheers,
>
> Eelco
>
>
>

Re: Hotplug support for VFIO

[Alejandro Lucero]

Hi Eelco,

On Wed, Feb 8, 2017 at 9:41 AM, Eelco Chaudron <echaudron@redhat.com> wrote:

> Hi Anatoly,
>
> This will be great... If you want I can do some testing on (early) patches
> if required.
>
> Also do you know why it is currently not supported, i.e. what are the
> limitations?
>
> Thanks,
>

I assume your reply was to my previous email and not to Anatoly's one. My
apologies if this assumption is wrong.
By the way, you did not reply to the list, but I'm doing to do so because
this could be interesting to other people. Also, I'm CCing hotplug UIO
developer, Tetsuya Mukawa, since there is no a specific maintainer for this
functionality.

I think the main problem for supporting VFIO hotplug is to deal with what
VFIO does, this is IOMMU mappings.
There is no code for doing so, and although the implementation would use
most of the code already there for UIO hotplug, which is basically related
to unmapping PCI resources, and the interrupt part will be likely quite
similar, the IOMMU unmapping requires more work.

I dare to say this is the main reason for not having VFIO hotplug support
right now. Maybe Tetsuya can confirm this or give us other reasons.


>
> Eelco
>
>
>
> On 07/02/17 10:29, Alejandro Lucero wrote:
>
> It seems none is working on this VFIO support.
>
> I will work on this if there is no reply to this thread saying the
> opposite the next days.
>
> On Thu, Feb 2, 2017 at 12:58 PM, Eelco Chaudron <echaudron@redhat.com>
> wrote:
>
>> On 02/02/17 13:05, Burakov, Anatoly wrote:
>>
>>> Hi Eelco,
>>>
>>> Please forgive me my ignorance on this matter, but doesn't it work at
>>> the moment? I would assume that if regular PCI hotplug works (with
>>> igb_uio), then so would hotplug with VFIO, as it basically utilizes the
>>> same PCI infrastructure igb_uio does. That said, I'm not aware of any
>>> patches submitted that had to do with VFIO and hotplug, so I guess the
>>> answer is, not at the moment.
>>>
>>> Thanks,
>>> Anatoly
>>>
>> I was asking as the documentation explicitly mentions its not supported.
>> http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug
>>
>> - "To detach a port, the port should be backed by a device that igb_uio
>> manages. VFIO is not supported."
>>
>> I could not find any specific reason why it's not supported, so if some
>> one can explain this it would help also...
>>
>> Cheers,
>>
>> Eelco
>>
>>
>>
>
>

Re: Hotplug support for VFIO

[Alejandro Lucero]

I just wanted to clarify the hotplug VFIO is not the problem, as I can see
it, but the unplug. When attaching a device the current VFIO code will be
used, but there is no code for doing the IOMMU unmapping when unplugging.

On Wed, Feb 8, 2017 at 3:43 PM, Alejandro Lucero <
alejandro.lucero@netronome.com> wrote:

> Hi Eelco,
>
> On Wed, Feb 8, 2017 at 9:41 AM, Eelco Chaudron <echaudron@redhat.com>
> wrote:
>
>> Hi Anatoly,
>>
>> This will be great... If you want I can do some testing on (early)
>> patches if required.
>>
>> Also do you know why it is currently not supported, i.e. what are the
>> limitations?
>>
>> Thanks,
>>
>
> I assume your reply was to my previous email and not to Anatoly's one. My
> apologies if this assumption is wrong.
> By the way, you did not reply to the list, but I'm doing to do so because
> this could be interesting to other people. Also, I'm CCing hotplug UIO
> developer, Tetsuya Mukawa, since there is no a specific maintainer for this
> functionality.
>
> I think the main problem for supporting VFIO hotplug is to deal with what
> VFIO does, this is IOMMU mappings.
> There is no code for doing so, and although the implementation would use
> most of the code already there for UIO hotplug, which is basically related
> to unmapping PCI resources, and the interrupt part will be likely quite
> similar, the IOMMU unmapping requires more work.
>
> I dare to say this is the main reason for not having VFIO hotplug support
> right now. Maybe Tetsuya can confirm this or give us other reasons.
>
>
>>
>> Eelco
>>
>>
>>
>> On 07/02/17 10:29, Alejandro Lucero wrote:
>>
>> It seems none is working on this VFIO support.
>>
>> I will work on this if there is no reply to this thread saying the
>> opposite the next days.
>>
>> On Thu, Feb 2, 2017 at 12:58 PM, Eelco Chaudron <echaudron@redhat.com>
>> wrote:
>>
>>> On 02/02/17 13:05, Burakov, Anatoly wrote:
>>>
>>>> Hi Eelco,
>>>>
>>>> Please forgive me my ignorance on this matter, but doesn't it work at
>>>> the moment? I would assume that if regular PCI hotplug works (with
>>>> igb_uio), then so would hotplug with VFIO, as it basically utilizes the
>>>> same PCI infrastructure igb_uio does. That said, I'm not aware of any
>>>> patches submitted that had to do with VFIO and hotplug, so I guess the
>>>> answer is, not at the moment.
>>>>
>>>> Thanks,
>>>> Anatoly
>>>>
>>> I was asking as the documentation explicitly mentions its not supported.
>>> http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework
>>> .html#hotplug
>>>
>>> - "To detach a port, the port should be backed by a device that igb_uio
>>> manages. VFIO is not supported."
>>>
>>> I could not find any specific reason why it's not supported, so if some
>>> one can explain this it would help also...
>>>
>>> Cheers,
>>>
>>> Eelco
>>>
>>>
>>>
>>
>>
>

Re: Hotplug support for VFIO

[Tetsuya Mukawa]

2017-02-09 0:48 GMT+09:00 Alejandro Lucero <alejandro.lucero@netronome.com>:
> I just wanted to clarify the hotplug VFIO is not the problem, as I can see
> it, but the unplug. When attaching a device the current VFIO code will be
> used, but there is no code for doing the IOMMU unmapping when unplugging.
>
> On Wed, Feb 8, 2017 at 3:43 PM, Alejandro Lucero
> <alejandro.lucero@netronome.com> wrote:
>>
>> Hi Eelco,
>>
>> On Wed, Feb 8, 2017 at 9:41 AM, Eelco Chaudron <echaudron@redhat.com>
>> wrote:
>>>
>>> Hi Anatoly,
>>>
>>> This will be great... If you want I can do some testing on (early)
>>> patches if required.
>>>
>>> Also do you know why it is currently not supported, i.e. what are the
>>> limitations?
>>>
>>> Thanks,
>>
>>
>> I assume your reply was to my previous email and not to Anatoly's one. My
>> apologies if this assumption is wrong.
>> By the way, you did not reply to the list, but I'm doing to do so because
>> this could be interesting to other people. Also, I'm CCing hotplug UIO
>> developer, Tetsuya Mukawa, since there is no a specific maintainer for this
>> functionality.
>>

Hi,

Sorry for late reply.
Honestly, I am not so familiar with VFIO, so I may be wrong.
But please check my below comments.

>> I think the main problem for supporting VFIO hotplug is to deal with what
>> VFIO does, this is IOMMU mappings.
>> There is no code for doing so, and although the implementation would use
>> most of the code already there for UIO hotplug, which is basically related
>> to unmapping PCI resources, and the interrupt part will be likely quite
>> similar, the IOMMU unmapping requires more work.
>>
>> I dare to say this is the main reason for not having VFIO hotplug support
>> right now. Maybe Tetsuya can confirm this or give us other reasons.

Yes, this is the main reason.
When I implement hot plugging feature, I also needed to implement UIO
detaching code.
And at the time, I also tried to implement VFIO code as well, but I
gave up with below reasons.
 - Patch will be bigger and more difficult to be merged in mainline code.
 - I don't have enough time to check actually what kind of code is
needed to detach it.

So above were not technical reasons.

One thing I am not clear is whether we may need to do something
special for multi process case.
For example, if primary process is died suddenly, what kind of error
will be happen to slave process, and what is good way to handle it.

Thanks,
Tetsuya

>>
>>>
>>>
>>> Eelco
>>>
>>>
>>>
>>> On 07/02/17 10:29, Alejandro Lucero wrote:
>>>
>>> It seems none is working on this VFIO support.
>>>
>>> I will work on this if there is no reply to this thread saying the
>>> opposite the next days.
>>>
>>> On Thu, Feb 2, 2017 at 12:58 PM, Eelco Chaudron <echaudron@redhat.com>
>>> wrote:
>>>>
>>>> On 02/02/17 13:05, Burakov, Anatoly wrote:
>>>>>
>>>>> Hi Eelco,
>>>>>
>>>>> Please forgive me my ignorance on this matter, but doesn't it work at
>>>>> the moment? I would assume that if regular PCI hotplug works (with igb_uio),
>>>>> then so would hotplug with VFIO, as it basically utilizes the same PCI
>>>>> infrastructure igb_uio does. That said, I'm not aware of any patches
>>>>> submitted that had to do with VFIO and hotplug, so I guess the answer is,
>>>>> not at the moment.
>>>>>
>>>>> Thanks,
>>>>> Anatoly
>>>>
>>>> I was asking as the documentation explicitly mentions its not supported.
>>>>
>>>> http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html#hotplug
>>>>
>>>> - "To detach a port, the port should be backed by a device that igb_uio
>>>> manages. VFIO is not supported."
>>>>
>>>> I could not find any specific reason why it's not supported, so if some
>>>> one can explain this it would help also...
>>>>
>>>> Cheers,
>>>>
>>>> Eelco
>>>>
>>>>
>>>
>>>
>>
>

Re: Hotplug support for VFIO

[Alejandro Lucero]

On Thu, Feb 9, 2017 at 3:16 AM, Tetsuya Mukawa <mtetsuyah@gmail.com> wrote:

> 2017-02-09 0:48 GMT+09:00 Alejandro Lucero <alejandro.lucero@netronome.com
> >:
> > I just wanted to clarify the hotplug VFIO is not the problem, as I can
> see
> > it, but the unplug. When attaching a device the current VFIO code will be
> > used, but there is no code for doing the IOMMU unmapping when unplugging.
> >
> > On Wed, Feb 8, 2017 at 3:43 PM, Alejandro Lucero
> > <alejandro.lucero@netronome.com> wrote:
> >>
> >> Hi Eelco,
> >>
> >> On Wed, Feb 8, 2017 at 9:41 AM, Eelco Chaudron <echaudron@redhat.com>
> >> wrote:
> >>>
> >>> Hi Anatoly,
> >>>
> >>> This will be great... If you want I can do some testing on (early)
> >>> patches if required.
> >>>
> >>> Also do you know why it is currently not supported, i.e. what are the
> >>> limitations?
> >>>
> >>> Thanks,
> >>
> >>
> >> I assume your reply was to my previous email and not to Anatoly's one.
> My
> >> apologies if this assumption is wrong.
> >> By the way, you did not reply to the list, but I'm doing to do so
> because
> >> this could be interesting to other people. Also, I'm CCing hotplug UIO
> >> developer, Tetsuya Mukawa, since there is no a specific maintainer for
> this
> >> functionality.
>
> Hi,
>
> Sorry for late reply.
> Honestly, I am not so familiar with VFIO, so I may be wrong.
> But please check my below comments.
>
> >>
> >> I think the main problem for supporting VFIO hotplug is to deal with
> what
> >> VFIO does, this is IOMMU mappings.
> >> There is no code for doing so, and although the implementation would use
> >> most of the code already there for UIO hotplug, which is basically
> related
> >> to unmapping PCI resources, and the interrupt part will be likely quite
> >> similar, the IOMMU unmapping requires more work.
> >>
> >> I dare to say this is the main reason for not having VFIO hotplug
> support
> >> right now. Maybe Tetsuya can confirm this or give us other reasons.
>
> Yes, this is the main reason.
> When I implement hot plugging, I also needed to implement UIO detaching
> code.
> And at the time, I also tried to implement VFIO code as well, but I
> gave up with below reasons.
>  - Patch will be bigger and more difficult to be merged in mainline code.
>  - I don't have enough time to check actually what kind of code is
> needed to detach it.
>
>
I'm quite familiar with the VFIO code so I know what it requires. At least
the main part. You know there are details not so trivial to see initially
until you start doing the work.


> So above were not technical reasons.
>
> One thing I am not clear is whether we may need to do something
> special for multi process case.
> For example, if primary process is died suddenly, what kind of error
> will be happen to slave process, and what is good way to handle it.
>
>
I do no think this is a problem specific to the hotplug mechanism, but
thanks for the heads up.

I will start working on this. Hopefully it could be part of the 17.05
release.

Thank you


> Thanks,
> Tetsuya
>
> >>
> >>>
> >>>
> >>> Eelco
> >>>
> >>>
> >>>
> >>> On 07/02/17 10:29, Alejandro Lucero wrote:
> >>>
> >>> It seems none is working on this VFIO support.
> >>>
> >>> I will work on this if there is no reply to this thread saying the
> >>> opposite the next days.
> >>>
> >>> On Thu, Feb 2, 2017 at 12:58 PM, Eelco Chaudron <echaudron@redhat.com>
> >>> wrote:
> >>>>
> >>>> On 02/02/17 13:05, Burakov, Anatoly wrote:
> >>>>>
> >>>>> Hi Eelco,
> >>>>>
> >>>>> Please forgive me my ignorance on this matter, but doesn't it work at
> >>>>> the moment? I would assume that if regular PCI hotplug works (with
> igb_uio),
> >>>>> then so would hotplug with VFIO, as it basically utilizes the same
> PCI
> >>>>> infrastructure igb_uio does. That said, I'm not aware of any patches
> >>>>> submitted that had to do with VFIO and hotplug, so I guess the
> answer is,
> >>>>> not at the moment.
> >>>>>
> >>>>> Thanks,
> >>>>> Anatoly
> >>>>
> >>>> I was asking as the documentation explicitly mentions its not
> supported.
> >>>>
> >>>> http://dpdk.org/doc/guides/prog_guide/port_hotplug_
> framework.html#hotplug
> >>>>
> >>>> - "To detach a port, the port should be backed by a device that
> igb_uio
> >>>> manages. VFIO is not supported."
> >>>>
> >>>> I could not find any specific reason why it's not supported, so if
> some
> >>>> one can explain this it would help also...
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Eelco
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
>