How to handle Hotplug with UIO userspace driver

How to handle Hotplug with UIO userspace driver

[Mandeep Sandhu]

Hi All,

We're working on a custom PCIe based hardware, which can be hotplugged into
the system.

We're using the UIO approach of having a minimal kernel driver which
basically handles the interrupt and provides a mmap of the device memory to
userspace and userspace driver which mmap's the provided memory and manages
the device.

We're using systemd and udev rules to spawn off a user process whenever the
hardware is plugged into the system.

Now my question is, how can we gracefully inform the userspace process if
someone pulls out the hardware (sitting on a PCI slot). Even with full
hotplug support, there can always be a case of "surprise removal", where
someone yanks the board without going through the whole "attention button"
routine.

While the kernel driver has no problem with hot-removal, I see that if I
"unregister" our UIO driver in response to the removal, then the mapping
for the hardware's physical memory would become "invalid" (done by UIO on
unregsiter). This could possibly cause the user-space process to crash due
to illegal memory access.

Is there a "standard" way to handle this scenario, i.e for hotpluggable
hardware using UIO?

Basically, we'd want to "unregister" the UIO device only after the
userspace process is informed of the removal so that it could unmap/close
the device and exit gracefully.

We could try doing it using some systemd trick, where we put a dependency
on one of the UIO created files (/dev/uioX), so that when the file is
destroyed, systemd could kill/shutdown the process. Although, there's no
guarantee that the userspace process won't try to access the mmaped space
during that interval, so this isn't a reliable solution either.

Any hints, thoughts?

Thanks.,
-mandeep
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20141126/6b6775ca/attachment.html 

How to handle Hotplug with UIO userspace driver

[Greg KH]

On Wed, Nov 26, 2014 at 11:27:07AM -0800, Mandeep Sandhu wrote:
> Hi All,
> 
> We're working on a custom PCIe based hardware, which can be hotplugged into the
> system.
> 
> We're using the UIO approach of having a minimal kernel driver which basically
> handles the interrupt and provides a mmap of the device memory to userspace and
> userspace driver which mmap's the provided memory and manages the device.
> 
> We're using systemd and udev rules to spawn off a user process whenever the
> hardware is plugged into the system.
> 
> Now my question is, how can we gracefully inform the userspace process if
> someone pulls out the hardware (sitting on a PCI slot). Even with full hotplug
> support, there can always be a case of "surprise removal", where someone yanks
> the board without going through the whole "attention button" routine.
> 
> While the kernel driver has no problem with hot-removal, I see that if I
> "unregister" our UIO driver in response to the removal, then the mapping for
> the hardware's physical memory would become "invalid" (done by UIO on
> unregsiter). This could possibly cause the user-space process to crash due to
> illegal memory access.

You should just get 0xff on the memory reads, right?  You can catch the
signal for invalid memory accesses.

> Is there a "standard" way to handle this scenario, i.e for hotpluggable
> hardware using UIO?

Your kernel driver knows when the device is removed, so have it tell
userspace this.  Or, just tie into libudev and have your userspace
program be notified when the device goes away.

Do you have a pointer to the source of your uio kernel driver anywhere?

thanks,

greg k-h

How to handle Hotplug with UIO userspace driver

[Mandeep Sandhu]

>
> > While the kernel driver has no problem with hot-removal, I see that if I
> > "unregister" our UIO driver in response to the removal, then the mapping
> for
> > the hardware's physical memory would become "invalid" (done by UIO on
> > unregsiter). This could possibly cause the user-space process to crash
> due to
> > illegal memory access.
>
> You should just get 0xff on the memory reads, right?  You can catch the
> signal for invalid memory accesses.
>

Wouldn't I get an illegal memory access, as UIO would've removed the
mappings that it setup when I registered the UIO driver? I saw the
userspace process segfault after removal.


>
> > Is there a "standard" way to handle this scenario, i.e for hotpluggable
> > hardware using UIO?
>
> Your kernel driver knows when the device is removed, so have it tell
> userspace this.  Or, just tie into libudev and have your userspace
> program be notified when the device goes away.
>

Right. This is what I'm trying to do using a udev "remove" rule. Although
the question still remains, what will happen if the device is being
actively accessed _before_ the signal reaches the process. Is it safe to do
"uio_unregister_device" while a userspace process is accessing the device?

I saw a patch for this very situation (UIO & hotplug) being discussed on
LKML almost 4 yrs back, although I don't see it in my kernel version -
3.16.0 (Ubuntu 3.16.0-24-generic).
The LKML link:

https://lkml.org/lkml/2010/9/20/21

Wouldn't this solve the issue? I wonder why it didn't make it into mainline?


>
> Do you have a pointer to the source of your uio kernel driver anywhere?
>

Not yet, as this is still under active development (we don't have the h/w
to test yet :/ ). Maybe once things stabilize a little, we'll probably
open-source it along with the h/w too! :)

Thanks,
-mandeep



>
> thanks,
>
> greg k-h
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20141126/8e70a5e2/attachment.html 

How to handle Hotplug with UIO userspace driver

[Greg KH]

On Wed, Nov 26, 2014 at 12:33:41PM -0800, Mandeep Sandhu wrote:
>     > While the kernel driver has no problem with hot-removal, I see that if I
>     > "unregister" our UIO driver in response to the removal, then the mapping
>     for
>     > the hardware's physical memory would become "invalid" (done by UIO on
>     > unregsiter). This could possibly cause the user-space process to crash
>     due to
>     > illegal memory access.
> 
>     You should just get 0xff on the memory reads, right?? You can catch the
>     signal for invalid memory accesses.
> 
> 
> Wouldn't I get an illegal memory access, as UIO would've removed the mappings
> that it setup when I registered the UIO driver? I saw the userspace process
> segfault after removal.

I don't know, never tried it.  What about catching the signal for this?

>     > Is there a "standard" way to handle this scenario, i.e for hotpluggable
>     > hardware using UIO?
> 
>     Your kernel driver knows when the device is removed, so have it tell
>     userspace this.? Or, just tie into libudev and have your userspace
>     program be notified when the device goes away.
> 
> 
> Right. This is what I'm trying to do using a udev "remove" rule.

No, just have libudev tell you about the problem, don't create a whole
new rule for this, that's overkill and messy and not dynamic at all.

> Although the
> question still remains, what will happen if the device is being actively
> accessed _before_ the signal reaches the process. Is it safe to do
> "uio_unregister_device" while a userspace process is accessing the device?

I don't know, try it and see :)

> I saw a patch for this very situation (UIO & hotplug) being discussed on LKML
> almost 4 yrs back, although I don't see it in my kernel version - 3.16.0
> (Ubuntu?3.16.0-24-generic).
> The LKML link:
> 
> https://lkml.org/lkml/2010/9/20/21
> 
> Wouldn't this solve the issue? I wonder why it didn't make it into mainline?

No idea, that UIO maintainer must be really lazy and never apply patches :)

Try that series on your kernel and see what happens.  If it works,
please resend that patch series.

thanks,

greg k-h

How to handle Hotplug with UIO userspace driver

[Mandeep Sandhu]

>
>
> Try that series on your kernel and see what happens.  If it works,
> please resend that patch series.
>
>
Sure. Will do.

Thanks
-mandeep


> thanks,
>
> greg k-h
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20141126/fae06df7/attachment.html 

How to handle Hotplug with UIO userspace driver

[Mandeep Sandhu]

Hi Greg,

>> I saw a patch for this very situation (UIO & hotplug) being discussed on LKML
>> almost 4 yrs back, although I don't see it in my kernel version - 3.16.0
>> (Ubuntu 3.16.0-24-generic).
>> The LKML link:
>>
>> https://lkml.org/lkml/2010/9/20/21
>>
>> Wouldn't this solve the issue? I wonder why it didn't make it into mainline?
>
> No idea, that UIO maintainer must be really lazy and never apply patches :)
>
> Try that series on your kernel and see what happens.  If it works,
> please resend that patch series.

I tried out a dummy uio driver with a userspace program accessing the
mmaped space (attached). On doing an unregister of the uio device, I
see a crash/bug being triggered. Sometimes the kernel continues to
run, sometimes failing during rmmod of my module, sometimes locks up
during reboot (trying this on VirtualBox VM) and other times there's
no crash at all (but I'm just getting lucky).

Although with the patch applied, the user process gets a SIGBUS and
exits when I do an uio_unregister_device() call in my driver. So I
guess this patch is needed.


A generic question, if I have to apply the patches and do a sanity
build, which branch should I be applying the patch to? linux-next?
linux-stable?

Thanks,
-mandeep


>
> thanks,
>
> greg k-h

How to handle Hotplug with UIO userspace driver

[Mandeep Sandhu]

Sorry, sent it too soon.

Here's the attached driver and userspace app for simulating a crash.
You can run make to build both.

Thanks,
-mandeep


On Mon, Dec 8, 2014 at 3:24 PM, Mandeep Sandhu
<mandeepsandhu.chd@gmail.com> wrote:
> Hi Greg,
>
>>> I saw a patch for this very situation (UIO & hotplug) being discussed on LKML
>>> almost 4 yrs back, although I don't see it in my kernel version - 3.16.0
>>> (Ubuntu 3.16.0-24-generic).
>>> The LKML link:
>>>
>>> https://lkml.org/lkml/2010/9/20/21
>>>
>>> Wouldn't this solve the issue? I wonder why it didn't make it into mainline?
>>
>> No idea, that UIO maintainer must be really lazy and never apply patches :)
>>
>> Try that series on your kernel and see what happens.  If it works,
>> please resend that patch series.
>
> I tried out a dummy uio driver with a userspace program accessing the
> mmaped space (attached). On doing an unregister of the uio device, I
> see a crash/bug being triggered. Sometimes the kernel continues to
> run, sometimes failing during rmmod of my module, sometimes locks up
> during reboot (trying this on VirtualBox VM) and other times there's
> no crash at all (but I'm just getting lucky).
>
> Although with the patch applied, the user process gets a SIGBUS and
> exits when I do an uio_unregister_device() call in my driver. So I
> guess this patch is needed.
>
>
> A generic question, if I have to apply the patches and do a sanity
> build, which branch should I be applying the patch to? linux-next?
> linux-stable?
>
> Thanks,
> -mandeep
>
>
>>
>> thanks,
>>
>> greg k-h
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Makefile
Type: application/octet-stream
Size: 281 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20141208/43b8df28/attachment-0001.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: uio_user.c
Type: text/x-csrc
Size: 1244 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20141208/43b8df28/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: uio_dummy.c
Type: text/x-csrc
Size: 3529 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20141208/43b8df28/attachment-0003.bin 

How to handle Hotplug with UIO userspace driver

[Divakar]

Hi Mandeep,
Recently i encountered similar issue with UIO framework in hotplug
scenario. I reviewed the earlier patch from 2010 and your 4 patch
series. 4 patch series seems to work. I checked the latest kernel
source and i dont think your patches have been merged. Hopefully
i am not looking at the wrong place. If they are not merged is there
a plan to merge them ? I do think these patches are required.

I also found a bug in your patch set which is discussed here
https://lkml.org/lkml/2017/6/14/30. I was able to workaround/fix it.
I can upload a patch if needed.

Please advice if you plan to get the patchset merged.

Thanks
Divakar

How to handle Hotplug with UIO userspace driver

[Mandeep Sandhu]

On Thu, Sep 28, 2017 at 11:51 AM, Divakar <divakar.chitturi@hpe.com> wrote:

> Hi Mandeep,
> Recently i encountered similar issue with UIO framework in hotplug
> scenario. I reviewed the earlier patch from 2010 and your 4 patch
> series. 4 patch series seems to work. I checked the latest kernel
> source and i dont think your patches have been merged. Hopefully
> i am not looking at the wrong place. If they are not merged is there
> a plan to merge them ? I do think these patches are required.
>
>
Thats correct. The patches have not been merged. I no longer have the
hardware available and have moved away from the project that needed it,
therefore I was unable to carry with the patches (they were still under
review). It'd be great if you can resubmit those. Make sure they apply
cleanly to the latest staging branch (?). I think Greg had asked some
question regarding during the review, it would be good if you can pick it
up from there. GKH (who's on this list too) can weigh in more on this.

Good luck.
-mandeep





> I also found a bug in your patch set which is discussed here
> https://lkml.org/lkml/2017/6/14/30. I was able to workaround/fix it.
> I can upload a patch if needed.
>
> Please advice if you plan to get the patchset merged.
>
> Thanks
> Divakar
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170928/e32cbb6a/attachment.html