Send SFP event from kernel driver to user space (UDEV)

Send SFP event from kernel driver to user space (UDEV)

[Shay Drory]

Today, SFP inserted / removal event impacts only the kernel space drivers.
There are users who wishes to get SFP insert / removal in a udev-event
format for their application / daemons / monitors.
The naive way to implement this feature would be to create a sysfs file
that represents device SFP, to expose it under the netdev sysfs, and
to raise a udev event over it.
However, it is not reasonable to create a sysfs for each net-device.
In this letter, I would like to offer a new mechanism that will add a
support to send SFP events from the kernel driver to user space.
This suggestion is built upon a new netlink infrastructure for ethtool
currently being written by Michal Kubeckwhich called “ethtool-netlink”[1].
My suggestion is to do it by adding a function
(ethtool_sfp_insterted/removed(...)) to ethtool API, This function will
raise a netlink event to be caught in user space.
The design:

- SFP event from NIC caught by the driver
- Driver call ethtool_sfp_inserted/removed()
- Ethtool generated netlink event with relevant data
- This event-message will be handled in the user-space library of UDEV
(for this purpose we would like to add a netlink infrastructure to UDEV
user-space library).

the flow in scheme:

UDEV (in systemd)
                 ↑
ethtool_netlink (in ethtool)
                 ↑
driver (mlx5_core for example)
                 ↑
NIC (SFP event)

Would like to hear your opinion on this suggestion, or on alternative
designs.

Thanks
Shay Drory

[1]
https://patchwork.ozlabs.org/project/netdev/list/?series=&submitter=11892&state=*&q=&archive=&delegate=

Re: Send SFP event from kernel driver to user space (UDEV)

[Shay Drory]

+relevant people

On 11/17/2019 13:46, Shay Drory wrote:
> Today, SFP inserted / removal event impacts only the kernel space drivers.
> There are users who wishes to get SFP insert / removal in a udev-event
> format for their application / daemons / monitors.
> The naive way to implement this feature would be to create a sysfs file
> that represents device SFP, to expose it under the netdev sysfs, and
> to raise a udev event over it.
> However, it is not reasonable to create a sysfs for each net-device.
> In this letter, I would like to offer a new mechanism that will add a
> support to send SFP events from the kernel driver to user space.
> This suggestion is built upon a new netlink infrastructure for ethtool
> currently being written by Michal Kubeckwhich called “ethtool-netlink”[1].
> My suggestion is to do it by adding a function
> (ethtool_sfp_insterted/removed(...)) to ethtool API, This function will
> raise a netlink event to be caught in user space.
> The design:
>
> - SFP event from NIC caught by the driver
> - Driver call ethtool_sfp_inserted/removed()
> - Ethtool generated netlink event with relevant data
> - This event-message will be handled in the user-space library of UDEV
> (for this purpose we would like to add a netlink infrastructure to UDEV
> user-space library).
>
> the flow in scheme:
>
> UDEV (in systemd)
>                   ↑
> ethtool_netlink (in ethtool)
>                   ↑
> driver (mlx5_core for example)
>                   ↑
> NIC (SFP event)
>
> Would like to hear your opinion on this suggestion, or on alternative
> designs.
>
> Thanks
> Shay Drory
>
> [1]
> https://patchwork.ozlabs.org/project/netdev/list/?series=&submitter=11892&state=*&q=&archive=&delegate=
>

Re: Send SFP event from kernel driver to user space (UDEV)

[Andrew Lunn]

On Sun, Nov 17, 2019 at 11:46:15AM +0000, Shay Drory wrote:

Hi Shay

It would be good to Cc: the generic SFP code maintainers.

> Today, SFP inserted / removal event impacts only the kernel space drivers.
> There are users who wishes to get SFP insert / removal in a udev-event
> format for their application / daemons / monitors.
> The naive way to implement this feature would be to create a sysfs file
> that represents device SFP, to expose it under the netdev sysfs, and
> to raise a udev event over it.
> However, it is not reasonable to create a sysfs for each net-device.
> In this letter, I would like to offer a new mechanism that will add a
> support to send SFP events from the kernel driver to user space.
> This suggestion is built upon a new netlink infrastructure for ethtool
> currently being written by Michal Kubeckwhich called “ethtool-netlink”[1].

So you are in no rush to make use of this? ethtool-nl seems to be
making very slow progress.

> My suggestion is to do it by adding a function
> (ethtool_sfp_insterted/removed(...)) to ethtool API, This function will
> raise a netlink event to be caught in user space.

What about the case of the SFP is inserted before the SFP is
associated to a netdev? Similarly, the SFP is ejected when the SFP is
not connected to a MAC. You don't have a netdev, so you cannot send an
event. And SFF, which are never inserted or removed? SFPs have a
different life cycle to a netdev. Do you care about this?

> The design:
> 
> - SFP event from NIC caught by the driver
> - Driver call ethtool_sfp_inserted/removed()
> - Ethtool generated netlink event with relevant data
> - This event-message will be handled in the user-space library of UDEV
> (for this purpose we would like to add a netlink infrastructure to UDEV
> user-space library).

Would you add just SFP insert/eject to UDEV. Or all the events which
get sent via netlink? Link up/down, route add/remove, etc?

What sort of daemon is this anyway? Most networking daemons already
have the code to listen to netlink events. So why complicate things by
using UDEV?

Is UDEV name space aware? Do you run a udev daemon in each network
name space? I assume when you open a netlink socket, it is for just
the current network namespace?

    Andrew

Re: Send SFP event from kernel driver to user space (UDEV)

[Shay Drory]

On 11/18/2019 03:29, Andrew Lunn wrote:
> On Sun, Nov 17, 2019 at 11:46:15AM +0000, Shay Drory wrote:
>
> Hi Shay
>
> It would be good to Cc: the generic SFP code maintainers.

The suggested solution is not targeted for SFP drivers (see below),
but I added them to the Cc list.

>
>> Today, SFP inserted / removal event impacts only the kernel space drivers.
>> There are users who wishes to get SFP insert / removal in a udev-event
>> format for their application / daemons / monitors.
>> The naive way to implement this feature would be to create a sysfs file
>> that represents device SFP, to expose it under the netdev sysfs, and
>> to raise a udev event over it.
>> However, it is not reasonable to create a sysfs for each net-device.
>> In this letter, I would like to offer a new mechanism that will add a
>> support to send SFP events from the kernel driver to user space.
>> This suggestion is built upon a new netlink infrastructure for ethtool
>> currently being written by Michal Kubeckwhich called “ethtool-netlink”[1].
> So you are in no rush to make use of this? ethtool-nl seems to be
> making very slow progress.

I am looking for the correct solution that we can push to kernel open source.
The ethtool-nl seems like a good path. If you have another suggestion,
base on existing code, I will be glad to hear.

>
>> My suggestion is to do it by adding a function
>> (ethtool_sfp_insterted/removed(...)) to ethtool API, This function will
>> raise a netlink event to be caught in user space.
> What about the case of the SFP is inserted before the SFP is
> associated to a netdev? Similarly, the SFP is ejected when the SFP is
> not connected to a MAC. You don't have a netdev, so you cannot send an
> event. And SFF, which are never inserted or removed? SFPs have a
> different life cycle to a netdev. Do you care about this?

The feature is targeted to netdev users. It is expected from the user to query current state via
ethtool -m and afterwards monitor for changes over UDEV.

>
>> The design:
>>
>> - SFP event from NIC caught by the driver
>> - Driver call ethtool_sfp_inserted/removed()
>> - Ethtool generated netlink event with relevant data
>> - This event-message will be handled in the user-space library of UDEV
>> (for this purpose we would like to add a netlink infrastructure to UDEV
>> user-space library).
> Would you add just SFP insert/eject to UDEV. Or all the events which
> get sent via netlink? Link up/down, route add/remove, etc?

It makes sense to notify all events. What do you think?

>
> What sort of daemon is this anyway? Most networking daemons already
> have the code to listen to netlink events. So why complicate things by
> using UDEV?

That is a good point, we will discuss it internally.

>
> Is UDEV name space aware? Do you run a udev daemon in each network
> name space? I assume when you open a netlink socket, it is for just
> the current network namespace?

UDEV will follow netlink name-space.

>
>      Andrew

Thanks for your comments, Shay.

Re: Send SFP event from kernel driver to user space (UDEV)

[Andrew Lunn]

On Mon, Nov 18, 2019 at 11:54:31AM +0000, Shay Drory wrote:
> On 11/18/2019 03:29, Andrew Lunn wrote:
> > On Sun, Nov 17, 2019 at 11:46:15AM +0000, Shay Drory wrote:
> >
> > Hi Shay
> >
> > It would be good to Cc: the generic SFP code maintainers.
> 
> The suggested solution is not targeted for SFP drivers (see below),
> but I added them to the Cc list.

Hi Shay

So you are proposing something which won't work for the generic SFP
code?  That should be an automatic NACK. Whatever you propose needs to
be generic so that it can work for any NICs that do firmware support
for SFPs, and those who let Linux handle the SFP.

> The feature is targeted to netdev users. It is expected from the
> user to query current state via ethtool -m and afterwards monitor
> for changes over UDEV.

What exactly are you interested in? What are your use cases. When
hwmon devices are created, you should get udev events. Maybe that is
sufficient? Or are you interested in some other parts of the SFP than
the diagnostic sensors?

> > Would you add just SFP insert/eject to UDEV. Or all the events which
> > get sent via netlink? Link up/down, route add/remove, etc?
> 
> It makes sense to notify all events. What do you think?

I don't particularly like two ways to get the same information. It
means we have two APIs we need to maintain forever, when just one
should be sufficient.

> > Is UDEV name space aware? Do you run a udev daemon in each network
> > name space? I assume when you open a netlink socket, it is for just
> > the current network namespace?
> 
> UDEV will follow netlink name-space.

So you do plan to have a udev daemon running in every network name
space. Does udev even support that?

I'm sceptical using udev is a good idea. But having netlink events
does sounds reasonable. And if you are willing to wait a while,
ethtool-nl does seem like a good fit. But please make sure your
solution is generic.

	 Andrew

Re: Send SFP event from kernel driver to user space (UDEV)

[Shay Drory]

On 11/19/2019 01:19, Andrew Lunn wrote:
> On Mon, Nov 18, 2019 at 11:54:31AM +0000, Shay Drory wrote:
>> On 11/18/2019 03:29, Andrew Lunn wrote:
>>> On Sun, Nov 17, 2019 at 11:46:15AM +0000, Shay Drory wrote:
>>>
>>> Hi Shay
>>>
>>> It would be good to Cc: the generic SFP code maintainers.
>> The suggested solution is not targeted for SFP drivers (see below),
>> but I added them to the Cc list.
> Hi Shay
>
> So you are proposing something which won't work for the generic SFP
> code?  That should be an automatic NACK. Whatever you propose needs to
> be generic so that it can work for any NICs that do firmware support
> for SFPs, and those who let Linux handle the SFP.

The suggestion is targeted to support all types of NICs
that do firmware support for SFPs. But I want to do it via the ethtool-nl
interface and not by using phy drivers.

>
>> The feature is targeted to netdev users. It is expected from the
>> user to query current state via ethtool -m and afterwards monitor
>> for changes over UDEV.
> What exactly are you interested in? What are your use cases. When
> hwmon devices are created, you should get udev events. Maybe that is
> sufficient? Or are you interested in some other parts of the SFP than
> the diagnostic sensors?

It looks like the hwmon is not sufficient for out purpose. I am interesting
in getting events when the SFP is inserted or removed.

>
>>> Would you add just SFP insert/eject to UDEV. Or all the events which
>>> get sent via netlink? Link up/down, route add/remove, etc?
>> It makes sense to notify all events. What do you think?
> I don't particularly like two ways to get the same information. It
> means we have two APIs we need to maintain forever, when just one
> should be sufficient.
>
>>> Is UDEV name space aware? Do you run a udev daemon in each network
>>> name space? I assume when you open a netlink socket, it is for just
>>> the current network namespace?
>> UDEV will follow netlink name-space.
> So you do plan to have a udev daemon running in every network name
> space. Does udev even support that?
>
> I'm sceptical using udev is a good idea. But having netlink events
> does sounds reasonable. And if you are willing to wait a while,
> ethtool-nl does seem like a good fit. But please make sure your
> solution is generic.
>
> 	 Andrew

Thanks for your comments. The using of Udev is under internal discussion.