Bridge port userspace events broken?

Bridge port userspace events broken?

[Richard Weinberger]

Hi!

My userspace needs /sys/class/net/eth0/brport/group_fwd_mask, so I set
up udev rules
to wait for the sysfs file.
Without luck.
Also "udevadm monitor" does not show any event related to
/sys/class/net/eth0/brport when I assign eth0 to a bridge.

First I thought that the bridge code just misses to emit some events but
br_add_if() calls kobject_uevent() which is good.

Greg gave me the hint that the bridge code might not use the kobject model
correctly.

Enabling kobjekt debugging shows that all events are dropped:
[   36.904602] device eth0 entered promiscuous mode
[   36.904786] kobject: 'brport' (0000000028a47e33): kobject_uevent_env
[   36.904789] kobject: 'brport' (0000000028a47e33):
kobject_uevent_env: filter function caused the event to drop!

If I understood Greg correctly this is because the bridge code uses
plain kobjects which
have a parent object. Therefore all events are dropped.

Shouldn't brport be a kset just like net_device->queues_kset?

-- 
Thanks,
//richard

Re: Bridge port userspace events broken?

[nikolay]

On 14 October 2019 22:33:22 CEST, Richard Weinberger <richard.weinberger@gmail.com> wrote:
>Hi!
>
>My userspace needs /sys/class/net/eth0/brport/group_fwd_mask, so I set
>up udev rules
>to wait for the sysfs file.
>Without luck.
>Also "udevadm monitor" does not show any event related to
>/sys/class/net/eth0/brport when I assign eth0 to a bridge.
>
>First I thought that the bridge code just misses to emit some events
>but
>br_add_if() calls kobject_uevent() which is good.
>
>Greg gave me the hint that the bridge code might not use the kobject
>model
>correctly.
>
>Enabling kobjekt debugging shows that all events are dropped:
>[   36.904602] device eth0 entered promiscuous mode
>[   36.904786] kobject: 'brport' (0000000028a47e33): kobject_uevent_env
>[   36.904789] kobject: 'brport' (0000000028a47e33):
>kobject_uevent_env: filter function caused the event to drop!
>
>If I understood Greg correctly this is because the bridge code uses
>plain kobjects which
>have a parent object. Therefore all events are dropped.
>
>Shouldn't brport be a kset just like net_device->queues_kset?


Hi Richard, 
I'm currently traveling and will be out of reach until mid-next week when I'll be
able to take a closer look, but one thing which comes to mind is that on
any bridge/port option change there should also be a netlink notification which
you could use. I'll look into this and will probably cook a fix, if anyone hasn't
beaten me to it by then. :) 

Cheers,
  Nik

Re: Bridge port userspace events broken?

[nikolay]

On 15 October 2019 12:48:58 CEST, nikolay@cumulusnetworks.com wrote:
>On 14 October 2019 22:33:22 CEST, Richard Weinberger
><richard.weinberger@gmail.com> wrote:
>>Hi!
>>
>>My userspace needs /sys/class/net/eth0/brport/group_fwd_mask, so I set
>>up udev rules
>>to wait for the sysfs file.
>>Without luck.
>>Also "udevadm monitor" does not show any event related to
>>/sys/class/net/eth0/brport when I assign eth0 to a bridge.
>>
>>First I thought that the bridge code just misses to emit some events
>>but
>>br_add_if() calls kobject_uevent() which is good.
>>
>>Greg gave me the hint that the bridge code might not use the kobject
>>model
>>correctly.
>>
>>Enabling kobjekt debugging shows that all events are dropped:
>>[   36.904602] device eth0 entered promiscuous mode
>>[   36.904786] kobject: 'brport' (0000000028a47e33):
>kobject_uevent_env
>>[   36.904789] kobject: 'brport' (0000000028a47e33):
>>kobject_uevent_env: filter function caused the event to drop!
>>
>>If I understood Greg correctly this is because the bridge code uses
>>plain kobjects which
>>have a parent object. Therefore all events are dropped.
>>
>>Shouldn't brport be a kset just like net_device->queues_kset?
>
>
>Hi Richard, 
>I'm currently traveling and will be out of reach until mid-next week
>when I'll be
>able to take a closer look, but one thing which comes to mind is that
>on
>any bridge/port option change there should also be a netlink
>notification which
>you could use. I'll look into this and will probably cook a fix, if
>anyone hasn't
>beaten me to it by then. :) 
>
>Cheers,
>  Nik

I meant the notifications could be used to configure the port mask once the
netdev is enslaved as well as for monitoring changes to them. 
Generally we prefer using netlink for configuration changes, some
of the bridge options are only accessible via netlink (e. g. vlan config). 

Re: Bridge port userspace events broken?

[Roopa Prabhu]

On Tue, Oct 15, 2019 at 3:53 AM <nikolay@cumulusnetworks.com> wrote:
>
> On 15 October 2019 12:48:58 CEST, nikolay@cumulusnetworks.com wrote:
> >On 14 October 2019 22:33:22 CEST, Richard Weinberger
> ><richard.weinberger@gmail.com> wrote:
> >>Hi!
> >>
> >>My userspace needs /sys/class/net/eth0/brport/group_fwd_mask, so I set
> >>up udev rules
> >>to wait for the sysfs file.
> >>Without luck.
> >>Also "udevadm monitor" does not show any event related to
> >>/sys/class/net/eth0/brport when I assign eth0 to a bridge.
> >>
> >>First I thought that the bridge code just misses to emit some events
> >>but
> >>br_add_if() calls kobject_uevent() which is good.
> >>
> >>Greg gave me the hint that the bridge code might not use the kobject
> >>model
> >>correctly.
> >>
> >>Enabling kobjekt debugging shows that all events are dropped:
> >>[   36.904602] device eth0 entered promiscuous mode
> >>[   36.904786] kobject: 'brport' (0000000028a47e33):
> >kobject_uevent_env
> >>[   36.904789] kobject: 'brport' (0000000028a47e33):
> >>kobject_uevent_env: filter function caused the event to drop!
> >>
> >>If I understood Greg correctly this is because the bridge code uses
> >>plain kobjects which
> >>have a parent object. Therefore all events are dropped.
> >>
> >>Shouldn't brport be a kset just like net_device->queues_kset?
> >
> >
> >Hi Richard,
> >I'm currently traveling and will be out of reach until mid-next week
> >when I'll be
> >able to take a closer look, but one thing which comes to mind is that
> >on
> >any bridge/port option change there should also be a netlink
> >notification which
> >you could use. I'll look into this and will probably cook a fix, if
> >anyone hasn't
> >beaten me to it by then. :)
> >
> >Cheers,
> >  Nik
>
> I meant the notifications could be used to configure the port mask once the
> netdev is enslaved as well as for monitoring changes to them.
> Generally we prefer using netlink for configuration changes, some
> of the bridge options are only accessible via netlink (e. g. vlan config).
>
>

+1,  this can be fixed....but in general all new bridge and link
attributes have better support with netlink.
In this case its IFLA_BRPORT_GROUP_FWD_MASK link attribute available
via ip monitor or bridge monitor.
 you probably cannot use it with udev today.

For the future, I think having udev listen to netlink link and devlink
events would make sense (Not sure if anybody is working on it).
AFAIK the sysfs uevent mechanism for link attributes don't  receive
the required attention and testing like the equivalent netlink events.

Re: Bridge port userspace events broken?

[Richard Weinberger]

Roopa, Nikolay,

----- Ursprüngliche Mail -----
> +1,  this can be fixed....but in general all new bridge and link
> attributes have better support with netlink.
> In this case its IFLA_BRPORT_GROUP_FWD_MASK link attribute available
> via ip monitor or bridge monitor.
> you probably cannot use it with udev today.
> 
> For the future, I think having udev listen to netlink link and devlink
> events would make sense (Not sure if anybody is working on it).
> AFAIK the sysfs uevent mechanism for link attributes don't  receive
> the required attention and testing like the equivalent netlink events.

I understand that netlink works best for you but sysfs notifications are still
useful.
Please let me explain my use case a little bit more.

The application I work on operates on network interfaces, in this case the
interface happens to be a bridge.
systemd-networkd sets up the bridge as soon all slave interfaces emerge.

Therefore the systemd service file of the application depends on the bridge.
i.e.
Requires=sys-subsystem-net-devices-br0.device

In one specific setup the bridge needs to forward more than usual and 
group_fwd_mask needs to be altered. Sadly this is nothing systemd-networkd
can do right now, so I added the following line to the service file of
the application:
ExecStartPre=/bin/bash -c "echo 0xfffd > /sys/class/net/eth0/brport/group_fwd_mask"

Here comes the problem, the unit is activated as soon br0 is created but
at this time eth0 is sometimes not yet a slave or br0. It takes some time.

So I need a way to model this dependency in a systemd environment.
A common approach to do so is setting up an udev rule which set a systemd notify
as soon a specific sysfs file arrives.

Teaching the application to listen for bridge specific netlink messages is
another possible approach but seems overkill to me.
Or maybe there is some nice wrapper/helper?

It would be nice to have sysfs notifications for bridge devices too.
I can understand that not everyone likes this approach but this is the way
how *many* systems out there work these day. Actually almost any (embedded)
system with systemd.

Thanks,
//richard

Re: Bridge port userspace events broken?

[Roopa Prabhu]

On Wed, Oct 16, 2019 at 2:16 PM Richard Weinberger <richard@nod.at> wrote:
>
> Roopa, Nikolay,
>
> ----- Ursprüngliche Mail -----
> > +1,  this can be fixed....but in general all new bridge and link
> > attributes have better support with netlink.
> > In this case its IFLA_BRPORT_GROUP_FWD_MASK link attribute available
> > via ip monitor or bridge monitor.
> > you probably cannot use it with udev today.
> >
> > For the future, I think having udev listen to netlink link and devlink
> > events would make sense (Not sure if anybody is working on it).
> > AFAIK the sysfs uevent mechanism for link attributes don't  receive
> > the required attention and testing like the equivalent netlink events.
>
> I understand that netlink works best for you but sysfs notifications are still
> useful.
> Please let me explain my use case a little bit more.
>
> The application I work on operates on network interfaces, in this case the
> interface happens to be a bridge.
> systemd-networkd sets up the bridge as soon all slave interfaces emerge.
>
> Therefore the systemd service file of the application depends on the bridge.
> i.e.
> Requires=sys-subsystem-net-devices-br0.device
>

interesting. We do have a lot of applications that depend on network
interfaces and we
 simply make them depend on the networking service eg with
"After=networking.service".

we don't use systemd-networkd today..but our network interface
management software (https://packages.debian.org/buster/ifupdown2)
registers itself as  networking.service. For unsupported options the
network interface manager provides hooks to invoke pre and post
commands
which takes care of the unsupported attributes case you mention above.
In absence of that, we would probably use systemd ExecStartPost


> In one specific setup the bridge needs to forward more than usual and
> group_fwd_mask needs to be altered. Sadly this is nothing systemd-networkd
> can do right now, so I added the following line to the service file of
> the application:
> ExecStartPre=/bin/bash -c "echo 0xfffd > /sys/class/net/eth0/brport/group_fwd_mask"

ok, yeah the right thing here will be for your network manager
(systemd-networkd)  to support this config attribute.

But understand that there will always be an attribute that your
network interface manager does not support :)

>
> Here comes the problem, the unit is activated as soon br0 is created but
> at this time eth0 is sometimes not yet a slave or br0. It takes some time.
>
> So I need a way to model this dependency in a systemd environment.
> A common approach to do so is setting up an udev rule which set a systemd notify
> as soon a specific sysfs file arrives.

That is cleaver. But, you now have systemd-networkd and udevd to get
your interfaces configured right.
You may be able to get past this for now to find more problems later.

>
> Teaching the application to listen for bridge specific netlink messages is
> another possible approach but seems overkill to me.

understood. In your case, the ideal thing to do is have all your
configuration done via systemd-networkd.
that will also make sure your config persists in a single place.
Agreed that having your application understand netlink to just deal
with this attribute is overkill.

> Or maybe there is some nice wrapper/helper?

There are many netlink libraries from libnl, python-nlmanager, pyroute2 etc

>
> It would be nice to have sysfs notifications for bridge devices too.
> I can understand that not everyone likes this approach but this is the way
> how *many* systems out there work these day. Actually almost any (embedded)
> system with systemd.
>

I think if you are using systemd, systemd-networkd which uses netlink
is the best
option to deal with interface link events.

Regardless, for existing bridge sysfs attribute files, Nikolay might
have a solution after he is back from vacation.
This should help your  immediate problem with group_fwd_mask.