From: Nikolay Aleksandrov <nikolay@nvidia.com>
To: Ido Schimmel <idosch@idosch.org>, Vladimir Oltean <olteanv@gmail.com>
Cc: Vladimir Oltean <vladimir.oltean@nxp.com>,
netdev@vger.kernel.org, Jakub Kicinski <kuba@kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Roopa Prabhu <roopa@nvidia.com>, Andrew Lunn <andrew@lunn.ch>,
Florian Fainelli <f.fainelli@gmail.com>,
Vivien Didelot <vivien.didelot@gmail.com>,
Vadym Kochan <vkochan@marvell.com>,
Taras Chornyi <tchornyi@marvell.com>,
Jiri Pirko <jiri@nvidia.com>, Ido Schimmel <idosch@nvidia.com>,
UNGLinuxDriver@microchip.com,
Grygorii Strashko <grygorii.strashko@ti.com>,
Marek Behun <kabel@blackhole.sk>,
DENG Qingfang <dqfext@gmail.com>,
Kurt Kanzenbach <kurt@linutronix.de>,
Hauke Mehrtens <hauke@hauke-m.de>,
Woojung Huh <woojung.huh@microchip.com>,
Sean Wang <sean.wang@mediatek.com>,
Landen Chao <Landen.Chao@mediatek.com>,
Claudiu Manoil <claudiu.manoil@nxp.com>,
Alexandre Belloni <alexandre.belloni@bootlin.com>,
George McCollister <george.mccollister@gmail.com>,
Ioana Ciornei <ioana.ciornei@nxp.com>,
Saeed Mahameed <saeedm@nvidia.com>,
Leon Romanovsky <leon@kernel.org>,
Lars Povlsen <lars.povlsen@microchip.com>,
Steen Hegelund <Steen.Hegelund@microchip.com>,
Julian Wiedmann <jwi@linux.ibm.com>,
Karsten Graul <kgraul@linux.ibm.com>,
Heiko Carstens <hca@linux.ibm.com>,
Vasily Gorbik <gor@linux.ibm.com>,
Christian Borntraeger <borntraeger@de.ibm.com>,
Ivan Vecera <ivecera@redhat.com>, Vlad Buslov <vladbu@nvidia.com>,
Jianbo Liu <jianbol@nvidia.com>, Mark Bloch <mbloch@nvidia.com>,
Roi Dayan <roid@nvidia.com>,
Tobias Waldekranz <tobias@waldekranz.com>,
Vignesh Raghavendra <vigneshr@ti.com>,
Jesse Brandeburg <jesse.brandeburg@intel.com>
Subject: Re: [PATCH v2 net-next 0/5] Make SWITCHDEV_FDB_{ADD,DEL}_TO_DEVICE blocking
Date: Mon, 23 Aug 2021 18:42:20 +0300 [thread overview]
Message-ID: <5d3190a8-b0f2-f695-564f-318f1d1e4a0c@nvidia.com> (raw)
In-Reply-To: <YSO8MK5Alv0yF9pr@shredder>
On 23/08/2021 18:18, Ido Schimmel wrote:
> On Mon, Aug 23, 2021 at 05:29:53PM +0300, Vladimir Oltean wrote:
>> On Mon, Aug 23, 2021 at 03:16:48PM +0300, Ido Schimmel wrote:
>>> I was thinking about the following case:
>>>
>>> t0 - <MAC1,VID1,P1> is added in syscall context under 'hash_lock'
>>> t1 - br_fdb_delete_by_port() flushes entries under 'hash_lock' in
>>> response to STP state. Notifications are added to 'deferred' list
>>> t2 - switchdev_deferred_process() is called in syscall context
>>> t3 - <MAC1,VID1,P1> is notified as blocking
>>>
>>> Updates to the SW FDB are protected by 'hash_lock', but updates to the
>>> HW FDB are not. In this case, <MAC1,VID1,P1> does not exist in SW, but
>>> it will exist in HW.
>>>
>>> Another case assuming switchdev_deferred_process() is called first:
>>>
>>> t0 - switchdev_deferred_process() is called in syscall context
>>> t1 - <MAC1,VID,P1> is learned under 'hash_lock'. Notification is added
>>> to 'deferred' list
>>> t2 - <MAC1,VID1,P1> is modified in syscall context under 'hash_lock' to
>>> <MAC1,VID1,P2>
>>> t3 - <MAC1,VID1,P2> is notified as blocking
>>> t4 - <MAC1,VID1,P1> is notified as blocking (next time the 'deferred'
>>> list is processed)
>>>
>>> In this case, the HW will have <MAC1,VID1,P1>, but SW will have
>>> <MAC1,VID1,P2>
>>
>> Ok, so if the hardware FDB entry needs to be updated under the same
>> hash_lock as the software FDB entry, then it seems that the goal of
>> updating the hardware FDB synchronously and in a sleepable manner is if
>> the data path defers the learning to sleepable context too. That in turn
>> means that there will be 'dead time' between the reception of a packet
>> from a given {MAC SA, VID} flow and the learning of that address. So I
>> don't think that is really desirable. So I don't know if it is actually
>> realistic to do this.
>>
>> Can we drop it from the requirements of this change, or do you feel like
>> it's not worth it to make my change if this problem is not solved?
>
> I didn't pose it as a requirement, but as a desirable goal that I don't
> know how to achieve w/o a surgery in the bridge driver that Nik and you
> (understandably) don't like.
>
> Regarding a more practical solution, earlier versions (not what you
> posted yesterday) have the undesirable properties of being both
> asynchronous (current state) and mandating RTNL to be held. If we are
> going with the asynchronous model, then I think we should have a model
> that doesn't force RTNL and allows batching.
>
> I have the following proposal, which I believe solves your problem and
> allows for batching without RTNL:
>
> The pattern of enqueuing a work item per-entry is not very smart.
> Instead, it is better to to add the notification info to a list
> (protected by a spin lock) and scheduling a single work item whose
> purpose is to dequeue entries from this list and batch process them.
>
> Inside the work item you would do something like:
>
> spin_lock_bh()
> list_splice_init()
> spin_unlock_bh()
>
> mutex_lock() // rtnl or preferably private lock
> list_for_each_entry_safe()
> // process entry
> cond_resched()
> mutex_unlock()
>
> In del_nbp(), after br_fdb_delete_by_port(), the bridge will emit some
> new blocking event (e.g., SWITCHDEV_FDB_FLUSH_TO_DEVICE) that will
> instruct the driver to flush all its pending FDB notifications. You
> don't strictly need this notification because of the
> netdev_upper_dev_unlink() that follows, but it helps in making things
> more structured.
>
I was also thinking about a solution along these lines, I like this proposition.
> Pros:
>
> 1. Solves your problem?
> 2. Pattern is not worse than what we currently have
> 3. Does not force RTNL
> 4. Allows for batching. For example, mlxsw has the ability to program up
> to 64 entries in one transaction with the device. I assume other devices
> in the same grade have similar capabilities
Batching would help a lot even if we don't remove rtnl, on loaded systems rtnl itself
is a bottleneck and we've seen crazy delays in commands because of contention. That
coupled with the ability to program multiple entries would be a nice win.
>
> Cons:
>
> 1. Asynchronous
> 2. Pattern we will see in multiple drivers? Can consider migrating it
> into switchdev itself at some point
> 3. Something I missed / overlooked
> >> There is of course the option of going half-way too, just like for
>> SWITCHDEV_PORT_ATTR_SET. You notify it once, synchronously, on the
>> atomic chain, the switchdev throws as many errors as it can reasonably
>> can, then you defer the actual installation which means a hardware access.
>
> Yes, the above proposal has the same property. You can throw errors
> before enqueueing the notification info on your list.
>
Thanks,
Nik
next prev parent reply other threads:[~2021-08-23 15:42 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-19 16:07 [PATCH v2 net-next 0/5] Make SWITCHDEV_FDB_{ADD,DEL}_TO_DEVICE blocking Vladimir Oltean
2021-08-19 16:07 ` [PATCH v2 net-next 1/5] net: switchdev: move SWITCHDEV_FDB_{ADD,DEL}_TO_DEVICE to the blocking notifier chain Vladimir Oltean
2021-08-19 18:15 ` Vlad Buslov
2021-08-19 23:18 ` Vladimir Oltean
2021-08-20 7:36 ` Vlad Buslov
2021-08-19 16:07 ` [PATCH v2 net-next 2/5] net: bridge: switchdev: make br_fdb_replay offer sleepable context to consumers Vladimir Oltean
2021-08-19 16:07 ` [PATCH v2 net-next 3/5] net: switchdev: drop the atomic notifier block from switchdev_bridge_port_{,un}offload Vladimir Oltean
2021-08-19 16:07 ` [PATCH v2 net-next 4/5] net: switchdev: don't assume RCU context in switchdev_handle_fdb_{add,del}_to_device Vladimir Oltean
2021-08-19 16:07 ` [PATCH v2 net-next 5/5] net: dsa: handle SWITCHDEV_FDB_{ADD,DEL}_TO_DEVICE synchronously Vladimir Oltean
2021-08-20 9:16 ` [PATCH v2 net-next 0/5] Make SWITCHDEV_FDB_{ADD,DEL}_TO_DEVICE blocking Ido Schimmel
2021-08-20 9:37 ` Vladimir Oltean
2021-08-20 16:09 ` Ido Schimmel
2021-08-20 17:06 ` Vladimir Oltean
2021-08-20 23:36 ` Nikolay Aleksandrov
2021-08-21 0:22 ` Vladimir Oltean
2021-08-22 6:48 ` Ido Schimmel
2021-08-22 9:12 ` Nikolay Aleksandrov
2021-08-22 13:31 ` Vladimir Oltean
2021-08-22 17:06 ` Ido Schimmel
2021-08-22 17:44 ` Vladimir Oltean
2021-08-23 10:47 ` Ido Schimmel
2021-08-23 11:00 ` Vladimir Oltean
2021-08-23 12:16 ` Ido Schimmel
2021-08-23 14:29 ` Vladimir Oltean
2021-08-23 15:18 ` Ido Schimmel
2021-08-23 15:42 ` Nikolay Aleksandrov [this message]
2021-08-23 15:42 ` Vladimir Oltean
2021-08-23 16:02 ` Ido Schimmel
2021-08-23 16:11 ` Vladimir Oltean
2021-08-23 16:23 ` Vladimir Oltean
2021-08-20 10:49 ` Vladimir Oltean
2021-08-20 16:11 ` Ido Schimmel
2021-08-21 19:09 ` Vladimir Oltean
2021-08-22 7:19 ` Ido Schimmel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5d3190a8-b0f2-f695-564f-318f1d1e4a0c@nvidia.com \
--to=nikolay@nvidia.com \
--cc=Landen.Chao@mediatek.com \
--cc=Steen.Hegelund@microchip.com \
--cc=UNGLinuxDriver@microchip.com \
--cc=alexandre.belloni@bootlin.com \
--cc=andrew@lunn.ch \
--cc=borntraeger@de.ibm.com \
--cc=claudiu.manoil@nxp.com \
--cc=davem@davemloft.net \
--cc=dqfext@gmail.com \
--cc=f.fainelli@gmail.com \
--cc=george.mccollister@gmail.com \
--cc=gor@linux.ibm.com \
--cc=grygorii.strashko@ti.com \
--cc=hauke@hauke-m.de \
--cc=hca@linux.ibm.com \
--cc=idosch@idosch.org \
--cc=idosch@nvidia.com \
--cc=ioana.ciornei@nxp.com \
--cc=ivecera@redhat.com \
--cc=jesse.brandeburg@intel.com \
--cc=jianbol@nvidia.com \
--cc=jiri@nvidia.com \
--cc=jwi@linux.ibm.com \
--cc=kabel@blackhole.sk \
--cc=kgraul@linux.ibm.com \
--cc=kuba@kernel.org \
--cc=kurt@linutronix.de \
--cc=lars.povlsen@microchip.com \
--cc=leon@kernel.org \
--cc=mbloch@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=olteanv@gmail.com \
--cc=roid@nvidia.com \
--cc=roopa@nvidia.com \
--cc=saeedm@nvidia.com \
--cc=sean.wang@mediatek.com \
--cc=tchornyi@marvell.com \
--cc=tobias@waldekranz.com \
--cc=vigneshr@ti.com \
--cc=vivien.didelot@gmail.com \
--cc=vkochan@marvell.com \
--cc=vladbu@nvidia.com \
--cc=vladimir.oltean@nxp.com \
--cc=woojung.huh@microchip.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).