Re: [PATCH v5 net 6/6] net/sched: qdisc_destroy() old ingress and clsact Qdiscs before grafting

From: Peilin Ye <yepeilin.cs@gmail.com>
To: Vlad Buslov <vladbu@nvidia.com>
Cc: Jamal Hadi Salim <jhs@mojatatu.com>,
	Jakub Kicinski <kuba@kernel.org>,
	Pedro Tammela <pctammela@mojatatu.com>,
	"David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@google.com>,
	Paolo Abeni <pabeni@redhat.com>,
	Cong Wang <xiyou.wangcong@gmail.com>,
	Jiri Pirko <jiri@resnulli.us>,
	Peilin Ye <peilin.ye@bytedance.com>,
	Daniel Borkmann <daniel@iogearbox.net>,
	John Fastabend <john.fastabend@gmail.com>,
	Hillf Danton <hdanton@sina.com>,
	netdev@vger.kernel.org, Cong Wang <cong.wang@bytedance.com>
Subject: Re: [PATCH v5 net 6/6] net/sched: qdisc_destroy() old ingress and clsact Qdiscs before grafting
Date: Tue, 30 May 2023 02:11:16 -0700	[thread overview]
Message-ID: <ZHW9tMw5oCkratfs@C02FL77VMD6R.googleapis.com> (raw)
In-Reply-To: <87fs7fxov6.fsf@nvidia.com>

On Mon, May 29, 2023 at 02:50:26PM +0300, Vlad Buslov wrote:
> After looking very carefully at the code I think I know what the issue
> might be:
>
>    Task 1 graft Qdisc   Task 2 new filter
>            +                    +
>            |                    |
>            v                    v
>         rtnl_lock()       take  q->refcnt
>            +                    +
>            |                    |
>            v                    v
> Spin while q->refcnt!=1   Block on rtnl_lock() indefinitely due to -EAGAIN
>
> This will cause a real deadlock with the proposed patch. I'll try to
> come up with a better approach. Sorry for not seeing it earlier.

Thanks a lot for pointing this out!  The reproducers add flower filters to
ingress Qdiscs so I didn't think of rtnl_lock()'ed filter requests...

On Mon, May 29, 2023 at 03:58:50PM +0300, Vlad Buslov wrote:
> - Account for such cls_api behavior in sch_api by dropping and
>   re-tacking the lock before replaying. This actually seems to be quite
>   straightforward since 'replay' functionality that we are reusing for
>   this is designed for similar behavior - it releases rtnl lock before
>   loading a sch module, takes the lock again and safely replays the
>   function by re-obtaining all the necessary data.

Yes, I've tested this using that reproducer Pedro posted.

On Mon, May 29, 2023 at 03:58:50PM +0300, Vlad Buslov wrote:
> If livelock with concurrent filters insertion is an issue, then it can
> be remedied by setting a new Qdisc->flags bit
> "DELETED-REJECT-NEW-FILTERS" and checking for it together with
> QDISC_CLASS_OPS_DOIT_UNLOCKED in order to force any concurrent filter
> insertion coming after the flag is set to synchronize on rtnl lock.

Thanks for the suggestion!  I'll try this approach.

Currently QDISC_CLASS_OPS_DOIT_UNLOCKED is checked after taking a refcnt of
the "being-deleted" Qdisc.  I'll try forcing "late" requests (that arrive
later than Qdisc is flagged as being-deleted) sync on RTNL lock without
(before) taking the Qdisc refcnt (otherwise I think Task 1 will replay for
even longer?).

Thanks,
Peilin Ye