Re: [PATCH v3] driver core: Fix SYNC_STATE_ONLY device link implementation

[Saravana Kannan <saravanak@google.com>]

On Mon, May 25, 2020 at 12:05 PM Michael Walle <michael@walle.cc> wrote:
>
> Am 2020-05-25 20:39, schrieb Saravana Kannan:
> > On Mon, May 25, 2020 at 4:31 AM Michael Walle <michael@walle.cc> wrote:
> >>
> >> Am 2020-05-23 00:47, schrieb Michael Walle:
> >> > Am 2020-05-23 00:21, schrieb Saravana Kannan:
> >> >> On Fri, May 22, 2020 at 11:41 AM Michael Walle <michael@walle.cc>
> >> >> wrote:
> >> >>>
> >> >>> Am Mon, 18 May 2020 23:30:00 -0700
> >> >>> schrieb Saravana Kannan <saravanak@google.com>:
> >> >>>
> >> >>> > When SYNC_STATE_ONLY support was added in commit 05ef983e0d65 ("driver
> >> >>> > core: Add device link support for SYNC_STATE_ONLY flag"),
> >> >>> > device_link_add() incorrectly skipped adding the new SYNC_STATE_ONLY
> >> >>> > device link to the supplier's and consumer's "device link" list.
> >> >>> >
> >> >>> > This causes multiple issues:
> >> >>> > - The device link is lost forever from driver core if the caller
> >> >>> >   didn't keep track of it (caller typically isn't expected to). This
> >> >>> > is a memory leak.
> >> >>> > - The device link is also never visible to any other code path after
> >> >>> >   device_link_add() returns.
> >> >>> >
> >> >>> > If we fix the "device link" list handling, that exposes a bunch of
> >> >>> > issues.
> >> >>> >
> >> >>> > 1. The device link "status" state management code rightfully doesn't
> >> >>> > handle the case where a DL_FLAG_MANAGED device link exists between a
> >> >>> > supplier and consumer, but the consumer manages to probe successfully
> >> >>> > before the supplier. The addition of DL_FLAG_SYNC_STATE_ONLY links
> >> >>> > break this assumption. This causes device_links_driver_bound() to
> >> >>> > throw a warning when this happens.
> >> >>> >
> >> >>> > Since DL_FLAG_SYNC_STATE_ONLY device links are mainly used for
> >> >>> > creating proxy device links for child device dependencies and aren't
> >> >>> > useful once the consumer device probes successfully, this patch just
> >> >>> > deletes DL_FLAG_SYNC_STATE_ONLY device links once its consumer device
> >> >>> > probes. This way, we avoid the warning, free up some memory and avoid
> >> >>> > complicating the device links "status" state management code.
> >> >>> >
> >> >>> > 2. Creating a DL_FLAG_STATELESS device link between two devices that
> >> >>> > already have a DL_FLAG_SYNC_STATE_ONLY device link will result in the
> >> >>> > DL_FLAG_STATELESS flag not getting set correctly. This patch also
> >> >>> > fixes this.
> >> >>> >
> >> >>> > Lastly, this patch also fixes minor whitespace issues.
> >> >>>
> >> >>> My board triggers the
> >> >>>   WARN_ON(link->status != DL_STATE_CONSUMER_PROBE);
> >> >>>
> >> >>> Full bootlog:
> >> > [..]
> >> >
> >> >> Thanks for the log and report. I haven't spent too much time thinking
> >> >> about this, but can you give this a shot?
> >> >> https://lore.kernel.org/lkml/20200520043626.181820-1-saravanak@google.com/
> >> >
> >> > I've already tried that, as this is already in linux-next. Doesn't fix
> >> > it,
> >> > though.
> >>
> >> btw. this only happens on linux-next (tested with next-20200522), not
> >> on
> >> 5.7-rc7 (which has the same two patches of yours)
> >
> > I wouldn't be surprised if the difference is due to
> > fw_devlink_pause/resume() calls in driver/of/property.c. It chops off
> > ~1s in boot time by changing the order in which device links are
> > created from DT. So, I think it's just masking the issue.
> >
> > On linux-next where you see the issue, can you get the logs with this
> > change:
> > +++ b/drivers/base/core.c
> > @@ -907,7 +907,10 @@ void device_links_driver_bound(struct device *dev)
> >                          */
> >                         device_link_drop_managed(link);
> >                 } else {
> > -                       WARN_ON(link->status !=
> > DL_STATE_CONSUMER_PROBE);
> > +                       WARN(link->status != DL_STATE_CONSUMER_PROBE,
> > +                               "sup:%s - con:%s f:%d s:%d\n",
> > +                               dev_name(supplier),
> > dev_name(link->consumer),
> > +                               link->flags, link->status);
> >                         WRITE_ONCE(link->status, DL_STATE_ACTIVE);
> >                 }
> >
> > My goal is to figure out the order in which the device links between
> > the supplier and consumers devices are created and how that's changing
> > the flag and status. Then I can come up with a fix.
>
> Here we go (hopefully, my mail client won't screw up the line wrapping):

Thanks for the logs!

Ok, that definitely gave me some more info.
1. It's happening only for this iommu which in some cases can create
device links before fw_devlink through the use of
BUS_NOTIFY_ADD_DEVICE.
2. The issue doesn't seem to be between STATELESS and SYNC_STATE_ONLY
flags (because STATELESS flag is not set).
3. There seems to be a MANAGED link created by arm-smmu.c before/after
the SYNC_STATE_ONLY link is created.

In which case, the SYNC_STATE_ONLY link is supposed to be a NOP, but
that doesn't seem to be the case for some reason.

Can you add these debug messages and give me the logs? Hopefully
this'll be my last log request. I tried reproducing this in hardware I
have, but I couldn't reproduce it.

--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -348,6 +348,10 @@ struct device_link *device_link_add(struct device
*consumer,
        if (flags & DL_FLAG_AUTOREMOVE_SUPPLIER)
                flags &= ~DL_FLAG_AUTOREMOVE_CONSUMER;

+       if (strstr(dev_name(supplier), "5000000.iommu")) {
+               dev_info(consumer, "Link attempted to %s 0x%x\n",
dev_name(supplier), flags);
+       }
+
        list_for_each_entry(link, &supplier->links.consumers, s_node) {
                if (link->consumer != consumer)
                        continue;
@@ -460,6 +464,10 @@ struct device_link *device_link_add(struct device
*consumer,
        dev_dbg(consumer, "Linked as a consumer to %s\n", dev_name(supplier));

 out:
+       if (strstr(dev_name(supplier), "5000000.iommu") && link) {
+               dev_info(consumer, "Link done to %s 0x%x %d\n",
dev_name(supplier), link->flags, link->status);
+       }
+
        device_pm_unlock();
        device_links_write_unlock();

Thanks,
Saravana