Re: [PATCH v5 net-next 3/5] net: wwan: t7xx: PCIe reset rescan

Re: [PATCH v5 net-next 3/5] net: wwan: t7xx: PCIe reset rescan

[Jakub Kicinski]

Hi Bjorn,

any objections to the kind of shenanigans this is playing?


On Sat, 21 Jan 2023 19:03:23 +0530 m.chetan.kumar@linux.intel.com wrote:
> From: M Chetan Kumar <m.chetan.kumar@linux.intel.com>
> 
> PCI rescan module implements "rescan work queue".
> In firmware flashing or coredump collection procedure
> WWAN device is programmed to boot in fastboot mode and
> a work item is scheduled for removal & detection.
> 
> The WWAN device is reset using APCI call as part driver
> removal flow. Work queue rescans pci bus at fixed interval
> for device detection, later when device is detect work queue
> exits.
> 
> Signed-off-by: Haijun Liu <haijun.liu@mediatek.com>
> Co-developed-by: Madhusmita Sahu <madhusmita.sahu@intel.com>
> Signed-off-by: Madhusmita Sahu <madhusmita.sahu@intel.com>
> Signed-off-by: Ricardo Martinez <ricardo.martinez@linux.intel.com>
> Signed-off-by: M Chetan Kumar <m.chetan.kumar@linux.intel.com>
> Signed-off-by: Devegowda Chandrashekar <chandrashekar.devegowda@intel.com>
> --
> v5:
>  * No Change.
> v4:
>  * Change "is still exist" to "still exists".
> v3:
>  * No Change.
> v2:
>  * Drop empty line inside critical sections.
>  * Correct log message.
>  * Correct logic inside t7xx_always_match().
>  * Drop hp_enable changes.
>  * Drop g_ prefix from t7xx_rescan_ctx.
>  * Use tab before comment in struct decl.
>  * Remove extra white space.
>  * Drop modem exception state check.
>  * Crit section newlines.
>  * Remove unnecessary header files inclusion.
>  * Drop spinlock around reset and rescan flow.
> ---
>  drivers/net/wwan/t7xx/Makefile          |  3 +-
>  drivers/net/wwan/t7xx/t7xx_modem_ops.c  |  3 +
>  drivers/net/wwan/t7xx/t7xx_pci.c        | 56 ++++++++++++++-
>  drivers/net/wwan/t7xx/t7xx_pci_rescan.c | 96 +++++++++++++++++++++++++
>  drivers/net/wwan/t7xx/t7xx_pci_rescan.h | 28 ++++++++
>  5 files changed, 184 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/net/wwan/t7xx/t7xx_pci_rescan.c
>  create mode 100644 drivers/net/wwan/t7xx/t7xx_pci_rescan.h
> 
> diff --git a/drivers/net/wwan/t7xx/Makefile b/drivers/net/wwan/t7xx/Makefile
> index 268ff9e87e5b..ba5c607404a4 100644
> --- a/drivers/net/wwan/t7xx/Makefile
> +++ b/drivers/net/wwan/t7xx/Makefile
> @@ -17,7 +17,8 @@ mtk_t7xx-y:=	t7xx_pci.o \
>  		t7xx_hif_dpmaif_tx.o \
>  		t7xx_hif_dpmaif_rx.o  \
>  		t7xx_dpmaif.o \
> -		t7xx_netdev.o
> +		t7xx_netdev.o \
> +		t7xx_pci_rescan.o
>  
>  mtk_t7xx-$(CONFIG_WWAN_DEBUGFS) += \
>  		t7xx_port_trace.o \
> diff --git a/drivers/net/wwan/t7xx/t7xx_modem_ops.c b/drivers/net/wwan/t7xx/t7xx_modem_ops.c
> index cbd65aa48721..2fcaea4694ba 100644
> --- a/drivers/net/wwan/t7xx/t7xx_modem_ops.c
> +++ b/drivers/net/wwan/t7xx/t7xx_modem_ops.c
> @@ -37,6 +37,7 @@
>  #include "t7xx_modem_ops.h"
>  #include "t7xx_netdev.h"
>  #include "t7xx_pci.h"
> +#include "t7xx_pci_rescan.h"
>  #include "t7xx_pcie_mac.h"
>  #include "t7xx_port.h"
>  #include "t7xx_port_proxy.h"
> @@ -194,6 +195,8 @@ static irqreturn_t t7xx_rgu_isr_thread(int irq, void *data)
>  
>  	msleep(RGU_RESET_DELAY_MS);
>  	t7xx_reset_device_via_pmic(t7xx_dev);
> +	t7xx_rescan_queue_work(t7xx_dev->pdev);
> +
>  	return IRQ_HANDLED;
>  }
>  
> diff --git a/drivers/net/wwan/t7xx/t7xx_pci.c b/drivers/net/wwan/t7xx/t7xx_pci.c
> index 871f2a27a398..3f5ebbc11b82 100644
> --- a/drivers/net/wwan/t7xx/t7xx_pci.c
> +++ b/drivers/net/wwan/t7xx/t7xx_pci.c
> @@ -38,6 +38,7 @@
>  #include "t7xx_mhccif.h"
>  #include "t7xx_modem_ops.h"
>  #include "t7xx_pci.h"
> +#include "t7xx_pci_rescan.h"
>  #include "t7xx_pcie_mac.h"
>  #include "t7xx_reg.h"
>  #include "t7xx_state_monitor.h"
> @@ -715,6 +716,7 @@ static int t7xx_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>  		return ret;
>  	}
>  
> +	t7xx_rescan_done();
>  	t7xx_pcie_mac_set_int(t7xx_dev, MHCCIF_INT);
>  	t7xx_pcie_mac_interrupts_en(t7xx_dev);
>  
> @@ -754,7 +756,59 @@ static struct pci_driver t7xx_pci_driver = {
>  	.shutdown = t7xx_pci_shutdown,
>  };
>  
> -module_pci_driver(t7xx_pci_driver);
> +static int __init t7xx_pci_init(void)
> +{
> +	int ret;
> +
> +	t7xx_pci_dev_rescan();
> +	ret = t7xx_rescan_init();
> +	if (ret) {
> +		pr_err("Failed to init t7xx rescan work\n");
> +		return ret;
> +	}
> +
> +	return pci_register_driver(&t7xx_pci_driver);
> +}
> +module_init(t7xx_pci_init);
> +
> +static int t7xx_always_match(struct device *dev, const void *data)
> +{
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +	const struct pci_device_id *id = data;
> +
> +	if (pci_match_id(id, pdev))
> +		return 1;
> +
> +	return 0;
> +}
> +
> +static void __exit t7xx_pci_cleanup(void)
> +{
> +	int remove_flag = 0;
> +	struct device *dev;
> +
> +	dev = driver_find_device(&t7xx_pci_driver.driver, NULL, &t7xx_pci_table[0],
> +				 t7xx_always_match);
> +	if (dev) {
> +		pr_debug("unregister t7xx PCIe driver while device still exists.\n");
> +		put_device(dev);
> +		remove_flag = 1;
> +	} else {
> +		pr_debug("no t7xx PCIe driver found.\n");
> +	}
> +
> +	pci_lock_rescan_remove();
> +	pci_unregister_driver(&t7xx_pci_driver);
> +	pci_unlock_rescan_remove();
> +
> +	t7xx_rescan_deinit();
> +	if (remove_flag) {
> +		pr_debug("remove t7xx PCI device\n");
> +		pci_stop_and_remove_bus_device_locked(to_pci_dev(dev));
> +	}
> +}
> +
> +module_exit(t7xx_pci_cleanup);
>  
>  MODULE_AUTHOR("MediaTek Inc");
>  MODULE_DESCRIPTION("MediaTek PCIe 5G WWAN modem T7xx driver");
> diff --git a/drivers/net/wwan/t7xx/t7xx_pci_rescan.c b/drivers/net/wwan/t7xx/t7xx_pci_rescan.c
> new file mode 100644
> index 000000000000..67f13c035846
> --- /dev/null
> +++ b/drivers/net/wwan/t7xx/t7xx_pci_rescan.c
> @@ -0,0 +1,96 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (c) 2021, MediaTek Inc.
> + * Copyright (c) 2021-2023, Intel Corporation.
> + */
> +
> +#define pr_fmt(fmt) KBUILD_MODNAME ":t7xx:%s: " fmt, __func__
> +#define dev_fmt(fmt) "t7xx: " fmt
> +
> +#include <linux/delay.h>
> +#include <linux/pci.h>
> +#include <linux/spinlock.h>
> +#include <linux/workqueue.h>
> +
> +#include "t7xx_pci.h"
> +#include "t7xx_pci_rescan.h"
> +
> +static struct remove_rescan_context t7xx_rescan_ctx;
> +
> +void t7xx_pci_dev_rescan(void)
> +{
> +	struct pci_bus *b = NULL;
> +
> +	pci_lock_rescan_remove();
> +	while ((b = pci_find_next_bus(b)))
> +		pci_rescan_bus(b);
> +	pci_unlock_rescan_remove();
> +}
> +
> +void t7xx_rescan_done(void)
> +{
> +	if (!atomic_read(&t7xx_rescan_ctx.rescan_done)) {
> +		atomic_set(&t7xx_rescan_ctx.rescan_done, 1);
> +		pr_debug("Rescan probe\n");
> +	} else {
> +		pr_debug("Init probe\n");
> +	}
> +}
> +
> +static void t7xx_remove_rescan(struct work_struct *work)
> +{
> +	int num_retries = RESCAN_RETRIES;
> +	struct pci_dev *pdev;
> +
> +	atomic_set(&t7xx_rescan_ctx.rescan_done, 0);
> +	pdev = t7xx_rescan_ctx.dev;
> +
> +	if (pdev) {
> +		pci_stop_and_remove_bus_device_locked(pdev);
> +		pr_debug("start remove and rescan flow\n");
> +	}
> +
> +	do {
> +		t7xx_pci_dev_rescan();
> +
> +		if (atomic_read(&t7xx_rescan_ctx.rescan_done))
> +			break;
> +
> +		msleep(DELAY_RESCAN_MTIME);
> +	} while (num_retries--);
> +}
> +
> +void t7xx_rescan_queue_work(struct pci_dev *pdev)
> +{
> +	if (!atomic_read(&t7xx_rescan_ctx.rescan_done)) {
> +		dev_err(&pdev->dev, "Rescan failed\n");
> +		return;
> +	}
> +
> +	t7xx_rescan_ctx.dev = pdev;
> +	queue_work(t7xx_rescan_ctx.pcie_rescan_wq, &t7xx_rescan_ctx.service_task);
> +}
> +
> +int t7xx_rescan_init(void)
> +{
> +	atomic_set(&t7xx_rescan_ctx.rescan_done, 1);
> +	t7xx_rescan_ctx.dev = NULL;
> +
> +	t7xx_rescan_ctx.pcie_rescan_wq = create_singlethread_workqueue(MTK_RESCAN_WQ);
> +	if (!t7xx_rescan_ctx.pcie_rescan_wq) {
> +		pr_err("Failed to create workqueue: %s\n", MTK_RESCAN_WQ);
> +		return -ENOMEM;
> +	}
> +
> +	INIT_WORK(&t7xx_rescan_ctx.service_task, t7xx_remove_rescan);
> +
> +	return 0;
> +}
> +
> +void t7xx_rescan_deinit(void)
> +{
> +	t7xx_rescan_ctx.dev = NULL;
> +	atomic_set(&t7xx_rescan_ctx.rescan_done, 0);
> +	cancel_work_sync(&t7xx_rescan_ctx.service_task);
> +	destroy_workqueue(t7xx_rescan_ctx.pcie_rescan_wq);
> +}
> diff --git a/drivers/net/wwan/t7xx/t7xx_pci_rescan.h b/drivers/net/wwan/t7xx/t7xx_pci_rescan.h
> new file mode 100644
> index 000000000000..80b25c44151c
> --- /dev/null
> +++ b/drivers/net/wwan/t7xx/t7xx_pci_rescan.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0-only
> + *
> + * Copyright (c) 2021, MediaTek Inc.
> + * Copyright (c) 2021-2023, Intel Corporation.
> + */
> +
> +#ifndef __T7XX_PCI_RESCAN_H__
> +#define __T7XX_PCI_RESCAN_H__
> +
> +#define MTK_RESCAN_WQ "mtk_rescan_wq"
> +
> +#define DELAY_RESCAN_MTIME 1000
> +#define RESCAN_RETRIES 35
> +
> +struct remove_rescan_context {
> +	struct work_struct service_task;
> +	struct workqueue_struct *pcie_rescan_wq;
> +	struct pci_dev *dev;
> +	atomic_t rescan_done;
> +};
> +
> +void t7xx_pci_dev_rescan(void);
> +void t7xx_rescan_queue_work(struct pci_dev *pdev);
> +int t7xx_rescan_init(void);
> +void t7xx_rescan_deinit(void);
> +void t7xx_rescan_done(void);
> +
> +#endif	/* __T7XX_PCI_RESCAN_H__ */

Re: [PATCH v5 net-next 3/5] net: wwan: t7xx: PCIe reset rescan

[Bjorn Helgaas]

On Tue, Jan 24, 2023 at 08:45:43PM -0800, Jakub Kicinski wrote:
> Hi Bjorn,
> 
> any objections to the kind of shenanigans this is playing?

Yes, thanks for asking.  Drivers definitely should not have to do this
sort of thing.

> On Sat, 21 Jan 2023 19:03:23 +0530 m.chetan.kumar@linux.intel.com wrote:
> > From: M Chetan Kumar <m.chetan.kumar@linux.intel.com>
> > 
> > PCI rescan module implements "rescan work queue".
> > In firmware flashing or coredump collection procedure
> > WWAN device is programmed to boot in fastboot mode and
> > a work item is scheduled for removal & detection.
> > 
> > The WWAN device is reset using APCI call as part driver
> > removal flow. Work queue rescans pci bus at fixed interval
> > for device detection, later when device is detect work queue
> > exits.

I'm not sure what's going on here.  Do we need to reset the device
when the t7xx driver is loaded so the device will load new firmware
when it comes out of reset?

There are a few drivers that do that, e.g., with pci_reset_function().

> > +static struct remove_rescan_context t7xx_rescan_ctx;

Apparently this only supports a single t7xx instance in a system?  Not
good.

> > +void t7xx_pci_dev_rescan(void)
> > +{
> > +	struct pci_bus *b = NULL;
> > +
> > +	pci_lock_rescan_remove();
> > +	while ((b = pci_find_next_bus(b)))
> > +		pci_rescan_bus(b);

No, this driver absolutely cannot rescan and assign unassigned
resources for all the PCI buses in the system.

Bjorn

Re: [PATCH v5 net-next 3/5] net: wwan: t7xx: PCIe reset rescan

[Kumar, M Chetan]

On 1/26/2023 8:55 PM, Bjorn Helgaas wrote:
> On Tue, Jan 24, 2023 at 08:45:43PM -0800, Jakub Kicinski wrote:
>> Hi Bjorn,
>>
>> any objections to the kind of shenanigans this is playing?
> 
> Yes, thanks for asking.  Drivers definitely should not have to do this
> sort of thing.
> 
>> On Sat, 21 Jan 2023 19:03:23 +0530 m.chetan.kumar@linux.intel.com wrote:
>>> From: M Chetan Kumar <m.chetan.kumar@linux.intel.com>
>>>
>>> PCI rescan module implements "rescan work queue".
>>> In firmware flashing or coredump collection procedure
>>> WWAN device is programmed to boot in fastboot mode and
>>> a work item is scheduled for removal & detection.
>>>
>>> The WWAN device is reset using APCI call as part driver
>>> removal flow. Work queue rescans pci bus at fixed interval
>>> for device detection, later when device is detect work queue
>>> exits.
> 
> I'm not sure what's going on here.  Do we need to reset the device
> when the t7xx driver is loaded so the device will load new firmware
> when it comes out of reset?

Flow is, Reset the device to get into firmware download mode then update
the firmware and later reset it to go back to normal mode.

> 
> There are a few drivers that do that, e.g., with pci_reset_function().
> 

Thanks for the suggestion.
I will explore a bit and also try to use pci_reset_function() interface 
and see if it serves the purpose.


>>> +static struct remove_rescan_context t7xx_rescan_ctx;
> 
> Apparently this only supports a single t7xx instance in a system?  Not
> good.
> 
>>> +void t7xx_pci_dev_rescan(void)
>>> +{
>>> +	struct pci_bus *b = NULL;
>>> +
>>> +	pci_lock_rescan_remove();
>>> +	while ((b = pci_find_next_bus(b)))
>>> +		pci_rescan_bus(b);
> 
> No, this driver absolutely cannot rescan and assign unassigned
> resources for all the PCI buses in the system.

T7xx device falls off the bus due to ACPI reset.
Would you please suggest how we can bring device back on the bus without 
such changes inside driver ?  Will pci_reset_function() help in this 
regard ?

If there is no alternate option, then we shall try using remove and 
rescan from user space and drop this patch.

> 
> Bjorn

-- 
Chetan

Re: [PATCH v5 net-next 3/5] net: wwan: t7xx: PCIe reset rescan

[Bjorn Helgaas]

On Fri, Jan 27, 2023 at 03:57:16PM +0530, Kumar, M Chetan wrote:
> On 1/26/2023 8:55 PM, Bjorn Helgaas wrote:
> > On Tue, Jan 24, 2023 at 08:45:43PM -0800, Jakub Kicinski wrote:
> > > On Sat, 21 Jan 2023 19:03:23 +0530 m.chetan.kumar@linux.intel.com wrote:
> > > > From: M Chetan Kumar <m.chetan.kumar@linux.intel.com>
> > > > 
> > > > PCI rescan module implements "rescan work queue".
> > > > In firmware flashing or coredump collection procedure
> > > > WWAN device is programmed to boot in fastboot mode and
> > > > a work item is scheduled for removal & detection.
> > > > 
> > > > The WWAN device is reset using APCI call as part driver
> > > > removal flow. Work queue rescans pci bus at fixed interval
> > > > for device detection, later when device is detect work queue
> > > > exits.
> > 
> > I'm not sure what's going on here.  Do we need to reset the device
> > when the t7xx driver is loaded so the device will load new
> > firmware when it comes out of reset?
> 
> Flow is, Reset the device to get into firmware download mode then
> update the firmware and later reset it to go back to normal mode.

Thanks, that makes sense, and I'm confident that t7xx is not the only
driver that needs to do something like this, so we should be able to
figure out a nice way to do it.

> > > > +void t7xx_pci_dev_rescan(void)
> > > > +{
> > > > +	struct pci_bus *b = NULL;
> > > > +
> > > > +	pci_lock_rescan_remove();
> > > > +	while ((b = pci_find_next_bus(b)))
> > > > +		pci_rescan_bus(b);
> > 
> > No, this driver absolutely cannot rescan and assign unassigned
> > resources for all the PCI buses in the system.
> 
> T7xx device falls off the bus due to ACPI reset.
> Would you please suggest how we can bring device back on the bus
> without such changes inside driver ?  Will pci_reset_function() help
> in this regard ?

"Falling off the bus" is not very precise language, but it usually
means the device stops responding to PCI transactions like config,
memory, or I/O accesses.

Any kind of reset, whether it's done via ACPI _RST or one of the
mechanisms used by pci_reset_function(), causes a PCI device to stop
responding temporarily.  When the device exits reset, it does some
internal initialization and eventually becomes ready to respond to PCI
transactions again.

The PCI core doesn't do anything to the device to "bring it back on
the bus" other than powering on the device or deasserting whatever
signal initiated the reset in the first place.

For example, if we do the reset via pci_reset_secondary_bus(), we set
the Secondary Bus Reset (PCI_BRIDGE_CTL_BUS_RESET) bit in a bridge,
which triggers a reset for devices below the bridge.  When we clear
Secondary Bus Reset, those devices reinitialize themselves and start
responding to PCI transactions.  pci_reset_secondary_bus() contains a
ssleep(1) to give the device time for that initialization.

The t7xx_pci_dev_rescan() loop calls pci_rescan_bus(), which does
nothing to bring devices back on the bus.  It merely issues config
reads to all the possible device addresses to see which respond.

If t7xx_pci_dev_rescan() seems to bring the device back on the bus, it
is probably simply because it takes time and gives the device time to
finish its initialization.  It doesn't actually *do* anything to the
device other than do a config read to it.

I notice that t7xx_remove_rescan() would actually *remove* the
pci_dev, and pci_rescan_bus() would create a new pci_dev if the t7xx
device responds to a config read.  But this a real mess.  When you
remove the device, the driver is detached from it, and we should no
longer be running any driver code.

If you can use pci_reset_function(), there's no need to remove and
re-enumerate the device, so that should let you get rid of the whole
t7xx_remove_rescan() workqueue.

Bjorn