[PATCH net] net: dsa: microchip: Added the condition for scheduling ksz_mib_read_work

[PATCH net] net: dsa: microchip: Added the condition for scheduling ksz_mib_read_work

[Arun Ramadoss]

When the ksz module is installed and removed using rmmod, kernel crashes
with null pointer dereferrence error. During rmmod, ksz_switch_remove
function tries to cancel the mib_read_workqueue using
cancel_delayed_work_sync routine.

At the end of  mib_read_workqueue execution, it again reschedule the
workqueue unconditionally. Due to which queue rescheduled after
mib_interval, during this execution it tries to access dp->slave. But
the slave is unregistered in the ksz_switch_remove function. Hence
kernel crashes.

To avoid this crash, before canceling the workqueue, resetted the
mib_interval to 0. In the work queue execution, it schedules the
workqueue next time only if the mib_interval is non zero.

Fixes: 469b390e1ba3 ("net: dsa: microchip: use delayed_work instead of timer + work")
Signed-off-by: Arun Ramadoss <arun.ramadoss@microchip.com>
---
 drivers/net/dsa/microchip/ksz_common.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/dsa/microchip/ksz_common.c b/drivers/net/dsa/microchip/ksz_common.c
index 1542bfb8b5e5..ffc8e6fb300a 100644
--- a/drivers/net/dsa/microchip/ksz_common.c
+++ b/drivers/net/dsa/microchip/ksz_common.c
@@ -94,7 +94,8 @@ static void ksz_mib_read_work(struct work_struct *work)
 		mutex_unlock(&mib->cnt_mutex);
 	}
 
-	schedule_delayed_work(&dev->mib_read, dev->mib_read_interval);
+	if (dev->mib_read_interval)
+		schedule_delayed_work(&dev->mib_read, dev->mib_read_interval);
 }
 
 void ksz_init_mib_timer(struct ksz_device *dev)
@@ -449,8 +450,10 @@ EXPORT_SYMBOL(ksz_switch_register);
 void ksz_switch_remove(struct ksz_device *dev)
 {
 	/* timer started */
-	if (dev->mib_read_interval)
+	if (dev->mib_read_interval) {
+		dev->mib_read_interval = 0;
 		cancel_delayed_work_sync(&dev->mib_read);
+	}
 
 	dev->dev_ops->exit(dev);
 	dsa_unregister_switch(dev->ds);
-- 
2.33.0

Re: [PATCH net] net: dsa: microchip: Added the condition for scheduling ksz_mib_read_work

[Andrew Lunn]

On Fri, Oct 08, 2021 at 02:13:48PM +0530, Arun Ramadoss wrote:
> When the ksz module is installed and removed using rmmod, kernel crashes
> with null pointer dereferrence error. During rmmod, ksz_switch_remove
> function tries to cancel the mib_read_workqueue using
> cancel_delayed_work_sync routine.
> 
> At the end of  mib_read_workqueue execution, it again reschedule the
> workqueue unconditionally. Due to which queue rescheduled after
> mib_interval, during this execution it tries to access dp->slave. But
> the slave is unregistered in the ksz_switch_remove function. Hence
> kernel crashes.

Something not correct here.

https://www.kernel.org/doc/html/latest/core-api/workqueue.html?highlight=delayed_work#c.cancel_delayed_work_sync

This is cancel_work_sync() for delayed works.

and

https://www.kernel.org/doc/html/latest/core-api/workqueue.html?highlight=delayed_work#c.cancel_work_sync

This function can be used even if the work re-queues itself or
migrates to another workqueue.

Maybe the real problem is a missing call to destroy_worker()?

      Andrew

Re: [PATCH net] net: dsa: microchip: Added the condition for scheduling ksz_mib_read_work

[Jakub Kicinski]

On Fri, 8 Oct 2021 15:58:16 +0200 Andrew Lunn wrote:
> On Fri, Oct 08, 2021 at 02:13:48PM +0530, Arun Ramadoss wrote:
> > When the ksz module is installed and removed using rmmod, kernel crashes
> > with null pointer dereferrence error. During rmmod, ksz_switch_remove
> > function tries to cancel the mib_read_workqueue using
> > cancel_delayed_work_sync routine.
> > 
> > At the end of  mib_read_workqueue execution, it again reschedule the
> > workqueue unconditionally. Due to which queue rescheduled after
> > mib_interval, during this execution it tries to access dp->slave. But
> > the slave is unregistered in the ksz_switch_remove function. Hence
> > kernel crashes.  
> 
> Something not correct here.
> 
> https://www.kernel.org/doc/html/latest/core-api/workqueue.html?highlight=delayed_work#c.cancel_delayed_work_sync
> 
> This is cancel_work_sync() for delayed works.
> 
> and
> 
> https://www.kernel.org/doc/html/latest/core-api/workqueue.html?highlight=delayed_work#c.cancel_work_sync
> 
> This function can be used even if the work re-queues itself or
> migrates to another workqueue.
> 
> Maybe the real problem is a missing call to destroy_worker()?

Also the cancel_delayed_work_sync() is suspiciously early in the remove
flow. There is a schedule_work call in ksz_mac_link_down() which may 
schedule the work back in. That'd also explain why the patch helps since
ksz_mac_link_down() only schedules if (dev->mib_read_interval).

Re: [PATCH net] net: dsa: microchip: Added the condition for scheduling ksz_mib_read_work

[Arun.Ramadoss]

On Fri, 2021-10-08 at 11:34 -0700, Jakub Kicinski wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you
> know the content is safe
> 
> On Fri, 8 Oct 2021 15:58:16 +0200 Andrew Lunn wrote:
> > On Fri, Oct 08, 2021 at 02:13:48PM +0530, Arun Ramadoss wrote:
> > > When the ksz module is installed and removed using rmmod, kernel
> > > crashes
> > > with null pointer dereferrence error. During rmmod,
> > > ksz_switch_remove
> > > function tries to cancel the mib_read_workqueue using
> > > cancel_delayed_work_sync routine.
> > > 
> > > At the end of  mib_read_workqueue execution, it again reschedule
> > > the
> > > workqueue unconditionally. Due to which queue rescheduled after
> > > mib_interval, during this execution it tries to access dp->slave. 
> > > But
> > > the slave is unregistered in the ksz_switch_remove function.
> > > Hence
> > > kernel crashes.
> > 
> > Something not correct here.
> > 
> > 
https://www.kernel.org/doc/html/latest/core-api/workqueue.html?highlight=delayed_work#c.cancel_delayed_work_sync
> > 
> > This is cancel_work_sync() for delayed works.
> > 
> > and
> > 
> > 
https://www.kernel.org/doc/html/latest/core-api/workqueue.html?highlight=delayed_work#c.cancel_work_sync
> > 
> > This function can be used even if the work re-queues itself or
> > migrates to another workqueue.
> > 
> > Maybe the real problem is a missing call to destroy_worker()?
> 
> Also the cancel_delayed_work_sync() is suspiciously early in the
> remove
> flow. There is a schedule_work call in ksz_mac_link_down() which may
> schedule the work back in. That'd also explain why the patch helps
> since
> ksz_mac_link_down() only schedules if (dev->mib_read_interval).
In this patch, I did two things. Added the if condition for
rescheduling the queue and other is resetted the mib_read_interval to
zero.
As per the cancel_delayed_queue_sync() functionality, Now I tried rmod
after removing the if condition for resheduling the queue,kernel didn't
crash. So, concluded that it is not rearm in ksz_mib_read_work  is
causing the problem but it is due to scheduling in the
ksz_mac_link_down function. This function is called due to the
dsa_unregister_switch. Due to resetting of the mib_read_interval to
zero in switch_remove, the queue is not scheduled in mac_link_down, so
kernel didn't crash.

And also, as per suggestion on cancel_delayed_work_sync() is
suspiciously placed in switch_remove. I undo this patch, and tried just
by moving the canceling of delayed_work after the dsa_unregister_switch
function. As expected dsa_unregister_switch calls the
ksz_mac_link_down, which schedules the mib_read_work. Now, when
cancel_delayed_work_sync is called, it cancels all the workqueue. As a
result, module is removed successfully without kernel crash.

Can I send the updated patch as v1 or new patch with updated commit
message and description.

Re: [PATCH net] net: dsa: microchip: Added the condition for scheduling ksz_mib_read_work

[Jakub Kicinski]

On Mon, 11 Oct 2021 09:41:55 +0000 Arun.Ramadoss@microchip.com wrote:
> On Fri, 2021-10-08 at 11:34 -0700, Jakub Kicinski wrote:
> > Also the cancel_delayed_work_sync() is suspiciously early in the
> > remove
> > flow. There is a schedule_work call in ksz_mac_link_down() which may
> > schedule the work back in. That'd also explain why the patch helps
> > since
> > ksz_mac_link_down() only schedules if (dev->mib_read_interval).  
> In this patch, I did two things. Added the if condition for
> rescheduling the queue and other is resetted the mib_read_interval to
> zero.
> As per the cancel_delayed_queue_sync() functionality, Now I tried rmod
> after removing the if condition for resheduling the queue,kernel didn't
> crash. So, concluded that it is not rearm in ksz_mib_read_work  is
> causing the problem but it is due to scheduling in the
> ksz_mac_link_down function. This function is called due to the
> dsa_unregister_switch. Due to resetting of the mib_read_interval to
> zero in switch_remove, the queue is not scheduled in mac_link_down, so
> kernel didn't crash.
> 
> And also, as per suggestion on cancel_delayed_work_sync() is
> suspiciously placed in switch_remove. I undo this patch, and tried just
> by moving the canceling of delayed_work after the dsa_unregister_switch
> function. As expected dsa_unregister_switch calls the
> ksz_mac_link_down, which schedules the mib_read_work. Now, when
> cancel_delayed_work_sync is called, it cancels all the workqueue. As a
> result, module is removed successfully without kernel crash.
> 
> Can I send the updated patch as v1 or new patch with updated commit
> message and description.

Please send a patch with just the second chunk (zeroing
mib_read_interval), you can mark it as v2.