Re: watchdog timeout panic in e1000 driver

[Auke Kok <auke-jan.h.kok@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2835 bytes --]

Kenzo Iwami wrote:
> Hi,
> 
>>>>> Even if the total lock time can be reduced, it's possible that interrupt
>>>>> handler is executed while the interrupted code is still holding the 
>>>>> semaphore.
>>>>> I think your method only decrease the frequency of this problem.
>>>>> Why does reducing the lock time solve this problem?
>>>> there are several problems here that need addressing. It's not acceptable 
>>>> for our driver to wait up to 15 seconds, and we can (presumably) reduce it 
>>>> to milliseconds, so that would help a lot. We should in no case at all hold 
>>>> it for any period longer than (give or take) half a second, so working 
>>>> towards that is a very good step in the right direction.
>>>>
>>>> Adding the timer task back may also help, as we are no longer trying to 
>>>> aqcuire the sw_fw_semaphore in interrupt context, but we removed it for a 
>>>> reason, and I need to dig up what reason this exactly was before we can 
>>>> revert it. Jesse might know, so I'll talk to him. But this will not fix the 
>>>> fact that the semaphore is held for a long time :)
> [...]
>> I think this problem occurs because interrupt handler is executed in same
>> CPU as process that acquires semaphore.
>> How about disabling interrupt while the process is holding the semaphore?
>> I think this is possible, if the total lock time has been reduced.
> 
> I created the attached patch based on the method described above.
> This patch disables interrupt while the process is holding the semaphore.
> 
> I measured how long interrupts are being disabled 10,000 times using the
> following method. TSC was read by rdtscll when interrupt was disabled
> and interrupt was enabled again, then I subtract these two value.
> 
> The longest period interrupt was disabled is under 10usec, which seems
> acceptable. The evaluation environment is;
>   CPU    : Intel Xeon CPU 3.73GHz
>   kernel : 2.6.19-rc5
> 
> I also ran the reproduction TP I sent previously and confirmed that the
> system didn't panic.
>   http://marc.theaimsgroup.com/?l=linux-netdev&m=116125332319850&w=2
> 
> How about this method?

I'm not sure why you would have to disable interrupts when freeing the semaphore, but 
more importantly I don't want to introduce irq code in the swfw handling functions.

Since the major (only) user running this piece of code in intterupt context is the 
watchdog, we might as well see if we can only disable interrupts for that code path, 
which would only be once per 2 seconds. We don't need to protect the ethtool path into 
this code as it doesn't run in irq context.

Would you mind giving attached patch a try and let me know if it works for you? It will 
disable irqs for a bit longer time than your patch, and it begs for a special 
check-link-in-watchdog function that doesn't take so damn long :(

Thanks,

Auke


[-- Attachment #2: e1000_git_disable_irqs_in_watchdog_link_status_check.patch --]
[-- Type: text/x-patch, Size: 673 bytes --]


Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>

diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
index a2f1464..0b52ded 100644
--- a/drivers/net/e1000/e1000_main.c
+++ b/drivers/net/e1000/e1000_main.c
@@ -2430,8 +2430,12 @@ e1000_watchdog(unsigned long data)
 	struct e1000_tx_ring *txdr = adapter->tx_ring;
 	uint32_t link, tctl;
 	int32_t ret_val;
+	unsigned long flags;
 
+	local_irq_save(flags);
 	ret_val = e1000_check_for_link(&adapter->hw);
+	local_irq_restore(flags);
+
 	if ((ret_val == E1000_ERR_PHY) &&
 	    (adapter->hw.phy_type == e1000_phy_igp_3) &&
 	    (E1000_READ_REG(&adapter->hw, CTRL) & E1000_PHY_CTRL_GBE_DISABLE)) {