From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751763AbdIMNdU (ORCPT ); Wed, 13 Sep 2017 09:33:20 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:38766 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751019AbdIMNdT (ORCPT ); Wed, 13 Sep 2017 09:33:19 -0400 Date: Wed, 13 Sep 2017 15:33:01 +0200 (CEST) From: Thomas Gleixner To: Kashyap Desai cc: Hannes Reinecke , YASUAKI ISHIMATSU , Marc Zyngier , Christoph Hellwig , axboe@kernel.dk, mpe@ellerman.id.au, keith.busch@intel.com, peterz@infradead.org, LKML , linux-scsi@vger.kernel.org, Sumit Saxena , Shivasharan Srikanteshwara Subject: RE: system hung up when offlining CPUs In-Reply-To: <8cb26204cb5402824496bbb6b636e0af@mail.gmail.com> Message-ID: References: <20170809124213.0d9518bb@why.wild-wind.fr.eu.org> <20170821131809.GA17564@lst.de> <8e0d76cd-7cd4-3a98-12ba-815f00d4d772@gmail.com> <2f2ae1bc-4093-d083-6a18-96b9aaa090c9@gmail.com> <8cb26204cb5402824496bbb6b636e0af@mail.gmail.com> User-Agent: Alpine 2.20 (DEB 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 13 Sep 2017, Kashyap Desai wrote: > > On 09/12/2017 08:15 PM, YASUAKI ISHIMATSU wrote: > > > + linux-scsi and maintainers of megasas > > >> In my server, IRQ#66-89 are sent to CPU#24-29. And if I offline > > >> CPU#24-29, I/O does not work, showing the following messages. .... > > This indeed looks like a problem. > > We're going to great lengths to submit and complete I/O on the same CPU, > > so > > if the CPU is offlined while I/O is in flight we won't be getting a > > completion for > > this particular I/O. > > However, the megasas driver should be able to cope with this situation; > > after > > all, the firmware maintains completions queues, so it would be dead easy > > to > > look at _other_ completions queues, too, if a timeout occurs. > In case of IO timeout, megaraid_sas driver is checking other queues as well. > That is why IO was completed in this case and further IOs were resumed. > > Driver complete commands as below code executed from > megasas_wait_for_outstanding_fusion(). > for (MSIxIndex = 0 ; MSIxIndex < count; MSIxIndex++) > complete_cmd_fusion(instance, MSIxIndex); > > Because of above code executed in driver, we see only one print as below in > this logs. > megaraid_sas 0000:02:00.0: [ 0]waiting for 2 commands to complete for scsi0 > > As per below link CPU hotplug will take care- "All interrupts targeted to > this CPU are migrated to a new CPU" > https://www.kernel.org/doc/html/v4.11/core-api/cpu_hotplug.html > > BTW - We are also able reproduce this issue locally. Reason for IO timeout > is -" IO is completed, but corresponding interrupt did not arrived on Online > CPU. Either missed due to CPU is in transient state of being OFFLINED. I am > not sure which component should take care this." > > Question - "what happens once __cpu_disable is called and some of the queued > interrupt has affinity to that particular CPU ?" > I assume ideally those pending/queued Interrupt should be migrated to > remaining online CPUs. It should not be unhandled if we want to avoid such > IO timeout. Can you please provide the following information, before and after offlining the last CPU in the affinity set: # cat /proc/irq/$IRQNUM/smp_affinity_list # cat /proc/irq/$IRQNUM/effective_affinity # cat /sys/kernel/debug/irq/irqs/$IRQNUM The last one requires: CONFIG_GENERIC_IRQ_DEBUGFS=y Thanks, tglx