From: Bart Van Assche <bart.vanassche@sandisk.com>
To: Sreekanth Reddy <sreekanth.reddy@broadcom.com>,
"Elliott, Robert (Persistent Memory)" <elliott@hpe.com>
Cc: "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"irqbalance@lists.infradead.org" <irqbalance@lists.infradead.org>,
"Kashyap Desai" <kashyap.desai@broadcom.com>,
Sathya Prakash Veerichetty <sathya.prakash@broadcom.com>,
Chaitra Basappa <chaitra.basappa@broadcom.com>,
Suganath Prabu Subramani <suganath-prabu.subramani@broadcom.com>
Subject: Re: Observing Softlockup's while running heavy IOs
Date: Fri, 19 Aug 2016 08:56:34 -0700 [thread overview]
Message-ID: <6b7930ca-092c-a03c-d745-b49153aa174c@sandisk.com> (raw)
In-Reply-To: <CAK=zhgrOQ_LAbM3RKfq_MtveygqU3vPtChCB9Jdf6AUfFnr0HQ@mail.gmail.com>
On 08/19/2016 04:44 AM, Sreekanth Reddy wrote:
> [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms
> [ ... ]
> [ +0.000005] [<ffffffff810c392b>] ? finish_task_switch+0x6b/0x200
> [ +0.000006] [<ffffffff8176dabc>] __schedule+0x36c/0x950
> [ +0.000002] [<ffffffff8176e0d7>] schedule+0x37/0x80
> [ +0.000006] [<ffffffff81116a1c>] futex_wait_queue_me+0xbc/0x120
> [ +0.000004] [<ffffffff81116da9>] futex_wait+0x119/0x270
> [ +0.000004] [<ffffffff81116800>] ? futex_wake+0x90/0x180
> [ +0.000003] [<ffffffff81118d6b>] do_futex+0x12b/0xb00
> [ +0.000005] [<ffffffff810d348e>] ? set_next_entity+0x23e/0x440
> [ +0.000007] [<ffffffff810136f1>] ? __switch_to+0x261/0x4b0
> [ +0.000004] [<ffffffff811197c1>] SyS_futex+0x81/0x180
> [ +0.000002] [<ffffffff8176e0d7>] ? schedule+0x37/0x80
> [ +0.000004] [<ffffffff817721ae>] entry_SYSCALL_64_fastpath+0x12/0x71
Hello Sreekanth,
If a "soft lockup" is reported that often means that kernel code is
iterating too long in a loop without giving up the CPU. Inserting a
cond_resched() call in such loops usually resolves these soft lockup
complaints. However, your latest e-mail shows that the soft lockup
complaint was reported on other code than __blk_mq_run_hw_queue(). I'm
afraid this means that the CPU on which the soft lockup was reported is
hammered so hard with interrupts that hardly any time remains for the
scheduler to run code on that CPU. You will have to follow Robert
Elliott's advice and reduce the time that is spent per CPU in interrupt
context.
Bart.
next prev parent reply other threads:[~2016-08-19 15:56 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-08-18 5:55 Observing Softlockup's while running heavy IOs Sreekanth Reddy
2016-08-18 5:55 ` Sreekanth Reddy
2016-08-18 14:59 ` Bart Van Assche
2016-08-18 14:59 ` Bart Van Assche
2016-08-18 21:08 ` Elliott, Robert (Persistent Memory)
2016-08-18 21:08 ` Elliott, Robert (Persistent Memory)
2016-08-19 11:44 ` Sreekanth Reddy
2016-08-19 11:44 ` Sreekanth Reddy
2016-08-19 15:56 ` Bart Van Assche [this message]
2016-08-19 15:56 ` Bart Van Assche
2016-09-01 10:31 ` Sreekanth Reddy
2016-09-01 10:31 ` Sreekanth Reddy
2016-09-01 23:04 ` Bart Van Assche
2016-09-01 23:04 ` Bart Van Assche
[not found] ` <CAK=zhgrLL22stCfwKdpJkN=PkxPVxL=K9RgpP1USEbg_xx5TEg@mail.gmail.com>
2016-09-06 15:06 ` Neil Horman
2016-09-06 15:06 ` Neil Horman
2016-09-07 6:00 ` Sreekanth Reddy
2016-09-07 6:00 ` Sreekanth Reddy
2016-09-07 13:24 ` Neil Horman
2016-09-07 13:24 ` Neil Horman
2016-09-08 5:42 ` Sreekanth Reddy
2016-09-08 5:42 ` Sreekanth Reddy
2016-09-08 13:39 ` Neil Horman
2016-09-08 13:39 ` Neil Horman
2016-09-12 8:18 ` Sreekanth Reddy
2016-09-12 8:18 ` Sreekanth Reddy
2016-09-12 12:03 ` Neil Horman
2016-09-12 12:03 ` Neil Horman
2016-08-19 21:27 ` Elliott, Robert (Persistent Memory)
2016-08-19 21:27 ` Elliott, Robert (Persistent Memory)
2016-08-23 9:52 ` Kashyap Desai
2016-08-23 9:52 ` Kashyap Desai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6b7930ca-092c-a03c-d745-b49153aa174c@sandisk.com \
--to=bart.vanassche@sandisk.com \
--cc=chaitra.basappa@broadcom.com \
--cc=elliott@hpe.com \
--cc=irqbalance@lists.infradead.org \
--cc=kashyap.desai@broadcom.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=sathya.prakash@broadcom.com \
--cc=sreekanth.reddy@broadcom.com \
--cc=suganath-prabu.subramani@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.