From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757416AbbCMQjA (ORCPT ); Fri, 13 Mar 2015 12:39:00 -0400 Received: from www.linutronix.de ([62.245.132.108]:57494 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757195AbbCMQiw convert rfc822-to-8bit (ORCPT ); Fri, 13 Mar 2015 12:38:52 -0400 Date: Fri, 13 Mar 2015 17:38:49 +0100 From: Sebastian Andrzej Siewior To: Brian Silverman Cc: linux-kernel@vger.kernel.org, linux-rt-users@vger.kernel.org, Austin Schuh , Thomas Gleixner , Ingo Molnar Subject: Re: CONFIG_PREEMPT_RT local_softirq_pending warning when ISR blocks Message-ID: <20150313163849.GF5592@linutronix.de> References: <20150309160845.GE13768@linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8BIT In-Reply-To: X-Key-Id: 2A8CF5D1 X-Key-Fingerprint: 6425 4695 FFF0 AA44 66CC 19E6 7B96 E816 2A8C F5D1 User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org * Brian Silverman | 2015-03-09 20:36:27 [-0400]: >> It looks like your softirq for net_rx is getting a packet and then after >> raising NET_RX (again?) it blocks on a lock. In order to get this lock >> it boosts and schedules bash. It gets runable but on the other CPU. On >> CPU1 there is nothig going is nothing going and the only runable task is >> the idle thread. And this is probably where the warning is written >> because we go to idle while we should process a softirq instead. > >That sounds like the issue. Doing the softirq instead of going idle in >this situation seems like it means calling thread_do_softirq() from >__schedule, but I don't know where the right place is. Can anybody >give me some help on where exactly to check for softirqs from? I was slightly wrong. > irq/18-can1-7228 [001] d.....2 6854.629276: _raw_spin_lock <-enqueue_to_backlog > irq/18-can1-7228 [001] d...1.2 6854.629276: __raise_softirq_irqoff <-enqueue_to_backlog > irq/18-can1-7228 [001] d...1.2 6854.629276: do_raise_softirq_irqoff <-__raise_softirq_irqoff > irq/18-can1-7228 [001] d...2.2 6854.629276: softirq_raise: vec=3 [action=NET_RX] >... # continues handling the can1 interrupt > irq/18-can1-7228 [001] ......6 6854.629286: rt_spin_lock <-get_page_from_freelist > irq/18-can1-7228 [001] ......6 6854.629287: rt_spin_lock_slowlock <-get_page_from_freelist > irq/18-can1-7228 [001] ......6 6854.629287: _raw_spin_lock <-rt_spin_lock_slowlock > irq/18-can1-7228 [001] ....1.6 6854.629287: __try_to_take_rt_mutex <-rt_spin_lock_slowlock > irq/18-can1-7228 [001] ....1.6 6854.629287: _raw_spin_lock_irq <-rt_spin_lock_slowlock > irq/18-can1-7228 [001] d...2.6 6854.629288: _raw_spin_unlock_irq <-rt_spin_lock_slowlock > irq/18-can1-7228 [001] ....1.6 6854.629288: task_blocks_on_rt_mutex <-rt_spin_lock_slowlock You raised the softirq but you did not wakeup softirqd. It is expected that you process softirq(s) in irq-thread context once you leave the interrupt thread (that is on local_bh_enable() because the thread is run with BH off). But this did not happen yet. While you are in your interrupt thread you got blocked on a lock. And since your CPU is idle otherwise, the scheduler puts the idle task on. softirq_check_pending_idle() has a check for this kind of things but only if the ksoftirqd itself got blocked. In your case it is a process with BH switched off. You wouldn't see the warning if you start a task in userland that just loops and keeps the CPU busy :) So. One thing I noticed from looking at the code, is that if a thread is marked IRQF_NO_SOFTIRQ_CALL() then it won't raise process softirqs at all. This is a bug but since nobody uses IRQF_NO_SOFTIRQ_CALL nobody noticed it so far (or nobody uses it because it is not working). And for you. I'm not sure yet what is the best thing to do here. We could - teach softirq_check_pending_idle() to ignore these things because once the irq thread is unblocked, it will process the softirq. - utilize the otherwise idle CPU and schedule ksoftirq with the proper mask. Both isn't that easy, I think… >Thanks, >Brian Sebastian