interrupt context

interrupt context

[Iain Fraser]

Hello,

I understand the interrupts and softirq's run in interrupt context (
as opposed to process context ). But what I
don't understand is why you cannot sleep in interrupt context?

What I have read it states that it doesn't have a process to schedule
out. But interrupts use the interrupted processes
kernel stack just like a syscall. So surely it is possible to sleep
using that stack. Understandably It would be unfair on the process
that blocked through no fault of its own.

Also if you are not allowed to sleep / schedule during interrupt
context. Then how does the system timer pre-empt processes by
calling schedule?

I understand there are many reasons why you shouldn't: irq lines being
disabled, quick completion, etc. But I don't understand
why you cannot.

Thanks for the help
Iain

Re: interrupt context

[Borislav Petkov]

On Fri, Oct 05, 2012 at 09:51:55AM +0100, Iain Fraser wrote:
> Hello,
> 
> I understand the interrupts and softirq's run in interrupt context (
> as opposed to process context ). But what I
> don't understand is why you cannot sleep in interrupt context?
> 
> What I have read it states that it doesn't have a process to schedule
> out. But interrupts use the interrupted processes
> kernel stack just like a syscall. So surely it is possible to sleep
> using that stack. Understandably It would be unfair on the process
> that blocked through no fault of its own.
> 
> Also if you are not allowed to sleep / schedule during interrupt
> context. Then how does the system timer pre-empt processes by
> calling schedule?
> 
> I understand there are many reasons why you shouldn't: irq lines being
> disabled, quick completion, etc. But I don't understand
> why you cannot.

Let's imagine for a second we could sleep in IRQ context and we had some
dummy process entity we can schedule out from.

So, we schedule out and run another process which enters the kernel and
grabs a lock L.

At that exact moment, another IRQ on the same line is raised, we execute
the same IRQ handler which purely coincidentally starts busy-waiting on
the same lock L.

How long do you think we'll be busy waiting in IRQ context on that lock?

:-)

HTH.

-- 
Regards/Gruss,
    Boris.

Re: interrupt context

[Iain Fraser]

Hi,

Thanks for the response. But it appears your example illustrates why its bad
to sleep while you have a lock. Which I understand, what I wanted to know is
why in interrupt context under no circumstances are you allowed to sleep?

Btw I have no intention of doing this, I'm just curious. After
thinking about it further
I'm starting to think its a "design" decision as opposed to a side effect of the
kernel implementation.

If you could answer:

"How can the system timer ( obviously in interrupt context ) call
schedule when it is impossible to sleep/schedule in interrupt context?"

That would help me greatly.

Thanks again
iain

On Fri, Oct 5, 2012 at 10:32 AM, Borislav Petkov <bp@alien8.de> wrote:
> On Fri, Oct 05, 2012 at 09:51:55AM +0100, Iain Fraser wrote:
>> Hello,
>>
>> I understand the interrupts and softirq's run in interrupt context (
>> as opposed to process context ). But what I
>> don't understand is why you cannot sleep in interrupt context?
>>
>> What I have read it states that it doesn't have a process to schedule
>> out. But interrupts use the interrupted processes
>> kernel stack just like a syscall. So surely it is possible to sleep
>> using that stack. Understandably It would be unfair on the process
>> that blocked through no fault of its own.
>>
>> Also if you are not allowed to sleep / schedule during interrupt
>> context. Then how does the system timer pre-empt processes by
>> calling schedule?
>>
>> I understand there are many reasons why you shouldn't: irq lines being
>> disabled, quick completion, etc. But I don't understand
>> why you cannot.
>
> Let's imagine for a second we could sleep in IRQ context and we had some
> dummy process entity we can schedule out from.
>
> So, we schedule out and run another process which enters the kernel and
> grabs a lock L.
>
> At that exact moment, another IRQ on the same line is raised, we execute
> the same IRQ handler which purely coincidentally starts busy-waiting on
> the same lock L.
>
> How long do you think we'll be busy waiting in IRQ context on that lock?
>
> :-)
>
> HTH.
>
> --
> Regards/Gruss,
>     Boris.

Re: interrupt context

[Borislav Petkov]

On Fri, Oct 05, 2012 at 11:20:20AM +0100, Iain Fraser wrote:
> Hi,
> 
> Thanks for the response. But it appears your example illustrates why its bad
> to sleep while you have a lock. Which I understand, what I wanted to know is
> why in interrupt context under no circumstances are you allowed to sleep?
> 
> Btw I have no intention of doing this, I'm just curious. After
> thinking about it further
> I'm starting to think its a "design" decision as opposed to a side effect of the
> kernel implementation.

There are more examples where sleeping in IRQ context is simply
problematic. So prohibiting sleeping in such contexts turns out to be
the simplest solution concerning code complexity, design, handling, etc.

Btw, you can trigger some yourself by adding sleeping functions in an
IRQ handler used on your system, disable the "sleeping-while-atomic"
warnings and try to run it to see what happens.

> If you could answer:
>
> "How can the system timer ( obviously in interrupt context ) call
> schedule when it is impossible to sleep/schedule in interrupt
> context?"

I think the best way to learn these things is to stare at the code until
you've understood exactly what it does when it does it, so I'll let you
answer that.

And you want to learn because you wouldn't be asking those questions in
the first place. :-)

HTH.

-- 
Regards/Gruss,
    Boris.

Re: interrupt context

[Theodore Ts'o]

On Fri, Oct 05, 2012 at 09:51:55AM +0100, Iain Fraser wrote:
> 
> I understand the interrupts and softirq's run in interrupt context (
> as opposed to process context ). But what I
> don't understand is why you cannot sleep in interrupt context?

Consider what happens with nested locks (and yes, we definitely need
nested locks).  In order to prevent deadlocks, it is critical to have
lock ordering; that is, you always take locks in a certain order.  If
all processes take lock A, and then lock B, etc., then you won't have
a problem where one process as lock A, and tries to get lock B, and
another process has lock B, and tries to take lock A, and they wait
for each other forever.

If a process has a lock when it gets interrupted, the interrupt
handler has no idea what locks may have already been taken.  So if a
process has taken a mutex (or some other sleeping lock) B, and then
the interrupt handler tries to take lock A, that's a perscription for
deadlock.

In addition, you must never sleep while holding a (non-sleeping)
spinlock.  If the interrupt handler has interrupted a process which is
holding a spinlock, then it simply may not sleep without triggering
all sorts of other problems.

> What I have read it states that it doesn't have a process to schedule
> out. But interrupts use the interrupted processes
> kernel stack just like a syscall. So surely it is possible to sleep
> using that stack. Understandably It would be unfair on the process
> that blocked through no fault of its own.
> 
> Also if you are not allowed to sleep / schedule during interrupt
> context. Then how does the system timer pre-empt processes by
> calling schedule?

The system timer sets the "need to reschedule" flag for that
particular process.  Then as the system timer returns from the
interrupt, there is a common code path which is checked on the way out
of any interrupt handler or system call.  This code path checks to see
if the "need to schedule" flag is set, and if so, at that point
instead of returning to the original process, the kernel will simply
return to some other process.

I would suggest that you get a good introductory Linux book, such as
"Linux Kernel Development" by Robert Love.  You might also check out
the kernelnewbies.org website and mailing list, where you are more
likely to get answers to basic introductory questions like this.

Regards,

						- Ted

Re: interrupt context

[Iain Fraser]

Thanks for the response Ted. I've actual got that book, in fact it is
the source of this question :) .

I knew it wasn't a "nice" thing to do to the poor process. But your
explanation explains why
its also a dangerous thing so thanks.

Also another problem with async sleeping a process, is that you could
end up causing the idle
thread to sleep. And end up in a scenario where there are no runnable tasks.

Thanks for the help
Iain


On Fri, Oct 5, 2012 at 2:27 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Fri, Oct 05, 2012 at 09:51:55AM +0100, Iain Fraser wrote:
>>
>> I understand the interrupts and softirq's run in interrupt context (
>> as opposed to process context ). But what I
>> don't understand is why you cannot sleep in interrupt context?
>
> Consider what happens with nested locks (and yes, we definitely need
> nested locks).  In order to prevent deadlocks, it is critical to have
> lock ordering; that is, you always take locks in a certain order.  If
> all processes take lock A, and then lock B, etc., then you won't have
> a problem where one process as lock A, and tries to get lock B, and
> another process has lock B, and tries to take lock A, and they wait
> for each other forever.
>
> If a process has a lock when it gets interrupted, the interrupt
> handler has no idea what locks may have already been taken.  So if a
> process has taken a mutex (or some other sleeping lock) B, and then
> the interrupt handler tries to take lock A, that's a perscription for
> deadlock.
>
> In addition, you must never sleep while holding a (non-sleeping)
> spinlock.  If the interrupt handler has interrupted a process which is
> holding a spinlock, then it simply may not sleep without triggering
> all sorts of other problems.
>
>> What I have read it states that it doesn't have a process to schedule
>> out. But interrupts use the interrupted processes
>> kernel stack just like a syscall. So surely it is possible to sleep
>> using that stack. Understandably It would be unfair on the process
>> that blocked through no fault of its own.
>>
>> Also if you are not allowed to sleep / schedule during interrupt
>> context. Then how does the system timer pre-empt processes by
>> calling schedule?
>
> The system timer sets the "need to reschedule" flag for that
> particular process.  Then as the system timer returns from the
> interrupt, there is a common code path which is checked on the way out
> of any interrupt handler or system call.  This code path checks to see
> if the "need to schedule" flag is set, and if so, at that point
> instead of returning to the original process, the kernel will simply
> return to some other process.
>
> I would suggest that you get a good introductory Linux book, such as
> "Linux Kernel Development" by Robert Love.  You might also check out
> the kernelnewbies.org website and mailing list, where you are more
> likely to get answers to basic introductory questions like this.
>
> Regards,
>
>                                                 - Ted

Re: interrupt context

[anish kumar]

On Fri, 2012-10-05 at 09:27 -0400, Theodore Ts'o wrote:
> On Fri, Oct 05, 2012 at 09:51:55AM +0100, Iain Fraser wrote:
> > 
> > I understand the interrupts and softirq's run in interrupt context (
> > as opposed to process context ). But what I
> > don't understand is why you cannot sleep in interrupt context?
> 
> Consider what happens with nested locks (and yes, we definitely need
> nested locks).  In order to prevent deadlocks, it is critical to have
> lock ordering; that is, you always take locks in a certain order.  If
> all processes take lock A, and then lock B, etc., then you won't have
> a problem where one process as lock A, and tries to get lock B, and
> another process has lock B, and tries to take lock A, and they wait
> for each other forever.
> 
> If a process has a lock when it gets interrupted, the interrupt
> handler has no idea what locks may have already been taken.  So if a
> process has taken a mutex (or some other sleeping lock) B, and then
> the interrupt handler tries to take lock A, that's a perscription for
> deadlock.
> 
> In addition, you must never sleep while holding a (non-sleeping)
> spinlock.  If the interrupt handler has interrupted a process which is
Just to get it right for me.Most of the time spinlock disables the local
core interrupts(or all cores interrupts) and when you call the sleep
function after holding the spinlock the interrupts will be  enabled
again(Am I right?).This is not desired and will have unknown results.
> holding a spinlock, then it simply may not sleep without triggering
> all sorts of other problems.
> 
> > What I have read it states that it doesn't have a process to schedule
> > out. But interrupts use the interrupted processes
> > kernel stack just like a syscall. So surely it is possible to sleep
> > using that stack. Understandably It would be unfair on the process
> > that blocked through no fault of its own.
> > 
> > Also if you are not allowed to sleep / schedule during interrupt
> > context. Then how does the  system timer pre-empt processes by
> > calling schedule?
> 
> The system timer sets the "need to reschedule" flag for that
> particular process.  Then as the system timer returns from the
> interrupt, there is a common code path which is checked on the way out
set_tsk_need_resched is the call?
> of any interrupt handler or system call.  This code path checks to see
> if the "need to schedule" flag is set, and if so, at that point
> instead of returning to the original process, the kernel will simply
> return to some other process.
> 
> I would suggest that you get a good introductory Linux book, such as
> "Linux Kernel Development" by Robert Love.  You might also check out
> the kernelnewbies.org website and mailing list, where you are more
> likely to get answers to basic introductory questions like this.
> 
> Regards,
> 
> 						- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: interrupt context

[Iain Fraser]

Actually Theodore I've been thinking about your reply. Surely the
kernel doesn't care about a processes locks. Because a process can be
pre-empted at any-time. Isn't that the same as allowing
a process to be scheduled in interrupt context?

I understand the other reason, i.e. not allowing the idle process to
sleep is critical. And also appreciate that async scheduling a process
( other than preemption ) is very unfair to say the least :)

Cheers
Iain

On Fri, Oct 5, 2012 at 2:27 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Fri, Oct 05, 2012 at 09:51:55AM +0100, Iain Fraser wrote:
>>
>> I understand the interrupts and softirq's run in interrupt context (
>> as opposed to process context ). But what I
>> don't understand is why you cannot sleep in interrupt context?
>
> Consider what happens with nested locks (and yes, we definitely need
> nested locks).  In order to prevent deadlocks, it is critical to have
> lock ordering; that is, you always take locks in a certain order.  If
> all processes take lock A, and then lock B, etc., then you won't have
> a problem where one process as lock A, and tries to get lock B, and
> another process has lock B, and tries to take lock A, and they wait
> for each other forever.
>
> If a process has a lock when it gets interrupted, the interrupt
> handler has no idea what locks may have already been taken.  So if a
> process has taken a mutex (or some other sleeping lock) B, and then
> the interrupt handler tries to take lock A, that's a perscription for
> deadlock.
>
> In addition, you must never sleep while holding a (non-sleeping)
> spinlock.  If the interrupt handler has interrupted a process which is
> holding a spinlock, then it simply may not sleep without triggering
> all sorts of other problems.
>
>> What I have read it states that it doesn't have a process to schedule
>> out. But interrupts use the interrupted processes
>> kernel stack just like a syscall. So surely it is possible to sleep
>> using that stack. Understandably It would be unfair on the process
>> that blocked through no fault of its own.
>>
>> Also if you are not allowed to sleep / schedule during interrupt
>> context. Then how does the system timer pre-empt processes by
>> calling schedule?
>
> The system timer sets the "need to reschedule" flag for that
> particular process.  Then as the system timer returns from the
> interrupt, there is a common code path which is checked on the way out
> of any interrupt handler or system call.  This code path checks to see
> if the "need to schedule" flag is set, and if so, at that point
> instead of returning to the original process, the kernel will simply
> return to some other process.
>
> I would suggest that you get a good introductory Linux book, such as
> "Linux Kernel Development" by Robert Love.  You might also check out
> the kernelnewbies.org website and mailing list, where you are more
> likely to get answers to basic introductory questions like this.
>
> Regards,
>
>                                                 - Ted