[Ksummit-discuss] No more module removal -- Unconference track

[Ksummit-discuss] No more module removal -- Unconference track

[Theodore Ts'o]

This has been scheduled at 2pm at the request of Rusty.  (Reminder: if
you're going to propose a topic, please send e-mail to start a thread
on ksummit-discuss).

						- Ted

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Guenter Roeck]

On Tue, Aug 19, 2014 at 10:48:39AM -0400, Theodore Ts'o wrote:
> This has been scheduled at 2pm at the request of Rusty.  (Reminder: if
> you're going to propose a topic, please send e-mail to start a thread
> on ksummit-discuss).
> 
Do we have a context ? I am using insert/remove module a lot during testing,
and would hate to see it go. It also permits module updates without having to
reboot the kernel. There must be lots of other reasons to support module
removal. So I would really dislike if it was no longer available, and I don't
really see the point.

Count me in.

Guenter

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Theodore Ts'o]

On Tue, Aug 19, 2014 at 07:55:47AM -0700, Guenter Roeck wrote:
> On Tue, Aug 19, 2014 at 10:48:39AM -0400, Theodore Ts'o wrote:
> > This has been scheduled at 2pm at the request of Rusty.  (Reminder: if
> > you're going to propose a topic, please send e-mail to start a thread
> > on ksummit-discuss).
> > 
> Do we have a context ? I am using insert/remove module a lot during testing,
> and would hate to see it go. It also permits module updates without having to
> reboot the kernel. There must be lots of other reasons to support module
> removal. So I would really dislike if it was no longer available, and I don't
> really see the point.

Rusty has been trying to nuke module removal for years.

Unfortunately, it's been incredibly useful for many reasons.

In addition to the reasons you've suggested, it's the only way that I
can reset a malfunctioning sound driver without rebooting, and I'd
hate to have to regress to windows style "reboot to fix the problem".

It was also noted during the kernel summit core day that because so
many people depend on module removal working, the bind and unbind
functions are much more reliable, and so kexec should perhaps consider
migrating to using bind/unbind.

As a result, what tends to happen is that officially, module unload is
not supported, and if the driver is actively in use, it may oops the
kernel.  However, in practice, this feature is heavily used, which is
perhaps why Rusty wants to make another try at removing this feature.  :-)

	    	  	   		    - Ted

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Dan Carpenter]

Why can't we just taint the kernel on module removal?

regards,
dan carpenter

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Guenter Roeck]

On Tue, Aug 19, 2014 at 06:40:29PM +0300, Dan Carpenter wrote:
> Why can't we just taint the kernel on module removal?
> 
Why not just fix its bugs ?

If you want to taint the kernel on module removal because it is known that many
drivers have bugs in their removal code, you should taint the kernel if any code
is used which is known to have a bug. Or, in other words, just taint the kernel,
period.

Guenter

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Takashi Iwai]

At Tue, 19 Aug 2014 11:23:50 -0400,
Theodore Ts'o wrote:
> 
> On Tue, Aug 19, 2014 at 07:55:47AM -0700, Guenter Roeck wrote:
> > On Tue, Aug 19, 2014 at 10:48:39AM -0400, Theodore Ts'o wrote:
> > > This has been scheduled at 2pm at the request of Rusty.  (Reminder: if
> > > you're going to propose a topic, please send e-mail to start a thread
> > > on ksummit-discuss).
> > > 
> > Do we have a context ? I am using insert/remove module a lot during testing,
> > and would hate to see it go. It also permits module updates without having to
> > reboot the kernel. There must be lots of other reasons to support module
> > removal. So I would really dislike if it was no longer available, and I don't
> > really see the point.
> 
> Rusty has been trying to nuke module removal for years.
> 
> Unfortunately, it's been incredibly useful for many reasons.
> 
> In addition to the reasons you've suggested, it's the only way that I
> can reset a malfunctioning sound driver without rebooting, and I'd
> hate to have to regress to windows style "reboot to fix the problem".

Oh, please report such a problem.

Or wait, maybe we shouldn't debug it for keeping one more vote for the
election "save module removal" :)


Takashi

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Dan Carpenter]

On Tue, Aug 19, 2014 at 08:47:38AM -0700, Guenter Roeck wrote:
> On Tue, Aug 19, 2014 at 06:40:29PM +0300, Dan Carpenter wrote:
> > Why can't we just taint the kernel on module removal?
> > 
> Why not just fix its bugs ?
> 
> If you want to taint the kernel on module removal because it is known that many
> drivers have bugs in their removal code, you should taint the kernel if any code
> is used which is known to have a bug. Or, in other words, just taint the kernel,
> period.

If you rmmod a module it probably means that your code was buggy before
you started the rmmod.  We already taint the kernel when the kernel
oopses.  It's the same thing.

regards,
dan carpenter

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Martin K. Petersen]

>>>>> "Dan" == Dan Carpenter <dan.carpenter@oracle.com> writes:

Dan> If you rmmod a module it probably means that your code was buggy
Dan> before you started the rmmod.  We already taint the kernel when the
Dan> kernel oopses.  It's the same thing.

Buggy does not imply that the rest of the kernel is left in a mangled
state. Also, I load and unload device driver modules many, many times
per day. A significant win on large systems that take half an hour to
boot.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [Ksummit-discuss] No more module removal -- Unconference track

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Tue, 2014-08-19 at 08:47 -0700, Guenter Roeck wrote:
> 
> If you want to taint the kernel on module removal because it is known that many
> drivers have bugs in their removal code, 

Most drivers don't have "removal code" per se. They have "unbind" code.
Which you can still exercise without actually unloading the driver.

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Rik van Riel]

On 08/19/2014 12:09 PM, Dan Carpenter wrote:
> On Tue, Aug 19, 2014 at 08:47:38AM -0700, Guenter Roeck wrote:
>> On Tue, Aug 19, 2014 at 06:40:29PM +0300, Dan Carpenter wrote:
>>> Why can't we just taint the kernel on module removal?
>>>
>> Why not just fix its bugs ?
>>
>> If you want to taint the kernel on module removal because it is known that many
>> drivers have bugs in their removal code, you should taint the kernel if any code
>> is used which is known to have a bug. Or, in other words, just taint the kernel,
>> period.
>
> If you rmmod a module it probably means that your code was buggy before
> you started the rmmod.

Not necessarily.

I often rmmod the kvm module (and then load a new one) to
test performance enhancements.

For tracing, tools like systemtap insert kernel modules
to trace kernel code, and will remove them again when
the tracing script exits.

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Grant Likely]

On Tue, Aug 19, 2014 at 11:43 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Tue, 2014-08-19 at 08:47 -0700, Guenter Roeck wrote:
>>
>> If you want to taint the kernel on module removal because it is known that many
>> drivers have bugs in their removal code,
>
> Most drivers don't have "removal code" per se. They have "unbind" code.
> Which you can still exercise without actually unloading the driver.

Many platform, i2c, spi drivers have moved to macro generated module
removal code that does nothing but unregister the struct
device_driver.

g.

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Guenter Roeck]

On Tue, Aug 19, 2014 at 07:09:17PM +0300, Dan Carpenter wrote:
> On Tue, Aug 19, 2014 at 08:47:38AM -0700, Guenter Roeck wrote:
> > On Tue, Aug 19, 2014 at 06:40:29PM +0300, Dan Carpenter wrote:
> > > Why can't we just taint the kernel on module removal?
> > > 
> > Why not just fix its bugs ?
> > 
> > If you want to taint the kernel on module removal because it is known that many
> > drivers have bugs in their removal code, you should taint the kernel if any code
> > is used which is known to have a bug. Or, in other words, just taint the kernel,
> > period.
> 
> If you rmmod a module it probably means that your code was buggy before
> you started the rmmod.  We already taint the kernel when the kernel
> oopses.  It's the same thing.
> 
I am also unloading modules for testing purposes. And if/when the respective
hardware has been removed. It is quite useful with hardware supporting OIR.
Sure, it is also used to replace it with a version which does have bug fixes,
but that doesn't mean those bugs, if hit, would actually crash the kernel. It
may be a stabilization patch, for example, or one with a performacne improvement.
But in any case, even if a module is replaced to fix a bug, that doesn't mean
it _hit_ that bug. That is quite different to an OOPS, where it is implied
that a bug was hit.

Again, if your logic is that module removal should cause the kernel to be tained
because it _could_ be that the removal occurred because of a bug, you might as
well taint the kernel all the time. If you want module removal functionality
removed because it is buggy, you might as well remove the entire kernel because
it is buggy.

So, sorry, I don't really understand your logic.

Guenter

Re: [Ksummit-discuss] No more module removal -- Unconference track

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

On Tue, 2014-08-19 at 12:13 -0500, Grant Likely wrote:
> On Tue, Aug 19, 2014 at 11:43 AM, David Woodhouse <dwmw2@infradead.org> wrote:
> > On Tue, 2014-08-19 at 08:47 -0700, Guenter Roeck wrote:
> >>
> >> If you want to taint the kernel on module removal because it is known that many
> >> drivers have bugs in their removal code,
> >
> > Most drivers don't have "removal code" per se. They have "unbind" code.
> > Which you can still exercise without actually unloading the driver.
> 
> Many platform, i2c, spi drivers have moved to macro generated module
> removal code that does nothing but unregister the struct
> device_driver.

That's exactly what I mean. There is no interesting code for unloading
the module. There is only the code for unbinding the device from the
driver (as in hot-unplug). Which doesn't go away if you disable module
unloading. It just gets harder to test.

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Masami Hiramatsu]

(2014/08/19 23:55), Guenter Roeck wrote:
> On Tue, Aug 19, 2014 at 10:48:39AM -0400, Theodore Ts'o wrote:
>> This has been scheduled at 2pm at the request of Rusty.  (Reminder: if
>> you're going to propose a topic, please send e-mail to start a thread
>> on ksummit-discuss).
>>
> Do we have a context ? I am using insert/remove module a lot during testing,
> and would hate to see it go. It also permits module updates without having to
> reboot the kernel. There must be lots of other reasons to support module
> removal. So I would really dislike if it was no longer available, and I don't
> really see the point.

I have to explain why, since I asked Rusty to improving removing.

What I found is that the module unloading involving 2 stop_machine()s
for each module removing. It must not be needed. However, since the
module's ref-counter is over-optimized for BIG SMP machine, we can't
remove it without replacing it. But it means some performance
regression can happen on such big-scale SMP machines (not the laptop nor
normal smp machine).
So I asked him to introduce something like lock-up option which locks
up the given module, and the kernel skips ref-counting on that module.

Anyway, I sent the series right now :)

https://lkml.org/lkml/2014/8/25/142

Please check it if it is good for your use-cases.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Jiri Kosina]

On Mon, 25 Aug 2014, Masami Hiramatsu wrote:

> What I found is that the module unloading involving 2 stop_machine()s 
> for each module removing. It must not be needed. However, since the 
> module's ref-counter is over-optimized for BIG SMP machine, we can't 
> remove it without replacing it. But it means some performance regression 
> can happen on such big-scale SMP machines (not the laptop nor normal smp 
> machine).

Is that really a big problem in practice?

I.e. are there valid usecase scenarios where module load / unload should 
be considered a hotpath where every ms of performance would matter?

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Masami Hiramatsu]

(2014/08/25 20:05), Jiri Kosina wrote:
> On Mon, 25 Aug 2014, Masami Hiramatsu wrote:
> 
>> What I found is that the module unloading involving 2 stop_machine()s 
>> for each module removing. It must not be needed. However, since the 
>> module's ref-counter is over-optimized for BIG SMP machine, we can't 
>> remove it without replacing it. But it means some performance regression 
>> can happen on such big-scale SMP machines (not the laptop nor normal smp 
>> machine).
> 
> Is that really a big problem in practice?
> 
> I.e. are there valid usecase scenarios where module load / unload should 
> be considered a hotpath where every ms of performance would matter?

There could be. However, in usual usecases, I guess people will not want
to unload it. Systemtap or something "additional" module users will
need to unload modules. (kpatch could be one of them, but I also think
no one want to remove applied patches anyway.)

I just tried to show that the kmodule unload is the one who uses
stop_machine heavily but it is not necessary :)

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] No more module removal -- Unconference track

[David Howells]

Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:

> What I found is that the module unloading involving 2 stop_machine()s
> for each module removing. It must not be needed.

IIRC, at least one of the stop_machine() calls was to prevent problems between
module removal oopsing and the symbol lookup trying to look up whilst dumping
the oops.

David

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Jiri Kosina]

On Tue, 26 Aug 2014, David Howells wrote:

> > What I found is that the module unloading involving 2 stop_machine()s
> > for each module removing. It must not be needed.
> 
> IIRC, at least one of the stop_machine() calls was to prevent problems between
> module removal oopsing and the symbol lookup trying to look up whilst dumping
> the oops.

This would be solved if symbol lookup would be RCU-ized though.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Masami Hiramatsu]

(2014/08/27 6:45), Jiri Kosina wrote:
> On Tue, 26 Aug 2014, David Howells wrote:
> 
>>> What I found is that the module unloading involving 2 stop_machine()s
>>> for each module removing. It must not be needed.
>>
>> IIRC, at least one of the stop_machine() calls was to prevent problems between
>> module removal oopsing and the symbol lookup trying to look up whilst dumping
>> the oops.
> 
> This would be solved if symbol lookup would be RCU-ized though.
> 

Yes, I did that :)

https://lkml.org/lkml/2014/8/25/143
https://lkml.org/lkml/2014/8/25/144
https://lkml.org/lkml/2014/8/25/145

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] No more module removal -- Unconference track

[Theodore Ts'o]

On Tue, Aug 26, 2014 at 02:24:33PM +0900, Masami Hiramatsu wrote:
> There could be. However, in usual usecases, I guess people will not want
> to unload it. Systemtap or something "additional" module users will
> need to unload modules. (kpatch could be one of them, but I also think
> no one want to remove applied patches anyway.)
> 
> I just tried to show that the kmodule unload is the one who uses
> stop_machine heavily but it is not necessary :)

Right, but if the argument that kmodule unload is slow on huge NUMA
machines is that Rusty submits a patch which gets rid of the support
for module unload entirely, then that would be very sad.  (And
**really** screw over systemtap...)

							- Ted