Re: [RFE] Who's using a module?

Re: [RFE] Who's using a module?

[Lucas De Marchi]

+Jessica +Steve +lkml

On Wed, Mar 11, 2020 at 6:33 AM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
>
> Once in a while there's a need to remove a module (for example because you rebuilt it, or to reload it with different parameters, or whatever…). And then doing `rmmod modulename` and `modprobe -r modulename` gives:
>
>         rmmod: ERROR: Module modulename is in use
>
> If you're lucky, firing up `lsmod | grep modulename` will get you offenders inside "used by" column. But often there's nothing except the count above zero. It is very easy to reproduce if you check `lsmod` output for your graphics driver. I checked it on `i915` and `amdgpu`: when graphics session is opened you can't remove it and `lsmod` doesn't show who's using it.
>
> There's very popular and old question on SO¹ that at the moment has over 55k views, and the only answer that seem to work for people is insanely big and convoluted; it is using a custom kernel driver and kernel tracing capabilities. I guess this amount of research means: no, currently there's no easy way to get who's using a module.
>
> It would be amazing if kernel has capability to figure out who's using a module.

Yeah, right now this would need some work on the kernel side to record
the callers of try_module_get()/__module_get()... usually done e.g on
fops-like structs in a owner field.
The only thing we have there right now is the trace. The trace is not
so bad since it can be added in the kernel command line, but would
usually only be enabled while debugging.

For implementing such a feature I think we would need to add/remove
module owner into the mod struct whenever we have a _get()/_put().
Maybe it's worth it, but it doesn't
come without overhead. I'd like to hear what other people think.

Thanks
Lucas De Marchi


>
> 1: https://stackoverflow.com/questions/448999/is-there-a-way-to-figure-out-what-is-using-a-linux-kernel-module
>
> P.S.: please, add me to CC when replying, I'm not subscribed to the list.



--
Lucas De Marchi

Re: [RFE] Who's using a module?

[Steven Rostedt]

On Fri, 13 Mar 2020 09:22:33 -0700
Lucas De Marchi <lucas.de.marchi@gmail.com> wrote:

> +Jessica +Steve +lkml
> 
> On Wed, Mar 11, 2020 at 6:33 AM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
> >
> > Once in a while there's a need to remove a module (for example because you rebuilt it, or to reload it with different parameters, or whatever…). And then doing `rmmod modulename` and `modprobe -r modulename` gives:
> >
> >         rmmod: ERROR: Module modulename is in use
> >
> > If you're lucky, firing up `lsmod | grep modulename` will get you offenders inside "used by" column. But often there's nothing except the count above zero. It is very easy to reproduce if you check `lsmod` output for your graphics driver. I checked it on `i915` and `amdgpu`: when graphics session is opened you can't remove it and `lsmod` doesn't show who's using it.
> >
> > There's very popular and old question on SO¹ that at the moment has over 55k views, and the only answer that seem to work for people is insanely big and convoluted; it is using a custom kernel driver and kernel tracing capabilities. I guess this amount of research means: no, currently there's no easy way to get who's using a module.
> >
> > It would be amazing if kernel has capability to figure out who's using a module.  
> 
> Yeah, right now this would need some work on the kernel side to record
> the callers of try_module_get()/__module_get()... usually done e.g on
> fops-like structs in a owner field.
> The only thing we have there right now is the trace. The trace is not
> so bad since it can be added in the kernel command line, but would
> usually only be enabled while debugging.
> 
> For implementing such a feature I think we would need to add/remove
> module owner into the mod struct whenever we have a _get()/_put().
> Maybe it's worth it, but it doesn't
> come without overhead. I'd like to hear what other people think.

I believe that the issue is a task did a get on the module and hasn't
released it. As it is using that module. If anything, you can trace all the
function of the module if it isn't loaded yet. Simply do the following:

 # trace-cmd start -p function -l :mod:modname -e module_get -f 'name == "modname"' -R stacktrace
 # modprobe modname
 # trace-cmd show

For example I did the following:

 # trace-cmd start -p function -l :mod:ebtables -e module_get -f 'name == "ebtables"' -R stacktrace
  plugin 'function'
 # modprobe ebtable_filter
 # trace-cmd show
# tracer: function
#
# entries-in-buffer/entries-written: 7/7   #P:8
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
        modprobe-1720  [001] ....   823.724284: ebtables_init <-do_one_initcall
        modprobe-1720  [001] ...1   823.724989: module_get: ebtables call_site=ref_module refcnt=2
        modprobe-1720  [001] ...2   823.724995: <stack trace>
 => trace_event_raw_event_module_refcnt
 => try_module_get.part.50
 => ref_module
 => resolve_symbol
 => load_module
 => __do_sys_finit_module
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
 => 0xc032e68fffffffff
 => 0x2946ffffffff
 => 0x68600000001
 => hp_wmi_perform_query
 => ebtables_init
 => 0x1020e9006
 => 0xc08e900000000686
 => 0xc032e6baffffffff
        modprobe-1720  [001] ....   823.725147: ebt_register_table <-ops_init
        modprobe-1720  [001] ....   823.725153: translate_table <-ebt_register_table
        modprobe-1720  [001] ....   823.725296: ebt_register_table <-ops_init
        modprobe-1720  [001] ....   823.725301: translate_table <-ebt_register_table

Not sure if this is something you are looking for.

-- Steve


> 
> 
> >
> > 1: https://stackoverflow.com/questions/448999/is-there-a-way-to-figure-out-what-is-using-a-linux-kernel-module
> >
> > P.S.: please, add me to CC when replying, I'm not subscribed to the list.  
> 
> 
> 
> --
> Lucas De Marchi

RE: [RFE] Who's using a module?

[David Laight]

From: Lucas De Marchi
> Sent: 13 March 2020 16:23
> On Wed, Mar 11, 2020 at 6:33 AM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
> >
> > Once in a while there's a need to remove a module (for example because you rebuilt it, or to reload
> it with different parameters, or whatever…). And then doing `rmmod modulename` and `modprobe -r
> modulename` gives:
> >
> >         rmmod: ERROR: Module modulename is in use
> >
> > If you're lucky, firing up `lsmod | grep modulename` will get you offenders inside "used by" column.
> But often there's nothing except the count above zero. It is very easy to reproduce if you check
> `lsmod` output for your graphics driver. I checked it on `i915` and `amdgpu`: when graphics session is
> opened you can't remove it and `lsmod` doesn't show who's using it.
> >
> > There's very popular and old question on SO¹ that at the moment has over 55k views, and the only
> answer that seem to work for people is insanely big and convoluted; it is using a custom kernel driver
> and kernel tracing capabilities. I guess this amount of research means: no, currently there's no easy
> way to get who's using a module.
> >
> > It would be amazing if kernel has capability to figure out who's using a module.
> 
> Yeah, right now this would need some work on the kernel side to record
> the callers of try_module_get()/__module_get()... usually done e.g on
> fops-like structs in a owner field.
> The only thing we have there right now is the trace. The trace is not
> so bad since it can be added in the kernel command line, but would
> usually only be enabled while debugging.
> 
> For implementing such a feature I think we would need to add/remove
> module owner into the mod struct whenever we have a _get()/_put().
> Maybe it's worth it, but it doesn't
> come without overhead. I'd like to hear what other people think.

Related would be a standard entry point into a module that indicates
that someone would like to unload it.
This would let the module code either error the request (EBUSY)
or start a tidy up sequence that should make it possible to unload
the module sometime later.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: [RFE] Who's using a module?

[Jessica Yu]

+++ Lucas De Marchi [13/03/20 09:22 -0700]:
>+Jessica +Steve +lkml
>
>On Wed, Mar 11, 2020 at 6:33 AM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
>>
>> Once in a while there's a need to remove a module (for example because you rebuilt it, or to reload it with different parameters, or whatever…). And then doing `rmmod modulename` and `modprobe -r modulename` gives:
>>
>>         rmmod: ERROR: Module modulename is in use
>>
>> If you're lucky, firing up `lsmod | grep modulename` will get you offenders inside "used by" column. But often there's nothing except the count above zero. It is very easy to reproduce if you check `lsmod` output for your graphics driver. I checked it on `i915` and `amdgpu`: when graphics session is opened you can't remove it and `lsmod` doesn't show who's using it.
>>
>> There's very popular and old question on SO¹ that at the moment has over 55k views, and the only answer that seem to work for people is insanely big and convoluted; it is using a custom kernel driver and kernel tracing capabilities. I guess this amount of research means: no, currently there's no easy way to get who's using a module.
>>
>> It would be amazing if kernel has capability to figure out who's using a module.
>
>Yeah, right now this would need some work on the kernel side to record
>the callers of try_module_get()/__module_get()... usually done e.g on
>fops-like structs in a owner field.
>The only thing we have there right now is the trace. The trace is not
>so bad since it can be added in the kernel command line, but would
>usually only be enabled while debugging.
>
>For implementing such a feature I think we would need to add/remove
>module owner into the mod struct whenever we have a _get()/_put().
>Maybe it's worth it, but it doesn't
>come without overhead. I'd like to hear what other people think.

Hmm, we technically have mod->source_list and mod->target_list
already, which keeps track of modules that depend on THIS_MODULE and
modules that THIS_MODULE depends on, respectively. However, this is
limited to symbol dependencies only, which is what is shown in lsmod
in the Used By field. In other words, we currently only keep track of
module dependencies when a module uses a symbol from another module.

Indeed, it could be nice to keep track of try_module_get() callers,
but I'm afraid this is not as trivial as simply adding owner to
mod->source_list when try_module_get() is called. try_module_get() and
module_put() are designed to be quick and callable from interrupt and
process context, and mod->{source,target}_list are protected by the
module_mutex. Even if we could use module_mutex, it may increase lock
contention considerably. So the locking scheme would have to be
reworked to make this happen. I suppose this could be introduced as a
DEBUG option, and I'd welcome a patch if there's someone out there
interested in picking up the legwork. But I don't get the feeling
that the demand for this is that high (perhaps I'm totally wrong?),
especially for something that ftrace could report on much better. I
could for example trace all module_get's and put's from boot and see
which processes and even the call sites where these calls stem from.
E.g., if I enable tracing in /sys/kernel/debug/tracing/events/module/module_{get,put}
(and filter name == "i915"), you'd see output like:

               X-2404  [000] ....  2334.643888: module_put: i915 call_site=dma_buf_release refcnt=33
               X-2404  [000] ....  2334.643990: module_put: i915 call_site=dma_buf_release refcnt=32
        kwin_x11-2877  [007] ....  2334.645605: module_get: i915 call_site=dma_buf_export refcnt=33
        kwin_x11-2877  [007] ....  2334.650827: module_get: i915 call_site=dma_buf_export refcnt=34

Of course, you'd have to know to enable tracing before the problem
occurs..

Hope that helps clarify things a bit,

Jessica