5.14-rc breaks iotop swap io monitoring.

5.14-rc breaks iotop swap io monitoring.

[Dave Jones]

I just noticed that in 5.14-rc running iotop, it complains on startup
that "CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine
SWAPIN and IO %" and sure enough a bunch of columns show up as
'unavailable'.

My suspicion is this commit, which I've not confirmed yet as it doesn't
cleanly revert.  I'll poke at it some more in the morning, but figured
I'd bring it up sans-evidence in the hope that I'll wake up to someone
having an Ah-ha moment.

	Dave


commit c5895d3f06cbb80ccb311f1dcb37074651030cb6
Author: Peter Zijlstra <peterz@infradead.org>
Date:   Tue May 4 22:43:42 2021 +0200

    sched: Simplify sched_info_on()
    
    The situation around sched_info is somewhat complicated, it is used by
    sched_stats and delayacct and, indirectly, kvm.
    
    If SCHEDSTATS=Y (but disabled by default) sched_info_on() is
    unconditionally true -- this is the case for all distro kernel configs
    I checked.
    
    If for some reason SCHEDSTATS=N, but TASK_DELAY_ACCT=Y, then
    sched_info_on() can return false when delayacct is disabled,
    presumably because there would be no other users left; except kvm is.
    
    Instead of complicating matters further by accurately accounting
    sched_stat and kvm state, simply unconditionally enable when
    SCHED_INFO=Y, matching the common distro case.
    
    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
    Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
    Reviewed-by: Ingo Molnar <mingo@kernel.org>
    Acked-by: Johannes Weiner <hannes@cmpxchg.org>
    Link: https://lkml.kernel.org/r/20210505111525.121458839@infradead.org

Re: 5.14-rc breaks iotop swap io monitoring.

[Dave Jones]

On Mon, Aug 23, 2021 at 11:16:48PM -0400, Dave Jones wrote:
 > I just noticed that in 5.14-rc running iotop, it complains on startup
 > that "CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine
 > SWAPIN and IO %" and sure enough a bunch of columns show up as
 > 'unavailable'.
 > 
 > My suspicion is this commit, which I've not confirmed yet as it doesn't
 > cleanly revert.  I'll poke at it some more in the morning, but figured
 > I'd bring it up sans-evidence in the hope that I'll wake up to someone
 > having an Ah-ha moment.

Forgot to mention, my config has both SCHEDSTATS, SCHED_INFO and TASK_DELAY_ACCT set
to =y

 	Dave
 

Re: 5.14-rc breaks iotop swap io monitoring.

[Peter Zijlstra]

On Mon, Aug 23, 2021 at 11:16:48PM -0400, Dave Jones wrote:
> I just noticed that in 5.14-rc running iotop, it complains on startup
> that "CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine
> SWAPIN and IO %" and sure enough a bunch of columns show up as
> 'unavailable'.
> 
> My suspicion is this commit, which I've not confirmed yet as it doesn't
> cleanly revert.  I'll poke at it some more in the morning, but figured
> I'd bring it up sans-evidence in the hope that I'll wake up to someone
> having an Ah-ha moment.

If you need DELAYACCT I'm thinking:

  e4042ad49235 ("delayacct: Default disabled")

and

  0cd7c741f01d ("delayacct: Add sysctl to enable at runtime")

might be more relevant..

Re: 5.14-rc breaks iotop swap io monitoring.

[Dave Jones]

On Tue, Aug 24, 2021 at 10:31:16AM +0200, Peter Zijlstra wrote:
 > On Mon, Aug 23, 2021 at 11:16:48PM -0400, Dave Jones wrote:
 > > I just noticed that in 5.14-rc running iotop, it complains on startup
 > > that "CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine
 > > SWAPIN and IO %" and sure enough a bunch of columns show up as
 > > 'unavailable'.
 > > 
 > > My suspicion is this commit, which I've not confirmed yet as it doesn't
 > > cleanly revert.  I'll poke at it some more in the morning, but figured
 > > I'd bring it up sans-evidence in the hope that I'll wake up to someone
 > > having an Ah-ha moment.
 > 
 > If you need DELAYACCT I'm thinking:
 > 
 >   e4042ad49235 ("delayacct: Default disabled")
 > 
 > and
 > 
 >   0cd7c741f01d ("delayacct: Add sysctl to enable at runtime")

That does sound more relevant. however, even after echo 1  > /proc/sys/kernel/task_delayacct,
it still fails in the same way.

	Dave

Re: 5.14-rc breaks iotop swap io monitoring.

[Linus Torvalds]

On Tue, Aug 24, 2021 at 8:19 AM Dave Jones <davej@codemonkey.org.uk> wrote:
>
> On Tue, Aug 24, 2021 at 10:31:16AM +0200, Peter Zijlstra wrote:
>  >
>  > If you need DELAYACCT I'm thinking:
>  >
>  >   e4042ad49235 ("delayacct: Default disabled")
>  >
>  > and
>  >
>  >   0cd7c741f01d ("delayacct: Add sysctl to enable at runtime")
>
> That does sound more relevant. however, even after echo 1  > /proc/sys/kernel/task_delayacct,
> it still fails in the same way.

Hmm.

What happens if you boot with the 'delayacct' kernel parameter.

Even if you enable it at run-time, processes that have been started
before it was enabled won't actually have the 'tsk->delays'
allocation. So I'm not sure how effective the run-time thing is.

            Linus

Re: 5.14-rc breaks iotop swap io monitoring.

[Peter Zijlstra]

On Tue, Aug 24, 2021 at 09:34:02AM -0700, Linus Torvalds wrote:

> Even if you enable it at run-time, processes that have been started
> before it was enabled won't actually have the 'tsk->delays'
> allocation. So I'm not sure how effective the run-time thing is.

Right; we *could* fix that by looping through the tasklist; grabbing a
task ref, doing the alloc and restarting until we've iterated the whole
tasklist and not found a task without a delays thing.

Didn't think it would be worth it, but can be done..

Re: 5.14-rc breaks iotop swap io monitoring.

[Dave Jones]

On Tue, Aug 24, 2021 at 09:34:02AM -0700, Linus Torvalds wrote:
 > On Tue, Aug 24, 2021 at 8:19 AM Dave Jones <davej@codemonkey.org.uk> wrote:
 > >
 > > On Tue, Aug 24, 2021 at 10:31:16AM +0200, Peter Zijlstra wrote:
 > >  >
 > >  > If you need DELAYACCT I'm thinking:
 > >  >
 > >  >   e4042ad49235 ("delayacct: Default disabled")
 > >  >
 > >  > and
 > >  >
 > >  >   0cd7c741f01d ("delayacct: Add sysctl to enable at runtime")
 > >
 > > That does sound more relevant. however, even after echo 1  > /proc/sys/kernel/task_delayacct,
 > > it still fails in the same way.
 > 
 > Hmm.
 > 
 > What happens if you boot with the 'delayacct' kernel parameter.
 > 
 > Even if you enable it at run-time, processes that have been started
 > before it was enabled won't actually have the 'tsk->delays'
 > allocation. So I'm not sure how effective the run-time thing is.

Yeah that boot option does make iotop work again.

	Dave

Re: 5.14-rc breaks iotop swap io monitoring.

[Linus Torvalds]

On Tue, Aug 24, 2021 at 2:41 PM Dave Jones <davej@codemonkey.org.uk> wrote:
>
> On Tue, Aug 24, 2021 at 09:34:02AM -0700, Linus Torvalds wrote:
>  >
>  > What happens if you boot with the 'delayacct' kernel parameter.
>  >
>  > Even if you enable it at run-time, processes that have been started
>  > before it was enabled won't actually have the 'tsk->delays'
>  > allocation. So I'm not sure how effective the run-time thing is.
>
> Yeah that boot option does make iotop work again.

Ok, so it's that delay allocation thing is it.

I'm inclined to let it be and see if somebody else notices, and how
big of a deal it is.

The 'delayacct' kernel command line is an acceptable workaround if
this is something only a few people will even notice or care about.

I wonder how much people care about some statistics and iotop these days.

I also assume the swap stats still show up in "vmstat" etc, and that
it's just that iotop ended up using fancier interfaces?

Or do I assume wrongly?

           Linus

Re: 5.14-rc breaks iotop swap io monitoring.

[Dave Jones]

On Tue, Aug 24, 2021 at 04:26:59PM -0700, Linus Torvalds wrote:
 > >  > What happens if you boot with the 'delayacct' kernel parameter.
 > >  >
 > > Yeah that boot option does make iotop work again.
 > 
 > Ok, so it's that delay allocation thing is it.
 > 
 > I'm inclined to let it be and see if somebody else notices, and how
 > big of a deal it is.
 >
 > The 'delayacct' kernel command line is an acceptable workaround if
 > this is something only a few people will even notice or care about.
 > 
 > I wonder how much people care about some statistics and iotop these days.
 > 
 > I also assume the swap stats still show up in "vmstat" etc, and that
 > it's just that iotop ended up using fancier interfaces?

afaict, vmstat is fine (though none of my machines are using swap, so
they're just reporting 0)

	Dave

Re: 5.14-rc breaks iotop swap io monitoring.

[Peter Zijlstra]

On Tue, Aug 24, 2021 at 04:26:59PM -0700, Linus Torvalds wrote:
> I also assume the swap stats still show up in "vmstat" etc, and that
> it's just that iotop ended up using fancier interfaces?

vmstat still works; I just started a fair number of stupid large
programs and I see swpd si and so changing.