[GIT PULL] bitmap changes for v6.0-rc1

[GIT PULL] bitmap changes for v6.0-rc1

[Yury Norov]

The following changes since commit 7e18e42e4b280c85b76967a9106a13ca61c16179:

  Linux 6.0-rc4 (2022-09-04 13:10:01 -0700)

are available in the Git repository at:

  https://github.com/norov/linux.git tags/bitmap-6.1-rc1

for you to fetch changes up to 78e5a3399421ad79fc024e6d78e2deb7809d26af:

  cpumask: fix checking valid cpu range (2022-10-01 10:22:58 -0700)

----------------------------------------------------------------
bitmap patches for v6.1-rc1

Hi Linus,

Please pull this patches. They spent more than a week in -next without
major problems. The only problem with warnings generated by cpumask_check(),
when robots do bisection, is fixed by moving the patch "cpumask: fix
checking valid cpu range" to the very end of the series.

Also, it looks like Phil's patch is already in master, merged through
Greg's tree.

Thanks,
Yury

This branch is based on v6.0-rc4 and includes:

drivers/base: Fix unsigned comparison to -1 in CPUMAP_FILE_MAX_BYTES
From Phil Auld
https://lore.kernel.org/lkml/20220906203542.1796629-1-pauld@redhat.com/

cpumask: cleanup nr_cpu_ids vs nr_cpumask_bits mess
From me
https://lore.kernel.org/lkml/20220905230820.3295223-4-yury.norov@gmail.com/T/#m96ffe122721893471fd3470d911a8f2fad6d03b3

lib: optimize find_bit() functions
From me
https://lore.kernel.org/lkml/20220915020730.852234-2-yury.norov@gmail.com/T/

lib/find: add find_nth_bit()
From me
https://lore.kernel.org/lkml/20220918030716.1252285-1-yury.norov@gmail.com/

cpumask: repair cpumask_check()
From me
https://lore.kernel.org/netdev/xhsmhwn9k3ibb.mognet@vschneid.remote.csb/T/

----------------------------------------------------------------
Phil Auld (1):
      drivers/base: Fix unsigned comparison to -1 in CPUMAP_FILE_MAX_BYTES

Yury Norov (23):
      smp: don't declare nr_cpu_ids if NR_CPUS == 1
      smp: add set_nr_cpu_ids()
      lib/cpumask: delete misleading comment
      lib/cpumask: deprecate nr_cpumask_bits
      powerpc/64: don't refer nr_cpu_ids in asm code when it's undefined
      lib/cpumask: add FORCE_NR_CPUS config option
      lib/find_bit: introduce FIND_FIRST_BIT() macro
      lib/find_bit: create find_first_zero_bit_le()
      lib/find_bit: optimize find_next_bit() functions
      tools: sync find_bit() implementation
      lib/bitmap: don't call __bitmap_weight() in kernel code
      lib/bitmap: add bitmap_weight_and()
      lib: add find_nth{,_and,_andnot}_bit()
      lib/bitmap: add tests for find_nth_bit()
      lib/bitmap: remove bitmap_ord_to_pos
      cpumask: add cpumask_nth_{,and,andnot}
      net: fix cpu_max_bits_warn() usage in netif_attrmask_next{,_and}
      cpumask: switch for_each_cpu{,_not} to use for_each_bit()
      lib/find_bit: add find_next{,_and}_bit_wrap
      lib/bitmap: introduce for_each_set_bit_wrap() macro
      lib/find: optimize for_each() macros
      lib/bitmap: add tests for for_each() loops
      cpumask: fix checking valid cpu range

 arch/loongarch/kernel/setup.c |   2 +-
 arch/mips/kernel/setup.c      |   2 +-
 arch/powerpc/kernel/head_64.S |   4 +
 arch/x86/kernel/smpboot.c     |   4 +-
 arch/x86/xen/smp_pv.c         |   2 +-
 fs/ntfs3/bitmap.c             |   4 +-
 include/linux/bitmap.h        |  13 +-
 include/linux/bitops.h        |  19 +++
 include/linux/cpumask.h       | 119 +++++++++++------
 include/linux/find.h          | 272 +++++++++++++++++++++++++++++++++------
 include/linux/netdevice.h     |  10 +-
 include/linux/nodemask.h      |   3 +-
 kernel/smp.c                  |   6 +-
 lib/Kconfig                   |   9 ++
 lib/bitmap.c                  |  68 ++++------
 lib/cpumask.c                 |  40 +++---
 lib/find_bit.c                | 224 +++++++++++++++++++++-----------
 lib/find_bit_benchmark.c      |  18 +++
 lib/test_bitmap.c             | 291 +++++++++++++++++++++++++++++++++++++++++-
 tools/include/linux/find.h    |  61 ++-------
 tools/lib/find_bit.c          | 149 ++++++++++-----------
 21 files changed, 953 insertions(+), 367 deletions(-)

Re: [GIT PULL] bitmap changes for v6.0-rc1

[Linus Torvalds]

On Sun, Oct 2, 2022 at 7:03 PM Yury Norov <yury.norov@gmail.com> wrote:
>
> Please pull this patches. They spent more than a week in -next without
> major problems. The only problem with warnings generated by cpumask_check(),
> when robots do bisection, is fixed by moving the patch "cpumask: fix
> checking valid cpu range" to the very end of the series.

So I appreciate the commentary about process just to explain what's going on.

But what I want for the merge message is a summary of what the pull
actually *does* - the process commentary is good addition, but it's
not useful for the merge commit message about what the merge actually
brings in.

And for that, I do *not* want explanations like this:

> This branch is based on v6.0-rc4 and includes:
>
> drivers/base: Fix unsigned comparison to -1 in CPUMAP_FILE_MAX_BYTES
> From Phil Auld
> https://lore.kernel.org/lkml/20220906203542.1796629-1-pauld@redhat.com/
...

where the explanations just point to external data.

Commit messages are supposed to be self-sufficient. External links can
be acceptable as a "this has subtle issues that are too boring and
extensive to explain here", but even then it's only for _additional_
commentary, not a replacement for explaining what is going on.

And those external links aren't even that. They are literally just the
"this is where the patches were posted originally". Not useful.

So I'm dropping this as not having enough explanations of why I should
pull them. It has explanations, yes, but they are about secondary
things, not about the actual expected improvements brought in by the
pull.

               Linus

Re: [GIT PULL] bitmap changes for v6.0-rc1

[Yury Norov]

On Tue, Oct 04, 2022 at 10:55:51AM -0700, Linus Torvalds wrote:
> On Sun, Oct 2, 2022 at 7:03 PM Yury Norov <yury.norov@gmail.com> wrote:
> >
> > Please pull this patches. They spent more than a week in -next without
> > major problems. The only problem with warnings generated by cpumask_check(),
> > when robots do bisection, is fixed by moving the patch "cpumask: fix
> > checking valid cpu range" to the very end of the series.
> 
> So I appreciate the commentary about process just to explain what's going on.
> 
> But what I want for the merge message is a summary of what the pull
> actually *does* - the process commentary is good addition, but it's
> not useful for the merge commit message about what the merge actually
> brings in.
> 
> And for that, I do *not* want explanations like this:
> 
> > This branch is based on v6.0-rc4 and includes:
> >
> > drivers/base: Fix unsigned comparison to -1 in CPUMAP_FILE_MAX_BYTES
> > From Phil Auld
> > https://lore.kernel.org/lkml/20220906203542.1796629-1-pauld@redhat.com/
> ...
> 
> where the explanations just point to external data.
> 
> Commit messages are supposed to be self-sufficient. External links can
> be acceptable as a "this has subtle issues that are too boring and
> extensive to explain here", but even then it's only for _additional_
> commentary, not a replacement for explaining what is going on.
> 
> And those external links aren't even that. They are literally just the
> "this is where the patches were posted originally". Not useful.
> 
> So I'm dropping this as not having enough explanations of why I should
> pull them. It has explanations, yes, but they are about secondary
> things, not about the actual expected improvements brought in by the
> pull.

OK, I'll resend shortly

Thanks,
Yury

Re: [GIT PULL] bitmap changes for v6.0-rc1

[pr-tracker-bot]

The pull request you sent on Sun, 2 Oct 2022 19:03:38 -0700:

> https://github.com/norov/linux.git tags/bitmap-6.1-rc1

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/90d482908eedd56f01a707325aa541cf9c40f936

Thank you!

-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html

Re: [GIT PULL] bitmap changes for v6.0-rc1

[Linus Torvalds]

On Sun, Oct 2, 2022 at 7:03 PM Yury Norov <yury.norov@gmail.com> wrote:
>
>       lib/cpumask: add FORCE_NR_CPUS config option

Side note: I think we should make this option depend on EXPERT.

It doesn't make much sense in any generic kernel, so it's mostly an
embedded thing. And the question is very confusing.

                 Linus

Re: [GIT PULL] bitmap changes for v6.0-rc1

[Linus Torvalds]

On Mon, Oct 10, 2022 at 2:12 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Sun, Oct 2, 2022 at 7:03 PM Yury Norov <yury.norov@gmail.com> wrote:
> >
> >       lib/cpumask: add FORCE_NR_CPUS config option
>
> Side note: I think we should make this option depend on EXPERT.

Oh, and if a constant 'nr_cpu_ids' is such an important thing from a
code generation standpoint, maybe we'd be better off saying 'for small
NR_CPUS, just make nr_cpu_ids be that small constant".

Rather than confusing people with a question that is a bit strange.
Afaik, all our kernel code has to work with "nr_cpu_id's doesn't match
the number of CPU's we actually have", because of cpu hotplug (and
suspend/resume) issues anyway.

So I _think_ all the nr_cpu_id's games are mainly because we don't
want to allocate huge arrays when  you can't actually have that many
CPU's - but for a small NR_CPUS situation that just isn't an issue
anyway.

Yes? No?

              Linus

Re: [GIT PULL] bitmap changes for v6.0-rc1

[Yury Norov]

On Mon, Oct 10, 2022 at 02:18:46PM -0700, Linus Torvalds wrote:
> On Mon, Oct 10, 2022 at 2:12 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > On Sun, Oct 2, 2022 at 7:03 PM Yury Norov <yury.norov@gmail.com> wrote:
> > >
> > >       lib/cpumask: add FORCE_NR_CPUS config option
> >
> > Side note: I think we should make this option depend on EXPERT.

OK. I'll send a patch.
 
> Oh, and if a constant 'nr_cpu_ids' is such an important thing from a
> code generation standpoint, maybe we'd be better off saying 'for small
> NR_CPUS, just make nr_cpu_ids be that small constant".

From what I see, 'FORCE' effects on size 3x more for bigger NR_CPU_IDS.
For x86_64 allmodconfig build (w/o MAXSMP) on top of today's master:

NR_CPUS            Diff    	          Absolute values
                        	  FORCE=n         FORCE=y
8192            -222429 	373299266       373076837
256             -221799 	363679392       363457593
4                -88922 	363387049       363298127

> Rather than confusing people with a question that is a bit strange.
> Afaik, all our kernel code has to work with "nr_cpu_id's doesn't match
> the number of CPU's we actually have", because of cpu hotplug (and
> suspend/resume) issues anyway.

nr_cpu_ids is a number of possible cpus. Once set, it's never changed.
What system has now is online cpus. FORCE_NR_CPU_IDS doesn't break
hotplug because it only controls when we set nr_cpu_ids - at boot time
or compile time.

> So I _think_ all the nr_cpu_id's games are mainly because we don't
> want to allocate huge arrays when  you can't actually have that many
> CPU's - but for a small NR_CPUS situation that just isn't an issue
> anyway.
> 
> Yes? No?

You're talking about .data section. FORCE_NR_CPUS is about .text and
code generation.

Compile-time allocations for cpumasks, per-cpu variables etc. shouldn't
change when we build kernel with FORCE_NR_CPUS on and off against the
same NR_CPUS, right?

I originally expected that 'FORCE' will improve code generation for
NR_CPUS < BITS_PER_LONG because it allows small_const_nbits()
optimization in cpumasks. But with big NR_CPUS it works even better.
I don't know exactly why. nr_cpu_ids is used widely beyond cpumasks,
after all.

Thanks,
Yury

Re: [GIT PULL] bitmap changes for v6.0-rc1

[Linus Torvalds]

On Mon, Oct 10, 2022 at 8:09 PM Yury Norov <yury.norov@gmail.com> wrote:
>
> > So I _think_ all the nr_cpu_id's games are mainly because we don't
> > want to allocate huge arrays when  you can't actually have that many
> > CPU's - but for a small NR_CPUS situation that just isn't an issue
> > anyway.
> >
> > Yes? No?
>
> You're talking about .data section. FORCE_NR_CPUS is about .text and
> code generation.

No, it's more than just code size that you compare.

It's also the "walk over big arrays".

It doesn't matter if the *code* is simpler, if it's a fixed walk over
64 long-words because the code uses a fixed 4096-bit bitmap.

Yes, that constant 64 is simpler than loading a variable, but it's not
*better*. It's worse. It has allocated more memory, and then it wastes
time walking over it.

So just looking at code size and "simplicity" is bogus.

Yes, a constant loop is smaller and simpler, but unless the constant
is also *small*, it sure isn't better.

              Linus