All of lore.kernel.org
 help / color / mirror / Atom feed
* [Question] Documentation/features: More automation/scripting help?
@ 2018-03-28 12:22 ` Andrea Parri
  0 siblings, 0 replies; 4+ messages in thread
From: Andrea Parri @ 2018-03-28 12:22 UTC (permalink / raw)
  To: Jonathan Corbet, Ingo Molnar, Andy Whitcroft, Joe Perches,
	Linus Torvalds
  Cc: linux-doc, linux-kernel, linux-arch

Hi all,

The directory (not yet three years old although, I freely admit, I've
only recently become aware of it) provides arch. support matrices for
more than 40 generic kernel features that need per-arch. support:

This is a superb project! ;-)  and not a simple one given that, to be
effective, this requires the prompt collaboration between (the intere-
sted) features maintainers/developers, every architecture maintainers,
and documentation maintainers.

There currently appear to be some mismatches between such doc and the
the actual state of the code (e.g., missing architecture, feature not
existing anymore, status flags out-of-date). Realized this, I started
to patch the doc, but this process became soon tedious (consider also
that I barely know what some of these features are about...).

Hence this post. I am wondering if it would make sense to script some
of these matrices.  So, rather than (or together with) using the cur-
rently hard-coded matrices, try to (automatically) generate them from
the sources/configs. Consider the sketch below (sorry for the raw sh).

diff --git a/Documentation/features/list-arch.sh b/Documentation/features/list-arch.sh
index c16b5b5956889..cdec0c1db9db2 100755
--- a/Documentation/features/list-arch.sh
+++ b/Documentation/features/list-arch.sh
@@ -18,7 +18,13 @@ for F in */*/arch-support.txt; do
   C=$(grep -h "^#         Kconfig:"     $F | cut -c25-)
   D=$(grep -h "^#         description:" $F | cut -c25-)
   S=$(grep -hw $ARCH $F | cut -d\| -f3)
+  myS=$(grep -h $C ../../arch/$ARCH/Kconfig)
+  if [ -z "$myS" ]; then
+	  myS=" -- "
+  else
+	  myS=" ok "
+  fi
 
-  printf "%10s/%-22s:%s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$C" "$D"
+  printf "%10s/%-22s:%s VS. %s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$myS" "$C" "$D"
 done
 

With this diff.,

andrea@andrea:~$ ./Documentation/features/list-arch.sh riscv > /tmp/riscv.txt
grep: asm/rwsem.h: No such file or directory
andrea@andrea:~$ cat /tmp/riscv.txt
#
# Kernel feature support matrix of the 'riscv' architecture:
#
      core/ BPF-JIT              : VS.  -- |                        HAVE_BPF_JIT #  arch supports BPF JIT optimizations
      core/ generic-idle-thread  : VS.  ok |             GENERIC_SMP_IDLE_THREAD #  arch makes use of the generic SMP idle thread facility
      core/ jump-labels          : VS.  -- |                HAVE_ARCH_JUMP_LABEL #  arch supports live patched, high efficiency branches
      core/ tracehook            : VS.  ok |                 HAVE_ARCH_TRACEHOOK #  arch supports tracehook (ptrace) register handling APIs
     debug/ gcov-profile-all     : VS.  -- |           ARCH_HAS_GCOV_PROFILE_ALL #  arch supports whole-kernel GCOV code coverage profiling
     debug/ KASAN                : VS.  -- |                     HAVE_ARCH_KASAN #  arch supports the KASAN runtime memory checker
     debug/ kgdb                 : VS.  -- |                      HAVE_ARCH_KGDB #  arch supports the kGDB kernel debugger
     debug/ kprobes              : VS.  ok |                        HAVE_KPROBES #  arch supports live patched kernel probe
     debug/ kprobes-on-ftrace    : VS.  -- |              HAVE_KPROBES_ON_FTRACE #  arch supports combined kprobes and ftrace live patching
     debug/ kretprobes           : VS.  -- |                     HAVE_KRETPROBES #  arch supports kernel function-return probes
     debug/ optprobes            : VS.  -- |                      HAVE_OPTPROBES #  arch supports live patched optprobes
     debug/ stackprotector       : VS.  -- |              HAVE_CC_STACKPROTECTOR #  arch supports compiler driven stack overflow protection
     debug/ uprobes              : VS.  -- |               ARCH_SUPPORTS_UPROBES #  arch supports live patched user probes
     debug/ user-ret-profiler    : VS.  -- |           HAVE_USER_RETURN_NOTIFIER #  arch supports user-space return from system call profiler
        io/ dma-api-debug        : VS.  ok |                  HAVE_DMA_API_DEBUG #  arch supports DMA debug facilities
        io/ dma-contiguous       : VS.  ok |                 HAVE_DMA_CONTIGUOUS #  arch supports the DMA CMA (continuous memory allocator)
        io/ sg-chain             : VS.  -- |                   ARCH_HAS_SG_CHAIN #  arch supports chained scatter-gather lists
       lib/ strncasecmp          : VS.  -- |             __HAVE_ARCH_STRNCASECMP #  arch provides an optimized strncasecmp() function
   locking/ cmpxchg-local        : VS.  -- |                  HAVE_CMPXCHG_LOCAL #  arch supports the this_cpu_cmpxchg() API
   locking/ lockdep              : VS.  -- |                     LOCKDEP_SUPPORT #  arch supports the runtime locking correctness debug facility
   locking/ queued-rwlocks       : VS.  -- |             ARCH_USE_QUEUED_RWLOCKS #  arch supports queued rwlocks
   locking/ queued-spinlocks     : VS.  -- |           ARCH_USE_QUEUED_SPINLOCKS #  arch supports queued spinlocks
   locking/ rwsem-optimized      : VS.  -- |               Optimized asm/rwsem.h #  arch provides optimized rwsem APIs
      perf/ kprobes-event        : VS.  -- |      HAVE_REGS_AND_STACK_ACCESS_API #  arch supports kprobes with perf events
      perf/ perf-regs            : VS.  -- |                      HAVE_PERF_REGS #  arch supports perf events register access
      perf/ perf-stackdump       : VS.  -- |           HAVE_PERF_USER_STACK_DUMP #  arch supports perf events stack dumps
     sched/ membarrier-sync-core : VS.  -- |       ARCH_HAS_MEMBARRIER_SYNC_CORE #  arch supports core serializing membarrier
     sched/ numa-balancing       : VS.  -- |        ARCH_SUPPORTS_NUMA_BALANCING #  arch supports NUMA balancing
   seccomp/ seccomp-filter       : VS.  -- |            HAVE_ARCH_SECCOMP_FILTER #  arch supports seccomp filters
      time/ arch-tick-broadcast  : VS.  -- |             ARCH_HAS_TICK_BROADCAST #  arch provides tick_broadcast()
      time/ clockevents          : VS.  ok |                 GENERIC_CLOCKEVENTS #  arch support generic clock events
      time/ context-tracking     : VS.  -- |               HAVE_CONTEXT_TRACKING #  arch supports context tracking for NO_HZ_FULL
      time/ irq-time-acct        : VS.  -- |            HAVE_IRQ_TIME_ACCOUNTING #  arch supports precise IRQ time accounting
      time/ modern-timekeeping   : VS.  -- |            !ARCH_USES_GETTIMEOFFSET #  arch does not use arch_gettimeoffset() anymore
      time/ virt-cpuacct         : VS.  -- |            HAVE_VIRT_CPU_ACCOUNTING #  arch supports precise virtual CPU time accounting
        vm/ ELF-ASLR             : VS.  -- |              ARCH_HAS_ELF_RANDOMIZE #  arch randomizes the stack, heap and binary images of ELF binaries
        vm/ huge-vmap            : VS.  -- |                 HAVE_ARCH_HUGE_VMAP #  arch supports the ioremap_pud_enabled() and ioremap_pmd_enabled() VM APIs
        vm/ ioremap_prot         : VS.  -- |                   HAVE_IOREMAP_PROT #  arch has ioremap_prot()
        vm/ numa-memblock        : VS.  ok |              HAVE_MEMBLOCK_NODE_MAP #  arch supports NUMA aware memblocks
        vm/ PG_uncached          : VS.  -- |               ARCH_USES_PG_UNCACHED #  arch supports the PG_uncached page flag
        vm/ pte_special          : VS.  -- |             __HAVE_ARCH_PTE_SPECIAL #  arch supports the pte_special()/pte_mkspecial() VM APIs
        vm/ THP                  : VS.  -- |      HAVE_ARCH_TRANSPARENT_HUGEPAGE #  arch supports transparent hugepages
        vm/ batch-unmap-tlb-flush: VS.  -- |   ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH #  arch supports deferral of TLB flush until multiple pages are unmapped


Suggesting that riscv is not included within the hard-coded matrices.
This also shows a first limitation of the proposed approach, i.e., it
doesn't parse "Optimized asm/rwsem.h" (wants "true" configs). It also
won't find "__HAVE_ARCH_PTE_SPECIAL" (reads Kconfig files) but things
like "!ARCH_USES_GETTIMEOFFSET" could still be handled...

We could try switching to another architectures: none of these should
result in "core/ BPF-JIT" set: the "HAVE_BPF_JIT" config was splitted
into cBPF and eBPF variants more recently. The comparison of the hard
coded status flags with the new/generated flags should also highlight
out-of-date values.

Alternative/Additional help could probably be provided by checkpatch,
say, warn when a patch touches/adds to Kconfig without updating such
documentation...

Thoughts?

  Andrea

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [Question] Documentation/features: More automation/scripting help?
@ 2018-03-28 12:22 ` Andrea Parri
  0 siblings, 0 replies; 4+ messages in thread
From: Andrea Parri @ 2018-03-28 12:22 UTC (permalink / raw)
  To: Jonathan Corbet, Ingo Molnar, Andy Whitcroft, Joe Perches,
	Linus Torvalds
  Cc: linux-doc, linux-kernel, linux-arch

Hi all,

The directory (not yet three years old although, I freely admit, I've
only recently become aware of it) provides arch. support matrices for
more than 40 generic kernel features that need per-arch. support:

This is a superb project! ;-)  and not a simple one given that, to be
effective, this requires the prompt collaboration between (the intere-
sted) features maintainers/developers, every architecture maintainers,
and documentation maintainers.

There currently appear to be some mismatches between such doc and the
the actual state of the code (e.g., missing architecture, feature not
existing anymore, status flags out-of-date). Realized this, I started
to patch the doc, but this process became soon tedious (consider also
that I barely know what some of these features are about...).

Hence this post. I am wondering if it would make sense to script some
of these matrices.  So, rather than (or together with) using the cur-
rently hard-coded matrices, try to (automatically) generate them from
the sources/configs. Consider the sketch below (sorry for the raw sh).

diff --git a/Documentation/features/list-arch.sh b/Documentation/features/list-arch.sh
index c16b5b5956889..cdec0c1db9db2 100755
--- a/Documentation/features/list-arch.sh
+++ b/Documentation/features/list-arch.sh
@@ -18,7 +18,13 @@ for F in */*/arch-support.txt; do
   C=$(grep -h "^#         Kconfig:"     $F | cut -c25-)
   D=$(grep -h "^#         description:" $F | cut -c25-)
   S=$(grep -hw $ARCH $F | cut -d\| -f3)
+  myS=$(grep -h $C ../../arch/$ARCH/Kconfig)
+  if [ -z "$myS" ]; then
+	  myS=" -- "
+  else
+	  myS=" ok "
+  fi
 
-  printf "%10s/%-22s:%s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$C" "$D"
+  printf "%10s/%-22s:%s VS. %s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$myS" "$C" "$D"
 done
 

With this diff.,

andrea@andrea:~$ ./Documentation/features/list-arch.sh riscv > /tmp/riscv.txt
grep: asm/rwsem.h: No such file or directory
andrea@andrea:~$ cat /tmp/riscv.txt
#
# Kernel feature support matrix of the 'riscv' architecture:
#
      core/ BPF-JIT              : VS.  -- |                        HAVE_BPF_JIT #  arch supports BPF JIT optimizations
      core/ generic-idle-thread  : VS.  ok |             GENERIC_SMP_IDLE_THREAD #  arch makes use of the generic SMP idle thread facility
      core/ jump-labels          : VS.  -- |                HAVE_ARCH_JUMP_LABEL #  arch supports live patched, high efficiency branches
      core/ tracehook            : VS.  ok |                 HAVE_ARCH_TRACEHOOK #  arch supports tracehook (ptrace) register handling APIs
     debug/ gcov-profile-all     : VS.  -- |           ARCH_HAS_GCOV_PROFILE_ALL #  arch supports whole-kernel GCOV code coverage profiling
     debug/ KASAN                : VS.  -- |                     HAVE_ARCH_KASAN #  arch supports the KASAN runtime memory checker
     debug/ kgdb                 : VS.  -- |                      HAVE_ARCH_KGDB #  arch supports the kGDB kernel debugger
     debug/ kprobes              : VS.  ok |                        HAVE_KPROBES #  arch supports live patched kernel probe
     debug/ kprobes-on-ftrace    : VS.  -- |              HAVE_KPROBES_ON_FTRACE #  arch supports combined kprobes and ftrace live patching
     debug/ kretprobes           : VS.  -- |                     HAVE_KRETPROBES #  arch supports kernel function-return probes
     debug/ optprobes            : VS.  -- |                      HAVE_OPTPROBES #  arch supports live patched optprobes
     debug/ stackprotector       : VS.  -- |              HAVE_CC_STACKPROTECTOR #  arch supports compiler driven stack overflow protection
     debug/ uprobes              : VS.  -- |               ARCH_SUPPORTS_UPROBES #  arch supports live patched user probes
     debug/ user-ret-profiler    : VS.  -- |           HAVE_USER_RETURN_NOTIFIER #  arch supports user-space return from system call profiler
        io/ dma-api-debug        : VS.  ok |                  HAVE_DMA_API_DEBUG #  arch supports DMA debug facilities
        io/ dma-contiguous       : VS.  ok |                 HAVE_DMA_CONTIGUOUS #  arch supports the DMA CMA (continuous memory allocator)
        io/ sg-chain             : VS.  -- |                   ARCH_HAS_SG_CHAIN #  arch supports chained scatter-gather lists
       lib/ strncasecmp          : VS.  -- |             __HAVE_ARCH_STRNCASECMP #  arch provides an optimized strncasecmp() function
   locking/ cmpxchg-local        : VS.  -- |                  HAVE_CMPXCHG_LOCAL #  arch supports the this_cpu_cmpxchg() API
   locking/ lockdep              : VS.  -- |                     LOCKDEP_SUPPORT #  arch supports the runtime locking correctness debug facility
   locking/ queued-rwlocks       : VS.  -- |             ARCH_USE_QUEUED_RWLOCKS #  arch supports queued rwlocks
   locking/ queued-spinlocks     : VS.  -- |           ARCH_USE_QUEUED_SPINLOCKS #  arch supports queued spinlocks
   locking/ rwsem-optimized      : VS.  -- |               Optimized asm/rwsem.h #  arch provides optimized rwsem APIs
      perf/ kprobes-event        : VS.  -- |      HAVE_REGS_AND_STACK_ACCESS_API #  arch supports kprobes with perf events
      perf/ perf-regs            : VS.  -- |                      HAVE_PERF_REGS #  arch supports perf events register access
      perf/ perf-stackdump       : VS.  -- |           HAVE_PERF_USER_STACK_DUMP #  arch supports perf events stack dumps
     sched/ membarrier-sync-core : VS.  -- |       ARCH_HAS_MEMBARRIER_SYNC_CORE #  arch supports core serializing membarrier
     sched/ numa-balancing       : VS.  -- |        ARCH_SUPPORTS_NUMA_BALANCING #  arch supports NUMA balancing
   seccomp/ seccomp-filter       : VS.  -- |            HAVE_ARCH_SECCOMP_FILTER #  arch supports seccomp filters
      time/ arch-tick-broadcast  : VS.  -- |             ARCH_HAS_TICK_BROADCAST #  arch provides tick_broadcast()
      time/ clockevents          : VS.  ok |                 GENERIC_CLOCKEVENTS #  arch support generic clock events
      time/ context-tracking     : VS.  -- |               HAVE_CONTEXT_TRACKING #  arch supports context tracking for NO_HZ_FULL
      time/ irq-time-acct        : VS.  -- |            HAVE_IRQ_TIME_ACCOUNTING #  arch supports precise IRQ time accounting
      time/ modern-timekeeping   : VS.  -- |            !ARCH_USES_GETTIMEOFFSET #  arch does not use arch_gettimeoffset() anymore
      time/ virt-cpuacct         : VS.  -- |            HAVE_VIRT_CPU_ACCOUNTING #  arch supports precise virtual CPU time accounting
        vm/ ELF-ASLR             : VS.  -- |              ARCH_HAS_ELF_RANDOMIZE #  arch randomizes the stack, heap and binary images of ELF binaries
        vm/ huge-vmap            : VS.  -- |                 HAVE_ARCH_HUGE_VMAP #  arch supports the ioremap_pud_enabled() and ioremap_pmd_enabled() VM APIs
        vm/ ioremap_prot         : VS.  -- |                   HAVE_IOREMAP_PROT #  arch has ioremap_prot()
        vm/ numa-memblock        : VS.  ok |              HAVE_MEMBLOCK_NODE_MAP #  arch supports NUMA aware memblocks
        vm/ PG_uncached          : VS.  -- |               ARCH_USES_PG_UNCACHED #  arch supports the PG_uncached page flag
        vm/ pte_special          : VS.  -- |             __HAVE_ARCH_PTE_SPECIAL #  arch supports the pte_special()/pte_mkspecial() VM APIs
        vm/ THP                  : VS.  -- |      HAVE_ARCH_TRANSPARENT_HUGEPAGE #  arch supports transparent hugepages
        vm/ batch-unmap-tlb-flush: VS.  -- |   ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH #  arch supports deferral of TLB flush until multiple pages are unmapped


Suggesting that riscv is not included within the hard-coded matrices.
This also shows a first limitation of the proposed approach, i.e., it
doesn't parse "Optimized asm/rwsem.h" (wants "true" configs). It also
won't find "__HAVE_ARCH_PTE_SPECIAL" (reads Kconfig files) but things
like "!ARCH_USES_GETTIMEOFFSET" could still be handled...

We could try switching to another architectures: none of these should
result in "core/ BPF-JIT" set: the "HAVE_BPF_JIT" config was splitted
into cBPF and eBPF variants more recently. The comparison of the hard
coded status flags with the new/generated flags should also highlight
out-of-date values.

Alternative/Additional help could probably be provided by checkpatch,
say, warn when a patch touches/adds to Kconfig without updating such
documentation...

Thoughts?

  Andrea
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [Question] Documentation/features: More automation/scripting help?
  2018-03-28 12:22 ` Andrea Parri
@ 2018-03-30 11:31   ` Ingo Molnar
  -1 siblings, 0 replies; 4+ messages in thread
From: Ingo Molnar @ 2018-03-30 11:31 UTC (permalink / raw)
  To: Andrea Parri
  Cc: Jonathan Corbet, Andy Whitcroft, Joe Perches, Linus Torvalds,
	linux-doc, linux-kernel, linux-arch


* Andrea Parri <andrea.parri@amarulasolutions.com> wrote:

> Hi all,
> 
> The directory (not yet three years old although, I freely admit, I've
> only recently become aware of it) provides arch. support matrices for
> more than 40 generic kernel features that need per-arch. support:
> 
> This is a superb project! ;-)  and not a simple one given that, to be
> effective, this requires the prompt collaboration between (the intere-
> sted) features maintainers/developers, every architecture maintainers,
> and documentation maintainers.
> 
> There currently appear to be some mismatches between such doc and the
> the actual state of the code (e.g., missing architecture, feature not
> existing anymore, status flags out-of-date). Realized this, I started
> to patch the doc, but this process became soon tedious (consider also
> that I barely know what some of these features are about...).
> 
> Hence this post. I am wondering if it would make sense to script some
> of these matrices.  So, rather than (or together with) using the cur-
> rently hard-coded matrices, try to (automatically) generate them from
> the sources/configs. Consider the sketch below (sorry for the raw sh).
> 
> diff --git a/Documentation/features/list-arch.sh b/Documentation/features/list-arch.sh
> index c16b5b5956889..cdec0c1db9db2 100755
> --- a/Documentation/features/list-arch.sh
> +++ b/Documentation/features/list-arch.sh
> @@ -18,7 +18,13 @@ for F in */*/arch-support.txt; do
>    C=$(grep -h "^#         Kconfig:"     $F | cut -c25-)
>    D=$(grep -h "^#         description:" $F | cut -c25-)
>    S=$(grep -hw $ARCH $F | cut -d\| -f3)
> +  myS=$(grep -h $C ../../arch/$ARCH/Kconfig)
> +  if [ -z "$myS" ]; then
> +	  myS=" -- "
> +  else
> +	  myS=" ok "
> +  fi
>  
> -  printf "%10s/%-22s:%s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$C" "$D"
> +  printf "%10s/%-22s:%s VS. %s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$myS" "$C" "$D"
>  done
>  
> 
> With this diff.,
> 
> andrea@andrea:~$ ./Documentation/features/list-arch.sh riscv > /tmp/riscv.txt
> grep: asm/rwsem.h: No such file or directory
> andrea@andrea:~$ cat /tmp/riscv.txt
> #
> # Kernel feature support matrix of the 'riscv' architecture:
> #
>       core/ BPF-JIT              : VS.  -- |                        HAVE_BPF_JIT #  arch supports BPF JIT optimizations
>       core/ generic-idle-thread  : VS.  ok |             GENERIC_SMP_IDLE_THREAD #  arch makes use of the generic SMP idle thread facility
>       core/ jump-labels          : VS.  -- |                HAVE_ARCH_JUMP_LABEL #  arch supports live patched, high efficiency branches
>       core/ tracehook            : VS.  ok |                 HAVE_ARCH_TRACEHOOK #  arch supports tracehook (ptrace) register handling APIs
>      debug/ gcov-profile-all     : VS.  -- |           ARCH_HAS_GCOV_PROFILE_ALL #  arch supports whole-kernel GCOV code coverage profiling
>      debug/ KASAN                : VS.  -- |                     HAVE_ARCH_KASAN #  arch supports the KASAN runtime memory checker
>      debug/ kgdb                 : VS.  -- |                      HAVE_ARCH_KGDB #  arch supports the kGDB kernel debugger
>      debug/ kprobes              : VS.  ok |                        HAVE_KPROBES #  arch supports live patched kernel probe
>      debug/ kprobes-on-ftrace    : VS.  -- |              HAVE_KPROBES_ON_FTRACE #  arch supports combined kprobes and ftrace live patching
>      debug/ kretprobes           : VS.  -- |                     HAVE_KRETPROBES #  arch supports kernel function-return probes
>      debug/ optprobes            : VS.  -- |                      HAVE_OPTPROBES #  arch supports live patched optprobes
>      debug/ stackprotector       : VS.  -- |              HAVE_CC_STACKPROTECTOR #  arch supports compiler driven stack overflow protection
>      debug/ uprobes              : VS.  -- |               ARCH_SUPPORTS_UPROBES #  arch supports live patched user probes
>      debug/ user-ret-profiler    : VS.  -- |           HAVE_USER_RETURN_NOTIFIER #  arch supports user-space return from system call profiler
>         io/ dma-api-debug        : VS.  ok |                  HAVE_DMA_API_DEBUG #  arch supports DMA debug facilities
>         io/ dma-contiguous       : VS.  ok |                 HAVE_DMA_CONTIGUOUS #  arch supports the DMA CMA (continuous memory allocator)
>         io/ sg-chain             : VS.  -- |                   ARCH_HAS_SG_CHAIN #  arch supports chained scatter-gather lists
>        lib/ strncasecmp          : VS.  -- |             __HAVE_ARCH_STRNCASECMP #  arch provides an optimized strncasecmp() function
>    locking/ cmpxchg-local        : VS.  -- |                  HAVE_CMPXCHG_LOCAL #  arch supports the this_cpu_cmpxchg() API
>    locking/ lockdep              : VS.  -- |                     LOCKDEP_SUPPORT #  arch supports the runtime locking correctness debug facility
>    locking/ queued-rwlocks       : VS.  -- |             ARCH_USE_QUEUED_RWLOCKS #  arch supports queued rwlocks
>    locking/ queued-spinlocks     : VS.  -- |           ARCH_USE_QUEUED_SPINLOCKS #  arch supports queued spinlocks
>    locking/ rwsem-optimized      : VS.  -- |               Optimized asm/rwsem.h #  arch provides optimized rwsem APIs
>       perf/ kprobes-event        : VS.  -- |      HAVE_REGS_AND_STACK_ACCESS_API #  arch supports kprobes with perf events
>       perf/ perf-regs            : VS.  -- |                      HAVE_PERF_REGS #  arch supports perf events register access
>       perf/ perf-stackdump       : VS.  -- |           HAVE_PERF_USER_STACK_DUMP #  arch supports perf events stack dumps
>      sched/ membarrier-sync-core : VS.  -- |       ARCH_HAS_MEMBARRIER_SYNC_CORE #  arch supports core serializing membarrier
>      sched/ numa-balancing       : VS.  -- |        ARCH_SUPPORTS_NUMA_BALANCING #  arch supports NUMA balancing
>    seccomp/ seccomp-filter       : VS.  -- |            HAVE_ARCH_SECCOMP_FILTER #  arch supports seccomp filters
>       time/ arch-tick-broadcast  : VS.  -- |             ARCH_HAS_TICK_BROADCAST #  arch provides tick_broadcast()
>       time/ clockevents          : VS.  ok |                 GENERIC_CLOCKEVENTS #  arch support generic clock events
>       time/ context-tracking     : VS.  -- |               HAVE_CONTEXT_TRACKING #  arch supports context tracking for NO_HZ_FULL
>       time/ irq-time-acct        : VS.  -- |            HAVE_IRQ_TIME_ACCOUNTING #  arch supports precise IRQ time accounting
>       time/ modern-timekeeping   : VS.  -- |            !ARCH_USES_GETTIMEOFFSET #  arch does not use arch_gettimeoffset() anymore
>       time/ virt-cpuacct         : VS.  -- |            HAVE_VIRT_CPU_ACCOUNTING #  arch supports precise virtual CPU time accounting
>         vm/ ELF-ASLR             : VS.  -- |              ARCH_HAS_ELF_RANDOMIZE #  arch randomizes the stack, heap and binary images of ELF binaries
>         vm/ huge-vmap            : VS.  -- |                 HAVE_ARCH_HUGE_VMAP #  arch supports the ioremap_pud_enabled() and ioremap_pmd_enabled() VM APIs
>         vm/ ioremap_prot         : VS.  -- |                   HAVE_IOREMAP_PROT #  arch has ioremap_prot()
>         vm/ numa-memblock        : VS.  ok |              HAVE_MEMBLOCK_NODE_MAP #  arch supports NUMA aware memblocks
>         vm/ PG_uncached          : VS.  -- |               ARCH_USES_PG_UNCACHED #  arch supports the PG_uncached page flag
>         vm/ pte_special          : VS.  -- |             __HAVE_ARCH_PTE_SPECIAL #  arch supports the pte_special()/pte_mkspecial() VM APIs
>         vm/ THP                  : VS.  -- |      HAVE_ARCH_TRANSPARENT_HUGEPAGE #  arch supports transparent hugepages
>         vm/ batch-unmap-tlb-flush: VS.  -- |   ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH #  arch supports deferral of TLB flush until multiple pages are unmapped
> 
> 
> Suggesting that riscv is not included within the hard-coded matrices.
> This also shows a first limitation of the proposed approach, i.e., it
> doesn't parse "Optimized asm/rwsem.h" (wants "true" configs). It also
> won't find "__HAVE_ARCH_PTE_SPECIAL" (reads Kconfig files) but things
> like "!ARCH_USES_GETTIMEOFFSET" could still be handled...
> 
> We could try switching to another architectures: none of these should
> result in "core/ BPF-JIT" set: the "HAVE_BPF_JIT" config was splitted
> into cBPF and eBPF variants more recently. The comparison of the hard
> coded status flags with the new/generated flags should also highlight
> out-of-date values.
> 
> Alternative/Additional help could probably be provided by checkpatch,
> say, warn when a patch touches/adds to Kconfig without updating such
> documentation...
> 
> Thoughts?

Yeah, so I think what should be done instead is to write a script that refreshes 
all the arch-support.txt files in-place.

(And no, I'd not add checkpatch support.)

It's OK for the script to have various quirks for weirdly implemented features and 
exceptions: i.e. basically whenever it gets a feature wrong, we can just tweak the 
script with quirks to make it all work out of box.
 
New features can then be added by basically just creating a header-only 
arch-support.txt file, such as:

  triton:~/tip/Documentation/features> cat foo/bar/arch-support.txt
  #
  # Feature name:          shiny new fubar kernel feature
  #         Kconfig:       ARCH_USE_FUBAR
  #         description:   arch supports the fubar feature
  #

And running Documentation/features/scripts/features-refresh.sh would auto-generate 
the arch support matrix.

Minimal changes to the existing arch-support.txt headers to make them more 
script-friendly would be OK as well I suppose, for example 
locking/rwsem-optimized/arch-support.txt right now has this:

  #         Kconfig:       Optimized asm/rwsem.h

A real Kconfig value could be used here - or even introduced, to make this all 
easier to support. I.e. it's OK to harmonize feature support within the kernel to 
make everything more clearly organized.

But in the end there should only be a single new script:

  Documentation/features/scripts/features-refresh.sh

... which operates on the arch-support.txt files and refreshes them in place, and 
which, after all the refreshes have been committed, should produce an empty
'git diff' result.

This way we soft- decouple the refreshing of the entries from the introduction of 
the features, while still making it all easy to keep sync and to extend.

I don't have the time to implement this feature myself, but if you'd like to try 
such an approach yourself, I can help merge the result upstream via the -tip tree.

Thanks,

	Ingo

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [Question] Documentation/features: More automation/scripting help?
@ 2018-03-30 11:31   ` Ingo Molnar
  0 siblings, 0 replies; 4+ messages in thread
From: Ingo Molnar @ 2018-03-30 11:31 UTC (permalink / raw)
  To: Andrea Parri
  Cc: Jonathan Corbet, Andy Whitcroft, Joe Perches, Linus Torvalds,
	linux-doc, linux-kernel, linux-arch


* Andrea Parri <andrea.parri@amarulasolutions.com> wrote:

> Hi all,
> 
> The directory (not yet three years old although, I freely admit, I've
> only recently become aware of it) provides arch. support matrices for
> more than 40 generic kernel features that need per-arch. support:
> 
> This is a superb project! ;-)  and not a simple one given that, to be
> effective, this requires the prompt collaboration between (the intere-
> sted) features maintainers/developers, every architecture maintainers,
> and documentation maintainers.
> 
> There currently appear to be some mismatches between such doc and the
> the actual state of the code (e.g., missing architecture, feature not
> existing anymore, status flags out-of-date). Realized this, I started
> to patch the doc, but this process became soon tedious (consider also
> that I barely know what some of these features are about...).
> 
> Hence this post. I am wondering if it would make sense to script some
> of these matrices.  So, rather than (or together with) using the cur-
> rently hard-coded matrices, try to (automatically) generate them from
> the sources/configs. Consider the sketch below (sorry for the raw sh).
> 
> diff --git a/Documentation/features/list-arch.sh b/Documentation/features/list-arch.sh
> index c16b5b5956889..cdec0c1db9db2 100755
> --- a/Documentation/features/list-arch.sh
> +++ b/Documentation/features/list-arch.sh
> @@ -18,7 +18,13 @@ for F in */*/arch-support.txt; do
>    C=$(grep -h "^#         Kconfig:"     $F | cut -c25-)
>    D=$(grep -h "^#         description:" $F | cut -c25-)
>    S=$(grep -hw $ARCH $F | cut -d\| -f3)
> +  myS=$(grep -h $C ../../arch/$ARCH/Kconfig)
> +  if [ -z "$myS" ]; then
> +	  myS=" -- "
> +  else
> +	  myS=" ok "
> +  fi
>  
> -  printf "%10s/%-22s:%s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$C" "$D"
> +  printf "%10s/%-22s:%s VS. %s| %35s # %s\n" "$SUBSYS" "$N" "$S" "$myS" "$C" "$D"
>  done
>  
> 
> With this diff.,
> 
> andrea@andrea:~$ ./Documentation/features/list-arch.sh riscv > /tmp/riscv.txt
> grep: asm/rwsem.h: No such file or directory
> andrea@andrea:~$ cat /tmp/riscv.txt
> #
> # Kernel feature support matrix of the 'riscv' architecture:
> #
>       core/ BPF-JIT              : VS.  -- |                        HAVE_BPF_JIT #  arch supports BPF JIT optimizations
>       core/ generic-idle-thread  : VS.  ok |             GENERIC_SMP_IDLE_THREAD #  arch makes use of the generic SMP idle thread facility
>       core/ jump-labels          : VS.  -- |                HAVE_ARCH_JUMP_LABEL #  arch supports live patched, high efficiency branches
>       core/ tracehook            : VS.  ok |                 HAVE_ARCH_TRACEHOOK #  arch supports tracehook (ptrace) register handling APIs
>      debug/ gcov-profile-all     : VS.  -- |           ARCH_HAS_GCOV_PROFILE_ALL #  arch supports whole-kernel GCOV code coverage profiling
>      debug/ KASAN                : VS.  -- |                     HAVE_ARCH_KASAN #  arch supports the KASAN runtime memory checker
>      debug/ kgdb                 : VS.  -- |                      HAVE_ARCH_KGDB #  arch supports the kGDB kernel debugger
>      debug/ kprobes              : VS.  ok |                        HAVE_KPROBES #  arch supports live patched kernel probe
>      debug/ kprobes-on-ftrace    : VS.  -- |              HAVE_KPROBES_ON_FTRACE #  arch supports combined kprobes and ftrace live patching
>      debug/ kretprobes           : VS.  -- |                     HAVE_KRETPROBES #  arch supports kernel function-return probes
>      debug/ optprobes            : VS.  -- |                      HAVE_OPTPROBES #  arch supports live patched optprobes
>      debug/ stackprotector       : VS.  -- |              HAVE_CC_STACKPROTECTOR #  arch supports compiler driven stack overflow protection
>      debug/ uprobes              : VS.  -- |               ARCH_SUPPORTS_UPROBES #  arch supports live patched user probes
>      debug/ user-ret-profiler    : VS.  -- |           HAVE_USER_RETURN_NOTIFIER #  arch supports user-space return from system call profiler
>         io/ dma-api-debug        : VS.  ok |                  HAVE_DMA_API_DEBUG #  arch supports DMA debug facilities
>         io/ dma-contiguous       : VS.  ok |                 HAVE_DMA_CONTIGUOUS #  arch supports the DMA CMA (continuous memory allocator)
>         io/ sg-chain             : VS.  -- |                   ARCH_HAS_SG_CHAIN #  arch supports chained scatter-gather lists
>        lib/ strncasecmp          : VS.  -- |             __HAVE_ARCH_STRNCASECMP #  arch provides an optimized strncasecmp() function
>    locking/ cmpxchg-local        : VS.  -- |                  HAVE_CMPXCHG_LOCAL #  arch supports the this_cpu_cmpxchg() API
>    locking/ lockdep              : VS.  -- |                     LOCKDEP_SUPPORT #  arch supports the runtime locking correctness debug facility
>    locking/ queued-rwlocks       : VS.  -- |             ARCH_USE_QUEUED_RWLOCKS #  arch supports queued rwlocks
>    locking/ queued-spinlocks     : VS.  -- |           ARCH_USE_QUEUED_SPINLOCKS #  arch supports queued spinlocks
>    locking/ rwsem-optimized      : VS.  -- |               Optimized asm/rwsem.h #  arch provides optimized rwsem APIs
>       perf/ kprobes-event        : VS.  -- |      HAVE_REGS_AND_STACK_ACCESS_API #  arch supports kprobes with perf events
>       perf/ perf-regs            : VS.  -- |                      HAVE_PERF_REGS #  arch supports perf events register access
>       perf/ perf-stackdump       : VS.  -- |           HAVE_PERF_USER_STACK_DUMP #  arch supports perf events stack dumps
>      sched/ membarrier-sync-core : VS.  -- |       ARCH_HAS_MEMBARRIER_SYNC_CORE #  arch supports core serializing membarrier
>      sched/ numa-balancing       : VS.  -- |        ARCH_SUPPORTS_NUMA_BALANCING #  arch supports NUMA balancing
>    seccomp/ seccomp-filter       : VS.  -- |            HAVE_ARCH_SECCOMP_FILTER #  arch supports seccomp filters
>       time/ arch-tick-broadcast  : VS.  -- |             ARCH_HAS_TICK_BROADCAST #  arch provides tick_broadcast()
>       time/ clockevents          : VS.  ok |                 GENERIC_CLOCKEVENTS #  arch support generic clock events
>       time/ context-tracking     : VS.  -- |               HAVE_CONTEXT_TRACKING #  arch supports context tracking for NO_HZ_FULL
>       time/ irq-time-acct        : VS.  -- |            HAVE_IRQ_TIME_ACCOUNTING #  arch supports precise IRQ time accounting
>       time/ modern-timekeeping   : VS.  -- |            !ARCH_USES_GETTIMEOFFSET #  arch does not use arch_gettimeoffset() anymore
>       time/ virt-cpuacct         : VS.  -- |            HAVE_VIRT_CPU_ACCOUNTING #  arch supports precise virtual CPU time accounting
>         vm/ ELF-ASLR             : VS.  -- |              ARCH_HAS_ELF_RANDOMIZE #  arch randomizes the stack, heap and binary images of ELF binaries
>         vm/ huge-vmap            : VS.  -- |                 HAVE_ARCH_HUGE_VMAP #  arch supports the ioremap_pud_enabled() and ioremap_pmd_enabled() VM APIs
>         vm/ ioremap_prot         : VS.  -- |                   HAVE_IOREMAP_PROT #  arch has ioremap_prot()
>         vm/ numa-memblock        : VS.  ok |              HAVE_MEMBLOCK_NODE_MAP #  arch supports NUMA aware memblocks
>         vm/ PG_uncached          : VS.  -- |               ARCH_USES_PG_UNCACHED #  arch supports the PG_uncached page flag
>         vm/ pte_special          : VS.  -- |             __HAVE_ARCH_PTE_SPECIAL #  arch supports the pte_special()/pte_mkspecial() VM APIs
>         vm/ THP                  : VS.  -- |      HAVE_ARCH_TRANSPARENT_HUGEPAGE #  arch supports transparent hugepages
>         vm/ batch-unmap-tlb-flush: VS.  -- |   ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH #  arch supports deferral of TLB flush until multiple pages are unmapped
> 
> 
> Suggesting that riscv is not included within the hard-coded matrices.
> This also shows a first limitation of the proposed approach, i.e., it
> doesn't parse "Optimized asm/rwsem.h" (wants "true" configs). It also
> won't find "__HAVE_ARCH_PTE_SPECIAL" (reads Kconfig files) but things
> like "!ARCH_USES_GETTIMEOFFSET" could still be handled...
> 
> We could try switching to another architectures: none of these should
> result in "core/ BPF-JIT" set: the "HAVE_BPF_JIT" config was splitted
> into cBPF and eBPF variants more recently. The comparison of the hard
> coded status flags with the new/generated flags should also highlight
> out-of-date values.
> 
> Alternative/Additional help could probably be provided by checkpatch,
> say, warn when a patch touches/adds to Kconfig without updating such
> documentation...
> 
> Thoughts?

Yeah, so I think what should be done instead is to write a script that refreshes 
all the arch-support.txt files in-place.

(And no, I'd not add checkpatch support.)

It's OK for the script to have various quirks for weirdly implemented features and 
exceptions: i.e. basically whenever it gets a feature wrong, we can just tweak the 
script with quirks to make it all work out of box.
 
New features can then be added by basically just creating a header-only 
arch-support.txt file, such as:

  triton:~/tip/Documentation/features> cat foo/bar/arch-support.txt
  #
  # Feature name:          shiny new fubar kernel feature
  #         Kconfig:       ARCH_USE_FUBAR
  #         description:   arch supports the fubar feature
  #

And running Documentation/features/scripts/features-refresh.sh would auto-generate 
the arch support matrix.

Minimal changes to the existing arch-support.txt headers to make them more 
script-friendly would be OK as well I suppose, for example 
locking/rwsem-optimized/arch-support.txt right now has this:

  #         Kconfig:       Optimized asm/rwsem.h

A real Kconfig value could be used here - or even introduced, to make this all 
easier to support. I.e. it's OK to harmonize feature support within the kernel to 
make everything more clearly organized.

But in the end there should only be a single new script:

  Documentation/features/scripts/features-refresh.sh

... which operates on the arch-support.txt files and refreshes them in place, and 
which, after all the refreshes have been committed, should produce an empty
'git diff' result.

This way we soft- decouple the refreshing of the entries from the introduction of 
the features, while still making it all easy to keep sync and to extend.

I don't have the time to implement this feature myself, but if you'd like to try 
such an approach yourself, I can help merge the result upstream via the -tip tree.

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2018-03-30 11:31 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-28 12:22 [Question] Documentation/features: More automation/scripting help? Andrea Parri
2018-03-28 12:22 ` Andrea Parri
2018-03-30 11:31 ` Ingo Molnar
2018-03-30 11:31   ` Ingo Molnar

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.