* [GIT PULL rcu/next] RCU commits for 4.21/5.0
@ 2018-12-03 22:54 Paul E. McKenney
2018-12-04 8:08 ` Ingo Molnar
0 siblings, 1 reply; 4+ messages in thread
From: Paul E. McKenney @ 2018-12-03 22:54 UTC (permalink / raw)
To: mingo
Cc: linux-kernel, danymadden, Dennis.Krein, joe, joel, ldr709,
pierceagriffiths, rdunlap, w, zhouzhouyi, connor.shu
Hello, Ingo,
This pull request contains the following changes:
1. Convert RCU's BUG_ON() and similar calls to WARN_ON() and similar.
http://lkml.kernel.org/r/20181111193156.GA3666@linux.ibm.com
2. Replace of calls to RCU-bh and RCU-sched update-side functions
to their vanilla RCU counterparts. This series is a step
towards complete removal of the RCU-bh and RCU-sched update-side
functions.
http://lkml.kernel.org/r/20181111194104.GA4787@linux.ibm.com
Note that several of the patches sent out have been dropped from
this series due to having been pulled in by their respective
maintainers.
3. Documentation updates, including a number of flavor-consolidation
updates from Joel Fernandes.
http://lkml.kernel.org/r/20181111195619.GA6958@linux.ibm.com
4. Miscellaneous fixes.
http://lkml.kernel.org/r/20181111192839.GA32144@linux.ibm.com
5. Automate generation of the initrd filesystem used for
rcutorture testing.
http://lkml.kernel.org/r/20181111200127.GA9511@linux.ibm.com
6. Convert spin_is_locked() assertions to instead use lockdep.
http://lkml.kernel.org/r/20181111200421.GA10551@linux.ibm.com
Note that several of the patches sent out have been dropped from
this series due to having been pulled in by their respective
maintainers.
7. SRCU updates, especially including a fix from Dennis Krein
for a bag-on-head-class bug.
http://lkml.kernel.org/r/20181111200834.GA10983@linux.ibm.com
8. RCU torture-test updates.
http://lkml.kernel.org/r/20181111201956.GA11935@linux.ibm.com
All of these changes have been subjected to 0day Test Robot and -next
testing, and are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git for-mingo
for you to fetch changes up to 5ac7cdc29897e5fc3f5e214f3f8c8b03ef8d7029:
rcutorture: Don't do busted forward-progress testing (2018-12-01 12:45:42 -0800)
----------------------------------------------------------------
Connor Shu (1):
rcutorture: Automatically create initrd directory
Dennis Krein (1):
srcu: Lock srcu_data structure in srcu_gp_start()
Joe Perches (1):
checkpatch: Create table of obsolete APIs and apply to RCU
Joel Fernandes (Google) (19):
rcu: Remove unused rcu_state externs
rcu: Fix rcu_{node,data} comments about gp_seq_needed
doc: Clarify RCU data-structure comment about rcu_tree fanout
doc: Remove rcu_preempt_state reference in stallwarn
doc: Update information about resched_cpu
doc: Remove rcu_dynticks from Data-Structures
doc: rcu: Update Data-Structures for RCU flavor consolidation
doc: rcu: Better clarify the rcu_segcblist ->len field
doc: rcu: Update description of gp_seq fields in rcu_data
doc: rcu: Update core and full API in whatisRCU
doc: rcu: Add more rationale for using rcu_read_lock_sched in checklist
doc: rcu: Remove obsolete suggestion from checklist
doc: rcu: Remove obsolete checklist item about synchronize_rcu usage
doc: rcu: Encourage use of rcu_barrier in checklist
doc: Make reader aware of rcu_dereference_protected
doc: Remove obsolete (non-)requirement about disabling preemption
doc: Make listing in RCU perf/scale requirements use rcu_assign_pointer()
doc: Correct parameter in stallwarn
doc: Fix "struction" typo in RCU memory-ordering documentation
Lance Roy (7):
x86/PCI: Replace spin_is_locked() with lockdep
sfc: Replace spin_is_locked() with lockdep
smsc: Replace spin_is_locked() with lockdep
userfaultfd: Replace spin_is_locked() with lockdep
locking/mutex: Replace spin_is_locked() with lockdep
mm: Replace spin_is_locked() with lockdep
KVM: arm/arm64: vgic: Replace spin_is_locked() with lockdep
Paul E. McKenney (77):
rcu: Eliminate BUG_ON() for sync.c
rcu: Eliminate BUG_ON() for kernel/rcu/tree.c
rcu: Eliminate synchronize_rcu_mult()
rcu: Consolidate the RCU update functions invoked by sync.c
sched/membarrier: Replace synchronize_sched() with synchronize_rcu()
sparc/oprofile: Convert timer_stop() to use synchronize_rcu()
s390/mm: Convert tlb_table_flush() to use call_rcu()
powerpc: Convert hugepd_free() to use call_rcu()
doc: Set down forward-progress requirements
rcutorture: Add initrd support for systems lacking dracut
rcutorture: Make initrd/init execute in userspace
rcutorture: Add cross-compile capability to initrd.sh
srcu: Prevent __call_srcu() counter wrap with read-side critical section
rcu: Stop expedited grace periods from relying on stop-machine
rcu: Eliminate BUG_ON() for kernel/rcu/tree_plugin.h
rcu: Eliminate BUG_ON() for kernel/rcu/update.c
doc: Document rcutorture forward-progress test kernel parameters
doc: RCU scheduler spinlock rcu_read_unlock() restriction remains
MAINTAINERS: Update from @linux.vnet.ibm.com to @linux.ibm.com
rcu: Avoid double multiply by HZ
rcu: Parameterize rcu_check_gp_start_stall()
rcu: Add state name to show_rcu_gp_kthreads() output
rcu: Add jiffies-since-GP-activity to show_rcu_gp_kthreads()
rcu: Trace end of grace period before end of grace period
rcu: Speed up expedited GPs when interrupting RCU reader
rcu: Replace this_cpu_ptr() with __this_cpu_read()
rcu: Avoid signed integer overflow in rcu_preempt_deferred_qs()
MAINTAINERS: Add Joel Fernandes as RCU reviewer
checkpatch.pl: Suggest lockdep instead of asserting !spin_is_locked()
crypto/pcrypt: Replace synchronize_rcu_bh() with synchronize_rcu()
drivers/ipmi: Replace synchronize_sched() with synchronize_rcu()
ethernet/sis: Replace synchronize_sched() with synchronize_rcu()
ethernet/realtek: Replace synchronize_sched() with synchronize_rcu()
drivers/vhost: Replace synchronize_rcu_bh() with synchronize_rcu()
cpufreq/intel_pstate: Replace synchronize_sched() with synchronize_rcu()
cpufreq/cpufreq_governor: Replace synchronize_sched() with synchronize_rcu()
fs/file: Replace synchronize_sched() with synchronize_rcu()
tracing: Replace synchronize_sched() and call_rcu_sched()
main: Replace rcu_barrier_sched() with rcu_barrier()
kprobes: Replace synchronize_sched() with synchronize_rcu()
lockdep: Replace synchronize_sched() with synchronize_rcu()
sched/membarrier: synchronize_sched() with synchronize_rcu()
modules: Replace synchronize_sched() and call_rcu_sched()
workqueue: Replace call_rcu_sched() with call_rcu()
events: Replace synchronize_sched() with synchronize_rcu()
percpu-refcount: Replace call_rcu_sched() with call_rcu()
slab: Replace synchronize_sched() with synchronize_rcu()
mm: Replace call_rcu_sched() with call_rcu()
srcu: Use "ssp" instead of "sp" for srcu_struct pointer
net/sched: Replace call_rcu_bh() and rcu_barrier_bh()
net/core: Replace call_rcu_bh() and synchronize_rcu_bh()
net/bridge: Replace call_rcu_bh() and rcu_barrier_bh()
percpu-rwsem: Replace synchronize_sched() with synchronize_rcu()
types: Remove call_rcu_bh() and call_rcu_sched()
cgroups: Replace synchronize_sched() with synchronize_rcu()
livepatch: Replace synchronize_sched() with synchronize_rcu()
net/core/skmsg: Replace call_rcu_sched() with call_rcu()
net/decnet: Replace rcu_barrier_bh() with rcu_barrier()
tools/kernel.h: Replace synchronize_sched() with synchronize_rcu()
rcutorture/formal: Replace synchronize_sched() with synchronize_rcu()
Merge branches 'bug.2018.11.12a', 'consolidate.2018.12.01a', 'doc.2018.11.12a', 'fixes.2018.11.12a', 'initrd.2018.11.08b', 'sil.2018.11.12a' and 'srcu.2018.11.27a' into HEAD
rcutorture: Add call_rcu() flooding forward-progress tests
torture: Bring any extra CPUs online during kernel startup
rcutorture: Remove cbflood facility
rcutorture: Break up too-long rcu_torture_fwd_prog() function
rcutorture: Affinity forward-progress test to avoid housekeeping CPUs
rcutorture: Prepare for asynchronous access to rcu_fwd_startat
rcutorture: Dump grace-period diagnostics upon forward-progress OOM
rcu: Account for nocb-CPU callback counts in RCU CPU stall warnings
rcu: Print per-CPU callback counts for forward-progress failures
rcutorture: Print GP age upon forward-progress failure
rcutorture: Print histogram of CB invocation at OOM time
rcutorture: Print time since GP end upon forward-progress failure
rcutorture: Print forward-progress test age upon failure
rcutorture: Recover from OOM during forward-progress tests
rcutorture: Use 100ms buckets for forward-progress callback histograms
rcutorture: Don't do busted forward-progress testing
Pierce Griffiths (1):
torture: Remove unnecessary "ret" variables
Randy Dunlap (1):
srcu: Fix kernel-doc missing notation
Willy Tarreau (4):
rcutorture: Always strip using the cross-compiler
rcutorture: Check initrd/init instead of initrd only
rcutorture: Import a copy of nolibc
rcutorture: Make use of nolibc when available
Zhouyi Zhou (1):
rcu: Adjust the comment of function rcu_is_watching
.../Design/Data-Structures/BigTreeClassicRCUBH.svg | 499 -----
.../Data-Structures/BigTreeClassicRCUBHdyntick.svg | 695 -------
.../Data-Structures/BigTreePreemptRCUBHdyntick.svg | 741 -------
.../BigTreePreemptRCUBHdyntickCB.svg | 834 +++-----
.../Design/Data-Structures/Data-Structures.html | 173 +-
.../RCU/Design/Data-Structures/blkd_task.svg | 676 +++---
.../Expedited-Grace-Periods.html | 6 +-
.../Memory-Ordering/Tree-RCU-Memory-Ordering.html | 2 +-
.../RCU/Design/Requirements/Requirements.html | 206 +-
Documentation/RCU/checklist.txt | 49 +-
Documentation/RCU/stallwarn.txt | 7 +-
Documentation/RCU/whatisRCU.txt | 70 +-
Documentation/admin-guide/kernel-parameters.txt | 35 +-
MAINTAINERS | 73 +-
arch/powerpc/mm/hugetlbpage.c | 2 +-
arch/s390/mm/pgalloc.c | 2 +-
arch/sparc/oprofile/init.c | 2 +-
arch/x86/pci/i386.c | 2 +-
crypto/pcrypt.c | 2 +-
drivers/char/ipmi/ipmi_si_intf.c | 2 +-
drivers/cpufreq/cpufreq_governor.c | 2 +-
drivers/cpufreq/intel_pstate.c | 2 +-
drivers/net/ethernet/realtek/8139too.c | 2 +-
drivers/net/ethernet/realtek/r8169.c | 4 +-
drivers/net/ethernet/sfc/efx.c | 2 +-
drivers/net/ethernet/sis/sis190.c | 2 +-
drivers/net/ethernet/smsc/smsc911x.h | 2 +-
drivers/vhost/net.c | 2 +-
fs/file.c | 2 +-
fs/userfaultfd.c | 2 +-
include/linux/percpu-rwsem.h | 2 +-
include/linux/rcupdate_wait.h | 17 -
include/linux/sched.h | 4 +-
include/linux/srcu.h | 79 +-
include/linux/srcutiny.h | 24 +-
include/linux/srcutree.h | 8 +-
include/linux/tracepoint.h | 2 +-
include/linux/types.h | 4 +-
init/main.c | 6 +-
kernel/cgroup/cgroup.c | 2 +-
kernel/events/core.c | 2 +-
kernel/kprobes.c | 10 +-
kernel/livepatch/patch.c | 4 +-
kernel/livepatch/transition.c | 4 +-
kernel/locking/lockdep.c | 2 +-
kernel/locking/mutex-debug.c | 4 +-
kernel/module.c | 14 +-
kernel/rcu/rcu.h | 4 +
kernel/rcu/rcutorture.c | 389 ++--
kernel/rcu/srcutiny.c | 120 +-
kernel/rcu/srcutree.c | 489 ++---
kernel/rcu/sync.c | 25 +-
kernel/rcu/tree.c | 114 +-
kernel/rcu/tree.h | 18 +-
kernel/rcu/tree_exp.h | 10 +-
kernel/rcu/tree_plugin.h | 81 +-
kernel/rcu/update.c | 9 +-
kernel/sched/core.c | 2 +-
kernel/sched/membarrier.c | 6 +-
kernel/torture.c | 34 +-
kernel/trace/ftrace.c | 24 +-
kernel/trace/ring_buffer.c | 12 +-
kernel/trace/trace.c | 10 +-
kernel/trace/trace_events_filter.c | 4 +-
kernel/trace/trace_kprobe.c | 2 +-
kernel/tracepoint.c | 4 +-
kernel/workqueue.c | 8 +-
lib/percpu-refcount.c | 2 +-
mm/khugepaged.c | 4 +-
mm/mmu_gather.c | 2 +-
mm/slab.c | 4 +-
mm/slab_common.c | 6 +-
mm/swap.c | 3 +-
net/bridge/br_mdb.c | 2 +-
net/bridge/br_multicast.c | 14 +-
net/core/netpoll.c | 4 +-
net/core/skmsg.c | 2 +-
net/decnet/af_decnet.c | 2 +-
net/sched/sch_api.c | 2 +-
net/sched/sch_generic.c | 8 +-
scripts/checkpatch.pl | 35 +
tools/include/linux/kernel.h | 2 +-
tools/testing/selftests/rcutorture/bin/kvm.sh | 8 +
tools/testing/selftests/rcutorture/bin/mkinitrd.sh | 136 ++
tools/testing/selftests/rcutorture/bin/nolibc.h | 2197 ++++++++++++++++++++
tools/testing/selftests/rcutorture/doc/initrd.txt | 99 +-
.../formal/srcu-cbmc/include/linux/types.h | 4 +-
virt/kvm/arm/vgic/vgic.c | 12 +-
88 files changed, 4187 insertions(+), 4014 deletions(-)
delete mode 100644 Documentation/RCU/Design/Data-Structures/BigTreeClassicRCUBH.svg
delete mode 100644 Documentation/RCU/Design/Data-Structures/BigTreeClassicRCUBHdyntick.svg
delete mode 100644 Documentation/RCU/Design/Data-Structures/BigTreePreemptRCUBHdyntick.svg
create mode 100755 tools/testing/selftests/rcutorture/bin/mkinitrd.sh
create mode 100644 tools/testing/selftests/rcutorture/bin/nolibc.h
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [GIT PULL rcu/next] RCU commits for 4.21/5.0
2018-12-03 22:54 [GIT PULL rcu/next] RCU commits for 4.21/5.0 Paul E. McKenney
@ 2018-12-04 8:08 ` Ingo Molnar
2018-12-04 13:38 ` Willy Tarreau
0 siblings, 1 reply; 4+ messages in thread
From: Ingo Molnar @ 2018-12-04 8:08 UTC (permalink / raw)
To: Paul E. McKenney, Arnaldo Carvalho de Melo, Linus Torvalds,
Willy Tarreau, H. Peter Anvin
Cc: linux-kernel, danymadden, Dennis.Krein, joe, joel, ldr709,
pierceagriffiths, rdunlap, w, zhouzhouyi, connor.shu
[ Added Linus and Arnaldo to the Cc:, for high level kernel side and
tooling side buy-in for the <nolibc.h> inclusion in the kernel tree
below: ]
* Paul E. McKenney <paulmck@linux.ibm.com> wrote:
> Hello, Ingo,
>
> This pull request contains the following changes:
>
> 1. Convert RCU's BUG_ON() and similar calls to WARN_ON() and similar.
>
> http://lkml.kernel.org/r/20181111193156.GA3666@linux.ibm.com
>
> 2. Replace of calls to RCU-bh and RCU-sched update-side functions
> to their vanilla RCU counterparts. This series is a step
> towards complete removal of the RCU-bh and RCU-sched update-side
> functions.
>
> http://lkml.kernel.org/r/20181111194104.GA4787@linux.ibm.com
>
> Note that several of the patches sent out have been dropped from
> this series due to having been pulled in by their respective
> maintainers.
>
> 3. Documentation updates, including a number of flavor-consolidation
> updates from Joel Fernandes.
>
> http://lkml.kernel.org/r/20181111195619.GA6958@linux.ibm.com
>
> 4. Miscellaneous fixes.
>
> http://lkml.kernel.org/r/20181111192839.GA32144@linux.ibm.com
>
> 5. Automate generation of the initrd filesystem used for
> rcutorture testing.
>
> http://lkml.kernel.org/r/20181111200127.GA9511@linux.ibm.com
>
> 6. Convert spin_is_locked() assertions to instead use lockdep.
>
> http://lkml.kernel.org/r/20181111200421.GA10551@linux.ibm.com
>
> Note that several of the patches sent out have been dropped from
> this series due to having been pulled in by their respective
> maintainers.
>
> 7. SRCU updates, especially including a fix from Dennis Krein
> for a bag-on-head-class bug.
>
> http://lkml.kernel.org/r/20181111200834.GA10983@linux.ibm.com
>
> 8. RCU torture-test updates.
>
> http://lkml.kernel.org/r/20181111201956.GA11935@linux.ibm.com
>
> All of these changes have been subjected to 0day Test Robot and -next
> testing, and are available in the git repository at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git for-mingo
>
> for you to fetch changes up to 5ac7cdc29897e5fc3f5e214f3f8c8b03ef8d7029:
>
> rcutorture: Don't do busted forward-progress testing (2018-12-01 12:45:42 -0800)
>
> ----------------------------------------------------------------
> Connor Shu (1):
> rcutorture: Automatically create initrd directory
>
> Dennis Krein (1):
> srcu: Lock srcu_data structure in srcu_gp_start()
>
> Joe Perches (1):
> checkpatch: Create table of obsolete APIs and apply to RCU
>
> Joel Fernandes (Google) (19):
> rcu: Remove unused rcu_state externs
> rcu: Fix rcu_{node,data} comments about gp_seq_needed
> doc: Clarify RCU data-structure comment about rcu_tree fanout
> doc: Remove rcu_preempt_state reference in stallwarn
> doc: Update information about resched_cpu
> doc: Remove rcu_dynticks from Data-Structures
> doc: rcu: Update Data-Structures for RCU flavor consolidation
> doc: rcu: Better clarify the rcu_segcblist ->len field
> doc: rcu: Update description of gp_seq fields in rcu_data
> doc: rcu: Update core and full API in whatisRCU
> doc: rcu: Add more rationale for using rcu_read_lock_sched in checklist
> doc: rcu: Remove obsolete suggestion from checklist
> doc: rcu: Remove obsolete checklist item about synchronize_rcu usage
> doc: rcu: Encourage use of rcu_barrier in checklist
> doc: Make reader aware of rcu_dereference_protected
> doc: Remove obsolete (non-)requirement about disabling preemption
> doc: Make listing in RCU perf/scale requirements use rcu_assign_pointer()
> doc: Correct parameter in stallwarn
> doc: Fix "struction" typo in RCU memory-ordering documentation
>
> Lance Roy (7):
> x86/PCI: Replace spin_is_locked() with lockdep
> sfc: Replace spin_is_locked() with lockdep
> smsc: Replace spin_is_locked() with lockdep
> userfaultfd: Replace spin_is_locked() with lockdep
> locking/mutex: Replace spin_is_locked() with lockdep
> mm: Replace spin_is_locked() with lockdep
> KVM: arm/arm64: vgic: Replace spin_is_locked() with lockdep
>
> Paul E. McKenney (77):
> rcu: Eliminate BUG_ON() for sync.c
> rcu: Eliminate BUG_ON() for kernel/rcu/tree.c
> rcu: Eliminate synchronize_rcu_mult()
> rcu: Consolidate the RCU update functions invoked by sync.c
> sched/membarrier: Replace synchronize_sched() with synchronize_rcu()
> sparc/oprofile: Convert timer_stop() to use synchronize_rcu()
> s390/mm: Convert tlb_table_flush() to use call_rcu()
> powerpc: Convert hugepd_free() to use call_rcu()
> doc: Set down forward-progress requirements
> rcutorture: Add initrd support for systems lacking dracut
> rcutorture: Make initrd/init execute in userspace
> rcutorture: Add cross-compile capability to initrd.sh
> srcu: Prevent __call_srcu() counter wrap with read-side critical section
> rcu: Stop expedited grace periods from relying on stop-machine
> rcu: Eliminate BUG_ON() for kernel/rcu/tree_plugin.h
> rcu: Eliminate BUG_ON() for kernel/rcu/update.c
> doc: Document rcutorture forward-progress test kernel parameters
> doc: RCU scheduler spinlock rcu_read_unlock() restriction remains
> MAINTAINERS: Update from @linux.vnet.ibm.com to @linux.ibm.com
> rcu: Avoid double multiply by HZ
> rcu: Parameterize rcu_check_gp_start_stall()
> rcu: Add state name to show_rcu_gp_kthreads() output
> rcu: Add jiffies-since-GP-activity to show_rcu_gp_kthreads()
> rcu: Trace end of grace period before end of grace period
> rcu: Speed up expedited GPs when interrupting RCU reader
> rcu: Replace this_cpu_ptr() with __this_cpu_read()
> rcu: Avoid signed integer overflow in rcu_preempt_deferred_qs()
> MAINTAINERS: Add Joel Fernandes as RCU reviewer
> checkpatch.pl: Suggest lockdep instead of asserting !spin_is_locked()
> crypto/pcrypt: Replace synchronize_rcu_bh() with synchronize_rcu()
> drivers/ipmi: Replace synchronize_sched() with synchronize_rcu()
> ethernet/sis: Replace synchronize_sched() with synchronize_rcu()
> ethernet/realtek: Replace synchronize_sched() with synchronize_rcu()
> drivers/vhost: Replace synchronize_rcu_bh() with synchronize_rcu()
> cpufreq/intel_pstate: Replace synchronize_sched() with synchronize_rcu()
> cpufreq/cpufreq_governor: Replace synchronize_sched() with synchronize_rcu()
> fs/file: Replace synchronize_sched() with synchronize_rcu()
> tracing: Replace synchronize_sched() and call_rcu_sched()
> main: Replace rcu_barrier_sched() with rcu_barrier()
> kprobes: Replace synchronize_sched() with synchronize_rcu()
> lockdep: Replace synchronize_sched() with synchronize_rcu()
> sched/membarrier: synchronize_sched() with synchronize_rcu()
> modules: Replace synchronize_sched() and call_rcu_sched()
> workqueue: Replace call_rcu_sched() with call_rcu()
> events: Replace synchronize_sched() with synchronize_rcu()
> percpu-refcount: Replace call_rcu_sched() with call_rcu()
> slab: Replace synchronize_sched() with synchronize_rcu()
> mm: Replace call_rcu_sched() with call_rcu()
> srcu: Use "ssp" instead of "sp" for srcu_struct pointer
> net/sched: Replace call_rcu_bh() and rcu_barrier_bh()
> net/core: Replace call_rcu_bh() and synchronize_rcu_bh()
> net/bridge: Replace call_rcu_bh() and rcu_barrier_bh()
> percpu-rwsem: Replace synchronize_sched() with synchronize_rcu()
> types: Remove call_rcu_bh() and call_rcu_sched()
> cgroups: Replace synchronize_sched() with synchronize_rcu()
> livepatch: Replace synchronize_sched() with synchronize_rcu()
> net/core/skmsg: Replace call_rcu_sched() with call_rcu()
> net/decnet: Replace rcu_barrier_bh() with rcu_barrier()
> tools/kernel.h: Replace synchronize_sched() with synchronize_rcu()
> rcutorture/formal: Replace synchronize_sched() with synchronize_rcu()
> Merge branches 'bug.2018.11.12a', 'consolidate.2018.12.01a', 'doc.2018.11.12a', 'fixes.2018.11.12a', 'initrd.2018.11.08b', 'sil.2018.11.12a' and 'srcu.2018.11.27a' into HEAD
> rcutorture: Add call_rcu() flooding forward-progress tests
> torture: Bring any extra CPUs online during kernel startup
> rcutorture: Remove cbflood facility
> rcutorture: Break up too-long rcu_torture_fwd_prog() function
> rcutorture: Affinity forward-progress test to avoid housekeeping CPUs
> rcutorture: Prepare for asynchronous access to rcu_fwd_startat
> rcutorture: Dump grace-period diagnostics upon forward-progress OOM
> rcu: Account for nocb-CPU callback counts in RCU CPU stall warnings
> rcu: Print per-CPU callback counts for forward-progress failures
> rcutorture: Print GP age upon forward-progress failure
> rcutorture: Print histogram of CB invocation at OOM time
> rcutorture: Print time since GP end upon forward-progress failure
> rcutorture: Print forward-progress test age upon failure
> rcutorture: Recover from OOM during forward-progress tests
> rcutorture: Use 100ms buckets for forward-progress callback histograms
> rcutorture: Don't do busted forward-progress testing
>
> Pierce Griffiths (1):
> torture: Remove unnecessary "ret" variables
>
> Randy Dunlap (1):
> srcu: Fix kernel-doc missing notation
>
> Willy Tarreau (4):
> rcutorture: Always strip using the cross-compiler
> rcutorture: Check initrd/init instead of initrd only
> rcutorture: Import a copy of nolibc
> rcutorture: Make use of nolibc when available
>
> Zhouyi Zhou (1):
> rcu: Adjust the comment of function rcu_is_watching
>
> .../Design/Data-Structures/BigTreeClassicRCUBH.svg | 499 -----
> .../Data-Structures/BigTreeClassicRCUBHdyntick.svg | 695 -------
> .../Data-Structures/BigTreePreemptRCUBHdyntick.svg | 741 -------
> .../BigTreePreemptRCUBHdyntickCB.svg | 834 +++-----
> .../Design/Data-Structures/Data-Structures.html | 173 +-
> .../RCU/Design/Data-Structures/blkd_task.svg | 676 +++---
> .../Expedited-Grace-Periods.html | 6 +-
> .../Memory-Ordering/Tree-RCU-Memory-Ordering.html | 2 +-
> .../RCU/Design/Requirements/Requirements.html | 206 +-
> Documentation/RCU/checklist.txt | 49 +-
> Documentation/RCU/stallwarn.txt | 7 +-
> Documentation/RCU/whatisRCU.txt | 70 +-
> Documentation/admin-guide/kernel-parameters.txt | 35 +-
> MAINTAINERS | 73 +-
> arch/powerpc/mm/hugetlbpage.c | 2 +-
> arch/s390/mm/pgalloc.c | 2 +-
> arch/sparc/oprofile/init.c | 2 +-
> arch/x86/pci/i386.c | 2 +-
> crypto/pcrypt.c | 2 +-
> drivers/char/ipmi/ipmi_si_intf.c | 2 +-
> drivers/cpufreq/cpufreq_governor.c | 2 +-
> drivers/cpufreq/intel_pstate.c | 2 +-
> drivers/net/ethernet/realtek/8139too.c | 2 +-
> drivers/net/ethernet/realtek/r8169.c | 4 +-
> drivers/net/ethernet/sfc/efx.c | 2 +-
> drivers/net/ethernet/sis/sis190.c | 2 +-
> drivers/net/ethernet/smsc/smsc911x.h | 2 +-
> drivers/vhost/net.c | 2 +-
> fs/file.c | 2 +-
> fs/userfaultfd.c | 2 +-
> include/linux/percpu-rwsem.h | 2 +-
> include/linux/rcupdate_wait.h | 17 -
> include/linux/sched.h | 4 +-
> include/linux/srcu.h | 79 +-
> include/linux/srcutiny.h | 24 +-
> include/linux/srcutree.h | 8 +-
> include/linux/tracepoint.h | 2 +-
> include/linux/types.h | 4 +-
> init/main.c | 6 +-
> kernel/cgroup/cgroup.c | 2 +-
> kernel/events/core.c | 2 +-
> kernel/kprobes.c | 10 +-
> kernel/livepatch/patch.c | 4 +-
> kernel/livepatch/transition.c | 4 +-
> kernel/locking/lockdep.c | 2 +-
> kernel/locking/mutex-debug.c | 4 +-
> kernel/module.c | 14 +-
> kernel/rcu/rcu.h | 4 +
> kernel/rcu/rcutorture.c | 389 ++--
> kernel/rcu/srcutiny.c | 120 +-
> kernel/rcu/srcutree.c | 489 ++---
> kernel/rcu/sync.c | 25 +-
> kernel/rcu/tree.c | 114 +-
> kernel/rcu/tree.h | 18 +-
> kernel/rcu/tree_exp.h | 10 +-
> kernel/rcu/tree_plugin.h | 81 +-
> kernel/rcu/update.c | 9 +-
> kernel/sched/core.c | 2 +-
> kernel/sched/membarrier.c | 6 +-
> kernel/torture.c | 34 +-
> kernel/trace/ftrace.c | 24 +-
> kernel/trace/ring_buffer.c | 12 +-
> kernel/trace/trace.c | 10 +-
> kernel/trace/trace_events_filter.c | 4 +-
> kernel/trace/trace_kprobe.c | 2 +-
> kernel/tracepoint.c | 4 +-
> kernel/workqueue.c | 8 +-
> lib/percpu-refcount.c | 2 +-
> mm/khugepaged.c | 4 +-
> mm/mmu_gather.c | 2 +-
> mm/slab.c | 4 +-
> mm/slab_common.c | 6 +-
> mm/swap.c | 3 +-
> net/bridge/br_mdb.c | 2 +-
> net/bridge/br_multicast.c | 14 +-
> net/core/netpoll.c | 4 +-
> net/core/skmsg.c | 2 +-
> net/decnet/af_decnet.c | 2 +-
> net/sched/sch_api.c | 2 +-
> net/sched/sch_generic.c | 8 +-
> scripts/checkpatch.pl | 35 +
> tools/include/linux/kernel.h | 2 +-
> tools/testing/selftests/rcutorture/bin/kvm.sh | 8 +
> tools/testing/selftests/rcutorture/bin/mkinitrd.sh | 136 ++
> tools/testing/selftests/rcutorture/bin/nolibc.h | 2197 ++++++++++++++++++++
> tools/testing/selftests/rcutorture/doc/initrd.txt | 99 +-
> .../formal/srcu-cbmc/include/linux/types.h | 4 +-
> virt/kvm/arm/vgic/vgic.c | 12 +-
> 88 files changed, 4187 insertions(+), 4014 deletions(-)
> delete mode 100644 Documentation/RCU/Design/Data-Structures/BigTreeClassicRCUBH.svg
> delete mode 100644 Documentation/RCU/Design/Data-Structures/BigTreeClassicRCUBHdyntick.svg
> delete mode 100644 Documentation/RCU/Design/Data-Structures/BigTreePreemptRCUBHdyntick.svg
> create mode 100755 tools/testing/selftests/rcutorture/bin/mkinitrd.sh
> create mode 100644 tools/testing/selftests/rcutorture/bin/nolibc.h
Pulled into tip:core/rcu, with the <nolibc.h> caveats below, thanks a lot
Paul!
I noticed this bit from Willy:
> tools/testing/selftests/rcutorture/bin/nolibc.h | 2197 ++++++++++++++++++++
So <nolibc.h> is a rather large header and it comes with very little
documentation - but once you read through the header it's obvious what it
does, the code is clean and it's pretty cool all around, and in hindsight
the name is a strong hint about what the header does as well. ;)
Still it would be nice to at least add a top level description to the
header to make people (like me) who are reading the code before the
changelogs wonder less. For tooling headers we require a similar
self-explanatory, feel-fuzzy structure as for kernel headers.
Beyond adding a bit more documentation it would also be useful to factor
nolibc.h out into tools/include/nolibc/ or so, no reason to hide it in
rcutorture, I bet there's a number of other testcases and smaller
utilities in tools/ that could make good use of it.
My long term hope would be that eventually we could even create a minimal
klibc from it (a minimal libc provided by the kernel itself), giving
minimalist binaries a mechanism to link against klibc.so:
- klibc would be an explicit opt-in mechanism, i.e. binaries that are
coupled with the kernel anyway (and initrd executables certainly are)
could use this.
- We could also add a way for the kernel to provide (non-swappable)
binaries via an automatic /klib/ mount point or so. This would allow
features like a minimal, console based rescue/debug shell that doesn't
rely on any filesystem state or external library dependencies, other
than the initial kernel+initrd image.
- In such a scheme klibc would be in /klib/klibc.so, with a mixed
static/dynamic linking model where klibc.so is linked to a particular
static address in all klibc binaries. This would allow executable
startup without dynamic linking overhead, there's basically just an
mmap() required to map klibc.so. I.e. we would get the best of the two
worlds: the simplicity and performance of static linking, with the
low memory overhead and centrally maintained mechanism of dynamic
linked libraries.
- klibc would also eventually allow deeper integration with the vDSO
proper: for example on klibc based embedded systems we could link klibc
and the vDSO into a single vDSO library, further simplifying and
optimizing it.
- klibc would also allow faster feature propagation from kernel to libc
as well, as we could prototype, test and benchmark new system calls and
new features on klibc - i.e. klibc integration and testcases could be a
requirement for new system calls.
- There's no upper limit to how such a minimal kernel-shell (root only)
environment would look like, beyond a minimal shell in principle it
could include bits like:
- a minimal editor,
- a minimal sshd server/client,
- a minimal 3D graphics and window system:
- we can use precompiled shaders for the minimal 3D library to
run the window system with high speed on modern GPU hardware on
GPUs with open-source drivers, such as amdgpu, intel or
nouveau,
- (a minimal font library using a single font, with the supported
font sizes rendered at build time, to avoid having to have a
minimal vector font library.)
- a minimal X terminal based on the 3D rendering environment,
running the shell and editor,
- a minimal version of 3D Tetris (just kidding)
- 3D graphics based perf binary (not kidding: it would have *far*
lower display overhead than the current console graphics we are
using)
- ... all of which would allow a fully integrated, self-contained (!)
solution with no external dependencies other than the build environment
to build the binaries, that allows the debugging of a system and the
eventual extraction of debug information, without having to interact
with the local filesystem or even the local X-window state for these
apps to run.
- I.e. a minimal Linux distribution done right, optimized for kernel
development, system bootstrap, rescue enviroment, etc. - far more
usable than a kdb session.
/me runs
Anyway, back to <nolibc.h>: wanted to ask Linus's and Arnaldo's opinion
about the merge of it, we can still re-base and re-try if there's any
objections.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [GIT PULL rcu/next] RCU commits for 4.21/5.0
2018-12-04 8:08 ` Ingo Molnar
@ 2018-12-04 13:38 ` Willy Tarreau
2018-12-04 20:43 ` Paul E. McKenney
0 siblings, 1 reply; 4+ messages in thread
From: Willy Tarreau @ 2018-12-04 13:38 UTC (permalink / raw)
To: Ingo Molnar
Cc: Paul E. McKenney, Arnaldo Carvalho de Melo, Linus Torvalds,
H. Peter Anvin, linux-kernel, danymadden, Dennis.Krein, joe,
joel, ldr709, pierceagriffiths, rdunlap, zhouzhouyi, connor.shu
Hi Ingo,
On Tue, Dec 04, 2018 at 09:08:37AM +0100, Ingo Molnar wrote:
> I noticed this bit from Willy:
>
> > tools/testing/selftests/rcutorture/bin/nolibc.h | 2197 ++++++++++++++++++++
>
> So <nolibc.h> is a rather large header and it comes with very little
> documentation - but once you read through the header it's obvious what it
> does, the code is clean and it's pretty cool all around, and in hindsight
> the name is a strong hint about what the header does as well. ;)
Thanks for the positive comment, as it was initially not designed to be
merged into the kernel and was just a local home project. I figured it
could be a perfect solution to Paul's executable size issues and offered
some help to get it in relatively quickly, but surely we can do much better!
> Still it would be nice to at least add a top level description to the
> header to make people (like me) who are reading the code before the
> changelogs wonder less. For tooling headers we require a similar
> self-explanatory, feel-fuzzy structure as for kernel headers.
I'm fine with doing this. I even wrote the very small header at the last
minute, without knowing if there was any chance it survives a review :-)
> Beyond adding a bit more documentation it would also be useful to factor
> nolibc.h out into tools/include/nolibc/ or so, no reason to hide it in
> rcutorture, I bet there's a number of other testcases and smaller
> utilities in tools/ that could make good use of it.
Fine as well. It's important however to keep in mind that I only covered
the few architectures I could test (i386/x86_64/arm/arm64/mips), and even
there the coverage is still limited. I don't want it to become too much of
a pain to use for other utilities just by lack of initial coverage. However
I agree that better exposure will help contributions come in.
> My long term hope would be that eventually we could even create a minimal
> klibc from it (a minimal libc provided by the kernel itself), giving
> minimalist binaries a mechanism to link against klibc.so:
>
> - klibc would be an explicit opt-in mechanism, i.e. binaries that are
> coupled with the kernel anyway (and initrd executables certainly are)
> could use this.
In fact it's very similar to my goal. I'm using it in initramfs and initrds
that do very little stuff and where it's acceptable to have a few #ifdef to
adapt to this or that libc. However I found it extremely convenient *not* to
require any external symbol, thus not to have to link against anything. But
I'm well aware that this position cannot last forever and that at some
point if we want to go further we'll possibly have a few layers (naked
syscalls returning -errno, decorated syscalls making use of an explicit
errno, libc-specific stuff like string functions). Possibly that in this
case only the naked version would remain in the .h and that the rest will
require linking with the .so/.a.
> - We could also add a way for the kernel to provide (non-swappable)
> binaries via an automatic /klib/ mount point or so. This would allow
> features like a minimal, console based rescue/debug shell that doesn't
> rely on any filesystem state or external library dependencies, other
> than the initial kernel+initrd image.
This could be convenient indeed, I never thought about this. I'm currently
doing something comparable using initramfs, so maybe in the end we don't
need the kernel to create anything beyond this, but instead just let the
user choose in the configuration what utilities should be added to the
initramfs sources.
(...)
> - klibc would also eventually allow deeper integration with the vDSO
> proper: for example on klibc based embedded systems we could link klibc
> and the vDSO into a single vDSO library, further simplifying and
> optimizing it.
I already looked how to implement vDSO. I figured it was not very difficult
but will require that I maintain variables with the AUXV, then I thought
that it went beyond the scope of this minimalist implementaiton and
postponed this.
> - klibc would also allow faster feature propagation from kernel to libc
> as well, as we could prototype, test and benchmark new system calls and
> new features on klibc - i.e. klibc integration and testcases could be a
> requirement for new system calls.
This actually is a good idea. There was already a discussion in another
thread about exposing syscalls better in the kernel for better interactions
with the libc, but it could start this way with test cases. It also increases
the likeliness that an awkward API is detected early when the person starts
to write his/her own part of the libc side.
> - There's no upper limit to how such a minimal kernel-shell (root only)
> environment would look like, beyond a minimal shell in principle it
> could include bits like:
(...)
That's more or less what the preinit present in my initramfs provides. I
just need a kernel and nothing else to start to manually find and mount
my rootfs while debugging, it's quite convenient. It's very limited
though. But I 100% agree with the benefits of such basic recovery
utilities shipped with the kernel images!
(...)
> - a minimal version of 3D Tetris (just kidding)
I thought you were already kidding when talking about 3D in fact but
apparently not! I think you really mean GPU-based acceleration rather
than 3D since I hardly see how 3D stuff may improve my debugging
abilities :-)
> - ... all of which would allow a fully integrated, self-contained (!)
> solution with no external dependencies other than the build environment
> to build the binaries, that allows the debugging of a system and the
> eventual extraction of debug information, without having to interact
> with the local filesystem or even the local X-window state for these
> apps to run.
In my case I also ship the modules within the kernel image. It's extremely
convenient to have the choice of a number of kernels for a given rootfs.
You never have to wonder if the modules are present on the roofs or not,
you never have to prepare any initrd, you just select the kernel you want
and you're done. For development, when combined with the preinit I'm
talking about, it's awesome, because you just want something which barely
boots to the preinit prompt, then you can start to investigate.
> - I.e. a minimal Linux distribution done right, optimized for kernel
> development, system bootstrap, rescue enviroment, etc. - far more
> usable than a kdb session.
The distro I'm using was initially not made for this but to design
reliable minimalist systems, and it turned out extremely effective for
kernel development for these reasons. The whole rootfs is an initrd
which contains all the tools I need, so I can only confirm the comfort
it brings!
> Anyway, back to <nolibc.h>: wanted to ask Linus's and Arnaldo's opinion
> about the merge of it, we can still re-base and re-try if there's any
> objections.
I'm fine with this as well. I just want to be sure we don't postpone the
coverage of Paul's rcutorture needs because it started for this initially.
If we see the discussion going too far, we could also at least cover just
rcutorture first which would buy us time to decide how it should be done
better, then remove it once we can port rcutorture to the newly designed
solution.
Thanks,
Willy
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [GIT PULL rcu/next] RCU commits for 4.21/5.0
2018-12-04 13:38 ` Willy Tarreau
@ 2018-12-04 20:43 ` Paul E. McKenney
0 siblings, 0 replies; 4+ messages in thread
From: Paul E. McKenney @ 2018-12-04 20:43 UTC (permalink / raw)
To: Willy Tarreau
Cc: Ingo Molnar, Arnaldo Carvalho de Melo, Linus Torvalds,
H. Peter Anvin, linux-kernel, danymadden, Dennis.Krein, joe,
joel, ldr709, pierceagriffiths, rdunlap, zhouzhouyi, connor.shu
On Tue, Dec 04, 2018 at 02:38:17PM +0100, Willy Tarreau wrote:
> Hi Ingo,
>
> On Tue, Dec 04, 2018 at 09:08:37AM +0100, Ingo Molnar wrote:
> > I noticed this bit from Willy:
> >
> > > tools/testing/selftests/rcutorture/bin/nolibc.h | 2197 ++++++++++++++++++++
> >
> > So <nolibc.h> is a rather large header and it comes with very little
> > documentation - but once you read through the header it's obvious what it
> > does, the code is clean and it's pretty cool all around, and in hindsight
> > the name is a strong hint about what the header does as well. ;)
>
> Thanks for the positive comment, as it was initially not designed to be
> merged into the kernel and was just a local home project. I figured it
> could be a perfect solution to Paul's executable size issues and offered
> some help to get it in relatively quickly, but surely we can do much better!
>
> > Still it would be nice to at least add a top level description to the
> > header to make people (like me) who are reading the code before the
> > changelogs wonder less. For tooling headers we require a similar
> > self-explanatory, feel-fuzzy structure as for kernel headers.
>
> I'm fine with doing this. I even wrote the very small header at the last
> minute, without knowing if there was any chance it survives a review :-)
>
> > Beyond adding a bit more documentation it would also be useful to factor
> > nolibc.h out into tools/include/nolibc/ or so, no reason to hide it in
> > rcutorture, I bet there's a number of other testcases and smaller
> > utilities in tools/ that could make good use of it.
>
> Fine as well. It's important however to keep in mind that I only covered
> the few architectures I could test (i386/x86_64/arm/arm64/mips), and even
> there the coverage is still limited. I don't want it to become too much of
> a pain to use for other utilities just by lack of initial coverage. However
> I agree that better exposure will help contributions come in.
>
> > My long term hope would be that eventually we could even create a minimal
> > klibc from it (a minimal libc provided by the kernel itself), giving
> > minimalist binaries a mechanism to link against klibc.so:
> >
> > - klibc would be an explicit opt-in mechanism, i.e. binaries that are
> > coupled with the kernel anyway (and initrd executables certainly are)
> > could use this.
>
> In fact it's very similar to my goal. I'm using it in initramfs and initrds
> that do very little stuff and where it's acceptable to have a few #ifdef to
> adapt to this or that libc. However I found it extremely convenient *not* to
> require any external symbol, thus not to have to link against anything. But
> I'm well aware that this position cannot last forever and that at some
> point if we want to go further we'll possibly have a few layers (naked
> syscalls returning -errno, decorated syscalls making use of an explicit
> errno, libc-specific stuff like string functions). Possibly that in this
> case only the naked version would remain in the .h and that the rest will
> require linking with the .so/.a.
>
> > - We could also add a way for the kernel to provide (non-swappable)
> > binaries via an automatic /klib/ mount point or so. This would allow
> > features like a minimal, console based rescue/debug shell that doesn't
> > rely on any filesystem state or external library dependencies, other
> > than the initial kernel+initrd image.
>
> This could be convenient indeed, I never thought about this. I'm currently
> doing something comparable using initramfs, so maybe in the end we don't
> need the kernel to create anything beyond this, but instead just let the
> user choose in the configuration what utilities should be added to the
> initramfs sources.
>
> (...)
> > - klibc would also eventually allow deeper integration with the vDSO
> > proper: for example on klibc based embedded systems we could link klibc
> > and the vDSO into a single vDSO library, further simplifying and
> > optimizing it.
>
> I already looked how to implement vDSO. I figured it was not very difficult
> but will require that I maintain variables with the AUXV, then I thought
> that it went beyond the scope of this minimalist implementaiton and
> postponed this.
>
> > - klibc would also allow faster feature propagation from kernel to libc
> > as well, as we could prototype, test and benchmark new system calls and
> > new features on klibc - i.e. klibc integration and testcases could be a
> > requirement for new system calls.
>
> This actually is a good idea. There was already a discussion in another
> thread about exposing syscalls better in the kernel for better interactions
> with the libc, but it could start this way with test cases. It also increases
> the likeliness that an awkward API is detected early when the person starts
> to write his/her own part of the libc side.
>
> > - There's no upper limit to how such a minimal kernel-shell (root only)
> > environment would look like, beyond a minimal shell in principle it
> > could include bits like:
> (...)
>
> That's more or less what the preinit present in my initramfs provides. I
> just need a kernel and nothing else to start to manually find and mount
> my rootfs while debugging, it's quite convenient. It's very limited
> though. But I 100% agree with the benefits of such basic recovery
> utilities shipped with the kernel images!
>
> (...)
> > - a minimal version of 3D Tetris (just kidding)
>
> I thought you were already kidding when talking about 3D in fact but
> apparently not! I think you really mean GPU-based acceleration rather
> than 3D since I hardly see how 3D stuff may improve my debugging
> abilities :-)
>
> > - ... all of which would allow a fully integrated, self-contained (!)
> > solution with no external dependencies other than the build environment
> > to build the binaries, that allows the debugging of a system and the
> > eventual extraction of debug information, without having to interact
> > with the local filesystem or even the local X-window state for these
> > apps to run.
>
> In my case I also ship the modules within the kernel image. It's extremely
> convenient to have the choice of a number of kernels for a given rootfs.
> You never have to wonder if the modules are present on the roofs or not,
> you never have to prepare any initrd, you just select the kernel you want
> and you're done. For development, when combined with the preinit I'm
> talking about, it's awesome, because you just want something which barely
> boots to the preinit prompt, then you can start to investigate.
>
> > - I.e. a minimal Linux distribution done right, optimized for kernel
> > development, system bootstrap, rescue enviroment, etc. - far more
> > usable than a kdb session.
>
> The distro I'm using was initially not made for this but to design
> reliable minimalist systems, and it turned out extremely effective for
> kernel development for these reasons. The whole rootfs is an initrd
> which contains all the tools I need, so I can only confirm the comfort
> it brings!
>
> > Anyway, back to <nolibc.h>: wanted to ask Linus's and Arnaldo's opinion
> > about the merge of it, we can still re-base and re-try if there's any
> > objections.
>
> I'm fine with this as well. I just want to be sure we don't postpone the
> coverage of Paul's rcutorture needs because it started for this initially.
> If we see the discussion going too far, we could also at least cover just
> rcutorture first which would buy us time to decide how it should be done
> better, then remove it once we can port rcutorture to the newly designed
> solution.
I of course prefer starting with what I have, but it would not be at
all difficult for me to rebase if that is what needs to happen.
Thanx, Paul
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-12-04 20:43 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-03 22:54 [GIT PULL rcu/next] RCU commits for 4.21/5.0 Paul E. McKenney
2018-12-04 8:08 ` Ingo Molnar
2018-12-04 13:38 ` Willy Tarreau
2018-12-04 20:43 ` Paul E. McKenney
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.