* Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
@ 2016-06-13 19:43 Geert Uytterhoeven
2016-06-14 6:24 ` Joonsoo Kim
2016-06-14 13:10 ` Geert Uytterhoeven
0 siblings, 2 replies; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-13 19:43 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel, Joonsoo Kim,
linux-renesas-soc, linux-arm-kernel
[-- Attachment #1: Type: text/plain, Size: 2932 bytes --]
Hi Joonsoo,
On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>
> To check whther free objects exist or not precisely, we need to grab a
> lock. But, accuracy isn't that important because race window would be
> even small and if there is too much free object, cache reaper would reap
> it. So, this patch makes the check for free object exisistence not to
> hold a lock. This will reduce lock contention in heavily allocation case.
>
> Note that until now, n->shared can be freed during the processing by
> writing slabinfo, but, with some trick in this patch, we can access it
> freely within interrupt disabled period.
>
> Below is the result of concurrent allocation/free in slab allocation
> benchmark made by Christoph a long time ago. I make the output simpler.
> The number shows cycle count during alloc/free respectively so less is
> better.
>
> * Before
> Kmalloc N*alloc N*free(32): Average=248/966
> Kmalloc N*alloc N*free(64): Average=261/949
> Kmalloc N*alloc N*free(128): Average=314/1016
> Kmalloc N*alloc N*free(256): Average=741/1061
> Kmalloc N*alloc N*free(512): Average=1246/1152
> Kmalloc N*alloc N*free(1024): Average=2437/1259
> Kmalloc N*alloc N*free(2048): Average=4980/1800
> Kmalloc N*alloc N*free(4096): Average=9000/2078
>
> * After
> Kmalloc N*alloc N*free(32): Average=344/792
> Kmalloc N*alloc N*free(64): Average=347/882
> Kmalloc N*alloc N*free(128): Average=390/959
> Kmalloc N*alloc N*free(256): Average=393/1067
> Kmalloc N*alloc N*free(512): Average=683/1229
> Kmalloc N*alloc N*free(1024): Average=1295/1325
> Kmalloc N*alloc N*free(2048): Average=2513/1664
> Kmalloc N*alloc N*free(4096): Average=4742/2172
>
> It shows that allocation performance decreases for the object size up to
> 128 and it may be due to extra checks in cache_alloc_refill(). But, with
> considering improvement of free performance, net result looks the same.
> Result for other size class looks very promising, roughly, 50% performance
> improvement.
>
> v2: replace kick_all_cpus_sync() with synchronize_sched().
>
> Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
(Renesas dual Cortex A9) to this patch, which is upstream commit
801faf0db8947e01877920e848a4d338dd7a99e7.
I've attached my .config. I don't know if it also happens with
shmobile_defconfig, as something went wrong with my remote access to the board,
preventing further testing. I also couldn't verify if the issue persists in
v4.7-rc3.
Do you have a clue?
Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
[-- Attachment #2: kzm9d_config --]
[-- Type: application/octet-stream, Size: 82729 bytes --]
#
# Automatically generated file; DO NOT EDIT.
# Linux/arm 4.6.0 Kernel Configuration
#
CONFIG_ARM=y
CONFIG_ARM_HAS_SG_CHAIN=y
CONFIG_MIGHT_HAVE_PCI=y
CONFIG_SYS_SUPPORTS_APM_EMULATION=y
CONFIG_HAVE_PROC_CPU=y
CONFIG_NO_IOPORT_MAP=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_VECTORS_BASE=0xffff0000
CONFIG_ARM_PATCH_PHYS_VIRT=y
CONFIG_GENERIC_BUG=y
CONFIG_PGTABLE_LEVELS=2
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION="-kzm9d"
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
# CONFIG_FHANDLE is not set
# CONFIG_USELIB is not set
# CONFIG_AUDIT is not set
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_IRQ_SHOW_LEVEL=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_HANDLE_DOMAIN_IRQ=y
CONFIG_IRQ_DOMAIN_DEBUG=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_ARCH_HAS_TICK_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
# CONFIG_TASKS_RCU is not set
CONFIG_RCU_STALL_COMMON=y
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_RCU_EXPEDITE_BOOT is not set
CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_GENERIC_SCHED_CLOCK=y
# CONFIG_CGROUPS is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE="/home/geert/Downloads/initramfs_data.cpio.gz "
CONFIG_INITRAMFS_ROOT_UID=0
CONFIG_INITRAMFS_ROOT_GID=0
CONFIG_RD_GZIP=y
# CONFIG_RD_BZIP2 is not set
# CONFIG_RD_LZMA is not set
# CONFIG_RD_XZ is not set
# CONFIG_RD_LZO is not set
# CONFIG_RD_LZ4 is not set
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_BPF=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_MULTIUSER=y
# CONFIG_SGETMASK_SYSCALL is not set
# CONFIG_SYSFS_SYSCALL is not set
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_ABSOLUTE_PERCPU is not set
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
# CONFIG_BPF_SYSCALL is not set
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_ADVISE_SYSCALLS=y
CONFIG_USERFAULTFD=y
CONFIG_MEMBARRIER=y
CONFIG_EMBEDDED=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_USE_VMALLOC=y
#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_COMPAT_BRK=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_SYSTEM_DATA_VERIFICATION is not set
# CONFIG_PROFILING is not set
CONFIG_KEXEC_CORE=y
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
# CONFIG_JUMP_LABEL is not set
# CONFIG_UPROBES is not set
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_GENERIC_IDLE_POLL_SETUP=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_CLK=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_HAVE_CC_STACKPROTECTOR=y
# CONFIG_CC_STACKPROTECTOR is not set
CONFIG_CC_STACKPROTECTOR_NONE=y
# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_REL=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_ARCH_MMAP_RND_BITS_MIN=8
CONFIG_ARCH_MMAP_RND_BITS_MAX=16
CONFIG_ARCH_MMAP_RND_BITS=8
CONFIG_CLONE_BACKWARDS=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_OLD_SIGACTION=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
# CONFIG_MODULE_COMPRESS is not set
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
CONFIG_LBDAF=y
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_BSGLIB is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
# CONFIG_BLK_CMDLINE_PARSER is not set
#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_FREEZER=y
#
# System Type
#
CONFIG_MMU=y
CONFIG_ARCH_MULTIPLATFORM=y
# CONFIG_ARCH_CLPS711X is not set
# CONFIG_ARCH_GEMINI is not set
# CONFIG_ARCH_EBSA110 is not set
# CONFIG_ARCH_EP93XX is not set
# CONFIG_ARCH_FOOTBRIDGE is not set
# CONFIG_ARCH_NETX is not set
# CONFIG_ARCH_IOP13XX is not set
# CONFIG_ARCH_IOP32X is not set
# CONFIG_ARCH_IOP33X is not set
# CONFIG_ARCH_IXP4XX is not set
# CONFIG_ARCH_DOVE is not set
# CONFIG_ARCH_KS8695 is not set
# CONFIG_ARCH_W90X900 is not set
# CONFIG_ARCH_LPC32XX is not set
# CONFIG_ARCH_PXA is not set
# CONFIG_ARCH_RPC is not set
# CONFIG_ARCH_SA1100 is not set
# CONFIG_ARCH_S3C24XX is not set
# CONFIG_ARCH_DAVINCI is not set
# CONFIG_ARCH_OMAP1 is not set
#
# Multiple platform selection
#
#
# CPU Core family selection
#
# CONFIG_ARCH_MULTI_V6 is not set
CONFIG_ARCH_MULTI_V7=y
CONFIG_ARCH_MULTI_V6_V7=y
# CONFIG_ARCH_MULTI_CPU_AUTO is not set
# CONFIG_ARCH_VIRT is not set
# CONFIG_ARCH_MVEBU is not set
# CONFIG_ARCH_ALPINE is not set
# CONFIG_ARCH_ARTPEC is not set
# CONFIG_ARCH_AT91 is not set
# CONFIG_ARCH_BCM is not set
# CONFIG_ARCH_BERLIN is not set
# CONFIG_ARCH_DIGICOLOR is not set
# CONFIG_ARCH_HIGHBANK is not set
# CONFIG_ARCH_HISI is not set
# CONFIG_ARCH_KEYSTONE is not set
# CONFIG_ARCH_MESON is not set
# CONFIG_ARCH_MXC is not set
# CONFIG_ARCH_MEDIATEK is not set
#
# TI OMAP/AM/DM/DRA Family
#
# CONFIG_ARCH_OMAP3 is not set
# CONFIG_ARCH_OMAP4 is not set
# CONFIG_SOC_OMAP5 is not set
# CONFIG_SOC_AM33XX is not set
# CONFIG_SOC_AM43XX is not set
# CONFIG_SOC_DRA7XX is not set
# CONFIG_ARCH_MMP is not set
# CONFIG_ARCH_QCOM is not set
# CONFIG_ARCH_REALVIEW is not set
# CONFIG_ARCH_ROCKCHIP is not set
# CONFIG_ARCH_SOCFPGA is not set
# CONFIG_PLAT_SPEAR is not set
# CONFIG_ARCH_STI is not set
# CONFIG_ARCH_S5PV210 is not set
# CONFIG_ARCH_EXYNOS is not set
CONFIG_ARCH_SHMOBILE=y
CONFIG_ARCH_SHMOBILE_MULTI=y
CONFIG_ARCH_RENESAS=y
CONFIG_ARCH_EMEV2=y
# CONFIG_ARCH_R7S72100 is not set
# CONFIG_ARCH_R8A73A4 is not set
# CONFIG_ARCH_R8A7740 is not set
# CONFIG_ARCH_R8A7778 is not set
# CONFIG_ARCH_R8A7779 is not set
# CONFIG_ARCH_R8A7790 is not set
# CONFIG_ARCH_R8A7791 is not set
# CONFIG_ARCH_R8A7793 is not set
# CONFIG_ARCH_R8A7794 is not set
# CONFIG_ARCH_SH73A0 is not set
# CONFIG_ARCH_SUNXI is not set
# CONFIG_ARCH_SIRF is not set
# CONFIG_ARCH_TANGO is not set
# CONFIG_ARCH_TEGRA is not set
# CONFIG_ARCH_UNIPHIER is not set
# CONFIG_ARCH_U8500 is not set
# CONFIG_ARCH_VEXPRESS is not set
# CONFIG_ARCH_WM8850 is not set
# CONFIG_ARCH_ZX is not set
# CONFIG_ARCH_ZYNQ is not set
#
# Processor Type
#
CONFIG_CPU_V7=y
CONFIG_CPU_32v6K=y
CONFIG_CPU_32v7=y
CONFIG_CPU_ABRT_EV7=y
CONFIG_CPU_PABRT_V7=y
CONFIG_CPU_CACHE_V7=y
CONFIG_CPU_CACHE_VIPT=y
CONFIG_CPU_COPY_V6=y
CONFIG_CPU_TLB_V7=y
CONFIG_CPU_HAS_ASID=y
CONFIG_CPU_CP15=y
CONFIG_CPU_CP15_MMU=y
#
# Processor Features
#
# CONFIG_ARM_LPAE is not set
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
CONFIG_ARM_THUMB=y
# CONFIG_ARM_THUMBEE is not set
CONFIG_ARM_VIRT_EXT=y
CONFIG_SWP_EMULATE=y
# CONFIG_CPU_ICACHE_DISABLE is not set
# CONFIG_CPU_BPREDICT_DISABLE is not set
CONFIG_KUSER_HELPERS=y
CONFIG_VDSO=y
CONFIG_OUTER_CACHE=y
CONFIG_OUTER_CACHE_SYNC=y
CONFIG_MIGHT_HAVE_CACHE_L2X0=y
CONFIG_CACHE_L2X0=y
# CONFIG_PL310_ERRATA_588369 is not set
# CONFIG_PL310_ERRATA_727915 is not set
# CONFIG_PL310_ERRATA_753970 is not set
CONFIG_PL310_ERRATA_769419=y
CONFIG_ARM_L1_CACHE_SHIFT_6=y
CONFIG_ARM_L1_CACHE_SHIFT=6
CONFIG_ARM_DMA_MEM_BUFFERABLE=y
CONFIG_ARM_HEAVY_MB=y
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_ALIGN_RODATA=y
CONFIG_MULTI_IRQ_HANDLER=y
# CONFIG_ARM_ERRATA_430973 is not set
# CONFIG_ARM_ERRATA_643719 is not set
# CONFIG_ARM_ERRATA_720789 is not set
CONFIG_ARM_ERRATA_754322=y
# CONFIG_ARM_ERRATA_754327 is not set
# CONFIG_ARM_ERRATA_764369 is not set
# CONFIG_ARM_ERRATA_775420 is not set
# CONFIG_ARM_ERRATA_798181 is not set
# CONFIG_ARM_ERRATA_773022 is not set
#
# Bus support
#
# CONFIG_PCI is not set
# CONFIG_PCI_DOMAINS_GENERIC is not set
# CONFIG_PCI_SYSCALL is not set
# CONFIG_PCCARD is not set
#
# Kernel Features
#
CONFIG_HAVE_SMP=y
CONFIG_SMP=y
CONFIG_SMP_ON_UP=y
CONFIG_ARM_CPU_TOPOLOGY=y
CONFIG_SCHED_MC=y
# CONFIG_SCHED_SMT is not set
CONFIG_HAVE_ARM_SCU=y
# CONFIG_HAVE_ARM_ARCH_TIMER is not set
CONFIG_HAVE_ARM_TWD=y
# CONFIG_MCPM is not set
# CONFIG_BIG_LITTLE is not set
CONFIG_VMSPLIT_3G=y
# CONFIG_VMSPLIT_3G_OPT is not set
# CONFIG_VMSPLIT_2G is not set
# CONFIG_VMSPLIT_1G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_NR_CPUS=4
CONFIG_HOTPLUG_CPU=y
# CONFIG_ARM_PSCI is not set
CONFIG_ARCH_NR_GPIO=1024
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_HZ_FIXED=0
CONFIG_HZ_100=y
# CONFIG_HZ_200 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_500 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
# CONFIG_THUMB2_KERNEL is not set
CONFIG_ARM_PATCH_IDIV=y
CONFIG_AEABI=y
# CONFIG_OABI_COMPAT is not set
# CONFIG_ARCH_SPARSEMEM_DEFAULT is not set
# CONFIG_ARCH_SELECT_MEMORY_MODEL is not set
CONFIG_HAVE_ARCH_PFN_VALID=y
CONFIG_HIGHMEM=y
# CONFIG_HIGHPTE is not set
CONFIG_CPU_SW_DOMAIN_PAN=y
CONFIG_HW_PERF_EVENTS=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
# CONFIG_ARM_MODULE_PLTS is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_NO_BOOTMEM=y
# CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=0
CONFIG_BOUNCE=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
# CONFIG_CLEANCACHE is not set
# CONFIG_FRONTSWAP is not set
# CONFIG_CMA is not set
# CONFIG_ZPOOL is not set
# CONFIG_ZBUD is not set
# CONFIG_ZSMALLOC is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
# CONFIG_IDLE_PAGE_TRACKING is not set
CONFIG_FORCE_MAX_ZONEORDER=11
CONFIG_ALIGNMENT_TRAP=y
# CONFIG_UACCESS_WITH_MEMCPY is not set
# CONFIG_SECCOMP is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_PARAVIRT is not set
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
# CONFIG_XEN is not set
#
# Boot options
#
CONFIG_USE_OF=y
CONFIG_ATAGS=y
# CONFIG_DEPRECATED_PARAM_STRUCT is not set
CONFIG_ZBOOT_ROM_TEXT=0x0
CONFIG_ZBOOT_ROM_BSS=0x0
CONFIG_ARM_APPENDED_DTB=y
# CONFIG_ARM_ATAG_DTB_COMPAT is not set
CONFIG_CMDLINE=""
CONFIG_KEXEC=y
CONFIG_ATAGS_PROC=y
# CONFIG_CRASH_DUMP is not set
CONFIG_AUTO_ZRELADDR=y
# CONFIG_EFI is not set
#
# CPU Power Management
#
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
# CONFIG_CPU_FREQ_GOV_SCHEDUTIL is not set
#
# CPU frequency scaling drivers
#
CONFIG_CPUFREQ_DT=y
CONFIG_CPUFREQ_DT_PLATDEV=y
# CONFIG_ARM_BIG_LITTLE_CPUFREQ is not set
# CONFIG_ARM_KIRKWOOD_CPUFREQ is not set
# CONFIG_QORIQ_CPUFREQ is not set
#
# CPU Idle
#
# CONFIG_CPU_IDLE is not set
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
#
# Floating point emulation
#
#
# At least one emulation must be selected
#
CONFIG_VFP=y
CONFIG_VFPv3=y
CONFIG_NEON=y
# CONFIG_KERNEL_MODE_NEON is not set
#
# Userspace binary formats
#
CONFIG_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_BINFMT_SCRIPT=y
# CONFIG_HAVE_AOUT is not set
# CONFIG_BINFMT_MISC is not set
CONFIG_COREDUMP=y
#
# Power management options
#
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_SUSPEND_SKIP_SYNC is not set
# CONFIG_HIBERNATION is not set
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_PM_TEST_SUSPEND=y
CONFIG_PM_SLEEP_DEBUG=y
# CONFIG_APM_EMULATION is not set
CONFIG_PM_OPP=y
CONFIG_PM_CLK=y
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
CONFIG_CPU_PM=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARM_CPU_SUSPEND=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_DIAG is not set
CONFIG_UNIX=y
# CONFIG_UNIX_DIAG is not set
# CONFIG_XFRM_USER is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
# CONFIG_IP_PNP_BOOTP is not set
# CONFIG_IP_PNP_RARP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
# CONFIG_NET_IP_TUNNEL is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_NET_UDP_TUNNEL is not set
# CONFIG_NET_FOU is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NET_PTP_CLASSIFY is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
# CONFIG_NETFILTER is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
# CONFIG_BRIDGE is not set
CONFIG_HAVE_NET_DSA=y
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
# CONFIG_VSOCKETS is not set
# CONFIG_NETLINK_DIAG is not set
# CONFIG_MPLS is not set
# CONFIG_HSR is not set
# CONFIG_NET_SWITCHDEV is not set
# CONFIG_NET_L3_MASTER_DEV is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
# CONFIG_SOCK_CGROUP_DATA is not set
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
# CONFIG_BPF_JIT is not set
CONFIG_NET_FLOW_LIMIT=y
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
CONFIG_IRDA=y
#
# IrDA protocols
#
# CONFIG_IRLAN is not set
# CONFIG_IRCOMM is not set
# CONFIG_IRDA_ULTRA is not set
#
# IrDA options
#
# CONFIG_IRDA_CACHE_LAST_LSAP is not set
# CONFIG_IRDA_FAST_RR is not set
# CONFIG_IRDA_DEBUG is not set
#
# Infrared-port device drivers
#
#
# SIR device drivers
#
# CONFIG_IRTTY_SIR is not set
#
# Dongle support
#
# CONFIG_KINGSUN_DONGLE is not set
# CONFIG_KSDAZZLE_DONGLE is not set
# CONFIG_KS959_DONGLE is not set
#
# FIR device drivers
#
# CONFIG_USB_IRDA is not set
# CONFIG_SIGMATEL_FIR is not set
# CONFIG_MCS_FIR is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_RFKILL_REGULATOR is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
# CONFIG_NFC is not set
# CONFIG_LWTUNNEL is not set
# CONFIG_DST_CACHE is not set
# CONFIG_NET_DEVLINK is not set
CONFIG_MAY_USE_DEVLINK=y
CONFIG_HAVE_CBPF_JIT=y
#
# Device Drivers
#
#
# Generic Driver Options
#
# CONFIG_UEVENT_HELPER is not set
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set
CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_SPI=y
CONFIG_REGMAP_IRQ=y
# CONFIG_DMA_SHARED_BUFFER is not set
#
# Bus devices
#
# CONFIG_ARM_CCI400_PMU is not set
# CONFIG_ARM_CCI5xx_PMU is not set
# CONFIG_ARM_CCN is not set
# CONFIG_BRCMSTB_GISB_ARB is not set
CONFIG_SIMPLE_PM_BUS=y
# CONFIG_VEXPRESS_CONFIG is not set
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
CONFIG_DTC=y
CONFIG_OF=y
# CONFIG_OF_UNITTEST is not set
CONFIG_OF_FLATTREE=y
CONFIG_OF_EARLY_FLATTREE=y
CONFIG_OF_DYNAMIC=y
CONFIG_OF_ADDRESS=y
CONFIG_OF_IRQ=y
CONFIG_OF_NET=y
CONFIG_OF_MDIO=y
CONFIG_OF_RESERVED_MEM=y
CONFIG_OF_RESOLVE=y
CONFIG_OF_OVERLAY=y
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
# CONFIG_PARPORT is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_NULL_BLK is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_RAM is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_MG_DISK is not set
# CONFIG_BLK_DEV_RBD is not set
#
# Misc devices
#
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_APDS9802ALS is not set
# CONFIG_ISL29003 is not set
# CONFIG_ISL29020 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_SENSORS_BH1780 is not set
# CONFIG_SENSORS_BH1770 is not set
# CONFIG_SENSORS_APDS990X is not set
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
# CONFIG_TI_DAC7512 is not set
# CONFIG_BMP085_I2C is not set
# CONFIG_BMP085_SPI is not set
# CONFIG_USB_SWITCH_FSA9480 is not set
# CONFIG_LATTICE_ECP3_CONFIG is not set
# CONFIG_SRAM is not set
# CONFIG_C2PORT is not set
#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_AT25 is not set
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_EEPROM_93XX46 is not set
#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
# CONFIG_SENSORS_LIS3_SPI is not set
# CONFIG_SENSORS_LIS3_I2C is not set
#
# Altera FPGA firmware download module
#
# CONFIG_ALTERA_STAPL is not set
#
# Intel MIC Bus Driver
#
#
# SCIF Bus Driver
#
#
# VOP Bus Driver
#
#
# Intel MIC Host Driver
#
#
# Intel MIC Card Driver
#
#
# SCIF Driver
#
#
# Intel MIC Coprocessor State Management (COSM) Drivers
#
#
# VOP Driver
#
# CONFIG_ECHO is not set
# CONFIG_CXL_BASE is not set
# CONFIG_CXL_KERNEL_API is not set
# CONFIG_CXL_EEH is not set
#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_NETLINK is not set
# CONFIG_SCSI_MQ_DEFAULT is not set
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
# CONFIG_CHR_DEV_SG is not set
# CONFIG_CHR_DEV_SCH is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_ISCSI_BOOT_SYSFS is not set
# CONFIG_SCSI_UFSHCD is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
# CONFIG_ATA is not set
# CONFIG_MD is not set
# CONFIG_TARGET_CORE is not set
CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y
# CONFIG_BONDING is not set
# CONFIG_DUMMY is not set
# CONFIG_EQUALIZER is not set
# CONFIG_NET_TEAM is not set
# CONFIG_MACVLAN is not set
# CONFIG_VXLAN is not set
# CONFIG_MACSEC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_TUN is not set
# CONFIG_TUN_VNET_CROSS_LE is not set
# CONFIG_VETH is not set
# CONFIG_NLMON is not set
#
# CAIF transport drivers
#
#
# Distributed Switch Architecture drivers
#
CONFIG_ETHERNET=y
# CONFIG_ALTERA_TSE is not set
CONFIG_NET_VENDOR_ARC=y
# CONFIG_ARC_EMAC is not set
# CONFIG_EMAC_ROCKCHIP is not set
# CONFIG_NET_VENDOR_AURORA is not set
CONFIG_NET_CADENCE=y
# CONFIG_MACB is not set
CONFIG_NET_VENDOR_BROADCOM=y
# CONFIG_B44 is not set
# CONFIG_BCMGENET is not set
# CONFIG_SYSTEMPORT is not set
CONFIG_NET_VENDOR_CIRRUS=y
# CONFIG_CS89x0 is not set
# CONFIG_DM9000 is not set
# CONFIG_DNET is not set
# CONFIG_NET_VENDOR_EZCHIP is not set
CONFIG_NET_VENDOR_FARADAY=y
# CONFIG_FTMAC100 is not set
# CONFIG_FTGMAC100 is not set
# CONFIG_NET_VENDOR_HISILICON is not set
CONFIG_NET_VENDOR_INTEL=y
CONFIG_NET_VENDOR_I825XX=y
CONFIG_NET_VENDOR_MARVELL=y
# CONFIG_MVMDIO is not set
# CONFIG_MVNETA_BM is not set
CONFIG_NET_VENDOR_MICREL=y
# CONFIG_KS8842 is not set
# CONFIG_KS8851 is not set
# CONFIG_KS8851_MLL is not set
# CONFIG_NET_VENDOR_MICROCHIP is not set
CONFIG_NET_VENDOR_NATSEMI=y
# CONFIG_NET_VENDOR_NETRONOME is not set
CONFIG_NET_VENDOR_8390=y
# CONFIG_AX88796 is not set
# CONFIG_ETHOC is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
CONFIG_NET_VENDOR_SAMSUNG=y
# CONFIG_SXGBE_ETH is not set
CONFIG_NET_VENDOR_SEEQ=y
CONFIG_NET_VENDOR_SMSC=y
# CONFIG_SMC91X is not set
# CONFIG_SMC911X is not set
CONFIG_SMSC911X=y
# CONFIG_SMSC911X_ARCH_HOOKS is not set
CONFIG_NET_VENDOR_STMICRO=y
# CONFIG_STMMAC_ETH is not set
# CONFIG_NET_VENDOR_SYNOPSYS is not set
CONFIG_NET_VENDOR_VIA=y
# CONFIG_VIA_RHINE is not set
# CONFIG_VIA_VELOCITY is not set
CONFIG_NET_VENDOR_WIZNET=y
# CONFIG_WIZNET_W5100 is not set
# CONFIG_WIZNET_W5300 is not set
CONFIG_PHYLIB=y
#
# MII PHY device drivers
#
# CONFIG_AQUANTIA_PHY is not set
# CONFIG_AT803X_PHY is not set
# CONFIG_AMD_PHY is not set
# CONFIG_MARVELL_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_TERANETICS_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_BCM7XXX_PHY is not set
# CONFIG_BCM87XX_PHY is not set
# CONFIG_ICPLUS_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_NATIONAL_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_DP83848_PHY is not set
# CONFIG_DP83867_PHY is not set
# CONFIG_MICROCHIP_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
# CONFIG_MDIO_BUS_MUX_GPIO is not set
# CONFIG_MDIO_BUS_MUX_MMIOREG is not set
# CONFIG_MDIO_BCM_UNIMAC is not set
# CONFIG_MICREL_KS8995MA is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_USB_NET_DRIVERS is not set
# CONFIG_WLAN is not set
#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
# CONFIG_ISDN is not set
# CONFIG_NVM is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_LEDS is not set
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
CONFIG_INPUT_SPARSEKMAP=y
# CONFIG_INPUT_MATRIXKMAP is not set
#
# Userland interfaces
#
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
# CONFIG_KEYBOARD_ATKBD is not set
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
CONFIG_KEYBOARD_GPIO=y
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_MATRIX is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_SAMSUNG is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_SH_KEYSC is not set
# CONFIG_KEYBOARD_OMAP4 is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_CAP11XX is not set
# CONFIG_KEYBOARD_BCM is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_PROPERTIES=y
# CONFIG_TOUCHSCREEN_ADS7846 is not set
# CONFIG_TOUCHSCREEN_AD7877 is not set
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_AR1021_I2C is not set
# CONFIG_TOUCHSCREEN_ATMEL_MXT is not set
# CONFIG_TOUCHSCREEN_AUO_PIXCIR is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_CHIPONE_ICN8318 is not set
# CONFIG_TOUCHSCREEN_CY8CTMG110 is not set
# CONFIG_TOUCHSCREEN_CYTTSP_CORE is not set
# CONFIG_TOUCHSCREEN_CYTTSP4_CORE is not set
# CONFIG_TOUCHSCREEN_DYNAPRO is not set
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_EGALAX is not set
# CONFIG_TOUCHSCREEN_EGALAX_SERIAL is not set
# CONFIG_TOUCHSCREEN_FT6236 is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
# CONFIG_TOUCHSCREEN_GOODIX is not set
# CONFIG_TOUCHSCREEN_ILI210X is not set
# CONFIG_TOUCHSCREEN_GUNZE is not set
# CONFIG_TOUCHSCREEN_ELAN is not set
# CONFIG_TOUCHSCREEN_ELO is not set
# CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
# CONFIG_TOUCHSCREEN_WACOM_I2C is not set
# CONFIG_TOUCHSCREEN_MAX11801 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
# CONFIG_TOUCHSCREEN_MMS114 is not set
# CONFIG_TOUCHSCREEN_MELFAS_MIP4 is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_IMX6UL_TSC is not set
# CONFIG_TOUCHSCREEN_INEXIO is not set
# CONFIG_TOUCHSCREEN_MK712 is not set
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_EDT_FT5X06 is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
# CONFIG_TOUCHSCREEN_PIXCIR is not set
# CONFIG_TOUCHSCREEN_WDT87XX_I2C is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC_SERIO is not set
# CONFIG_TOUCHSCREEN_TSC2004 is not set
# CONFIG_TOUCHSCREEN_TSC2005 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
CONFIG_TOUCHSCREEN_ST1232=y
# CONFIG_TOUCHSCREEN_SX8654 is not set
# CONFIG_TOUCHSCREEN_TPS6507X is not set
# CONFIG_TOUCHSCREEN_ZFORCE is not set
# CONFIG_TOUCHSCREEN_ROHM_BU21023 is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_BMA150 is not set
# CONFIG_INPUT_E3X0_BUTTON is not set
# CONFIG_INPUT_MMA8450 is not set
# CONFIG_INPUT_MPU3050 is not set
# CONFIG_INPUT_GP2A is not set
# CONFIG_INPUT_GPIO_BEEPER is not set
# CONFIG_INPUT_GPIO_TILT_POLLED is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_KXTJ9 is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_REGULATOR_HAPTIC is not set
# CONFIG_INPUT_UINPUT is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_GPIO_ROTARY_ENCODER is not set
CONFIG_INPUT_ADXL34X=y
CONFIG_INPUT_ADXL34X_I2C=y
# CONFIG_INPUT_ADXL34X_SPI is not set
# CONFIG_INPUT_IMS_PCU is not set
# CONFIG_INPUT_CMA3000 is not set
# CONFIG_INPUT_SOC_BUTTON_ARRAY is not set
# CONFIG_INPUT_DRV260X_HAPTICS is not set
# CONFIG_INPUT_DRV2665_HAPTICS is not set
# CONFIG_INPUT_DRV2667_HAPTICS is not set
# CONFIG_RMI4_CORE is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_LIBPS2 is not set
# CONFIG_SERIO_RAW is not set
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_SERIO_ARC_PS2 is not set
# CONFIG_SERIO_APBPS2 is not set
# CONFIG_USERIO is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
CONFIG_DEVMEM=y
CONFIG_DEVKMEM=y
#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
CONFIG_SERIAL_8250_CONSOLE=y
# CONFIG_SERIAL_8250_DMA is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
CONFIG_SERIAL_8250_FSL=y
# CONFIG_SERIAL_8250_DW is not set
CONFIG_SERIAL_8250_EM=y
# CONFIG_SERIAL_8250_RT288X is not set
# CONFIG_SERIAL_OF_PLATFORM is not set
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_EARLYCON_ARM_SEMIHOST is not set
# CONFIG_SERIAL_MAX3100 is not set
# CONFIG_SERIAL_MAX310X is not set
# CONFIG_SERIAL_UARTLITE is not set
# CONFIG_SERIAL_SH_SCI is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_SC16IS7XX is not set
# CONFIG_SERIAL_BCM63XX is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_IFX6X60 is not set
# CONFIG_SERIAL_XILINX_PS_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_CONEXANT_DIGICOLOR is not set
# CONFIG_SERIAL_ST_ASC is not set
# CONFIG_SERIAL_STM32 is not set
# CONFIG_SERIAL_MVEBU_UART is not set
# CONFIG_TTY_PRINTK is not set
# CONFIG_HVC_DCC is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_R3964 is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_XILLYBUS is not set
#
# I2C support
#
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
#
# I2C Hardware Bus support
#
#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_CBUS_GPIO is not set
# CONFIG_I2C_DESIGNWARE_PLATFORM is not set
CONFIG_I2C_EMEV2=y
# CONFIG_I2C_GPIO is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_PXA_PCI is not set
# CONFIG_I2C_RIIC is not set
# CONFIG_I2C_RK3X is not set
# CONFIG_I2C_SH_MOBILE is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set
# CONFIG_I2C_RCAR is not set
#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_ROBOTFUZZ_OSIF is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set
#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_STUB is not set
CONFIG_I2C_SLAVE=y
# CONFIG_I2C_SLAVE_EEPROM is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_SPI=y
CONFIG_SPI_DEBUG=y
CONFIG_SPI_MASTER=y
#
# SPI Master Controller Drivers
#
# CONFIG_SPI_ALTERA is not set
# CONFIG_SPI_AXI_SPI_ENGINE is not set
# CONFIG_SPI_BITBANG is not set
# CONFIG_SPI_CADENCE is not set
# CONFIG_SPI_DESIGNWARE is not set
# CONFIG_SPI_GPIO is not set
# CONFIG_SPI_FSL_SPI is not set
# CONFIG_SPI_OC_TINY is not set
# CONFIG_SPI_PXA2XX_PCI is not set
# CONFIG_SPI_ROCKCHIP is not set
# CONFIG_SPI_RSPI is not set
# CONFIG_SPI_SC18IS602 is not set
# CONFIG_SPI_SH_MSIOF is not set
# CONFIG_SPI_SH_HSPI is not set
# CONFIG_SPI_XCOMM is not set
# CONFIG_SPI_XILINX is not set
# CONFIG_SPI_ZYNQMP_GQSPI is not set
#
# SPI Protocol Masters
#
CONFIG_SPI_SPIDEV=y
# CONFIG_SPI_LOOPBACK_TEST is not set
# CONFIG_SPI_TLE62X0 is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
#
# PPS support
#
# CONFIG_PPS is not set
#
# PPS generators support
#
#
# PTP clock support
#
# CONFIG_PTP_1588_CLOCK is not set
#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
CONFIG_PINCTRL=y
#
# Pin controllers
#
CONFIG_PINMUX=y
CONFIG_PINCONF=y
CONFIG_GENERIC_PINCONF=y
# CONFIG_DEBUG_PINCTRL is not set
# CONFIG_PINCTRL_AMD is not set
# CONFIG_PINCTRL_SINGLE is not set
CONFIG_PINCTRL_SH_PFC=y
CONFIG_PINCTRL_PFC_EMEV2=y
CONFIG_ARCH_HAVE_CUSTOM_GPIO_H=y
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_ARCH_REQUIRE_GPIOLIB=y
CONFIG_GPIOLIB=y
CONFIG_GPIO_DEVRES=y
CONFIG_OF_GPIO=y
CONFIG_GPIOLIB_IRQCHIP=y
CONFIG_DEBUG_GPIO=y
CONFIG_GPIO_SYSFS=y
#
# Memory mapped GPIO drivers
#
# CONFIG_GPIO_74XX_MMIO is not set
# CONFIG_GPIO_ALTERA is not set
# CONFIG_GPIO_DWAPB is not set
CONFIG_GPIO_EM=y
# CONFIG_GPIO_GENERIC_PLATFORM is not set
# CONFIG_GPIO_GRGPIO is not set
# CONFIG_GPIO_MPC8XXX is not set
# CONFIG_GPIO_RCAR is not set
# CONFIG_GPIO_XILINX is not set
# CONFIG_GPIO_ZEVIO is not set
# CONFIG_GPIO_ZX is not set
#
# I2C GPIO expanders
#
# CONFIG_GPIO_ADP5588 is not set
# CONFIG_GPIO_ADNP is not set
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
CONFIG_GPIO_PCF857X=y
# CONFIG_GPIO_SX150X is not set
# CONFIG_GPIO_TPIC2810 is not set
#
# MFD GPIO expanders
#
#
# SPI GPIO expanders
#
CONFIG_GPIO_74X164=y
# CONFIG_GPIO_MAX7301 is not set
# CONFIG_GPIO_MC33880 is not set
# CONFIG_GPIO_PISOSR is not set
#
# SPI or I2C GPIO expanders
#
# CONFIG_GPIO_MCP23S08 is not set
#
# USB GPIO expanders
#
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
CONFIG_POWER_SUPPLY_DEBUG=y
# CONFIG_PDA_POWER is not set
# CONFIG_GENERIC_ADC_BATTERY is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2781 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_SBS is not set
# CONFIG_BATTERY_BQ27XXX is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_MAX8903 is not set
# CONFIG_CHARGER_LP8727 is not set
# CONFIG_CHARGER_GPIO is not set
# CONFIG_CHARGER_MANAGER is not set
# CONFIG_CHARGER_BQ2415X is not set
# CONFIG_CHARGER_BQ24190 is not set
# CONFIG_CHARGER_BQ24257 is not set
# CONFIG_CHARGER_BQ24735 is not set
# CONFIG_CHARGER_BQ25890 is not set
# CONFIG_CHARGER_SMB347 is not set
# CONFIG_BATTERY_GAUGE_LTC2941 is not set
# CONFIG_CHARGER_RT9455 is not set
CONFIG_POWER_RESET=y
# CONFIG_POWER_RESET_GPIO is not set
# CONFIG_POWER_RESET_GPIO_RESTART is not set
# CONFIG_POWER_RESET_LTC2952 is not set
# CONFIG_POWER_RESET_RESTART is not set
# CONFIG_POWER_RESET_SYSCON is not set
# CONFIG_POWER_RESET_SYSCON_POWEROFF is not set
# CONFIG_POWER_AVS is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
#
# Broadcom specific AMBA
#
# CONFIG_BCMA is not set
#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
# CONFIG_MFD_ACT8945A is not set
CONFIG_MFD_AS3711=y
# CONFIG_MFD_AS3722 is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_AAT2870_CORE is not set
# CONFIG_MFD_ATMEL_FLEXCOM is not set
# CONFIG_MFD_ATMEL_HLCDC is not set
# CONFIG_MFD_BCM590XX is not set
# CONFIG_MFD_AXP20X_I2C is not set
# CONFIG_MFD_CROS_EC is not set
# CONFIG_MFD_ASIC3 is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_SPI is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
# CONFIG_MFD_DA9062 is not set
# CONFIG_MFD_DA9063 is not set
# CONFIG_MFD_DA9150 is not set
# CONFIG_MFD_DLN2 is not set
# CONFIG_MFD_MC13XXX_SPI is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_MFD_HI6421_PMIC is not set
# CONFIG_HTC_EGPIO is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_HTC_I2CPLD is not set
# CONFIG_INTEL_SOC_PMIC is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_88PM800 is not set
# CONFIG_MFD_88PM805 is not set
# CONFIG_MFD_88PM860X is not set
# CONFIG_MFD_MAX14577 is not set
# CONFIG_MFD_MAX77686 is not set
# CONFIG_MFD_MAX77693 is not set
# CONFIG_MFD_MAX77843 is not set
# CONFIG_MFD_MAX8907 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_MENF21BMC is not set
# CONFIG_EZX_PCAP is not set
# CONFIG_MFD_VIPERBOARD is not set
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_MFD_PM8921_CORE is not set
# CONFIG_MFD_RT5033 is not set
# CONFIG_MFD_RTSX_USB is not set
# CONFIG_MFD_RC5T583 is not set
# CONFIG_MFD_RK808 is not set
# CONFIG_MFD_RN5T618 is not set
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_MFD_SKY81452 is not set
# CONFIG_MFD_SMSC is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_STMPE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_LP3943 is not set
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_PALMAS is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS65010 is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS65086 is not set
# CONFIG_MFD_TPS65090 is not set
# CONFIG_MFD_TPS65217 is not set
# CONFIG_MFD_TPS65218 is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS65910 is not set
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_MFD_TPS65912_SPI is not set
# CONFIG_MFD_TPS80031 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_TWL6040_CORE is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TC3589X is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_MFD_T7L66XB is not set
# CONFIG_MFD_TC6387XB is not set
# CONFIG_MFD_TC6393XB is not set
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_MFD_ARIZONA_SPI is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM831X_SPI is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
CONFIG_REGULATOR=y
# CONFIG_REGULATOR_DEBUG is not set
CONFIG_REGULATOR_FIXED_VOLTAGE=y
# CONFIG_REGULATOR_VIRTUAL_CONSUMER is not set
# CONFIG_REGULATOR_USERSPACE_CONSUMER is not set
# CONFIG_REGULATOR_ACT8865 is not set
# CONFIG_REGULATOR_AD5398 is not set
CONFIG_REGULATOR_AS3711=y
# CONFIG_REGULATOR_DA9210 is not set
# CONFIG_REGULATOR_DA9211 is not set
# CONFIG_REGULATOR_FAN53555 is not set
# CONFIG_REGULATOR_GPIO is not set
# CONFIG_REGULATOR_ISL9305 is not set
# CONFIG_REGULATOR_ISL6271A is not set
# CONFIG_REGULATOR_LP3971 is not set
# CONFIG_REGULATOR_LP3972 is not set
# CONFIG_REGULATOR_LP872X is not set
# CONFIG_REGULATOR_LP8755 is not set
# CONFIG_REGULATOR_LTC3589 is not set
# CONFIG_REGULATOR_MAX1586 is not set
# CONFIG_REGULATOR_MAX8649 is not set
# CONFIG_REGULATOR_MAX8660 is not set
# CONFIG_REGULATOR_MAX8952 is not set
# CONFIG_REGULATOR_MT6311 is not set
# CONFIG_REGULATOR_PFUZE100 is not set
# CONFIG_REGULATOR_PV88060 is not set
# CONFIG_REGULATOR_PV88080 is not set
# CONFIG_REGULATOR_PV88090 is not set
# CONFIG_REGULATOR_TPS51632 is not set
# CONFIG_REGULATOR_TPS62360 is not set
# CONFIG_REGULATOR_TPS65023 is not set
# CONFIG_REGULATOR_TPS6507X is not set
# CONFIG_REGULATOR_TPS6524X is not set
# CONFIG_MEDIA_SUPPORT is not set
#
# Graphics support
#
# CONFIG_DRM is not set
#
# ACP (Audio CoProcessor) Configuration
#
# CONFIG_DRM_AMD_ACP is not set
#
# Frame buffer Devices
#
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
# CONFIG_FB_DDC is not set
# CONFIG_FB_BOOT_VESA_SUPPORT is not set
# CONFIG_FB_CFB_FILLRECT is not set
# CONFIG_FB_CFB_COPYAREA is not set
# CONFIG_FB_CFB_IMAGEBLIT is not set
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
# CONFIG_FB_MODE_HELPERS is not set
# CONFIG_FB_TILEBLITTING is not set
#
# Frame buffer hardware drivers
#
# CONFIG_FB_OPENCORES is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_SH_MOBILE_LCDC is not set
# CONFIG_FB_SMSCUFX is not set
# CONFIG_FB_UDL is not set
# CONFIG_FB_IBM_GXT4500 is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_BROADSHEET is not set
# CONFIG_FB_AUO_K190X is not set
# CONFIG_FB_SIMPLE is not set
# CONFIG_FB_SH_MOBILE_MERAM is not set
# CONFIG_FB_SSD1307 is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=y
# CONFIG_LCD_L4F00242T03 is not set
# CONFIG_LCD_LMS283GF05 is not set
# CONFIG_LCD_LTV350QV is not set
# CONFIG_LCD_ILI922X is not set
# CONFIG_LCD_ILI9320 is not set
# CONFIG_LCD_TDO24M is not set
# CONFIG_LCD_VGG2432A4 is not set
# CONFIG_LCD_PLATFORM is not set
# CONFIG_LCD_S6E63M0 is not set
# CONFIG_LCD_LD9040 is not set
# CONFIG_LCD_AMS369FG06 is not set
# CONFIG_LCD_LMS501KF03 is not set
# CONFIG_LCD_HX8357 is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=y
# CONFIG_BACKLIGHT_PM8941_WLED is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_ADP8870 is not set
# CONFIG_BACKLIGHT_LM3639 is not set
CONFIG_BACKLIGHT_AS3711=y
# CONFIG_BACKLIGHT_GPIO is not set
# CONFIG_BACKLIGHT_LV5207LP is not set
# CONFIG_BACKLIGHT_BD6107 is not set
# CONFIG_VGASTATE is not set
#
# Console display driver support
#
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
# CONFIG_SOUND_OSS_CORE is not set
CONFIG_SND=y
CONFIG_SND_PCM=y
CONFIG_SND_JACK=y
CONFIG_SND_JACK_INPUT_DEV=y
# CONFIG_SND_SEQUENCER is not set
# CONFIG_SND_MIXER_OSS is not set
# CONFIG_SND_PCM_OSS is not set
# CONFIG_SND_PCM_TIMER is not set
# CONFIG_SND_HRTIMER is not set
# CONFIG_SND_DYNAMIC_MINORS is not set
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_PROC_FS=y
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
# CONFIG_SND_RAWMIDI_SEQ is not set
# CONFIG_SND_OPL3_LIB_SEQ is not set
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
# CONFIG_SND_EMU10K1_SEQ is not set
# CONFIG_SND_DRIVERS is not set
#
# HD-Audio
#
CONFIG_SND_HDA_PREALLOC_SIZE=64
# CONFIG_SND_ARM is not set
# CONFIG_SND_SPI is not set
# CONFIG_SND_USB is not set
CONFIG_SND_SOC=y
# CONFIG_SND_SOC_AMD_ACP is not set
# CONFIG_SND_ATMEL_SOC is not set
# CONFIG_SND_DESIGNWARE_I2S is not set
#
# SoC Audio for Freescale CPUs
#
#
# Common SoC Audio options for Freescale CPUs:
#
# CONFIG_SND_SOC_FSL_ASRC is not set
# CONFIG_SND_SOC_FSL_SAI is not set
# CONFIG_SND_SOC_FSL_SSI is not set
# CONFIG_SND_SOC_FSL_SPDIF is not set
# CONFIG_SND_SOC_FSL_ESAI is not set
# CONFIG_SND_SOC_IMX_AUDMUX is not set
# CONFIG_SND_SOC_IMG is not set
#
# SoC Audio support for SuperH
#
# CONFIG_SND_SOC_SH4_FSI is not set
# CONFIG_SND_SOC_RCAR is not set
# CONFIG_SND_SOC_RSRC_CARD is not set
#
# Allwinner SoC Audio support
#
# CONFIG_SND_SUN4I_CODEC is not set
# CONFIG_SND_SUN4I_SPDIF is not set
# CONFIG_SND_SOC_XTFPGA_I2S is not set
CONFIG_SND_SOC_I2C_AND_SPI=y
#
# CODEC drivers
#
# CONFIG_SND_SOC_AC97_CODEC is not set
# CONFIG_SND_SOC_ADAU1701 is not set
# CONFIG_SND_SOC_AK4104 is not set
# CONFIG_SND_SOC_AK4554 is not set
# CONFIG_SND_SOC_AK4613 is not set
CONFIG_SND_SOC_AK4642=y
# CONFIG_SND_SOC_AK5386 is not set
# CONFIG_SND_SOC_ALC5623 is not set
# CONFIG_SND_SOC_CS35L32 is not set
# CONFIG_SND_SOC_CS42L51_I2C is not set
# CONFIG_SND_SOC_CS42L52 is not set
# CONFIG_SND_SOC_CS42L56 is not set
# CONFIG_SND_SOC_CS42L73 is not set
# CONFIG_SND_SOC_CS4265 is not set
# CONFIG_SND_SOC_CS4270 is not set
# CONFIG_SND_SOC_CS4271_I2C is not set
# CONFIG_SND_SOC_CS4271_SPI is not set
# CONFIG_SND_SOC_CS42XX8_I2C is not set
# CONFIG_SND_SOC_CS4349 is not set
# CONFIG_SND_SOC_ES8328 is not set
# CONFIG_SND_SOC_GTM601 is not set
# CONFIG_SND_SOC_INNO_RK3036 is not set
# CONFIG_SND_SOC_PCM1681 is not set
# CONFIG_SND_SOC_PCM179X_I2C is not set
# CONFIG_SND_SOC_PCM179X_SPI is not set
# CONFIG_SND_SOC_PCM3168A_I2C is not set
# CONFIG_SND_SOC_PCM3168A_SPI is not set
# CONFIG_SND_SOC_PCM512x_I2C is not set
# CONFIG_SND_SOC_PCM512x_SPI is not set
# CONFIG_SND_SOC_RT5616 is not set
# CONFIG_SND_SOC_RT5631 is not set
# CONFIG_SND_SOC_RT5677_SPI is not set
# CONFIG_SND_SOC_SGTL5000 is not set
# CONFIG_SND_SOC_SIRF_AUDIO_CODEC is not set
# CONFIG_SND_SOC_SPDIF is not set
# CONFIG_SND_SOC_SSM2602_SPI is not set
# CONFIG_SND_SOC_SSM2602_I2C is not set
# CONFIG_SND_SOC_SSM4567 is not set
# CONFIG_SND_SOC_STA32X is not set
# CONFIG_SND_SOC_STA350 is not set
# CONFIG_SND_SOC_STI_SAS is not set
# CONFIG_SND_SOC_TAS2552 is not set
# CONFIG_SND_SOC_TAS5086 is not set
# CONFIG_SND_SOC_TAS571X is not set
# CONFIG_SND_SOC_TFA9879 is not set
# CONFIG_SND_SOC_TLV320AIC23_I2C is not set
# CONFIG_SND_SOC_TLV320AIC23_SPI is not set
# CONFIG_SND_SOC_TLV320AIC31XX is not set
# CONFIG_SND_SOC_TLV320AIC3X is not set
# CONFIG_SND_SOC_TS3A227E is not set
# CONFIG_SND_SOC_WM8510 is not set
# CONFIG_SND_SOC_WM8523 is not set
# CONFIG_SND_SOC_WM8580 is not set
# CONFIG_SND_SOC_WM8711 is not set
# CONFIG_SND_SOC_WM8728 is not set
# CONFIG_SND_SOC_WM8731 is not set
# CONFIG_SND_SOC_WM8737 is not set
# CONFIG_SND_SOC_WM8741 is not set
# CONFIG_SND_SOC_WM8750 is not set
# CONFIG_SND_SOC_WM8753 is not set
# CONFIG_SND_SOC_WM8770 is not set
# CONFIG_SND_SOC_WM8776 is not set
# CONFIG_SND_SOC_WM8804_I2C is not set
# CONFIG_SND_SOC_WM8804_SPI is not set
# CONFIG_SND_SOC_WM8903 is not set
# CONFIG_SND_SOC_WM8962 is not set
# CONFIG_SND_SOC_WM8974 is not set
# CONFIG_SND_SOC_WM8978 is not set
# CONFIG_SND_SOC_TPA6130A2 is not set
CONFIG_SND_SIMPLE_CARD=y
# CONFIG_SOUND_PRIME is not set
#
# HID support
#
CONFIG_HID=y
# CONFIG_HID_BATTERY_STRENGTH is not set
# CONFIG_HIDRAW is not set
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=y
#
# Special HID drivers
#
# CONFIG_HID_A4TECH is not set
# CONFIG_HID_ACRUX is not set
# CONFIG_HID_APPLE is not set
# CONFIG_HID_APPLEIR is not set
# CONFIG_HID_AUREAL is not set
# CONFIG_HID_BELKIN is not set
# CONFIG_HID_BETOP_FF is not set
# CONFIG_HID_CHERRY is not set
# CONFIG_HID_CHICONY is not set
# CONFIG_HID_CORSAIR is not set
# CONFIG_HID_PRODIKEYS is not set
# CONFIG_HID_CMEDIA is not set
# CONFIG_HID_CP2112 is not set
# CONFIG_HID_CYPRESS is not set
# CONFIG_HID_DRAGONRISE is not set
# CONFIG_HID_EMS_FF is not set
# CONFIG_HID_ELECOM is not set
# CONFIG_HID_ELO is not set
# CONFIG_HID_EZKEY is not set
# CONFIG_HID_GEMBIRD is not set
# CONFIG_HID_GFRM is not set
# CONFIG_HID_HOLTEK is not set
# CONFIG_HID_GT683R is not set
# CONFIG_HID_KEYTOUCH is not set
# CONFIG_HID_KYE is not set
# CONFIG_HID_UCLOGIC is not set
# CONFIG_HID_WALTOP is not set
# CONFIG_HID_GYRATION is not set
# CONFIG_HID_ICADE is not set
# CONFIG_HID_TWINHAN is not set
# CONFIG_HID_KENSINGTON is not set
# CONFIG_HID_LCPOWER is not set
# CONFIG_HID_LENOVO is not set
# CONFIG_HID_LOGITECH is not set
# CONFIG_HID_MAGICMOUSE is not set
# CONFIG_HID_MICROSOFT is not set
# CONFIG_HID_MONTEREY is not set
# CONFIG_HID_MULTITOUCH is not set
# CONFIG_HID_NTRIG is not set
# CONFIG_HID_ORTEK is not set
# CONFIG_HID_PANTHERLORD is not set
# CONFIG_HID_PENMOUNT is not set
# CONFIG_HID_PETALYNX is not set
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_PLANTRONICS is not set
# CONFIG_HID_PRIMAX is not set
# CONFIG_HID_ROCCAT is not set
# CONFIG_HID_SAITEK is not set
# CONFIG_HID_SAMSUNG is not set
# CONFIG_HID_SONY is not set
# CONFIG_HID_SPEEDLINK is not set
# CONFIG_HID_STEELSERIES is not set
# CONFIG_HID_SUNPLUS is not set
# CONFIG_HID_RMI is not set
# CONFIG_HID_GREENASIA is not set
# CONFIG_HID_SMARTJOYPLUS is not set
# CONFIG_HID_TIVO is not set
# CONFIG_HID_TOPSEED is not set
# CONFIG_HID_THINGM is not set
# CONFIG_HID_THRUSTMASTER is not set
# CONFIG_HID_WACOM is not set
# CONFIG_HID_WIIMOTE is not set
# CONFIG_HID_XINMO is not set
# CONFIG_HID_ZEROPLUS is not set
# CONFIG_HID_ZYDACRON is not set
# CONFIG_HID_SENSOR_HUB is not set
#
# USB HID support
#
CONFIG_USB_HID=y
# CONFIG_HID_PID is not set
# CONFIG_USB_HIDDEV is not set
#
# I2C HID support
#
# CONFIG_I2C_HID is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
#
# Miscellaneous USB options
#
CONFIG_USB_DEFAULT_PERSIST=y
CONFIG_USB_DYNAMIC_MINORS=y
# CONFIG_USB_OTG is not set
# CONFIG_USB_OTG_WHITELIST is not set
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
# CONFIG_USB_ULPI_BUS is not set
# CONFIG_USB_MON is not set
# CONFIG_USB_WUSB_CBAF is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
# CONFIG_USB_EHCI_HCD is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
# CONFIG_USB_FOTG210_HCD is not set
# CONFIG_USB_MAX3421_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
CONFIG_USB_R8A66597_HCD=y
# CONFIG_USB_HCD_TEST_MODE is not set
# CONFIG_USB_RENESAS_USBHS is not set
#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set
#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#
#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_STORAGE_ENE_UB6250 is not set
# CONFIG_USB_UAS is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USBIP_CORE is not set
# CONFIG_USB_MUSB_HDRC is not set
# CONFIG_USB_DWC3 is not set
# CONFIG_USB_DWC2 is not set
# CONFIG_USB_CHIPIDEA is not set
# CONFIG_USB_ISP1760 is not set
#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_EHSET_TEST_FIXTURE is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_YUREX is not set
# CONFIG_USB_EZUSB_FX2 is not set
# CONFIG_USB_HSIC_USB3503 is not set
# CONFIG_USB_LINK_LAYER_TEST is not set
#
# USB Physical Layer drivers
#
# CONFIG_USB_PHY is not set
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_AM335X_PHY_USB is not set
# CONFIG_USB_GPIO_VBUS is not set
# CONFIG_USB_ISP1301 is not set
# CONFIG_USB_ULPI is not set
CONFIG_USB_GADGET=y
# CONFIG_USB_GADGET_DEBUG is not set
# CONFIG_USB_GADGET_DEBUG_FILES is not set
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_VBUS_DRAW=2
CONFIG_USB_GADGET_STORAGE_NUM_BUFFERS=2
#
# USB Peripheral Controller
#
# CONFIG_USB_FUSB300 is not set
# CONFIG_USB_FOTG210_UDC is not set
# CONFIG_USB_GR_UDC is not set
# CONFIG_USB_R8A66597 is not set
# CONFIG_USB_RENESAS_USB3 is not set
# CONFIG_USB_PXA27X is not set
# CONFIG_USB_MV_UDC is not set
# CONFIG_USB_MV_U3D is not set
# CONFIG_USB_M66592 is not set
# CONFIG_USB_BDC_UDC is not set
# CONFIG_USB_NET2272 is not set
# CONFIG_USB_GADGET_XILINX is not set
# CONFIG_USB_DUMMY_HCD is not set
CONFIG_USB_LIBCOMPOSITE=m
CONFIG_USB_U_ETHER=m
CONFIG_USB_F_ECM=m
CONFIG_USB_F_SUBSET=m
CONFIG_USB_F_RNDIS=m
CONFIG_USB_F_MASS_STORAGE=m
# CONFIG_USB_CONFIGFS is not set
# CONFIG_USB_ZERO is not set
# CONFIG_USB_AUDIO is not set
CONFIG_USB_ETH=m
CONFIG_USB_ETH_RNDIS=y
# CONFIG_USB_ETH_EEM is not set
# CONFIG_USB_G_NCM is not set
# CONFIG_USB_GADGETFS is not set
# CONFIG_USB_FUNCTIONFS is not set
CONFIG_USB_MASS_STORAGE=m
# CONFIG_USB_G_SERIAL is not set
# CONFIG_USB_MIDI_GADGET is not set
# CONFIG_USB_G_PRINTER is not set
# CONFIG_USB_CDC_COMPOSITE is not set
# CONFIG_USB_G_ACM_MS is not set
# CONFIG_USB_G_MULTI is not set
# CONFIG_USB_G_HID is not set
# CONFIG_USB_G_DBGP is not set
CONFIG_USB_LED_TRIG=y
# CONFIG_UWB is not set
CONFIG_MMC=y
# CONFIG_MMC_DEBUG is not set
CONFIG_PWRSEQ_EMMC=y
CONFIG_PWRSEQ_SIMPLE=y
#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=y
CONFIG_MMC_BLOCK_MINORS=8
# CONFIG_MMC_BLOCK_BOUNCE is not set
# CONFIG_SDIO_UART is not set
# CONFIG_MMC_TEST is not set
#
# MMC/SD/SDIO Host Controller Drivers
#
# CONFIG_MMC_SDHCI is not set
CONFIG_MMC_TMIO_CORE=y
CONFIG_MMC_SDHI=y
# CONFIG_MMC_DW is not set
# CONFIG_MMC_SH_MMCIF is not set
# CONFIG_MMC_VUB300 is not set
# CONFIG_MMC_USHC is not set
# CONFIG_MMC_USDHI6ROL0 is not set
# CONFIG_MMC_MTK is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
# CONFIG_LEDS_CLASS_FLASH is not set
#
# LED drivers
#
# CONFIG_LEDS_BCM6328 is not set
# CONFIG_LEDS_BCM6358 is not set
# CONFIG_LEDS_LM3530 is not set
# CONFIG_LEDS_LM3642 is not set
# CONFIG_LEDS_PCA9532 is not set
CONFIG_LEDS_GPIO=y
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_LP5521 is not set
# CONFIG_LEDS_LP5523 is not set
# CONFIG_LEDS_LP5562 is not set
# CONFIG_LEDS_LP8501 is not set
# CONFIG_LEDS_LP8860 is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_PCA963X is not set
# CONFIG_LEDS_DAC124S085 is not set
# CONFIG_LEDS_REGULATOR is not set
# CONFIG_LEDS_BD2802 is not set
# CONFIG_LEDS_LT3593 is not set
# CONFIG_LEDS_TCA6507 is not set
# CONFIG_LEDS_TLC591XX is not set
# CONFIG_LEDS_LM355x is not set
# CONFIG_LEDS_IS31FL32XX is not set
#
# LED driver for blink(1) USB RGB LED is under Special HID drivers (HID_THINGM)
#
# CONFIG_LEDS_BLINKM is not set
#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=y
CONFIG_LEDS_TRIGGER_ONESHOT=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
CONFIG_LEDS_TRIGGER_CPU=y
CONFIG_LEDS_TRIGGER_GPIO=y
CONFIG_LEDS_TRIGGER_DEFAULT_ON=y
#
# iptables trigger is under Netfilter config (LED target)
#
CONFIG_LEDS_TRIGGER_TRANSIENT=y
# CONFIG_LEDS_TRIGGER_CAMERA is not set
CONFIG_LEDS_TRIGGER_PANIC=y
# CONFIG_ACCESSIBILITY is not set
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
CONFIG_RTC_SYSTOHC=y
CONFIG_RTC_SYSTOHC_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_ABB5ZES3 is not set
# CONFIG_RTC_DRV_ABX80X is not set
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_HYM8563 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
CONFIG_RTC_DRV_RS5C372=y
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_ISL12022 is not set
# CONFIG_RTC_DRV_ISL12057 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8523 is not set
# CONFIG_RTC_DRV_PCF85063 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_BQ32K is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8010 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set
# CONFIG_RTC_DRV_EM3027 is not set
# CONFIG_RTC_DRV_RV3029C2 is not set
# CONFIG_RTC_DRV_RV8803 is not set
#
# SPI RTC drivers
#
# CONFIG_RTC_DRV_M41T93 is not set
# CONFIG_RTC_DRV_M41T94 is not set
# CONFIG_RTC_DRV_DS1305 is not set
# CONFIG_RTC_DRV_DS1343 is not set
# CONFIG_RTC_DRV_DS1347 is not set
# CONFIG_RTC_DRV_DS1390 is not set
# CONFIG_RTC_DRV_R9701 is not set
# CONFIG_RTC_DRV_RX4581 is not set
# CONFIG_RTC_DRV_RX6110 is not set
# CONFIG_RTC_DRV_RS5C348 is not set
# CONFIG_RTC_DRV_MAX6902 is not set
# CONFIG_RTC_DRV_PCF2123 is not set
# CONFIG_RTC_DRV_MCP795 is not set
CONFIG_RTC_I2C_AND_SPI=y
#
# SPI and I2C RTC drivers
#
# CONFIG_RTC_DRV_DS3232 is not set
# CONFIG_RTC_DRV_PCF2127 is not set
#
# Platform RTC drivers
#
# CONFIG_RTC_DRV_CMOS is not set
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1685_FAMILY is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_DS2404 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set
# CONFIG_RTC_DRV_V3020 is not set
# CONFIG_RTC_DRV_ZYNQMP is not set
#
# on-CPU RTC drivers
#
# CONFIG_RTC_DRV_SNVS is not set
#
# HID Sensor RTC drivers
#
# CONFIG_RTC_DRV_HID_SENSOR_TIME is not set
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set
#
# DMA Devices
#
CONFIG_DMA_ENGINE=y
CONFIG_DMA_OF=y
# CONFIG_FSL_EDMA is not set
# CONFIG_INTEL_IDMA64 is not set
# CONFIG_NBPFAXI_DMA is not set
# CONFIG_QCOM_HIDMA_MGMT is not set
# CONFIG_QCOM_HIDMA is not set
# CONFIG_DW_DMAC is not set
# CONFIG_SH_DMAE_BASE is not set
# CONFIG_RCAR_DMAC is not set
# CONFIG_RENESAS_USB_DMAC is not set
#
# DMA Clients
#
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_VIRT_DRIVERS is not set
#
# Virtio drivers
#
# CONFIG_VIRTIO_MMIO is not set
#
# Microsoft Hyper-V guest support
#
CONFIG_STAGING=y
# CONFIG_COMEDI is not set
#
# IIO staging drivers
#
#
# Accelerometers
#
# CONFIG_ADIS16201 is not set
# CONFIG_ADIS16203 is not set
# CONFIG_ADIS16204 is not set
# CONFIG_ADIS16209 is not set
# CONFIG_ADIS16220 is not set
# CONFIG_ADIS16240 is not set
# CONFIG_LIS3L02DQ is not set
# CONFIG_SCA3000 is not set
#
# Analog to digital converters
#
# CONFIG_AD7606 is not set
# CONFIG_AD7780 is not set
# CONFIG_AD7816 is not set
# CONFIG_AD7192 is not set
# CONFIG_AD7280 is not set
#
# Analog digital bi-direction converters
#
# CONFIG_ADT7316 is not set
#
# Capacitance to digital converters
#
# CONFIG_AD7150 is not set
# CONFIG_AD7152 is not set
# CONFIG_AD7746 is not set
#
# Direct Digital Synthesis
#
# CONFIG_AD9832 is not set
# CONFIG_AD9834 is not set
#
# Digital gyroscope sensors
#
# CONFIG_ADIS16060 is not set
#
# Network Analyzer, Impedance Converters
#
# CONFIG_AD5933 is not set
#
# Light sensors
#
# CONFIG_SENSORS_ISL29018 is not set
# CONFIG_SENSORS_ISL29028 is not set
# CONFIG_TSL2583 is not set
# CONFIG_TSL2x7x is not set
#
# Active energy metering IC
#
# CONFIG_ADE7753 is not set
# CONFIG_ADE7754 is not set
# CONFIG_ADE7758 is not set
# CONFIG_ADE7759 is not set
# CONFIG_ADE7854 is not set
#
# Resolver to digital converters
#
# CONFIG_AD2S90 is not set
# CONFIG_AD2S1200 is not set
# CONFIG_AD2S1210 is not set
#
# Triggers - standalone
#
# CONFIG_USB_EMXX is not set
#
# Speakup console speech
#
# CONFIG_SPEAKUP is not set
# CONFIG_STAGING_MEDIA is not set
#
# Android
#
# CONFIG_STAGING_BOARD is not set
# CONFIG_LTE_GDM724X is not set
# CONFIG_LNET is not set
# CONFIG_GS_FPGABOOT is not set
# CONFIG_COMMON_CLK_XLNX_CLKWZRD is not set
# CONFIG_FB_TFT is not set
# CONFIG_MOST is not set
# CONFIG_GOLDFISH is not set
# CONFIG_CHROME_PLATFORMS is not set
CONFIG_CLKDEV_LOOKUP=y
CONFIG_HAVE_CLK_PREPARE=y
CONFIG_COMMON_CLK=y
#
# Common Clock Framework
#
# CONFIG_COMMON_CLK_SI5351 is not set
# CONFIG_COMMON_CLK_SI514 is not set
# CONFIG_COMMON_CLK_SI570 is not set
# CONFIG_COMMON_CLK_CDCE706 is not set
# CONFIG_COMMON_CLK_CDCE925 is not set
# CONFIG_COMMON_CLK_CS2000_CP is not set
# CONFIG_CLK_QORIQ is not set
# CONFIG_COMMON_CLK_NXP is not set
# CONFIG_COMMON_CLK_PXA is not set
#
# Hardware Spinlock drivers
#
#
# Clock Source drivers
#
CONFIG_CLKSRC_OF=y
CONFIG_CLKSRC_PROBE=y
# CONFIG_ARM_TIMER_SP804 is not set
# CONFIG_ATMEL_PIT is not set
CONFIG_SYS_SUPPORTS_EM_STI=y
# CONFIG_SH_TIMER_CMT is not set
# CONFIG_SH_TIMER_MTU2 is not set
# CONFIG_SH_TIMER_TMU is not set
CONFIG_EM_TIMER_STI=y
# CONFIG_MAILBOX is not set
# CONFIG_IOMMU_SUPPORT is not set
#
# Remoteproc drivers
#
# CONFIG_STE_MODEM_RPROC is not set
#
# Rpmsg drivers
#
#
# SOC (System On Chip) specific Drivers
#
# CONFIG_SOC_BRCMSTB is not set
# CONFIG_SUNXI_SRAM is not set
# CONFIG_SOC_TI is not set
# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
CONFIG_IIO=y
CONFIG_IIO_BUFFER=y
# CONFIG_IIO_BUFFER_CB is not set
CONFIG_IIO_KFIFO_BUF=y
CONFIG_IIO_CONFIGFS=y
CONFIG_IIO_TRIGGER=y
CONFIG_IIO_CONSUMERS_PER_TRIGGER=2
CONFIG_IIO_SW_TRIGGER=y
#
# Accelerometers
#
# CONFIG_BMA180 is not set
# CONFIG_BMC150_ACCEL is not set
# CONFIG_IIO_ST_ACCEL_3AXIS is not set
# CONFIG_KXSD9 is not set
# CONFIG_KXCJK1013 is not set
# CONFIG_MMA7455_I2C is not set
# CONFIG_MMA7455_SPI is not set
# CONFIG_MMA8452 is not set
# CONFIG_MMA9551 is not set
# CONFIG_MMA9553 is not set
# CONFIG_MXC4005 is not set
# CONFIG_MXC6255 is not set
# CONFIG_STK8312 is not set
# CONFIG_STK8BA50 is not set
#
# Analog to digital converters
#
# CONFIG_AD7266 is not set
# CONFIG_AD7291 is not set
# CONFIG_AD7298 is not set
# CONFIG_AD7476 is not set
# CONFIG_AD7791 is not set
# CONFIG_AD7793 is not set
# CONFIG_AD7887 is not set
# CONFIG_AD7923 is not set
# CONFIG_AD799X is not set
# CONFIG_CC10001_ADC is not set
# CONFIG_HI8435 is not set
# CONFIG_INA2XX_ADC is not set
# CONFIG_MAX1027 is not set
# CONFIG_MAX1363 is not set
# CONFIG_MCP320X is not set
# CONFIG_MCP3422 is not set
# CONFIG_NAU7802 is not set
# CONFIG_TI_ADC081C is not set
# CONFIG_TI_ADC0832 is not set
# CONFIG_TI_ADC128S052 is not set
# CONFIG_TI_ADS1015 is not set
# CONFIG_TI_ADS8688 is not set
# CONFIG_VF610_ADC is not set
#
# Amplifiers
#
# CONFIG_AD8366 is not set
#
# Chemical Sensors
#
# CONFIG_ATLAS_PH_SENSOR is not set
# CONFIG_IAQCORE is not set
# CONFIG_VZ89X is not set
#
# Hid Sensor IIO Common
#
#
# SSP Sensor Common
#
# CONFIG_IIO_SSP_SENSORHUB is not set
#
# Digital to analog converters
#
# CONFIG_AD5064 is not set
# CONFIG_AD5360 is not set
# CONFIG_AD5380 is not set
# CONFIG_AD5421 is not set
# CONFIG_AD5446 is not set
# CONFIG_AD5449 is not set
# CONFIG_AD5504 is not set
# CONFIG_AD5624R_SPI is not set
# CONFIG_AD5686 is not set
# CONFIG_AD5755 is not set
# CONFIG_AD5761 is not set
# CONFIG_AD5764 is not set
# CONFIG_AD5791 is not set
# CONFIG_AD7303 is not set
# CONFIG_M62332 is not set
# CONFIG_MAX517 is not set
# CONFIG_MAX5821 is not set
# CONFIG_MCP4725 is not set
# CONFIG_MCP4922 is not set
# CONFIG_VF610_DAC is not set
#
# IIO dummy driver
#
# CONFIG_IIO_SIMPLE_DUMMY is not set
#
# Frequency Synthesizers DDS/PLL
#
#
# Clock Generator/Distribution
#
# CONFIG_AD9523 is not set
#
# Phase-Locked Loop (PLL) frequency synthesizers
#
# CONFIG_ADF4350 is not set
#
# Digital gyroscope sensors
#
# CONFIG_ADIS16080 is not set
# CONFIG_ADIS16130 is not set
# CONFIG_ADIS16136 is not set
# CONFIG_ADIS16260 is not set
# CONFIG_ADXRS450 is not set
# CONFIG_BMG160 is not set
# CONFIG_IIO_ST_GYRO_3AXIS is not set
# CONFIG_ITG3200 is not set
#
# Health Sensors
#
#
# Heart Rate Monitors
#
# CONFIG_AFE4403 is not set
# CONFIG_AFE4404 is not set
# CONFIG_MAX30100 is not set
#
# Humidity sensors
#
# CONFIG_DHT11 is not set
# CONFIG_HDC100X is not set
# CONFIG_HTU21 is not set
# CONFIG_SI7005 is not set
# CONFIG_SI7020 is not set
#
# Inertial measurement units
#
# CONFIG_ADIS16400 is not set
# CONFIG_ADIS16480 is not set
# CONFIG_KMX61 is not set
# CONFIG_INV_MPU6050_SPI is not set
#
# Light sensors
#
# CONFIG_ADJD_S311 is not set
# CONFIG_AL3320A is not set
# CONFIG_APDS9300 is not set
# CONFIG_APDS9960 is not set
# CONFIG_BH1750 is not set
# CONFIG_CM32181 is not set
# CONFIG_CM3232 is not set
# CONFIG_CM3323 is not set
# CONFIG_CM36651 is not set
# CONFIG_GP2AP020A00F is not set
# CONFIG_ISL29125 is not set
# CONFIG_JSA1212 is not set
# CONFIG_RPR0521 is not set
# CONFIG_LTR501 is not set
# CONFIG_OPT3001 is not set
# CONFIG_PA12203001 is not set
# CONFIG_STK3310 is not set
# CONFIG_TCS3414 is not set
# CONFIG_TCS3472 is not set
# CONFIG_SENSORS_TSL2563 is not set
# CONFIG_TSL4531 is not set
# CONFIG_US5182D is not set
# CONFIG_VCNL4000 is not set
#
# Magnetometer sensors
#
CONFIG_AK8975=y
# CONFIG_AK09911 is not set
# CONFIG_BMC150_MAGN is not set
# CONFIG_MAG3110 is not set
# CONFIG_MMC35240 is not set
# CONFIG_IIO_ST_MAGN_3AXIS is not set
# CONFIG_SENSORS_HMC5843_I2C is not set
# CONFIG_SENSORS_HMC5843_SPI is not set
#
# Inclinometer sensors
#
#
# Triggers - standalone
#
# CONFIG_IIO_HRTIMER_TRIGGER is not set
# CONFIG_IIO_INTERRUPT_TRIGGER is not set
# CONFIG_IIO_SYSFS_TRIGGER is not set
#
# Digital potentiometers
#
# CONFIG_MCP4531 is not set
# CONFIG_TPL0102 is not set
#
# Pressure sensors
#
# CONFIG_BMP280 is not set
# CONFIG_MPL115_I2C is not set
# CONFIG_MPL115_SPI is not set
# CONFIG_MPL3115 is not set
# CONFIG_MS5611 is not set
# CONFIG_MS5637 is not set
# CONFIG_IIO_ST_PRESS is not set
# CONFIG_T5403 is not set
#
# Lightning sensors
#
# CONFIG_AS3935 is not set
#
# Proximity sensors
#
# CONFIG_LIDAR_LITE_V2 is not set
# CONFIG_SX9500 is not set
#
# Temperature sensors
#
# CONFIG_MLX90614 is not set
# CONFIG_TMP006 is not set
# CONFIG_TSYS01 is not set
# CONFIG_TSYS02D is not set
# CONFIG_PWM is not set
CONFIG_IRQCHIP=y
CONFIG_ARM_GIC=y
CONFIG_ARM_GIC_MAX_NR=1
# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set
# CONFIG_FMC is not set
#
# PHY Subsystem
#
# CONFIG_GENERIC_PHY is not set
# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
# CONFIG_PHY_RCAR_GEN3_USB2 is not set
# CONFIG_BCM_KONA_USB2_PHY is not set
# CONFIG_POWERCAP is not set
# CONFIG_MCB is not set
#
# Performance monitor support
#
CONFIG_ARM_PMU=y
# CONFIG_RAS is not set
#
# Android
#
# CONFIG_ANDROID is not set
# CONFIG_NVMEM is not set
CONFIG_STM=y
# CONFIG_STM_DUMMY is not set
# CONFIG_STM_SOURCE_CONSOLE is not set
# CONFIG_STM_SOURCE_HEARTBEAT is not set
# CONFIG_INTEL_TH is not set
#
# FPGA Configuration Support
#
# CONFIG_FPGA is not set
#
# Firmware Drivers
#
# CONFIG_FIRMWARE_MEMMAP is not set
CONFIG_HAVE_ARM_SMCCC=y
#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
# CONFIG_EXT4_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
# CONFIG_F2FS_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_FILE_LOCKING=y
# CONFIG_MANDATORY_FILE_LOCKING is not set
# CONFIG_FS_ENCRYPTION is not set
CONFIG_FSNOTIFY=y
# CONFIG_DNOTIFY is not set
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
# CONFIG_QUOTA is not set
# CONFIG_QUOTACTL is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
# CONFIG_OVERLAY_FS is not set
#
# Caches
#
# CONFIG_FSCACHE is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
# CONFIG_MSDOS_FS is not set
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_FAT_DEFAULT_UTF8 is not set
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_PROC_CHILDREN=y
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
# CONFIG_NFS_V4_2 is not set
CONFIG_PNFS_FILE_LAYOUT=y
CONFIG_PNFS_FLEXFILE_LAYOUT=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_V4_1_MIGRATION is not set
CONFIG_ROOT_NFS=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
# CONFIG_NFSD is not set
CONFIG_GRACE_PERIOD=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_SUNRPC_BACKCHANNEL=y
# CONFIG_SUNRPC_DEBUG is not set
# CONFIG_CEPH_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_MAC_ROMAN is not set
# CONFIG_NLS_MAC_CELTIC is not set
# CONFIG_NLS_MAC_CENTEURO is not set
# CONFIG_NLS_MAC_CROATIAN is not set
# CONFIG_NLS_MAC_CYRILLIC is not set
# CONFIG_NLS_MAC_GAELIC is not set
# CONFIG_NLS_MAC_GREEK is not set
# CONFIG_NLS_MAC_ICELAND is not set
# CONFIG_NLS_MAC_INUIT is not set
# CONFIG_NLS_MAC_ROMANIAN is not set
# CONFIG_NLS_MAC_TURKISH is not set
# CONFIG_NLS_UTF8 is not set
# CONFIG_DLM is not set
#
# Kernel hacking
#
#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_DYNAMIC_DEBUG is not set
#
# Compile-time checks and compiler options
#
# CONFIG_DEBUG_INFO is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_PAGE_OWNER is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_MAGIC_SYSRQ is not set
CONFIG_DEBUG_KERNEL=y
#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_MEMORY_INIT is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_DEBUG_HIGHMEM is not set
# CONFIG_DEBUG_SHIRQ is not set
#
# Debug Lockups and Hangs
#
CONFIG_LOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_WQ_WATCHDOG=y
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_PANIC_TIMEOUT=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHED_INFO=y
CONFIG_SCHEDSTATS=y
# CONFIG_SCHED_STACK_END_CHECK is not set
CONFIG_DEBUG_TIMEKEEPING=y
# CONFIG_TIMER_STATS is not set
#
# Lock Debugging (spinlocks, mutexes, etc...)
#
# CONFIG_DEBUG_RT_MUTEXES is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_PI_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
#
# RCU Debugging
#
# CONFIG_PROVE_RCU is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_TORTURE_TEST is not set
# CONFIG_RCU_PERF_TEST is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_EQS_DEBUG=y
# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_CPU_HOTPLUG_STATE_CONTROL is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set
#
# Runtime Testing
#
# CONFIG_LKDTM is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_RBTREE_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
# CONFIG_PERCPU_TEST is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_TEST_HEXDUMP is not set
# CONFIG_TEST_STRING_HELPERS is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_TEST_PRINTF is not set
# CONFIG_TEST_BITMAP is not set
# CONFIG_TEST_RHASHTABLE is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_TEST_LKM is not set
# CONFIG_TEST_USER_COPY is not set
# CONFIG_TEST_BPF is not set
# CONFIG_TEST_FIRMWARE is not set
# CONFIG_TEST_UDELAY is not set
CONFIG_MEMTEST=y
# CONFIG_TEST_STATIC_KEYS is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_UBSAN is not set
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
# CONFIG_STRICT_DEVMEM is not set
# CONFIG_ARM_PTDUMP is not set
CONFIG_ARM_UNWIND=y
# CONFIG_DEBUG_USER is not set
# CONFIG_DEBUG_LL is not set
CONFIG_DEBUG_LL_INCLUDE="mach/debug-macro.S"
# CONFIG_DEBUG_UART_8250 is not set
CONFIG_UNCOMPRESS_INCLUDE="debug/uncompress.h"
# CONFIG_PID_IN_CONTEXTIDR is not set
# CONFIG_DEBUG_SET_MODULE_RONX is not set
# CONFIG_CORESIGHT is not set
#
# Security options
#
CONFIG_KEYS=y
# CONFIG_PERSISTENT_KEYRINGS is not set
# CONFIG_BIG_KEYS is not set
# CONFIG_ENCRYPTED_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=m
CONFIG_CRYPTO_AKCIPHER2=y
# CONFIG_CRYPTO_RSA is not set
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_USER is not set
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
# CONFIG_CRYPTO_GF128MUL is not set
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_NULL2=y
# CONFIG_CRYPTO_PCRYPT is not set
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_MCRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set
#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_CHACHA20POLY1305 is not set
# CONFIG_CRYPTO_SEQIV is not set
CONFIG_CRYPTO_ECHAINIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set
# CONFIG_CRYPTO_KEYWRAP is not set
#
# Hash modes
#
# CONFIG_CRYPTO_CMAC is not set
CONFIG_CRYPTO_HMAC=m
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_VMAC is not set
#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_CRC32 is not set
# CONFIG_CRYPTO_CRCT10DIF is not set
# CONFIG_CRYPTO_GHASH is not set
# CONFIG_CRYPTO_POLY1305 is not set
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set
#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_CHACHA20 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_LZO is not set
# CONFIG_CRYPTO_842 is not set
# CONFIG_CRYPTO_LZ4 is not set
# CONFIG_CRYPTO_LZ4HC is not set
#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_DRBG_MENU=m
CONFIG_CRYPTO_DRBG_HMAC=y
# CONFIG_CRYPTO_DRBG_HASH is not set
# CONFIG_CRYPTO_DRBG_CTR is not set
CONFIG_CRYPTO_DRBG=m
CONFIG_CRYPTO_JITTERENTROPY=m
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
# CONFIG_CRYPTO_USER_API_RNG is not set
# CONFIG_CRYPTO_USER_API_AEAD is not set
CONFIG_CRYPTO_HW=y
# CONFIG_ASYMMETRIC_KEY_TYPE is not set
#
# Certificates for signature checking
#
# CONFIG_SYSTEM_TRUSTED_KEYRING is not set
CONFIG_ARM_CRYPTO=y
CONFIG_CRYPTO_SHA1_ARM=y
CONFIG_CRYPTO_SHA256_ARM=y
# CONFIG_CRYPTO_SHA512_ARM is not set
CONFIG_CRYPTO_AES_ARM=y
# CONFIG_BINARY_PRINTF is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_HAVE_ARCH_BITREVERSE=y
CONFIG_RATIONAL=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
# CONFIG_CRC_T10DIF is not set
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
# CONFIG_CRC8 is not set
# CONFIG_AUDIT_ARCH_COMPAT_GENERIC is not set
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
# CONFIG_XZ_DEC is not set
# CONFIG_XZ_DEC_BCJ is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_ASSOCIATIVE_ARRAY=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_DMA=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
# CONFIG_CORDIC is not set
# CONFIG_DDR is not set
# CONFIG_IRQ_POLL is not set
CONFIG_LIBFDT=y
CONFIG_OID_REGISTRY=y
CONFIG_FONT_SUPPORT=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_SG_SPLIT is not set
CONFIG_SG_POOL=y
CONFIG_ARCH_HAS_SG_CHAIN=y
# CONFIG_VIRTUALIZATION is not set
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-13 19:43 Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache) Geert Uytterhoeven
@ 2016-06-14 6:24 ` Joonsoo Kim
2016-06-14 7:31 ` Geert Uytterhoeven
2016-06-14 13:10 ` Geert Uytterhoeven
1 sibling, 1 reply; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-14 6:24 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> Hi Joonsoo,
Hello,
>
> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> >
> > To check whther free objects exist or not precisely, we need to grab a
> > lock. But, accuracy isn't that important because race window would be
> > even small and if there is too much free object, cache reaper would reap
> > it. So, this patch makes the check for free object exisistence not to
> > hold a lock. This will reduce lock contention in heavily allocation case.
> >
> > Note that until now, n->shared can be freed during the processing by
> > writing slabinfo, but, with some trick in this patch, we can access it
> > freely within interrupt disabled period.
> >
> > Below is the result of concurrent allocation/free in slab allocation
> > benchmark made by Christoph a long time ago. I make the output simpler.
> > The number shows cycle count during alloc/free respectively so less is
> > better.
> >
> > * Before
> > Kmalloc N*alloc N*free(32): Average=248/966
> > Kmalloc N*alloc N*free(64): Average=261/949
> > Kmalloc N*alloc N*free(128): Average=314/1016
> > Kmalloc N*alloc N*free(256): Average=741/1061
> > Kmalloc N*alloc N*free(512): Average=1246/1152
> > Kmalloc N*alloc N*free(1024): Average=2437/1259
> > Kmalloc N*alloc N*free(2048): Average=4980/1800
> > Kmalloc N*alloc N*free(4096): Average=9000/2078
> >
> > * After
> > Kmalloc N*alloc N*free(32): Average=344/792
> > Kmalloc N*alloc N*free(64): Average=347/882
> > Kmalloc N*alloc N*free(128): Average=390/959
> > Kmalloc N*alloc N*free(256): Average=393/1067
> > Kmalloc N*alloc N*free(512): Average=683/1229
> > Kmalloc N*alloc N*free(1024): Average=1295/1325
> > Kmalloc N*alloc N*free(2048): Average=2513/1664
> > Kmalloc N*alloc N*free(4096): Average=4742/2172
> >
> > It shows that allocation performance decreases for the object size up to
> > 128 and it may be due to extra checks in cache_alloc_refill(). But, with
> > considering improvement of free performance, net result looks the same.
> > Result for other size class looks very promising, roughly, 50% performance
> > improvement.
> >
> > v2: replace kick_all_cpus_sync() with synchronize_sched().
> >
> > Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>
> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> (Renesas dual Cortex A9) to this patch, which is upstream commit
> 801faf0db8947e01877920e848a4d338dd7a99e7.
>
> I've attached my .config. I don't know if it also happens with
> shmobile_defconfig, as something went wrong with my remote access to the board,
> preventing further testing. I also couldn't verify if the issue persists in
> v4.7-rc3.
>
> Do you have a clue?
I don't have yet. Could you help me to narrow down the problem?
Following diff is half-revert change to check that synchronize_sched()
has no problem.
Thanks.
----->8-----
diff --git a/mm/slab.c b/mm/slab.c
index 763096a..257a0eb 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3016,9 +3016,6 @@ static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)
n = get_node(cachep, node);
BUG_ON(ac->avail > 0 || !n);
- shared = READ_ONCE(n->shared);
- if (!n->free_objects && (!shared || !shared->avail))
- goto direct_grow;
spin_lock(&n->list_lock);
shared = READ_ONCE(n->shared);
@@ -3047,7 +3044,6 @@ alloc_done:
spin_unlock(&n->list_lock);
fixup_objfreelist_debug(cachep, &list);
-direct_grow:
if (unlikely(!ac->avail)) {
/* Check if we can use obj in pfmemalloc slab */
if (sk_memalloc_socks()) {
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-14 6:24 ` Joonsoo Kim
@ 2016-06-14 7:31 ` Geert Uytterhoeven
2016-06-14 8:11 ` Joonsoo Kim
0 siblings, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-14 7:31 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
Hi Joonsoo,
On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
>> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
>> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>> > To check whther free objects exist or not precisely, we need to grab a
>> > lock. But, accuracy isn't that important because race window would be
>> > even small and if there is too much free object, cache reaper would reap
>> > it. So, this patch makes the check for free object exisistence not to
>> > hold a lock. This will reduce lock contention in heavily allocation case.
>> >
>> > Note that until now, n->shared can be freed during the processing by
>> > writing slabinfo, but, with some trick in this patch, we can access it
>> > freely within interrupt disabled period.
>> >
>> > Below is the result of concurrent allocation/free in slab allocation
>> > benchmark made by Christoph a long time ago. I make the output simpler.
>> > The number shows cycle count during alloc/free respectively so less is
>> > better.
>> >
>> > * Before
>> > Kmalloc N*alloc N*free(32): Average=248/966
>> > Kmalloc N*alloc N*free(64): Average=261/949
>> > Kmalloc N*alloc N*free(128): Average=314/1016
>> > Kmalloc N*alloc N*free(256): Average=741/1061
>> > Kmalloc N*alloc N*free(512): Average=1246/1152
>> > Kmalloc N*alloc N*free(1024): Average=2437/1259
>> > Kmalloc N*alloc N*free(2048): Average=4980/1800
>> > Kmalloc N*alloc N*free(4096): Average=9000/2078
>> >
>> > * After
>> > Kmalloc N*alloc N*free(32): Average=344/792
>> > Kmalloc N*alloc N*free(64): Average=347/882
>> > Kmalloc N*alloc N*free(128): Average=390/959
>> > Kmalloc N*alloc N*free(256): Average=393/1067
>> > Kmalloc N*alloc N*free(512): Average=683/1229
>> > Kmalloc N*alloc N*free(1024): Average=1295/1325
>> > Kmalloc N*alloc N*free(2048): Average=2513/1664
>> > Kmalloc N*alloc N*free(4096): Average=4742/2172
>> >
>> > It shows that allocation performance decreases for the object size up to
>> > 128 and it may be due to extra checks in cache_alloc_refill(). But, with
>> > considering improvement of free performance, net result looks the same.
>> > Result for other size class looks very promising, roughly, 50% performance
>> > improvement.
>> >
>> > v2: replace kick_all_cpus_sync() with synchronize_sched().
>> >
>> > Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>>
>> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
>> (Renesas dual Cortex A9) to this patch, which is upstream commit
>> 801faf0db8947e01877920e848a4d338dd7a99e7.
>>
>> I've attached my .config. I don't know if it also happens with
>> shmobile_defconfig, as something went wrong with my remote access to the board,
>> preventing further testing. I also couldn't verify if the issue persists in
>> v4.7-rc3.
In the mean time, I've verified it also happens with shmobile_defconfig.
>>
>> Do you have a clue?
>
> I don't have yet. Could you help me to narrow down the problem?
> Following diff is half-revert change to check that synchronize_sched()
> has no problem.
Thanks!
Unfortunately the half revert is not sufficient. The full revert is.
> ----->8-----
> diff --git a/mm/slab.c b/mm/slab.c
> index 763096a..257a0eb 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -3016,9 +3016,6 @@ static void *cache_alloc_refill(struct kmem_cache *cachep, gfp_t flags)
> n = get_node(cachep, node);
>
> BUG_ON(ac->avail > 0 || !n);
> - shared = READ_ONCE(n->shared);
> - if (!n->free_objects && (!shared || !shared->avail))
> - goto direct_grow;
>
> spin_lock(&n->list_lock);
> shared = READ_ONCE(n->shared);
> @@ -3047,7 +3044,6 @@ alloc_done:
> spin_unlock(&n->list_lock);
> fixup_objfreelist_debug(cachep, &list);
>
> -direct_grow:
> if (unlikely(!ac->avail)) {
> /* Check if we can use obj in pfmemalloc slab */
> if (sk_memalloc_socks()) {
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-14 7:31 ` Geert Uytterhoeven
@ 2016-06-14 8:11 ` Joonsoo Kim
2016-06-14 10:45 ` Geert Uytterhoeven
0 siblings, 1 reply; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-14 8:11 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> Hi Joonsoo,
>
> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> >> > To check whther free objects exist or not precisely, we need to grab a
> >> > lock. But, accuracy isn't that important because race window would be
> >> > even small and if there is too much free object, cache reaper would reap
> >> > it. So, this patch makes the check for free object exisistence not to
> >> > hold a lock. This will reduce lock contention in heavily allocation case.
> >> >
> >> > Note that until now, n->shared can be freed during the processing by
> >> > writing slabinfo, but, with some trick in this patch, we can access it
> >> > freely within interrupt disabled period.
> >> >
> >> > Below is the result of concurrent allocation/free in slab allocation
> >> > benchmark made by Christoph a long time ago. I make the output simpler.
> >> > The number shows cycle count during alloc/free respectively so less is
> >> > better.
> >> >
> >> > * Before
> >> > Kmalloc N*alloc N*free(32): Average=248/966
> >> > Kmalloc N*alloc N*free(64): Average=261/949
> >> > Kmalloc N*alloc N*free(128): Average=314/1016
> >> > Kmalloc N*alloc N*free(256): Average=741/1061
> >> > Kmalloc N*alloc N*free(512): Average=1246/1152
> >> > Kmalloc N*alloc N*free(1024): Average=2437/1259
> >> > Kmalloc N*alloc N*free(2048): Average=4980/1800
> >> > Kmalloc N*alloc N*free(4096): Average=9000/2078
> >> >
> >> > * After
> >> > Kmalloc N*alloc N*free(32): Average=344/792
> >> > Kmalloc N*alloc N*free(64): Average=347/882
> >> > Kmalloc N*alloc N*free(128): Average=390/959
> >> > Kmalloc N*alloc N*free(256): Average=393/1067
> >> > Kmalloc N*alloc N*free(512): Average=683/1229
> >> > Kmalloc N*alloc N*free(1024): Average=1295/1325
> >> > Kmalloc N*alloc N*free(2048): Average=2513/1664
> >> > Kmalloc N*alloc N*free(4096): Average=4742/2172
> >> >
> >> > It shows that allocation performance decreases for the object size up to
> >> > 128 and it may be due to extra checks in cache_alloc_refill(). But, with
> >> > considering improvement of free performance, net result looks the same.
> >> > Result for other size class looks very promising, roughly, 50% performance
> >> > improvement.
> >> >
> >> > v2: replace kick_all_cpus_sync() with synchronize_sched().
> >> >
> >> > Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> >>
> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> >>
> >> I've attached my .config. I don't know if it also happens with
> >> shmobile_defconfig, as something went wrong with my remote access to the board,
> >> preventing further testing. I also couldn't verify if the issue persists in
> >> v4.7-rc3.
>
> In the mean time, I've verified it also happens with shmobile_defconfig.
>
> >>
> >> Do you have a clue?
> >
> > I don't have yet. Could you help me to narrow down the problem?
> > Following diff is half-revert change to check that synchronize_sched()
> > has no problem.
>
> Thanks!
>
> Unfortunately the half revert is not sufficient. The full revert is.
Thanks for quick testing!
Could I ask one more time to check that synchronize_sched() is root
cause of the problem? Testing following two diffs will be helpful to me.
Thanks.
------->8--------
diff --git a/mm/slab.c b/mm/slab.c
index 763096a..d892364 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -965,7 +965,7 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
* freed after synchronize_sched().
*/
if (force_change)
- synchronize_sched();
+ kick_all_cpus_sync();
fail:
kfree(old_shared);
------->8------
diff --git a/mm/slab.c b/mm/slab.c
index 763096a..38d99c2 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -964,8 +964,6 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
* guaranteed to be valid until irq is re-enabled, because it will be
* freed after synchronize_sched().
*/
- if (force_change)
- synchronize_sched();
fail:
kfree(old_shared);
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-14 8:11 ` Joonsoo Kim
@ 2016-06-14 10:45 ` Geert Uytterhoeven
2016-06-15 2:23 ` Joonsoo Kim
0 siblings, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-14 10:45 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
Hi Joonsoo,
On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
>> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
>> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
>> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
>> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>> >> > To check whther free objects exist or not precisely, we need to grab a
>> >> > lock. But, accuracy isn't that important because race window would be
>> >> > even small and if there is too much free object, cache reaper would reap
>> >> > it. So, this patch makes the check for free object exisistence not to
>> >> > hold a lock. This will reduce lock contention in heavily allocation case.
>> >> >
>> >> > Note that until now, n->shared can be freed during the processing by
>> >> > writing slabinfo, but, with some trick in this patch, we can access it
>> >> > freely within interrupt disabled period.
>> >> >
>> >> > Below is the result of concurrent allocation/free in slab allocation
>> >> > benchmark made by Christoph a long time ago. I make the output simpler.
>> >> > The number shows cycle count during alloc/free respectively so less is
>> >> > better.
>> >> >
>> >> > * Before
>> >> > Kmalloc N*alloc N*free(32): Average=248/966
>> >> > Kmalloc N*alloc N*free(64): Average=261/949
>> >> > Kmalloc N*alloc N*free(128): Average=314/1016
>> >> > Kmalloc N*alloc N*free(256): Average=741/1061
>> >> > Kmalloc N*alloc N*free(512): Average=1246/1152
>> >> > Kmalloc N*alloc N*free(1024): Average=2437/1259
>> >> > Kmalloc N*alloc N*free(2048): Average=4980/1800
>> >> > Kmalloc N*alloc N*free(4096): Average=9000/2078
>> >> >
>> >> > * After
>> >> > Kmalloc N*alloc N*free(32): Average=344/792
>> >> > Kmalloc N*alloc N*free(64): Average=347/882
>> >> > Kmalloc N*alloc N*free(128): Average=390/959
>> >> > Kmalloc N*alloc N*free(256): Average=393/1067
>> >> > Kmalloc N*alloc N*free(512): Average=683/1229
>> >> > Kmalloc N*alloc N*free(1024): Average=1295/1325
>> >> > Kmalloc N*alloc N*free(2048): Average=2513/1664
>> >> > Kmalloc N*alloc N*free(4096): Average=4742/2172
>> >> >
>> >> > It shows that allocation performance decreases for the object size up to
>> >> > 128 and it may be due to extra checks in cache_alloc_refill(). But, with
>> >> > considering improvement of free performance, net result looks the same.
>> >> > Result for other size class looks very promising, roughly, 50% performance
>> >> > improvement.
>> >> >
>> >> > v2: replace kick_all_cpus_sync() with synchronize_sched().
>> >> >
>> >> > Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>> >>
>> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
>> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
>> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
>> >>
>> >> I've attached my .config. I don't know if it also happens with
>> >> shmobile_defconfig, as something went wrong with my remote access to the board,
>> >> preventing further testing. I also couldn't verify if the issue persists in
>> >> v4.7-rc3.
>>
>> In the mean time, I've verified it also happens with shmobile_defconfig.
>>
>> >> Do you have a clue?
>> >
>> > I don't have yet. Could you help me to narrow down the problem?
>> > Following diff is half-revert change to check that synchronize_sched()
>> > has no problem.
>>
>> Thanks!
>>
>> Unfortunately the half revert is not sufficient. The full revert is.
>
> Thanks for quick testing!
>
> Could I ask one more time to check that synchronize_sched() is root
> cause of the problem? Testing following two diffs will be helpful to me.
>
> Thanks.
>
> ------->8--------
> diff --git a/mm/slab.c b/mm/slab.c
> index 763096a..d892364 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -965,7 +965,7 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> * freed after synchronize_sched().
> */
> if (force_change)
> - synchronize_sched();
> + kick_all_cpus_sync();
>
> fail:
> kfree(old_shared);
Works.
> ------->8------
> diff --git a/mm/slab.c b/mm/slab.c
> index 763096a..38d99c2 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -964,8 +964,6 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> * guaranteed to be valid until irq is re-enabled, because it will be
> * freed after synchronize_sched().
> */
> - if (force_change)
> - synchronize_sched();
>
> fail:
> kfree(old_shared);
>
Also works.
Note that I do not see this problem on any of the other boards I use, one
of which is also a dual Cortex A9.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-13 19:43 Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache) Geert Uytterhoeven
2016-06-14 6:24 ` Joonsoo Kim
@ 2016-06-14 13:10 ` Geert Uytterhoeven
1 sibling, 0 replies; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-14 13:10 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel, Joonsoo Kim,
linux-renesas-soc, linux-arm-kernel
Hi Jonsoo,
On Mon, Jun 13, 2016 at 9:43 PM, Geert Uytterhoeven
<geert@linux-m68k.org> wrote:
> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
>> From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>> To check whther free objects exist or not precisely, we need to grab a
>> lock. But, accuracy isn't that important because race window would be
>> even small and if there is too much free object, cache reaper would reap
>> it. So, this patch makes the check for free object exisistence not to
>> hold a lock. This will reduce lock contention in heavily allocation case.
>> Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>
> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> (Renesas dual Cortex A9) to this patch, which is upstream commit
> 801faf0db8947e01877920e848a4d338dd7a99e7.
BTW, when disabling SMP, the problem goes away.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-14 10:45 ` Geert Uytterhoeven
@ 2016-06-15 2:23 ` Joonsoo Kim
2016-06-15 8:39 ` Geert Uytterhoeven
0 siblings, 1 reply; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-15 2:23 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
> Hi Joonsoo,
>
> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> >> >> > To check whther free objects exist or not precisely, we need to grab a
> >> >> > lock. But, accuracy isn't that important because race window would be
> >> >> > even small and if there is too much free object, cache reaper would reap
> >> >> > it. So, this patch makes the check for free object exisistence not to
> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
> >> >> >
> >> >> > Note that until now, n->shared can be freed during the processing by
> >> >> > writing slabinfo, but, with some trick in this patch, we can access it
> >> >> > freely within interrupt disabled period.
> >> >> >
> >> >> > Below is the result of concurrent allocation/free in slab allocation
> >> >> > benchmark made by Christoph a long time ago. I make the output simpler.
> >> >> > The number shows cycle count during alloc/free respectively so less is
> >> >> > better.
> >> >> >
> >> >> > * Before
> >> >> > Kmalloc N*alloc N*free(32): Average=248/966
> >> >> > Kmalloc N*alloc N*free(64): Average=261/949
> >> >> > Kmalloc N*alloc N*free(128): Average=314/1016
> >> >> > Kmalloc N*alloc N*free(256): Average=741/1061
> >> >> > Kmalloc N*alloc N*free(512): Average=1246/1152
> >> >> > Kmalloc N*alloc N*free(1024): Average=2437/1259
> >> >> > Kmalloc N*alloc N*free(2048): Average=4980/1800
> >> >> > Kmalloc N*alloc N*free(4096): Average=9000/2078
> >> >> >
> >> >> > * After
> >> >> > Kmalloc N*alloc N*free(32): Average=344/792
> >> >> > Kmalloc N*alloc N*free(64): Average=347/882
> >> >> > Kmalloc N*alloc N*free(128): Average=390/959
> >> >> > Kmalloc N*alloc N*free(256): Average=393/1067
> >> >> > Kmalloc N*alloc N*free(512): Average=683/1229
> >> >> > Kmalloc N*alloc N*free(1024): Average=1295/1325
> >> >> > Kmalloc N*alloc N*free(2048): Average=2513/1664
> >> >> > Kmalloc N*alloc N*free(4096): Average=4742/2172
> >> >> >
> >> >> > It shows that allocation performance decreases for the object size up to
> >> >> > 128 and it may be due to extra checks in cache_alloc_refill(). But, with
> >> >> > considering improvement of free performance, net result looks the same.
> >> >> > Result for other size class looks very promising, roughly, 50% performance
> >> >> > improvement.
> >> >> >
> >> >> > v2: replace kick_all_cpus_sync() with synchronize_sched().
> >> >> >
> >> >> > Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> >> >>
> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> >> >>
> >> >> I've attached my .config. I don't know if it also happens with
> >> >> shmobile_defconfig, as something went wrong with my remote access to the board,
> >> >> preventing further testing. I also couldn't verify if the issue persists in
> >> >> v4.7-rc3.
> >>
> >> In the mean time, I've verified it also happens with shmobile_defconfig.
> >>
> >> >> Do you have a clue?
> >> >
> >> > I don't have yet. Could you help me to narrow down the problem?
> >> > Following diff is half-revert change to check that synchronize_sched()
> >> > has no problem.
> >>
> >> Thanks!
> >>
> >> Unfortunately the half revert is not sufficient. The full revert is.
> >
> > Thanks for quick testing!
> >
> > Could I ask one more time to check that synchronize_sched() is root
> > cause of the problem? Testing following two diffs will be helpful to me.
> >
> > Thanks.
> >
> > ------->8--------
> > diff --git a/mm/slab.c b/mm/slab.c
> > index 763096a..d892364 100644
> > --- a/mm/slab.c
> > +++ b/mm/slab.c
> > @@ -965,7 +965,7 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > * freed after synchronize_sched().
> > */
> > if (force_change)
> > - synchronize_sched();
> > + kick_all_cpus_sync();
> >
> > fail:
> > kfree(old_shared);
>
> Works.
>
> > ------->8------
> > diff --git a/mm/slab.c b/mm/slab.c
> > index 763096a..38d99c2 100644
> > --- a/mm/slab.c
> > +++ b/mm/slab.c
> > @@ -964,8 +964,6 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > * guaranteed to be valid until irq is re-enabled, because it will be
> > * freed after synchronize_sched().
> > */
> > - if (force_change)
> > - synchronize_sched();
> >
> > fail:
> > kfree(old_shared);
> >
>
> Also works.
>
> Note that I do not see this problem on any of the other boards I use, one
> of which is also a dual Cortex A9.
Thanks for your help!
It's curious that synchronize_sched() has some effect in this early
phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
First, I'd like to confirm that num_online_cpus() is correct.
Could you try following patch and give me a dmesg?
Thanks.
------->8----------
diff --git a/mm/slab.c b/mm/slab.c
index 763096a..5b7300a 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
* guaranteed to be valid until irq is re-enabled, because it will be
* freed after synchronize_sched().
*/
- if (force_change)
- synchronize_sched();
+ if (force_change) {
+ WARN_ON_ONCE(num_online_cpus() <= 1);
+ WARN_ON_ONCE(num_online_cpus() > 1);
+ }
fail:
kfree(old_shared);
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-15 2:23 ` Joonsoo Kim
@ 2016-06-15 8:39 ` Geert Uytterhoeven
2016-06-20 6:39 ` Joonsoo Kim
0 siblings, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-15 8:39 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
Hi Joonsoo,
On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
>> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
>> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
>> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
>> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
>> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
>> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
>> >> >> > To check whther free objects exist or not precisely, we need to grab a
>> >> >> > lock. But, accuracy isn't that important because race window would be
>> >> >> > even small and if there is too much free object, cache reaper would reap
>> >> >> > it. So, this patch makes the check for free object exisistence not to
>> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
>> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
>> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
>> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> It's curious that synchronize_sched() has some effect in this early
> phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
> it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
>
> It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
>
> First, I'd like to confirm that num_online_cpus() is correct.
> Could you try following patch and give me a dmesg?
>
> Thanks.
>
> ------->8----------
> diff --git a/mm/slab.c b/mm/slab.c
> index 763096a..5b7300a 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> * guaranteed to be valid until irq is re-enabled, because it will be
> * freed after synchronize_sched().
> */
> - if (force_change)
> - synchronize_sched();
> + if (force_change) {
> + WARN_ON_ONCE(num_online_cpus() <= 1);
> + WARN_ON_ONCE(num_online_cpus() > 1);
> + }
Full dmesg output below.
I also tested whether it's the call to synchronize_sched() before or after
secondary CPU bringup that hangs.
if (force_change && num_online_cpus() <= 1)
synchronize_sched();
boots.
if (force_change && num_online_cpus() > 1)
synchronize_sched();
hangs.
Booting Linux on physical CPU 0x0
Linux version 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty (geert@ramsan)
(gcc version 4.9.0 (GCC) ) #84 SMP Wed Jun 15 10:20:12 CEST 2016
CPU: ARMv7 Processor [411fc093] revision 3 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: EMEV2 KZM9D Board
debug: ignoring loglevel setting.
Memory policy: Data cache writealloc
On node 0 totalpages: 32768
free_area_init_node: node 0, pgdat c09286c0, node_mem_map c7efa000
Normal zone: 256 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 32768 pages, LIFO batch:7
percpu: Embedded 12 pages/cpu @c7ed9000 s19264 r8192 d21696 u49152
pcpu-alloc: s19264 r8192 d21696 u49152 alloc=12*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
Kernel command line: console=ttyS1,115200n81 ignore_loglevel
root=/dev/nfs ip=dhcp
PID hash table entries: 512 (order: -1, 2048 bytes)
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 121144K/131072K available (4243K kernel code, 165K rwdata,
1344K rodata, 2048K init, 264K bss, 9928K reserved, 0K cma-reserved,
0K highmem)
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xc8800000 - 0xff800000 ( 880 MB)
lowmem : 0xc0000000 - 0xc8000000 ( 128 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc0674eb8 (6580 kB)
.init : 0xc0700000 - 0xc0900000 (2048 kB)
.data : 0xc0900000 - 0xc0929420 ( 166 kB)
.bss : 0xc092b000 - 0xc096d1e8 ( 265 kB)
Hierarchical RCU implementation.
Build-time adjustment of leaf fanout to 32.
RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
NR_IRQS:16 nr_irqs:16 16
clocksource_probe: no matching clocksources found
sched_clock: 32 bits at 100 Hz, resolution 10000000ns, wraps every
21474836475000000ns
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at mm/slab.c:975 setup_kmem_cache_node+0x160/0x1c8
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted
4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
[<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
[<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
[<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
[<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
[<c011fad0>] (warn_slowpath_null) from [<c01ced4c>]
(setup_kmem_cache_node+0x160/0x1c8)
[<c01ced4c>] (setup_kmem_cache_node) from [<c01cf02c>]
(__do_tune_cpucache+0xf4/0x114)
[<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
[<c01cf0b0>] (enable_cpucache) from [<c0710324>]
(kmem_cache_init_late+0x40/0x84)
[<c0710324>] (kmem_cache_init_late) from [<c0700af8>] (start_kernel+0x238/0x36c)
[<c0700af8>] (start_kernel) from [<4000807c>] (0x4000807c)
---[ end trace cb88537fdc8fa200 ]---
Console: colour dummy device 80x30
Calibrating delay loop (skipped) preset value.. 1066.00 BogoMIPS (lpj=5330000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x40100000 - 0x40100058
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Brought up 2 CPUs
SMP: Total of 2 processors activated (2132.00 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
------------[ cut here ]------------
WARNING: CPU: 0 PID: 1 at mm/slab.c:976 setup_kmem_cache_node+0x198/0x1c8
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W
4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
[<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
[<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
[<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
[<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
[<c011fad0>] (warn_slowpath_null) from [<c01ced84>]
(setup_kmem_cache_node+0x198/0x1c8)
[<c01ced84>] (setup_kmem_cache_node) from [<c01cf02c>]
(__do_tune_cpucache+0xf4/0x114)
[<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
[<c01cf0b0>] (enable_cpucache) from [<c01cf4cc>]
(__kmem_cache_create+0x1a0/0x1c8)
[<c01cf4cc>] (__kmem_cache_create) from [<c01b2904>]
(kmem_cache_create+0xbc/0x190)
[<c01b2904>] (kmem_cache_create) from [<c070d724>] (shmem_init+0x34/0xb0)
[<c070d724>] (shmem_init) from [<c0700cc4>] (kernel_init_freeable+0x98/0x1ec)
[<c0700cc4>] (kernel_init_freeable) from [<c0497760>] (kernel_init+0x8/0x110)
[<c0497760>] (kernel_init) from [<c0106c78>] (ret_from_fork+0x14/0x3c)
---[ end trace cb88537fdc8fa201 ]---
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
max_idle_ns: 19112604462750000 ns
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
sh-pfc e0140200.pfc: emev2_pfc support registered
gpiochip_find_base: found new base at 992
gpio gpiochip0: (e0050000.gpio): created GPIO range 0->31 ==>
e0140200.pfc PIN 0->31
gpio gpiochip0: (e0050000.gpio): added GPIO chardev (254:0)
gpiochip_setup_dev: registered GPIOs 992 to 1023 on device: gpiochip0
(e0050000.gpio)
gpiochip_find_base: found new base at 960
gpio gpiochip1: (e0050080.gpio): created GPIO range 0->31 ==>
e0140200.pfc PIN 32->63
gpio gpiochip1: (e0050080.gpio): added GPIO chardev (254:1)
gpiochip_setup_dev: registered GPIOs 960 to 991 on device: gpiochip1
(e0050080.gpio)
gpiochip_find_base: found new base at 928
gpio gpiochip2: (e0050100.gpio): created GPIO range 0->31 ==>
e0140200.pfc PIN 64->95
gpio gpiochip2: (e0050100.gpio): added GPIO chardev (254:2)
gpiochip_setup_dev: registered GPIOs 928 to 959 on device: gpiochip2
(e0050100.gpio)
gpiochip_find_base: found new base at 896
gpio gpiochip3: (e0050180.gpio): created GPIO range 0->31 ==>
e0140200.pfc PIN 96->127
gpio gpiochip3: (e0050180.gpio): added GPIO chardev (254:3)
gpiochip_setup_dev: registered GPIOs 896 to 927 on device: gpiochip3
(e0050180.gpio)
gpiochip_find_base: found new base at 865
gpio gpiochip4: (e0050200.gpio): created GPIO range 0->30 ==>
e0140200.pfc PIN 128->158
gpio gpiochip4: (e0050200.gpio): added GPIO chardev (254:4)
gpiochip_setup_dev: registered GPIOs 865 to 895 on device: gpiochip4
(e0050200.gpio)
gio: map hw irq = 1, irq = 35
gio: sense irq = 1, mode = 8
No ATAGs?
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@0[0]'
of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@1[0]'
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
em_sti e0180000.timer: used for clock events
em_sti e0180000.timer: used for oneshot clock events
em_sti e0180000.timer: used as clock source
clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
0x1ef4687b1, max_idle_ns: 3697658158765000000 ns
Advanced Linux Sound Architecture Driver Initialized.
NET: Registered protocol family 23
clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
0x1ef4687b1, max_idle_ns: 112843571739654 ns
clocksource: Switched to clocksource e0180000.timer
NET: Registered protocol family 2
TCP established hash table entries: 1024 (order: 0, 4096 bytes)
TCP bind hash table entries: 1024 (order: 2, 20480 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
UDP hash table entries: 256 (order: 1, 12288 bytes)
UDP-Lite hash table entries: 256 (order: 1, 12288 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Clockevents: could not switch to one-shot mode:
Clockevents: could not switch to one-shot mode: dummy_timer is not functional.
Could not switch to high resolution mode on CPU 0
dummy_timer is not functional.
Could not switch to high resolution mode on CPU 1
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
futex hash table entries: 512 (order: 3, 32768 bytes)
workingset: timestamp_bits=28 max_order=15 bucket_order=0
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
nfs4filelayout_init: NFSv4 File Layout Driver Registering...
io scheduler noop registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
e1020000.serial: ttyS0 at MMIO 0xe1020000 (irq = 19, base_baud =
796444) is a 16550A
e1030000.serial: ttyS1 at MMIO 0xe1030000 (irq = 20, base_baud =
7168000) is a 16550A
console [ttyS1] enabled
e1040000.serial: ttyS2 at MMIO 0xe1040000 (irq = 21, base_baud =
14336000) is a 16550A
e1050000.serial: ttyS3 at MMIO 0xe1050000 (irq = 22, base_baud =
2389333) is a 16550A
gio: sense irq = 1, mode = 8
libphy: smsc911x-mdio: probed
Generic PHY 20000000.etherne:01: attached PHY driver [Generic PHY]
(mii_bus:phy_addr=20000000.etherne:01, irq=-1)
smsc911x 20000000.ethernet eth0: MAC Address: 00:01:9b:04:03:cf
usbcore: registered new interface driver usb-storage
i2c /dev entries driver
em-i2c e0070000.i2c: Added i2c controller 0, irq 33
em-i2c e10a0000.i2c: Added i2c controller 1, irq 34
cpu cpu0: failed to get clock: -2
cpufreq-dt: probe of cpufreq-dt failed with error -2
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
NET: Registered protocol family 17
Key type dns_resolver registered
Registering SWP/SWPB emulation handler
of_get_named_gpiod_flags: parsed 'gpios' property of node
'/gpio_keys/button@1[0]' - status (0)
of_get_named_gpiod_flags: parsed 'gpios' property of node
'/gpio_keys/button@2[0]' - status (0)
of_get_named_gpiod_flags: parsed 'gpios' property of node
'/gpio_keys/button@3[0]' - status (0)
of_get_named_gpiod_flags: parsed 'gpios' property of node
'/gpio_keys/button@4[0]' - status (0)
gpio-1006 (DSW2-1): gpiod_set_debounce: missing set() or
set_debounce() operations
gio: map hw irq = 14, irq = 36
gio: sense irq = 14, mode = 12
gpio-1007 (DSW2-2): gpiod_set_debounce: missing set() or
set_debounce() operations
gio: map hw irq = 15, irq = 37
gio: sense irq = 15, mode = 12
gpio-1008 (DSW2-3): gpiod_set_debounce: missing set() or
set_debounce() operations
gio: map hw irq = 16, irq = 38
gio: sense irq = 16, mode = 12
gpio-1009 (DSW2-4): gpiod_set_debounce: missing set() or
set_debounce() operations
gio: map hw irq = 17, irq = 39
gio: sense irq = 17, mode = 12
input: gpio_keys as /devices/platform/gpio_keys/input/input0
hctosys: unable to open rtc device (rtc0)
smsc911x 20000000.ethernet eth0: SMSC911x/921x identified at 0xc8880000, IRQ: 35
Sending DHCP requests .., OK
IP-Config: Got DHCP answer from 192.168.97.254, my address is 192.168.97.215
IP-Config: Complete:
device=eth0, hwaddr=00:01:9b:04:03:cf, ipaddr=192.168.97.215,
mask=255.255.255.0, gw=192.168.97.254
host=192.168.97.215, domain=of.borg, nis-domain=(none)
bootserver=192.168.97.254, rootserver=192.168.97.254, rootpath=
nameserver0=192.168.97.254
ALSA device list:
No soundcards found.
Freeing unused kernel memory: 2048K (c0700000 - c0900000)
sysctl: error: 'kernel.hotplug' is an unknown key
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-15 8:39 ` Geert Uytterhoeven
@ 2016-06-20 6:39 ` Joonsoo Kim
2016-06-20 13:12 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-20 6:39 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Andrew Morton, Christoph Lameter, Pekka Enberg, David Rientjes,
Jesper Dangaard Brouer, Paul E. McKenney, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
CCing Paul to ask some question.
On Wed, Jun 15, 2016 at 10:39:47AM +0200, Geert Uytterhoeven wrote:
> Hi Joonsoo,
>
> On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
> >> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> >> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> >> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> >> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> >> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> >> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> >> >> >> > To check whther free objects exist or not precisely, we need to grab a
> >> >> >> > lock. But, accuracy isn't that important because race window would be
> >> >> >> > even small and if there is too much free object, cache reaper would reap
> >> >> >> > it. So, this patch makes the check for free object exisistence not to
> >> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
>
> >> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> >> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> >> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
>
> > It's curious that synchronize_sched() has some effect in this early
> > phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
> > it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
> >
> > It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
> >
> > First, I'd like to confirm that num_online_cpus() is correct.
> > Could you try following patch and give me a dmesg?
> >
> > Thanks.
> >
> > ------->8----------
> > diff --git a/mm/slab.c b/mm/slab.c
> > index 763096a..5b7300a 100644
> > --- a/mm/slab.c
> > +++ b/mm/slab.c
> > @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > * guaranteed to be valid until irq is re-enabled, because it will be
> > * freed after synchronize_sched().
> > */
> > - if (force_change)
> > - synchronize_sched();
> > + if (force_change) {
> > + WARN_ON_ONCE(num_online_cpus() <= 1);
> > + WARN_ON_ONCE(num_online_cpus() > 1);
> > + }
>
> Full dmesg output below.
>
> I also tested whether it's the call to synchronize_sched() before or after
> secondary CPU bringup that hangs.
>
> if (force_change && num_online_cpus() <= 1)
> synchronize_sched();
>
> boots.
>
> if (force_change && num_online_cpus() > 1)
> synchronize_sched();
>
> hangs.
Hello, Paul.
I changed slab.c to use synchronize_sched() for full memory barrier. First
call happens on kmem_cache_init_late() and it would not be a problem
because, at this time, num_online_cpus() <= 1 and synchronize_sched()
would return immediately. Second call site would be shmem_init()
and it seems that system hangs on it. Since smp is already initialized
at that time, there would be some effect of synchronize_sched() but I
can't imagine what's wrong here. Is it invalid moment to call
synchronize_sched()?
Note that my x86 virtual machine works fine even if
synchronize_sched() is called in shmem_init() but Geert's some ARM
machines (not all ARM machine) don't work well with it.
Thanks.
>
> Booting Linux on physical CPU 0x0
> Linux version 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty (geert@ramsan)
> (gcc version 4.9.0 (GCC) ) #84 SMP Wed Jun 15 10:20:12 CEST 2016
> CPU: ARMv7 Processor [411fc093] revision 3 (ARMv7), cr=10c5387d
> CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
> Machine model: EMEV2 KZM9D Board
> debug: ignoring loglevel setting.
> Memory policy: Data cache writealloc
> On node 0 totalpages: 32768
> free_area_init_node: node 0, pgdat c09286c0, node_mem_map c7efa000
> Normal zone: 256 pages used for memmap
> Normal zone: 0 pages reserved
> Normal zone: 32768 pages, LIFO batch:7
> percpu: Embedded 12 pages/cpu @c7ed9000 s19264 r8192 d21696 u49152
> pcpu-alloc: s19264 r8192 d21696 u49152 alloc=12*4096
> pcpu-alloc: [0] 0 [0] 1
> Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
> Kernel command line: console=ttyS1,115200n81 ignore_loglevel
> root=/dev/nfs ip=dhcp
> PID hash table entries: 512 (order: -1, 2048 bytes)
> Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
> Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
> Memory: 121144K/131072K available (4243K kernel code, 165K rwdata,
> 1344K rodata, 2048K init, 264K bss, 9928K reserved, 0K cma-reserved,
> 0K highmem)
> Virtual kernel memory layout:
> vector : 0xffff0000 - 0xffff1000 ( 4 kB)
> fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
> vmalloc : 0xc8800000 - 0xff800000 ( 880 MB)
> lowmem : 0xc0000000 - 0xc8000000 ( 128 MB)
> pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
> modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
> .text : 0xc0008000 - 0xc0674eb8 (6580 kB)
> .init : 0xc0700000 - 0xc0900000 (2048 kB)
> .data : 0xc0900000 - 0xc0929420 ( 166 kB)
> .bss : 0xc092b000 - 0xc096d1e8 ( 265 kB)
> Hierarchical RCU implementation.
> Build-time adjustment of leaf fanout to 32.
> RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
> RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
> NR_IRQS:16 nr_irqs:16 16
> clocksource_probe: no matching clocksources found
> sched_clock: 32 bits at 100 Hz, resolution 10000000ns, wraps every
> 21474836475000000ns
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 0 at mm/slab.c:975 setup_kmem_cache_node+0x160/0x1c8
> Modules linked in:
> CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> [<c011fad0>] (warn_slowpath_null) from [<c01ced4c>]
> (setup_kmem_cache_node+0x160/0x1c8)
> [<c01ced4c>] (setup_kmem_cache_node) from [<c01cf02c>]
> (__do_tune_cpucache+0xf4/0x114)
> [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> [<c01cf0b0>] (enable_cpucache) from [<c0710324>]
> (kmem_cache_init_late+0x40/0x84)
> [<c0710324>] (kmem_cache_init_late) from [<c0700af8>] (start_kernel+0x238/0x36c)
> [<c0700af8>] (start_kernel) from [<4000807c>] (0x4000807c)
> ---[ end trace cb88537fdc8fa200 ]---
> Console: colour dummy device 80x30
> Calibrating delay loop (skipped) preset value.. 1066.00 BogoMIPS (lpj=5330000)
> pid_max: default: 32768 minimum: 301
> Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
> Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
> CPU: Testing write buffer coherency: ok
> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> Setting up static identity map for 0x40100000 - 0x40100058
> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> Brought up 2 CPUs
> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> CPU: All CPU(s) started in SVC mode.
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 1 at mm/slab.c:976 setup_kmem_cache_node+0x198/0x1c8
> Modules linked in:
> CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W
> 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> [<c011fad0>] (warn_slowpath_null) from [<c01ced84>]
> (setup_kmem_cache_node+0x198/0x1c8)
> [<c01ced84>] (setup_kmem_cache_node) from [<c01cf02c>]
> (__do_tune_cpucache+0xf4/0x114)
> [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> [<c01cf0b0>] (enable_cpucache) from [<c01cf4cc>]
> (__kmem_cache_create+0x1a0/0x1c8)
> [<c01cf4cc>] (__kmem_cache_create) from [<c01b2904>]
> (kmem_cache_create+0xbc/0x190)
> [<c01b2904>] (kmem_cache_create) from [<c070d724>] (shmem_init+0x34/0xb0)
> [<c070d724>] (shmem_init) from [<c0700cc4>] (kernel_init_freeable+0x98/0x1ec)
> [<c0700cc4>] (kernel_init_freeable) from [<c0497760>] (kernel_init+0x8/0x110)
> [<c0497760>] (kernel_init) from [<c0106c78>] (ret_from_fork+0x14/0x3c)
> ---[ end trace cb88537fdc8fa201 ]---
> devtmpfs: initialized
> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> max_idle_ns: 19112604462750000 ns
> pinctrl core: initialized pinctrl subsystem
> NET: Registered protocol family 16
> DMA: preallocated 256 KiB pool for atomic coherent allocations
> sh-pfc e0140200.pfc: emev2_pfc support registered
> gpiochip_find_base: found new base at 992
> gpio gpiochip0: (e0050000.gpio): created GPIO range 0->31 ==>
> e0140200.pfc PIN 0->31
> gpio gpiochip0: (e0050000.gpio): added GPIO chardev (254:0)
> gpiochip_setup_dev: registered GPIOs 992 to 1023 on device: gpiochip0
> (e0050000.gpio)
> gpiochip_find_base: found new base at 960
> gpio gpiochip1: (e0050080.gpio): created GPIO range 0->31 ==>
> e0140200.pfc PIN 32->63
> gpio gpiochip1: (e0050080.gpio): added GPIO chardev (254:1)
> gpiochip_setup_dev: registered GPIOs 960 to 991 on device: gpiochip1
> (e0050080.gpio)
> gpiochip_find_base: found new base at 928
> gpio gpiochip2: (e0050100.gpio): created GPIO range 0->31 ==>
> e0140200.pfc PIN 64->95
> gpio gpiochip2: (e0050100.gpio): added GPIO chardev (254:2)
> gpiochip_setup_dev: registered GPIOs 928 to 959 on device: gpiochip2
> (e0050100.gpio)
> gpiochip_find_base: found new base at 896
> gpio gpiochip3: (e0050180.gpio): created GPIO range 0->31 ==>
> e0140200.pfc PIN 96->127
> gpio gpiochip3: (e0050180.gpio): added GPIO chardev (254:3)
> gpiochip_setup_dev: registered GPIOs 896 to 927 on device: gpiochip3
> (e0050180.gpio)
> gpiochip_find_base: found new base at 865
> gpio gpiochip4: (e0050200.gpio): created GPIO range 0->30 ==>
> e0140200.pfc PIN 128->158
> gpio gpiochip4: (e0050200.gpio): added GPIO chardev (254:4)
> gpiochip_setup_dev: registered GPIOs 865 to 895 on device: gpiochip4
> (e0050200.gpio)
> gio: map hw irq = 1, irq = 35
> gio: sense irq = 1, mode = 8
> No ATAGs?
> hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
> hw-breakpoint: maximum watchpoint size is 4 bytes.
> of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@0[0]'
> of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@1[0]'
> SCSI subsystem initialized
> usbcore: registered new interface driver usbfs
> usbcore: registered new interface driver hub
> usbcore: registered new device driver usb
> em_sti e0180000.timer: used for clock events
> em_sti e0180000.timer: used for oneshot clock events
> em_sti e0180000.timer: used as clock source
> clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> 0x1ef4687b1, max_idle_ns: 3697658158765000000 ns
> Advanced Linux Sound Architecture Driver Initialized.
> NET: Registered protocol family 23
> clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> 0x1ef4687b1, max_idle_ns: 112843571739654 ns
> clocksource: Switched to clocksource e0180000.timer
> NET: Registered protocol family 2
> TCP established hash table entries: 1024 (order: 0, 4096 bytes)
> TCP bind hash table entries: 1024 (order: 2, 20480 bytes)
> TCP: Hash tables configured (established 1024 bind 1024)
> UDP hash table entries: 256 (order: 1, 12288 bytes)
> UDP-Lite hash table entries: 256 (order: 1, 12288 bytes)
> NET: Registered protocol family 1
> RPC: Registered named UNIX socket transport module.
> RPC: Registered udp transport module.
> RPC: Registered tcp transport module.
> RPC: Registered tcp NFSv4.1 backchannel transport module.
> Clockevents: could not switch to one-shot mode:
> Clockevents: could not switch to one-shot mode: dummy_timer is not functional.
> Could not switch to high resolution mode on CPU 0
> dummy_timer is not functional.
> Could not switch to high resolution mode on CPU 1
> hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
> futex hash table entries: 512 (order: 3, 32768 bytes)
> workingset: timestamp_bits=28 max_order=15 bucket_order=0
> NFS: Registering the id_resolver key type
> Key type id_resolver registered
> Key type id_legacy registered
> nfs4filelayout_init: NFSv4 File Layout Driver Registering...
> io scheduler noop registered (default)
> Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> e1020000.serial: ttyS0 at MMIO 0xe1020000 (irq = 19, base_baud =
> 796444) is a 16550A
> e1030000.serial: ttyS1 at MMIO 0xe1030000 (irq = 20, base_baud =
> 7168000) is a 16550A
> console [ttyS1] enabled
> e1040000.serial: ttyS2 at MMIO 0xe1040000 (irq = 21, base_baud =
> 14336000) is a 16550A
> e1050000.serial: ttyS3 at MMIO 0xe1050000 (irq = 22, base_baud =
> 2389333) is a 16550A
> gio: sense irq = 1, mode = 8
> libphy: smsc911x-mdio: probed
> Generic PHY 20000000.etherne:01: attached PHY driver [Generic PHY]
> (mii_bus:phy_addr=20000000.etherne:01, irq=-1)
> smsc911x 20000000.ethernet eth0: MAC Address: 00:01:9b:04:03:cf
> usbcore: registered new interface driver usb-storage
> i2c /dev entries driver
> em-i2c e0070000.i2c: Added i2c controller 0, irq 33
> em-i2c e10a0000.i2c: Added i2c controller 1, irq 34
> cpu cpu0: failed to get clock: -2
> cpufreq-dt: probe of cpufreq-dt failed with error -2
> ledtrig-cpu: registered to indicate activity on CPUs
> usbcore: registered new interface driver usbhid
> usbhid: USB HID core driver
> NET: Registered protocol family 17
> Key type dns_resolver registered
> Registering SWP/SWPB emulation handler
> of_get_named_gpiod_flags: parsed 'gpios' property of node
> '/gpio_keys/button@1[0]' - status (0)
> of_get_named_gpiod_flags: parsed 'gpios' property of node
> '/gpio_keys/button@2[0]' - status (0)
> of_get_named_gpiod_flags: parsed 'gpios' property of node
> '/gpio_keys/button@3[0]' - status (0)
> of_get_named_gpiod_flags: parsed 'gpios' property of node
> '/gpio_keys/button@4[0]' - status (0)
> gpio-1006 (DSW2-1): gpiod_set_debounce: missing set() or
> set_debounce() operations
> gio: map hw irq = 14, irq = 36
> gio: sense irq = 14, mode = 12
> gpio-1007 (DSW2-2): gpiod_set_debounce: missing set() or
> set_debounce() operations
> gio: map hw irq = 15, irq = 37
> gio: sense irq = 15, mode = 12
> gpio-1008 (DSW2-3): gpiod_set_debounce: missing set() or
> set_debounce() operations
> gio: map hw irq = 16, irq = 38
> gio: sense irq = 16, mode = 12
> gpio-1009 (DSW2-4): gpiod_set_debounce: missing set() or
> set_debounce() operations
> gio: map hw irq = 17, irq = 39
> gio: sense irq = 17, mode = 12
> input: gpio_keys as /devices/platform/gpio_keys/input/input0
> hctosys: unable to open rtc device (rtc0)
> smsc911x 20000000.ethernet eth0: SMSC911x/921x identified at 0xc8880000, IRQ: 35
> Sending DHCP requests .., OK
> IP-Config: Got DHCP answer from 192.168.97.254, my address is 192.168.97.215
> IP-Config: Complete:
> device=eth0, hwaddr=00:01:9b:04:03:cf, ipaddr=192.168.97.215,
> mask=255.255.255.0, gw=192.168.97.254
> host=192.168.97.215, domain=of.borg, nis-domain=(none)
> bootserver=192.168.97.254, rootserver=192.168.97.254, rootpath=
> nameserver0=192.168.97.254
> ALSA device list:
> No soundcards found.
> Freeing unused kernel memory: 2048K (c0700000 - c0900000)
> sysctl: error: 'kernel.hotplug' is an unknown key
>
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> -- Linus Torvalds
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-20 6:39 ` Joonsoo Kim
@ 2016-06-20 13:12 ` Paul E. McKenney
2016-06-21 6:43 ` Joonsoo Kim
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-20 13:12 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Mon, Jun 20, 2016 at 03:39:43PM +0900, Joonsoo Kim wrote:
> CCing Paul to ask some question.
>
> On Wed, Jun 15, 2016 at 10:39:47AM +0200, Geert Uytterhoeven wrote:
> > Hi Joonsoo,
> >
> > On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
> > >> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > >> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> > >> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > >> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> > >> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> > >> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> > >> >> >> > To check whther free objects exist or not precisely, we need to grab a
> > >> >> >> > lock. But, accuracy isn't that important because race window would be
> > >> >> >> > even small and if there is too much free object, cache reaper would reap
> > >> >> >> > it. So, this patch makes the check for free object exisistence not to
> > >> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
> >
> > >> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> > >> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> > >> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> >
> > > It's curious that synchronize_sched() has some effect in this early
> > > phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
> > > it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
> > >
> > > It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
> > >
> > > First, I'd like to confirm that num_online_cpus() is correct.
> > > Could you try following patch and give me a dmesg?
> > >
> > > Thanks.
> > >
> > > ------->8----------
> > > diff --git a/mm/slab.c b/mm/slab.c
> > > index 763096a..5b7300a 100644
> > > --- a/mm/slab.c
> > > +++ b/mm/slab.c
> > > @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > * freed after synchronize_sched().
> > > */
> > > - if (force_change)
> > > - synchronize_sched();
> > > + if (force_change) {
> > > + WARN_ON_ONCE(num_online_cpus() <= 1);
> > > + WARN_ON_ONCE(num_online_cpus() > 1);
> > > + }
> >
> > Full dmesg output below.
> >
> > I also tested whether it's the call to synchronize_sched() before or after
> > secondary CPU bringup that hangs.
> >
> > if (force_change && num_online_cpus() <= 1)
> > synchronize_sched();
> >
> > boots.
> >
> > if (force_change && num_online_cpus() > 1)
> > synchronize_sched();
> >
> > hangs.
>
> Hello, Paul.
>
> I changed slab.c to use synchronize_sched() for full memory barrier. First
> call happens on kmem_cache_init_late() and it would not be a problem
> because, at this time, num_online_cpus() <= 1 and synchronize_sched()
> would return immediately. Second call site would be shmem_init()
> and it seems that system hangs on it. Since smp is already initialized
> at that time, there would be some effect of synchronize_sched() but I
> can't imagine what's wrong here. Is it invalid moment to call
> synchronize_sched()?
>
> Note that my x86 virtual machine works fine even if
> synchronize_sched() is called in shmem_init() but Geert's some ARM
> machines (not all ARM machine) don't work well with it.
Color me confused.
Is Geert's ARM system somehow adding the second CPU before
rcu_spawn_gp_kthread() is called, that is, before or during
early_initcall() time?
RCU does assume that its kthreads are created before the second
CPU shows up.
Thanx, Paul
> Thanks.
>
> >
> > Booting Linux on physical CPU 0x0
> > Linux version 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty (geert@ramsan)
> > (gcc version 4.9.0 (GCC) ) #84 SMP Wed Jun 15 10:20:12 CEST 2016
> > CPU: ARMv7 Processor [411fc093] revision 3 (ARMv7), cr=10c5387d
> > CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
> > Machine model: EMEV2 KZM9D Board
> > debug: ignoring loglevel setting.
> > Memory policy: Data cache writealloc
> > On node 0 totalpages: 32768
> > free_area_init_node: node 0, pgdat c09286c0, node_mem_map c7efa000
> > Normal zone: 256 pages used for memmap
> > Normal zone: 0 pages reserved
> > Normal zone: 32768 pages, LIFO batch:7
> > percpu: Embedded 12 pages/cpu @c7ed9000 s19264 r8192 d21696 u49152
> > pcpu-alloc: s19264 r8192 d21696 u49152 alloc=12*4096
> > pcpu-alloc: [0] 0 [0] 1
> > Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
> > Kernel command line: console=ttyS1,115200n81 ignore_loglevel
> > root=/dev/nfs ip=dhcp
> > PID hash table entries: 512 (order: -1, 2048 bytes)
> > Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
> > Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
> > Memory: 121144K/131072K available (4243K kernel code, 165K rwdata,
> > 1344K rodata, 2048K init, 264K bss, 9928K reserved, 0K cma-reserved,
> > 0K highmem)
> > Virtual kernel memory layout:
> > vector : 0xffff0000 - 0xffff1000 ( 4 kB)
> > fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
> > vmalloc : 0xc8800000 - 0xff800000 ( 880 MB)
> > lowmem : 0xc0000000 - 0xc8000000 ( 128 MB)
> > pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
> > modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
> > .text : 0xc0008000 - 0xc0674eb8 (6580 kB)
> > .init : 0xc0700000 - 0xc0900000 (2048 kB)
> > .data : 0xc0900000 - 0xc0929420 ( 166 kB)
> > .bss : 0xc092b000 - 0xc096d1e8 ( 265 kB)
> > Hierarchical RCU implementation.
> > Build-time adjustment of leaf fanout to 32.
> > RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
> > RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
> > NR_IRQS:16 nr_irqs:16 16
> > clocksource_probe: no matching clocksources found
> > sched_clock: 32 bits at 100 Hz, resolution 10000000ns, wraps every
> > 21474836475000000ns
> > ------------[ cut here ]------------
> > WARNING: CPU: 0 PID: 0 at mm/slab.c:975 setup_kmem_cache_node+0x160/0x1c8
> > Modules linked in:
> > CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> > 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> > [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> > [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> > [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> > [<c011fad0>] (warn_slowpath_null) from [<c01ced4c>]
> > (setup_kmem_cache_node+0x160/0x1c8)
> > [<c01ced4c>] (setup_kmem_cache_node) from [<c01cf02c>]
> > (__do_tune_cpucache+0xf4/0x114)
> > [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> > [<c01cf0b0>] (enable_cpucache) from [<c0710324>]
> > (kmem_cache_init_late+0x40/0x84)
> > [<c0710324>] (kmem_cache_init_late) from [<c0700af8>] (start_kernel+0x238/0x36c)
> > [<c0700af8>] (start_kernel) from [<4000807c>] (0x4000807c)
> > ---[ end trace cb88537fdc8fa200 ]---
> > Console: colour dummy device 80x30
> > Calibrating delay loop (skipped) preset value.. 1066.00 BogoMIPS (lpj=5330000)
> > pid_max: default: 32768 minimum: 301
> > Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
> > Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
> > CPU: Testing write buffer coherency: ok
> > CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > Setting up static identity map for 0x40100000 - 0x40100058
> > CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > Brought up 2 CPUs
> > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > CPU: All CPU(s) started in SVC mode.
> > ------------[ cut here ]------------
> > WARNING: CPU: 0 PID: 1 at mm/slab.c:976 setup_kmem_cache_node+0x198/0x1c8
> > Modules linked in:
> > CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W
> > 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> > [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> > [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> > [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> > [<c011fad0>] (warn_slowpath_null) from [<c01ced84>]
> > (setup_kmem_cache_node+0x198/0x1c8)
> > [<c01ced84>] (setup_kmem_cache_node) from [<c01cf02c>]
> > (__do_tune_cpucache+0xf4/0x114)
> > [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> > [<c01cf0b0>] (enable_cpucache) from [<c01cf4cc>]
> > (__kmem_cache_create+0x1a0/0x1c8)
> > [<c01cf4cc>] (__kmem_cache_create) from [<c01b2904>]
> > (kmem_cache_create+0xbc/0x190)
> > [<c01b2904>] (kmem_cache_create) from [<c070d724>] (shmem_init+0x34/0xb0)
> > [<c070d724>] (shmem_init) from [<c0700cc4>] (kernel_init_freeable+0x98/0x1ec)
> > [<c0700cc4>] (kernel_init_freeable) from [<c0497760>] (kernel_init+0x8/0x110)
> > [<c0497760>] (kernel_init) from [<c0106c78>] (ret_from_fork+0x14/0x3c)
> > ---[ end trace cb88537fdc8fa201 ]---
> > devtmpfs: initialized
> > VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > max_idle_ns: 19112604462750000 ns
> > pinctrl core: initialized pinctrl subsystem
> > NET: Registered protocol family 16
> > DMA: preallocated 256 KiB pool for atomic coherent allocations
> > sh-pfc e0140200.pfc: emev2_pfc support registered
> > gpiochip_find_base: found new base at 992
> > gpio gpiochip0: (e0050000.gpio): created GPIO range 0->31 ==>
> > e0140200.pfc PIN 0->31
> > gpio gpiochip0: (e0050000.gpio): added GPIO chardev (254:0)
> > gpiochip_setup_dev: registered GPIOs 992 to 1023 on device: gpiochip0
> > (e0050000.gpio)
> > gpiochip_find_base: found new base at 960
> > gpio gpiochip1: (e0050080.gpio): created GPIO range 0->31 ==>
> > e0140200.pfc PIN 32->63
> > gpio gpiochip1: (e0050080.gpio): added GPIO chardev (254:1)
> > gpiochip_setup_dev: registered GPIOs 960 to 991 on device: gpiochip1
> > (e0050080.gpio)
> > gpiochip_find_base: found new base at 928
> > gpio gpiochip2: (e0050100.gpio): created GPIO range 0->31 ==>
> > e0140200.pfc PIN 64->95
> > gpio gpiochip2: (e0050100.gpio): added GPIO chardev (254:2)
> > gpiochip_setup_dev: registered GPIOs 928 to 959 on device: gpiochip2
> > (e0050100.gpio)
> > gpiochip_find_base: found new base at 896
> > gpio gpiochip3: (e0050180.gpio): created GPIO range 0->31 ==>
> > e0140200.pfc PIN 96->127
> > gpio gpiochip3: (e0050180.gpio): added GPIO chardev (254:3)
> > gpiochip_setup_dev: registered GPIOs 896 to 927 on device: gpiochip3
> > (e0050180.gpio)
> > gpiochip_find_base: found new base at 865
> > gpio gpiochip4: (e0050200.gpio): created GPIO range 0->30 ==>
> > e0140200.pfc PIN 128->158
> > gpio gpiochip4: (e0050200.gpio): added GPIO chardev (254:4)
> > gpiochip_setup_dev: registered GPIOs 865 to 895 on device: gpiochip4
> > (e0050200.gpio)
> > gio: map hw irq = 1, irq = 35
> > gio: sense irq = 1, mode = 8
> > No ATAGs?
> > hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
> > hw-breakpoint: maximum watchpoint size is 4 bytes.
> > of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@0[0]'
> > of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@1[0]'
> > SCSI subsystem initialized
> > usbcore: registered new interface driver usbfs
> > usbcore: registered new interface driver hub
> > usbcore: registered new device driver usb
> > em_sti e0180000.timer: used for clock events
> > em_sti e0180000.timer: used for oneshot clock events
> > em_sti e0180000.timer: used as clock source
> > clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> > 0x1ef4687b1, max_idle_ns: 3697658158765000000 ns
> > Advanced Linux Sound Architecture Driver Initialized.
> > NET: Registered protocol family 23
> > clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> > 0x1ef4687b1, max_idle_ns: 112843571739654 ns
> > clocksource: Switched to clocksource e0180000.timer
> > NET: Registered protocol family 2
> > TCP established hash table entries: 1024 (order: 0, 4096 bytes)
> > TCP bind hash table entries: 1024 (order: 2, 20480 bytes)
> > TCP: Hash tables configured (established 1024 bind 1024)
> > UDP hash table entries: 256 (order: 1, 12288 bytes)
> > UDP-Lite hash table entries: 256 (order: 1, 12288 bytes)
> > NET: Registered protocol family 1
> > RPC: Registered named UNIX socket transport module.
> > RPC: Registered udp transport module.
> > RPC: Registered tcp transport module.
> > RPC: Registered tcp NFSv4.1 backchannel transport module.
> > Clockevents: could not switch to one-shot mode:
> > Clockevents: could not switch to one-shot mode: dummy_timer is not functional.
> > Could not switch to high resolution mode on CPU 0
> > dummy_timer is not functional.
> > Could not switch to high resolution mode on CPU 1
> > hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
> > futex hash table entries: 512 (order: 3, 32768 bytes)
> > workingset: timestamp_bits=28 max_order=15 bucket_order=0
> > NFS: Registering the id_resolver key type
> > Key type id_resolver registered
> > Key type id_legacy registered
> > nfs4filelayout_init: NFSv4 File Layout Driver Registering...
> > io scheduler noop registered (default)
> > Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> > e1020000.serial: ttyS0 at MMIO 0xe1020000 (irq = 19, base_baud =
> > 796444) is a 16550A
> > e1030000.serial: ttyS1 at MMIO 0xe1030000 (irq = 20, base_baud =
> > 7168000) is a 16550A
> > console [ttyS1] enabled
> > e1040000.serial: ttyS2 at MMIO 0xe1040000 (irq = 21, base_baud =
> > 14336000) is a 16550A
> > e1050000.serial: ttyS3 at MMIO 0xe1050000 (irq = 22, base_baud =
> > 2389333) is a 16550A
> > gio: sense irq = 1, mode = 8
> > libphy: smsc911x-mdio: probed
> > Generic PHY 20000000.etherne:01: attached PHY driver [Generic PHY]
> > (mii_bus:phy_addr=20000000.etherne:01, irq=-1)
> > smsc911x 20000000.ethernet eth0: MAC Address: 00:01:9b:04:03:cf
> > usbcore: registered new interface driver usb-storage
> > i2c /dev entries driver
> > em-i2c e0070000.i2c: Added i2c controller 0, irq 33
> > em-i2c e10a0000.i2c: Added i2c controller 1, irq 34
> > cpu cpu0: failed to get clock: -2
> > cpufreq-dt: probe of cpufreq-dt failed with error -2
> > ledtrig-cpu: registered to indicate activity on CPUs
> > usbcore: registered new interface driver usbhid
> > usbhid: USB HID core driver
> > NET: Registered protocol family 17
> > Key type dns_resolver registered
> > Registering SWP/SWPB emulation handler
> > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > '/gpio_keys/button@1[0]' - status (0)
> > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > '/gpio_keys/button@2[0]' - status (0)
> > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > '/gpio_keys/button@3[0]' - status (0)
> > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > '/gpio_keys/button@4[0]' - status (0)
> > gpio-1006 (DSW2-1): gpiod_set_debounce: missing set() or
> > set_debounce() operations
> > gio: map hw irq = 14, irq = 36
> > gio: sense irq = 14, mode = 12
> > gpio-1007 (DSW2-2): gpiod_set_debounce: missing set() or
> > set_debounce() operations
> > gio: map hw irq = 15, irq = 37
> > gio: sense irq = 15, mode = 12
> > gpio-1008 (DSW2-3): gpiod_set_debounce: missing set() or
> > set_debounce() operations
> > gio: map hw irq = 16, irq = 38
> > gio: sense irq = 16, mode = 12
> > gpio-1009 (DSW2-4): gpiod_set_debounce: missing set() or
> > set_debounce() operations
> > gio: map hw irq = 17, irq = 39
> > gio: sense irq = 17, mode = 12
> > input: gpio_keys as /devices/platform/gpio_keys/input/input0
> > hctosys: unable to open rtc device (rtc0)
> > smsc911x 20000000.ethernet eth0: SMSC911x/921x identified at 0xc8880000, IRQ: 35
> > Sending DHCP requests .., OK
> > IP-Config: Got DHCP answer from 192.168.97.254, my address is 192.168.97.215
> > IP-Config: Complete:
> > device=eth0, hwaddr=00:01:9b:04:03:cf, ipaddr=192.168.97.215,
> > mask=255.255.255.0, gw=192.168.97.254
> > host=192.168.97.215, domain=of.borg, nis-domain=(none)
> > bootserver=192.168.97.254, rootserver=192.168.97.254, rootpath=
> > nameserver0=192.168.97.254
> > ALSA device list:
> > No soundcards found.
> > Freeing unused kernel memory: 2048K (c0700000 - c0900000)
> > sysctl: error: 'kernel.hotplug' is an unknown key
> >
> > Gr{oetje,eeting}s,
> >
> > Geert
> >
> > --
> > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> >
> > In personal conversations with technical people, I call myself a hacker. But
> > when I'm talking to journalists I just say "programmer" or something like that.
> > -- Linus Torvalds
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org. For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-20 13:12 ` Paul E. McKenney
@ 2016-06-21 6:43 ` Joonsoo Kim
2016-06-21 12:54 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-21 6:43 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Mon, Jun 20, 2016 at 06:12:54AM -0700, Paul E. McKenney wrote:
> On Mon, Jun 20, 2016 at 03:39:43PM +0900, Joonsoo Kim wrote:
> > CCing Paul to ask some question.
> >
> > On Wed, Jun 15, 2016 at 10:39:47AM +0200, Geert Uytterhoeven wrote:
> > > Hi Joonsoo,
> > >
> > > On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
> > > >> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > >> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> > > >> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > >> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> > > >> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> > > >> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> > > >> >> >> > To check whther free objects exist or not precisely, we need to grab a
> > > >> >> >> > lock. But, accuracy isn't that important because race window would be
> > > >> >> >> > even small and if there is too much free object, cache reaper would reap
> > > >> >> >> > it. So, this patch makes the check for free object exisistence not to
> > > >> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
> > >
> > > >> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> > > >> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> > > >> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> > >
> > > > It's curious that synchronize_sched() has some effect in this early
> > > > phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
> > > > it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
> > > >
> > > > It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
> > > >
> > > > First, I'd like to confirm that num_online_cpus() is correct.
> > > > Could you try following patch and give me a dmesg?
> > > >
> > > > Thanks.
> > > >
> > > > ------->8----------
> > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > index 763096a..5b7300a 100644
> > > > --- a/mm/slab.c
> > > > +++ b/mm/slab.c
> > > > @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > * freed after synchronize_sched().
> > > > */
> > > > - if (force_change)
> > > > - synchronize_sched();
> > > > + if (force_change) {
> > > > + WARN_ON_ONCE(num_online_cpus() <= 1);
> > > > + WARN_ON_ONCE(num_online_cpus() > 1);
> > > > + }
> > >
> > > Full dmesg output below.
> > >
> > > I also tested whether it's the call to synchronize_sched() before or after
> > > secondary CPU bringup that hangs.
> > >
> > > if (force_change && num_online_cpus() <= 1)
> > > synchronize_sched();
> > >
> > > boots.
> > >
> > > if (force_change && num_online_cpus() > 1)
> > > synchronize_sched();
> > >
> > > hangs.
> >
> > Hello, Paul.
> >
> > I changed slab.c to use synchronize_sched() for full memory barrier. First
> > call happens on kmem_cache_init_late() and it would not be a problem
> > because, at this time, num_online_cpus() <= 1 and synchronize_sched()
> > would return immediately. Second call site would be shmem_init()
> > and it seems that system hangs on it. Since smp is already initialized
> > at that time, there would be some effect of synchronize_sched() but I
> > can't imagine what's wrong here. Is it invalid moment to call
> > synchronize_sched()?
> >
> > Note that my x86 virtual machine works fine even if
> > synchronize_sched() is called in shmem_init() but Geert's some ARM
> > machines (not all ARM machine) don't work well with it.
>
> Color me confused.
>
> Is Geert's ARM system somehow adding the second CPU before
> rcu_spawn_gp_kthread() is called, that is, before or during
> early_initcall() time?
Hang would happen on shmem_init() which is called in do_basic_setup().
do_basic_setup() is called after early_initcall().
Hmm... Is it okay to call synchronize_sched() by kernel thread?
Thanks.
>
> RCU does assume that its kthreads are created before the second
> CPU shows up.
>
> Thanx, Paul
>
> > Thanks.
> >
> > >
> > > Booting Linux on physical CPU 0x0
> > > Linux version 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty (geert@ramsan)
> > > (gcc version 4.9.0 (GCC) ) #84 SMP Wed Jun 15 10:20:12 CEST 2016
> > > CPU: ARMv7 Processor [411fc093] revision 3 (ARMv7), cr=10c5387d
> > > CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
> > > Machine model: EMEV2 KZM9D Board
> > > debug: ignoring loglevel setting.
> > > Memory policy: Data cache writealloc
> > > On node 0 totalpages: 32768
> > > free_area_init_node: node 0, pgdat c09286c0, node_mem_map c7efa000
> > > Normal zone: 256 pages used for memmap
> > > Normal zone: 0 pages reserved
> > > Normal zone: 32768 pages, LIFO batch:7
> > > percpu: Embedded 12 pages/cpu @c7ed9000 s19264 r8192 d21696 u49152
> > > pcpu-alloc: s19264 r8192 d21696 u49152 alloc=12*4096
> > > pcpu-alloc: [0] 0 [0] 1
> > > Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
> > > Kernel command line: console=ttyS1,115200n81 ignore_loglevel
> > > root=/dev/nfs ip=dhcp
> > > PID hash table entries: 512 (order: -1, 2048 bytes)
> > > Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
> > > Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
> > > Memory: 121144K/131072K available (4243K kernel code, 165K rwdata,
> > > 1344K rodata, 2048K init, 264K bss, 9928K reserved, 0K cma-reserved,
> > > 0K highmem)
> > > Virtual kernel memory layout:
> > > vector : 0xffff0000 - 0xffff1000 ( 4 kB)
> > > fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
> > > vmalloc : 0xc8800000 - 0xff800000 ( 880 MB)
> > > lowmem : 0xc0000000 - 0xc8000000 ( 128 MB)
> > > pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
> > > modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
> > > .text : 0xc0008000 - 0xc0674eb8 (6580 kB)
> > > .init : 0xc0700000 - 0xc0900000 (2048 kB)
> > > .data : 0xc0900000 - 0xc0929420 ( 166 kB)
> > > .bss : 0xc092b000 - 0xc096d1e8 ( 265 kB)
> > > Hierarchical RCU implementation.
> > > Build-time adjustment of leaf fanout to 32.
> > > RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
> > > RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
> > > NR_IRQS:16 nr_irqs:16 16
> > > clocksource_probe: no matching clocksources found
> > > sched_clock: 32 bits at 100 Hz, resolution 10000000ns, wraps every
> > > 21474836475000000ns
> > > ------------[ cut here ]------------
> > > WARNING: CPU: 0 PID: 0 at mm/slab.c:975 setup_kmem_cache_node+0x160/0x1c8
> > > Modules linked in:
> > > CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> > > 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> > > [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> > > [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> > > [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> > > [<c011fad0>] (warn_slowpath_null) from [<c01ced4c>]
> > > (setup_kmem_cache_node+0x160/0x1c8)
> > > [<c01ced4c>] (setup_kmem_cache_node) from [<c01cf02c>]
> > > (__do_tune_cpucache+0xf4/0x114)
> > > [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> > > [<c01cf0b0>] (enable_cpucache) from [<c0710324>]
> > > (kmem_cache_init_late+0x40/0x84)
> > > [<c0710324>] (kmem_cache_init_late) from [<c0700af8>] (start_kernel+0x238/0x36c)
> > > [<c0700af8>] (start_kernel) from [<4000807c>] (0x4000807c)
> > > ---[ end trace cb88537fdc8fa200 ]---
> > > Console: colour dummy device 80x30
> > > Calibrating delay loop (skipped) preset value.. 1066.00 BogoMIPS (lpj=5330000)
> > > pid_max: default: 32768 minimum: 301
> > > Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
> > > Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
> > > CPU: Testing write buffer coherency: ok
> > > CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > > Setting up static identity map for 0x40100000 - 0x40100058
> > > CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > > Brought up 2 CPUs
> > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > CPU: All CPU(s) started in SVC mode.
> > > ------------[ cut here ]------------
> > > WARNING: CPU: 0 PID: 1 at mm/slab.c:976 setup_kmem_cache_node+0x198/0x1c8
> > > Modules linked in:
> > > CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W
> > > 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> > > [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> > > [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> > > [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> > > [<c011fad0>] (warn_slowpath_null) from [<c01ced84>]
> > > (setup_kmem_cache_node+0x198/0x1c8)
> > > [<c01ced84>] (setup_kmem_cache_node) from [<c01cf02c>]
> > > (__do_tune_cpucache+0xf4/0x114)
> > > [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> > > [<c01cf0b0>] (enable_cpucache) from [<c01cf4cc>]
> > > (__kmem_cache_create+0x1a0/0x1c8)
> > > [<c01cf4cc>] (__kmem_cache_create) from [<c01b2904>]
> > > (kmem_cache_create+0xbc/0x190)
> > > [<c01b2904>] (kmem_cache_create) from [<c070d724>] (shmem_init+0x34/0xb0)
> > > [<c070d724>] (shmem_init) from [<c0700cc4>] (kernel_init_freeable+0x98/0x1ec)
> > > [<c0700cc4>] (kernel_init_freeable) from [<c0497760>] (kernel_init+0x8/0x110)
> > > [<c0497760>] (kernel_init) from [<c0106c78>] (ret_from_fork+0x14/0x3c)
> > > ---[ end trace cb88537fdc8fa201 ]---
> > > devtmpfs: initialized
> > > VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > > clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > > max_idle_ns: 19112604462750000 ns
> > > pinctrl core: initialized pinctrl subsystem
> > > NET: Registered protocol family 16
> > > DMA: preallocated 256 KiB pool for atomic coherent allocations
> > > sh-pfc e0140200.pfc: emev2_pfc support registered
> > > gpiochip_find_base: found new base at 992
> > > gpio gpiochip0: (e0050000.gpio): created GPIO range 0->31 ==>
> > > e0140200.pfc PIN 0->31
> > > gpio gpiochip0: (e0050000.gpio): added GPIO chardev (254:0)
> > > gpiochip_setup_dev: registered GPIOs 992 to 1023 on device: gpiochip0
> > > (e0050000.gpio)
> > > gpiochip_find_base: found new base at 960
> > > gpio gpiochip1: (e0050080.gpio): created GPIO range 0->31 ==>
> > > e0140200.pfc PIN 32->63
> > > gpio gpiochip1: (e0050080.gpio): added GPIO chardev (254:1)
> > > gpiochip_setup_dev: registered GPIOs 960 to 991 on device: gpiochip1
> > > (e0050080.gpio)
> > > gpiochip_find_base: found new base at 928
> > > gpio gpiochip2: (e0050100.gpio): created GPIO range 0->31 ==>
> > > e0140200.pfc PIN 64->95
> > > gpio gpiochip2: (e0050100.gpio): added GPIO chardev (254:2)
> > > gpiochip_setup_dev: registered GPIOs 928 to 959 on device: gpiochip2
> > > (e0050100.gpio)
> > > gpiochip_find_base: found new base at 896
> > > gpio gpiochip3: (e0050180.gpio): created GPIO range 0->31 ==>
> > > e0140200.pfc PIN 96->127
> > > gpio gpiochip3: (e0050180.gpio): added GPIO chardev (254:3)
> > > gpiochip_setup_dev: registered GPIOs 896 to 927 on device: gpiochip3
> > > (e0050180.gpio)
> > > gpiochip_find_base: found new base at 865
> > > gpio gpiochip4: (e0050200.gpio): created GPIO range 0->30 ==>
> > > e0140200.pfc PIN 128->158
> > > gpio gpiochip4: (e0050200.gpio): added GPIO chardev (254:4)
> > > gpiochip_setup_dev: registered GPIOs 865 to 895 on device: gpiochip4
> > > (e0050200.gpio)
> > > gio: map hw irq = 1, irq = 35
> > > gio: sense irq = 1, mode = 8
> > > No ATAGs?
> > > hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
> > > hw-breakpoint: maximum watchpoint size is 4 bytes.
> > > of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@0[0]'
> > > of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@1[0]'
> > > SCSI subsystem initialized
> > > usbcore: registered new interface driver usbfs
> > > usbcore: registered new interface driver hub
> > > usbcore: registered new device driver usb
> > > em_sti e0180000.timer: used for clock events
> > > em_sti e0180000.timer: used for oneshot clock events
> > > em_sti e0180000.timer: used as clock source
> > > clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> > > 0x1ef4687b1, max_idle_ns: 3697658158765000000 ns
> > > Advanced Linux Sound Architecture Driver Initialized.
> > > NET: Registered protocol family 23
> > > clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> > > 0x1ef4687b1, max_idle_ns: 112843571739654 ns
> > > clocksource: Switched to clocksource e0180000.timer
> > > NET: Registered protocol family 2
> > > TCP established hash table entries: 1024 (order: 0, 4096 bytes)
> > > TCP bind hash table entries: 1024 (order: 2, 20480 bytes)
> > > TCP: Hash tables configured (established 1024 bind 1024)
> > > UDP hash table entries: 256 (order: 1, 12288 bytes)
> > > UDP-Lite hash table entries: 256 (order: 1, 12288 bytes)
> > > NET: Registered protocol family 1
> > > RPC: Registered named UNIX socket transport module.
> > > RPC: Registered udp transport module.
> > > RPC: Registered tcp transport module.
> > > RPC: Registered tcp NFSv4.1 backchannel transport module.
> > > Clockevents: could not switch to one-shot mode:
> > > Clockevents: could not switch to one-shot mode: dummy_timer is not functional.
> > > Could not switch to high resolution mode on CPU 0
> > > dummy_timer is not functional.
> > > Could not switch to high resolution mode on CPU 1
> > > hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
> > > futex hash table entries: 512 (order: 3, 32768 bytes)
> > > workingset: timestamp_bits=28 max_order=15 bucket_order=0
> > > NFS: Registering the id_resolver key type
> > > Key type id_resolver registered
> > > Key type id_legacy registered
> > > nfs4filelayout_init: NFSv4 File Layout Driver Registering...
> > > io scheduler noop registered (default)
> > > Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> > > e1020000.serial: ttyS0 at MMIO 0xe1020000 (irq = 19, base_baud =
> > > 796444) is a 16550A
> > > e1030000.serial: ttyS1 at MMIO 0xe1030000 (irq = 20, base_baud =
> > > 7168000) is a 16550A
> > > console [ttyS1] enabled
> > > e1040000.serial: ttyS2 at MMIO 0xe1040000 (irq = 21, base_baud =
> > > 14336000) is a 16550A
> > > e1050000.serial: ttyS3 at MMIO 0xe1050000 (irq = 22, base_baud =
> > > 2389333) is a 16550A
> > > gio: sense irq = 1, mode = 8
> > > libphy: smsc911x-mdio: probed
> > > Generic PHY 20000000.etherne:01: attached PHY driver [Generic PHY]
> > > (mii_bus:phy_addr=20000000.etherne:01, irq=-1)
> > > smsc911x 20000000.ethernet eth0: MAC Address: 00:01:9b:04:03:cf
> > > usbcore: registered new interface driver usb-storage
> > > i2c /dev entries driver
> > > em-i2c e0070000.i2c: Added i2c controller 0, irq 33
> > > em-i2c e10a0000.i2c: Added i2c controller 1, irq 34
> > > cpu cpu0: failed to get clock: -2
> > > cpufreq-dt: probe of cpufreq-dt failed with error -2
> > > ledtrig-cpu: registered to indicate activity on CPUs
> > > usbcore: registered new interface driver usbhid
> > > usbhid: USB HID core driver
> > > NET: Registered protocol family 17
> > > Key type dns_resolver registered
> > > Registering SWP/SWPB emulation handler
> > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > '/gpio_keys/button@1[0]' - status (0)
> > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > '/gpio_keys/button@2[0]' - status (0)
> > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > '/gpio_keys/button@3[0]' - status (0)
> > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > '/gpio_keys/button@4[0]' - status (0)
> > > gpio-1006 (DSW2-1): gpiod_set_debounce: missing set() or
> > > set_debounce() operations
> > > gio: map hw irq = 14, irq = 36
> > > gio: sense irq = 14, mode = 12
> > > gpio-1007 (DSW2-2): gpiod_set_debounce: missing set() or
> > > set_debounce() operations
> > > gio: map hw irq = 15, irq = 37
> > > gio: sense irq = 15, mode = 12
> > > gpio-1008 (DSW2-3): gpiod_set_debounce: missing set() or
> > > set_debounce() operations
> > > gio: map hw irq = 16, irq = 38
> > > gio: sense irq = 16, mode = 12
> > > gpio-1009 (DSW2-4): gpiod_set_debounce: missing set() or
> > > set_debounce() operations
> > > gio: map hw irq = 17, irq = 39
> > > gio: sense irq = 17, mode = 12
> > > input: gpio_keys as /devices/platform/gpio_keys/input/input0
> > > hctosys: unable to open rtc device (rtc0)
> > > smsc911x 20000000.ethernet eth0: SMSC911x/921x identified at 0xc8880000, IRQ: 35
> > > Sending DHCP requests .., OK
> > > IP-Config: Got DHCP answer from 192.168.97.254, my address is 192.168.97.215
> > > IP-Config: Complete:
> > > device=eth0, hwaddr=00:01:9b:04:03:cf, ipaddr=192.168.97.215,
> > > mask=255.255.255.0, gw=192.168.97.254
> > > host=192.168.97.215, domain=of.borg, nis-domain=(none)
> > > bootserver=192.168.97.254, rootserver=192.168.97.254, rootpath=
> > > nameserver0=192.168.97.254
> > > ALSA device list:
> > > No soundcards found.
> > > Freeing unused kernel memory: 2048K (c0700000 - c0900000)
> > > sysctl: error: 'kernel.hotplug' is an unknown key
> > >
> > > Gr{oetje,eeting}s,
> > >
> > > Geert
> > >
> > > --
> > > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> > >
> > > In personal conversations with technical people, I call myself a hacker. But
> > > when I'm talking to journalists I just say "programmer" or something like that.
> > > -- Linus Torvalds
> > >
> > > --
> > > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > > the body to majordomo@kvack.org. For more info on Linux MM,
> > > see: http://www.linux-mm.org/ .
> > > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> >
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-21 6:43 ` Joonsoo Kim
@ 2016-06-21 12:54 ` Paul E. McKenney
2016-06-22 0:52 ` Joonsoo Kim
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-21 12:54 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Tue, Jun 21, 2016 at 03:43:02PM +0900, Joonsoo Kim wrote:
> On Mon, Jun 20, 2016 at 06:12:54AM -0700, Paul E. McKenney wrote:
> > On Mon, Jun 20, 2016 at 03:39:43PM +0900, Joonsoo Kim wrote:
> > > CCing Paul to ask some question.
> > >
> > > On Wed, Jun 15, 2016 at 10:39:47AM +0200, Geert Uytterhoeven wrote:
> > > > Hi Joonsoo,
> > > >
> > > > On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
> > > > >> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > >> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> > > > >> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > >> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> > > > >> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> > > > >> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> > > > >> >> >> > To check whther free objects exist or not precisely, we need to grab a
> > > > >> >> >> > lock. But, accuracy isn't that important because race window would be
> > > > >> >> >> > even small and if there is too much free object, cache reaper would reap
> > > > >> >> >> > it. So, this patch makes the check for free object exisistence not to
> > > > >> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
> > > >
> > > > >> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> > > > >> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> > > > >> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> > > >
> > > > > It's curious that synchronize_sched() has some effect in this early
> > > > > phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
> > > > > it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
> > > > >
> > > > > It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
> > > > >
> > > > > First, I'd like to confirm that num_online_cpus() is correct.
> > > > > Could you try following patch and give me a dmesg?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > ------->8----------
> > > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > > index 763096a..5b7300a 100644
> > > > > --- a/mm/slab.c
> > > > > +++ b/mm/slab.c
> > > > > @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > > * freed after synchronize_sched().
> > > > > */
> > > > > - if (force_change)
> > > > > - synchronize_sched();
> > > > > + if (force_change) {
> > > > > + WARN_ON_ONCE(num_online_cpus() <= 1);
> > > > > + WARN_ON_ONCE(num_online_cpus() > 1);
> > > > > + }
> > > >
> > > > Full dmesg output below.
> > > >
> > > > I also tested whether it's the call to synchronize_sched() before or after
> > > > secondary CPU bringup that hangs.
> > > >
> > > > if (force_change && num_online_cpus() <= 1)
> > > > synchronize_sched();
> > > >
> > > > boots.
> > > >
> > > > if (force_change && num_online_cpus() > 1)
> > > > synchronize_sched();
> > > >
> > > > hangs.
> > >
> > > Hello, Paul.
> > >
> > > I changed slab.c to use synchronize_sched() for full memory barrier. First
> > > call happens on kmem_cache_init_late() and it would not be a problem
> > > because, at this time, num_online_cpus() <= 1 and synchronize_sched()
> > > would return immediately. Second call site would be shmem_init()
> > > and it seems that system hangs on it. Since smp is already initialized
> > > at that time, there would be some effect of synchronize_sched() but I
> > > can't imagine what's wrong here. Is it invalid moment to call
> > > synchronize_sched()?
> > >
> > > Note that my x86 virtual machine works fine even if
> > > synchronize_sched() is called in shmem_init() but Geert's some ARM
> > > machines (not all ARM machine) don't work well with it.
> >
> > Color me confused.
> >
> > Is Geert's ARM system somehow adding the second CPU before
> > rcu_spawn_gp_kthread() is called, that is, before or during
> > early_initcall() time?
>
> Hang would happen on shmem_init() which is called in do_basic_setup().
> do_basic_setup() is called after early_initcall().
Thank you for the info!
That should be lat enough that the RCU kthreads are alive and well.
Can you get sysalt-t output?
> Hmm... Is it okay to call synchronize_sched() by kernel thread?
Yes, it can, in fact, rcutorture does this all the time. As do any
number of other kthreads.
Thanx, Paul
> Thanks.
>
> >
> > RCU does assume that its kthreads are created before the second
> > CPU shows up.
> >
> > Thanx, Paul
> >
> > > Thanks.
> > >
> > > >
> > > > Booting Linux on physical CPU 0x0
> > > > Linux version 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty (geert@ramsan)
> > > > (gcc version 4.9.0 (GCC) ) #84 SMP Wed Jun 15 10:20:12 CEST 2016
> > > > CPU: ARMv7 Processor [411fc093] revision 3 (ARMv7), cr=10c5387d
> > > > CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
> > > > Machine model: EMEV2 KZM9D Board
> > > > debug: ignoring loglevel setting.
> > > > Memory policy: Data cache writealloc
> > > > On node 0 totalpages: 32768
> > > > free_area_init_node: node 0, pgdat c09286c0, node_mem_map c7efa000
> > > > Normal zone: 256 pages used for memmap
> > > > Normal zone: 0 pages reserved
> > > > Normal zone: 32768 pages, LIFO batch:7
> > > > percpu: Embedded 12 pages/cpu @c7ed9000 s19264 r8192 d21696 u49152
> > > > pcpu-alloc: s19264 r8192 d21696 u49152 alloc=12*4096
> > > > pcpu-alloc: [0] 0 [0] 1
> > > > Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
> > > > Kernel command line: console=ttyS1,115200n81 ignore_loglevel
> > > > root=/dev/nfs ip=dhcp
> > > > PID hash table entries: 512 (order: -1, 2048 bytes)
> > > > Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
> > > > Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
> > > > Memory: 121144K/131072K available (4243K kernel code, 165K rwdata,
> > > > 1344K rodata, 2048K init, 264K bss, 9928K reserved, 0K cma-reserved,
> > > > 0K highmem)
> > > > Virtual kernel memory layout:
> > > > vector : 0xffff0000 - 0xffff1000 ( 4 kB)
> > > > fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
> > > > vmalloc : 0xc8800000 - 0xff800000 ( 880 MB)
> > > > lowmem : 0xc0000000 - 0xc8000000 ( 128 MB)
> > > > pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
> > > > modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
> > > > .text : 0xc0008000 - 0xc0674eb8 (6580 kB)
> > > > .init : 0xc0700000 - 0xc0900000 (2048 kB)
> > > > .data : 0xc0900000 - 0xc0929420 ( 166 kB)
> > > > .bss : 0xc092b000 - 0xc096d1e8 ( 265 kB)
> > > > Hierarchical RCU implementation.
> > > > Build-time adjustment of leaf fanout to 32.
> > > > RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
> > > > RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
> > > > NR_IRQS:16 nr_irqs:16 16
> > > > clocksource_probe: no matching clocksources found
> > > > sched_clock: 32 bits at 100 Hz, resolution 10000000ns, wraps every
> > > > 21474836475000000ns
> > > > ------------[ cut here ]------------
> > > > WARNING: CPU: 0 PID: 0 at mm/slab.c:975 setup_kmem_cache_node+0x160/0x1c8
> > > > Modules linked in:
> > > > CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> > > > 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > > [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> > > > [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> > > > [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> > > > [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> > > > [<c011fad0>] (warn_slowpath_null) from [<c01ced4c>]
> > > > (setup_kmem_cache_node+0x160/0x1c8)
> > > > [<c01ced4c>] (setup_kmem_cache_node) from [<c01cf02c>]
> > > > (__do_tune_cpucache+0xf4/0x114)
> > > > [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> > > > [<c01cf0b0>] (enable_cpucache) from [<c0710324>]
> > > > (kmem_cache_init_late+0x40/0x84)
> > > > [<c0710324>] (kmem_cache_init_late) from [<c0700af8>] (start_kernel+0x238/0x36c)
> > > > [<c0700af8>] (start_kernel) from [<4000807c>] (0x4000807c)
> > > > ---[ end trace cb88537fdc8fa200 ]---
> > > > Console: colour dummy device 80x30
> > > > Calibrating delay loop (skipped) preset value.. 1066.00 BogoMIPS (lpj=5330000)
> > > > pid_max: default: 32768 minimum: 301
> > > > Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
> > > > Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
> > > > CPU: Testing write buffer coherency: ok
> > > > CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > > > Setting up static identity map for 0x40100000 - 0x40100058
> > > > CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > > > Brought up 2 CPUs
> > > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > > CPU: All CPU(s) started in SVC mode.
> > > > ------------[ cut here ]------------
> > > > WARNING: CPU: 0 PID: 1 at mm/slab.c:976 setup_kmem_cache_node+0x198/0x1c8
> > > > Modules linked in:
> > > > CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W
> > > > 4.6.0-kzm9d-05060-g801faf0db8947e01-dirty #84
> > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > > [<c010de08>] (unwind_backtrace) from [<c010a620>] (show_stack+0x10/0x14)
> > > > [<c010a620>] (show_stack) from [<c02b3178>] (dump_stack+0x7c/0x9c)
> > > > [<c02b3178>] (dump_stack) from [<c011f9fc>] (__warn+0xcc/0xfc)
> > > > [<c011f9fc>] (__warn) from [<c011fad0>] (warn_slowpath_null+0x1c/0x24)
> > > > [<c011fad0>] (warn_slowpath_null) from [<c01ced84>]
> > > > (setup_kmem_cache_node+0x198/0x1c8)
> > > > [<c01ced84>] (setup_kmem_cache_node) from [<c01cf02c>]
> > > > (__do_tune_cpucache+0xf4/0x114)
> > > > [<c01cf02c>] (__do_tune_cpucache) from [<c01cf0b0>] (enable_cpucache+0x64/0xb4)
> > > > [<c01cf0b0>] (enable_cpucache) from [<c01cf4cc>]
> > > > (__kmem_cache_create+0x1a0/0x1c8)
> > > > [<c01cf4cc>] (__kmem_cache_create) from [<c01b2904>]
> > > > (kmem_cache_create+0xbc/0x190)
> > > > [<c01b2904>] (kmem_cache_create) from [<c070d724>] (shmem_init+0x34/0xb0)
> > > > [<c070d724>] (shmem_init) from [<c0700cc4>] (kernel_init_freeable+0x98/0x1ec)
> > > > [<c0700cc4>] (kernel_init_freeable) from [<c0497760>] (kernel_init+0x8/0x110)
> > > > [<c0497760>] (kernel_init) from [<c0106c78>] (ret_from_fork+0x14/0x3c)
> > > > ---[ end trace cb88537fdc8fa201 ]---
> > > > devtmpfs: initialized
> > > > VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > > > clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > > > max_idle_ns: 19112604462750000 ns
> > > > pinctrl core: initialized pinctrl subsystem
> > > > NET: Registered protocol family 16
> > > > DMA: preallocated 256 KiB pool for atomic coherent allocations
> > > > sh-pfc e0140200.pfc: emev2_pfc support registered
> > > > gpiochip_find_base: found new base at 992
> > > > gpio gpiochip0: (e0050000.gpio): created GPIO range 0->31 ==>
> > > > e0140200.pfc PIN 0->31
> > > > gpio gpiochip0: (e0050000.gpio): added GPIO chardev (254:0)
> > > > gpiochip_setup_dev: registered GPIOs 992 to 1023 on device: gpiochip0
> > > > (e0050000.gpio)
> > > > gpiochip_find_base: found new base at 960
> > > > gpio gpiochip1: (e0050080.gpio): created GPIO range 0->31 ==>
> > > > e0140200.pfc PIN 32->63
> > > > gpio gpiochip1: (e0050080.gpio): added GPIO chardev (254:1)
> > > > gpiochip_setup_dev: registered GPIOs 960 to 991 on device: gpiochip1
> > > > (e0050080.gpio)
> > > > gpiochip_find_base: found new base at 928
> > > > gpio gpiochip2: (e0050100.gpio): created GPIO range 0->31 ==>
> > > > e0140200.pfc PIN 64->95
> > > > gpio gpiochip2: (e0050100.gpio): added GPIO chardev (254:2)
> > > > gpiochip_setup_dev: registered GPIOs 928 to 959 on device: gpiochip2
> > > > (e0050100.gpio)
> > > > gpiochip_find_base: found new base at 896
> > > > gpio gpiochip3: (e0050180.gpio): created GPIO range 0->31 ==>
> > > > e0140200.pfc PIN 96->127
> > > > gpio gpiochip3: (e0050180.gpio): added GPIO chardev (254:3)
> > > > gpiochip_setup_dev: registered GPIOs 896 to 927 on device: gpiochip3
> > > > (e0050180.gpio)
> > > > gpiochip_find_base: found new base at 865
> > > > gpio gpiochip4: (e0050200.gpio): created GPIO range 0->30 ==>
> > > > e0140200.pfc PIN 128->158
> > > > gpio gpiochip4: (e0050200.gpio): added GPIO chardev (254:4)
> > > > gpiochip_setup_dev: registered GPIOs 865 to 895 on device: gpiochip4
> > > > (e0050200.gpio)
> > > > gio: map hw irq = 1, irq = 35
> > > > gio: sense irq = 1, mode = 8
> > > > No ATAGs?
> > > > hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
> > > > hw-breakpoint: maximum watchpoint size is 4 bytes.
> > > > of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@0[0]'
> > > > of_get_named_gpiod_flags: can't parse 'gpio' property of node '/regulator@1[0]'
> > > > SCSI subsystem initialized
> > > > usbcore: registered new interface driver usbfs
> > > > usbcore: registered new interface driver hub
> > > > usbcore: registered new device driver usb
> > > > em_sti e0180000.timer: used for clock events
> > > > em_sti e0180000.timer: used for oneshot clock events
> > > > em_sti e0180000.timer: used as clock source
> > > > clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> > > > 0x1ef4687b1, max_idle_ns: 3697658158765000000 ns
> > > > Advanced Linux Sound Architecture Driver Initialized.
> > > > NET: Registered protocol family 23
> > > > clocksource: e0180000.timer: mask: 0xffffffffffff max_cycles:
> > > > 0x1ef4687b1, max_idle_ns: 112843571739654 ns
> > > > clocksource: Switched to clocksource e0180000.timer
> > > > NET: Registered protocol family 2
> > > > TCP established hash table entries: 1024 (order: 0, 4096 bytes)
> > > > TCP bind hash table entries: 1024 (order: 2, 20480 bytes)
> > > > TCP: Hash tables configured (established 1024 bind 1024)
> > > > UDP hash table entries: 256 (order: 1, 12288 bytes)
> > > > UDP-Lite hash table entries: 256 (order: 1, 12288 bytes)
> > > > NET: Registered protocol family 1
> > > > RPC: Registered named UNIX socket transport module.
> > > > RPC: Registered udp transport module.
> > > > RPC: Registered tcp transport module.
> > > > RPC: Registered tcp NFSv4.1 backchannel transport module.
> > > > Clockevents: could not switch to one-shot mode:
> > > > Clockevents: could not switch to one-shot mode: dummy_timer is not functional.
> > > > Could not switch to high resolution mode on CPU 0
> > > > dummy_timer is not functional.
> > > > Could not switch to high resolution mode on CPU 1
> > > > hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
> > > > futex hash table entries: 512 (order: 3, 32768 bytes)
> > > > workingset: timestamp_bits=28 max_order=15 bucket_order=0
> > > > NFS: Registering the id_resolver key type
> > > > Key type id_resolver registered
> > > > Key type id_legacy registered
> > > > nfs4filelayout_init: NFSv4 File Layout Driver Registering...
> > > > io scheduler noop registered (default)
> > > > Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> > > > e1020000.serial: ttyS0 at MMIO 0xe1020000 (irq = 19, base_baud =
> > > > 796444) is a 16550A
> > > > e1030000.serial: ttyS1 at MMIO 0xe1030000 (irq = 20, base_baud =
> > > > 7168000) is a 16550A
> > > > console [ttyS1] enabled
> > > > e1040000.serial: ttyS2 at MMIO 0xe1040000 (irq = 21, base_baud =
> > > > 14336000) is a 16550A
> > > > e1050000.serial: ttyS3 at MMIO 0xe1050000 (irq = 22, base_baud =
> > > > 2389333) is a 16550A
> > > > gio: sense irq = 1, mode = 8
> > > > libphy: smsc911x-mdio: probed
> > > > Generic PHY 20000000.etherne:01: attached PHY driver [Generic PHY]
> > > > (mii_bus:phy_addr=20000000.etherne:01, irq=-1)
> > > > smsc911x 20000000.ethernet eth0: MAC Address: 00:01:9b:04:03:cf
> > > > usbcore: registered new interface driver usb-storage
> > > > i2c /dev entries driver
> > > > em-i2c e0070000.i2c: Added i2c controller 0, irq 33
> > > > em-i2c e10a0000.i2c: Added i2c controller 1, irq 34
> > > > cpu cpu0: failed to get clock: -2
> > > > cpufreq-dt: probe of cpufreq-dt failed with error -2
> > > > ledtrig-cpu: registered to indicate activity on CPUs
> > > > usbcore: registered new interface driver usbhid
> > > > usbhid: USB HID core driver
> > > > NET: Registered protocol family 17
> > > > Key type dns_resolver registered
> > > > Registering SWP/SWPB emulation handler
> > > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > > '/gpio_keys/button@1[0]' - status (0)
> > > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > > '/gpio_keys/button@2[0]' - status (0)
> > > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > > '/gpio_keys/button@3[0]' - status (0)
> > > > of_get_named_gpiod_flags: parsed 'gpios' property of node
> > > > '/gpio_keys/button@4[0]' - status (0)
> > > > gpio-1006 (DSW2-1): gpiod_set_debounce: missing set() or
> > > > set_debounce() operations
> > > > gio: map hw irq = 14, irq = 36
> > > > gio: sense irq = 14, mode = 12
> > > > gpio-1007 (DSW2-2): gpiod_set_debounce: missing set() or
> > > > set_debounce() operations
> > > > gio: map hw irq = 15, irq = 37
> > > > gio: sense irq = 15, mode = 12
> > > > gpio-1008 (DSW2-3): gpiod_set_debounce: missing set() or
> > > > set_debounce() operations
> > > > gio: map hw irq = 16, irq = 38
> > > > gio: sense irq = 16, mode = 12
> > > > gpio-1009 (DSW2-4): gpiod_set_debounce: missing set() or
> > > > set_debounce() operations
> > > > gio: map hw irq = 17, irq = 39
> > > > gio: sense irq = 17, mode = 12
> > > > input: gpio_keys as /devices/platform/gpio_keys/input/input0
> > > > hctosys: unable to open rtc device (rtc0)
> > > > smsc911x 20000000.ethernet eth0: SMSC911x/921x identified at 0xc8880000, IRQ: 35
> > > > Sending DHCP requests .., OK
> > > > IP-Config: Got DHCP answer from 192.168.97.254, my address is 192.168.97.215
> > > > IP-Config: Complete:
> > > > device=eth0, hwaddr=00:01:9b:04:03:cf, ipaddr=192.168.97.215,
> > > > mask=255.255.255.0, gw=192.168.97.254
> > > > host=192.168.97.215, domain=of.borg, nis-domain=(none)
> > > > bootserver=192.168.97.254, rootserver=192.168.97.254, rootpath=
> > > > nameserver0=192.168.97.254
> > > > ALSA device list:
> > > > No soundcards found.
> > > > Freeing unused kernel memory: 2048K (c0700000 - c0900000)
> > > > sysctl: error: 'kernel.hotplug' is an unknown key
> > > >
> > > > Gr{oetje,eeting}s,
> > > >
> > > > Geert
> > > >
> > > > --
> > > > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> > > >
> > > > In personal conversations with technical people, I call myself a hacker. But
> > > > when I'm talking to journalists I just say "programmer" or something like that.
> > > > -- Linus Torvalds
> > > >
> > > > --
> > > > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > > > the body to majordomo@kvack.org. For more info on Linux MM,
> > > > see: http://www.linux-mm.org/ .
> > > > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> > >
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org. For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-21 12:54 ` Paul E. McKenney
@ 2016-06-22 0:52 ` Joonsoo Kim
2016-06-22 3:15 ` Paul E. McKenney
2016-06-22 15:01 ` Geert Uytterhoeven
0 siblings, 2 replies; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-22 0:52 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Tue, Jun 21, 2016 at 05:54:06AM -0700, Paul E. McKenney wrote:
> On Tue, Jun 21, 2016 at 03:43:02PM +0900, Joonsoo Kim wrote:
> > On Mon, Jun 20, 2016 at 06:12:54AM -0700, Paul E. McKenney wrote:
> > > On Mon, Jun 20, 2016 at 03:39:43PM +0900, Joonsoo Kim wrote:
> > > > CCing Paul to ask some question.
> > > >
> > > > On Wed, Jun 15, 2016 at 10:39:47AM +0200, Geert Uytterhoeven wrote:
> > > > > Hi Joonsoo,
> > > > >
> > > > > On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > > On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
> > > > > >> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > >> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> > > > > >> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > >> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> > > > > >> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> > > > > >> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> > > > > >> >> >> > To check whther free objects exist or not precisely, we need to grab a
> > > > > >> >> >> > lock. But, accuracy isn't that important because race window would be
> > > > > >> >> >> > even small and if there is too much free object, cache reaper would reap
> > > > > >> >> >> > it. So, this patch makes the check for free object exisistence not to
> > > > > >> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
> > > > >
> > > > > >> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> > > > > >> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> > > > > >> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> > > > >
> > > > > > It's curious that synchronize_sched() has some effect in this early
> > > > > > phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
> > > > > > it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
> > > > > >
> > > > > > It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
> > > > > >
> > > > > > First, I'd like to confirm that num_online_cpus() is correct.
> > > > > > Could you try following patch and give me a dmesg?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > ------->8----------
> > > > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > > > index 763096a..5b7300a 100644
> > > > > > --- a/mm/slab.c
> > > > > > +++ b/mm/slab.c
> > > > > > @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > > > * freed after synchronize_sched().
> > > > > > */
> > > > > > - if (force_change)
> > > > > > - synchronize_sched();
> > > > > > + if (force_change) {
> > > > > > + WARN_ON_ONCE(num_online_cpus() <= 1);
> > > > > > + WARN_ON_ONCE(num_online_cpus() > 1);
> > > > > > + }
> > > > >
> > > > > Full dmesg output below.
> > > > >
> > > > > I also tested whether it's the call to synchronize_sched() before or after
> > > > > secondary CPU bringup that hangs.
> > > > >
> > > > > if (force_change && num_online_cpus() <= 1)
> > > > > synchronize_sched();
> > > > >
> > > > > boots.
> > > > >
> > > > > if (force_change && num_online_cpus() > 1)
> > > > > synchronize_sched();
> > > > >
> > > > > hangs.
> > > >
> > > > Hello, Paul.
> > > >
> > > > I changed slab.c to use synchronize_sched() for full memory barrier. First
> > > > call happens on kmem_cache_init_late() and it would not be a problem
> > > > because, at this time, num_online_cpus() <= 1 and synchronize_sched()
> > > > would return immediately. Second call site would be shmem_init()
> > > > and it seems that system hangs on it. Since smp is already initialized
> > > > at that time, there would be some effect of synchronize_sched() but I
> > > > can't imagine what's wrong here. Is it invalid moment to call
> > > > synchronize_sched()?
> > > >
> > > > Note that my x86 virtual machine works fine even if
> > > > synchronize_sched() is called in shmem_init() but Geert's some ARM
> > > > machines (not all ARM machine) don't work well with it.
> > >
> > > Color me confused.
> > >
> > > Is Geert's ARM system somehow adding the second CPU before
> > > rcu_spawn_gp_kthread() is called, that is, before or during
> > > early_initcall() time?
> >
> > Hang would happen on shmem_init() which is called in do_basic_setup().
> > do_basic_setup() is called after early_initcall().
>
> Thank you for the info!
>
> That should be lat enough that the RCU kthreads are alive and well.
>
> Can you get sysalt-t output?
>
> > Hmm... Is it okay to call synchronize_sched() by kernel thread?
>
> Yes, it can, in fact, rcutorture does this all the time. As do any
> number of other kthreads.
Paul, thanks for confirmation.
Geert, we need to try more debugging.
Could you try below patch to check who causes the hang?
And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
used it before but Paul could find some culprit on it. :)
Thanks.
----->8-----
diff --git a/mm/slab.c b/mm/slab.c
index 763096a..9652d38 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
* guaranteed to be valid until irq is re-enabled, because it will be
* freed after synchronize_sched().
*/
- if (force_change)
+ if (force_change) {
+ if (num_online_cpus() > 1)
+ dump_stack();
synchronize_sched();
+ if (num_online_cpus() > 1)
+ dump_stack();
+ }
fail:
kfree(old_shared);
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-22 0:52 ` Joonsoo Kim
@ 2016-06-22 3:15 ` Paul E. McKenney
2016-06-22 15:01 ` Geert Uytterhoeven
1 sibling, 0 replies; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-22 3:15 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Wed, Jun 22, 2016 at 09:52:08AM +0900, Joonsoo Kim wrote:
> On Tue, Jun 21, 2016 at 05:54:06AM -0700, Paul E. McKenney wrote:
> > On Tue, Jun 21, 2016 at 03:43:02PM +0900, Joonsoo Kim wrote:
> > > On Mon, Jun 20, 2016 at 06:12:54AM -0700, Paul E. McKenney wrote:
> > > > On Mon, Jun 20, 2016 at 03:39:43PM +0900, Joonsoo Kim wrote:
> > > > > CCing Paul to ask some question.
> > > > >
> > > > > On Wed, Jun 15, 2016 at 10:39:47AM +0200, Geert Uytterhoeven wrote:
> > > > > > Hi Joonsoo,
> > > > > >
> > > > > > On Wed, Jun 15, 2016 at 4:23 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > > > On Tue, Jun 14, 2016 at 12:45:14PM +0200, Geert Uytterhoeven wrote:
> > > > > > >> On Tue, Jun 14, 2016 at 10:11 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > > >> > On Tue, Jun 14, 2016 at 09:31:23AM +0200, Geert Uytterhoeven wrote:
> > > > > > >> >> On Tue, Jun 14, 2016 at 8:24 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > > >> >> > On Mon, Jun 13, 2016 at 09:43:13PM +0200, Geert Uytterhoeven wrote:
> > > > > > >> >> >> On Tue, Apr 12, 2016 at 6:51 AM, <js1304@gmail.com> wrote:
> > > > > > >> >> >> > From: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> > > > > > >> >> >> > To check whther free objects exist or not precisely, we need to grab a
> > > > > > >> >> >> > lock. But, accuracy isn't that important because race window would be
> > > > > > >> >> >> > even small and if there is too much free object, cache reaper would reap
> > > > > > >> >> >> > it. So, this patch makes the check for free object exisistence not to
> > > > > > >> >> >> > hold a lock. This will reduce lock contention in heavily allocation case.
> > > > > >
> > > > > > >> >> >> I've bisected a boot failure (no output at all) in v4.7-rc2 on emev2/kzm9d
> > > > > > >> >> >> (Renesas dual Cortex A9) to this patch, which is upstream commit
> > > > > > >> >> >> 801faf0db8947e01877920e848a4d338dd7a99e7.
> > > > > >
> > > > > > > It's curious that synchronize_sched() has some effect in this early
> > > > > > > phase. In synchronize_sched(), rcu_blocking_is_gp() is called and
> > > > > > > it checks num_online_cpus <= 1. If so, synchronize_sched() does nothing.
> > > > > > >
> > > > > > > It would be related to might_sleep() in rcu_blocking_is_gp() but I'm not sure now.
> > > > > > >
> > > > > > > First, I'd like to confirm that num_online_cpus() is correct.
> > > > > > > Could you try following patch and give me a dmesg?
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > ------->8----------
> > > > > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > > > > index 763096a..5b7300a 100644
> > > > > > > --- a/mm/slab.c
> > > > > > > +++ b/mm/slab.c
> > > > > > > @@ -964,8 +964,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > > > > * freed after synchronize_sched().
> > > > > > > */
> > > > > > > - if (force_change)
> > > > > > > - synchronize_sched();
> > > > > > > + if (force_change) {
> > > > > > > + WARN_ON_ONCE(num_online_cpus() <= 1);
> > > > > > > + WARN_ON_ONCE(num_online_cpus() > 1);
> > > > > > > + }
> > > > > >
> > > > > > Full dmesg output below.
> > > > > >
> > > > > > I also tested whether it's the call to synchronize_sched() before or after
> > > > > > secondary CPU bringup that hangs.
> > > > > >
> > > > > > if (force_change && num_online_cpus() <= 1)
> > > > > > synchronize_sched();
> > > > > >
> > > > > > boots.
> > > > > >
> > > > > > if (force_change && num_online_cpus() > 1)
> > > > > > synchronize_sched();
> > > > > >
> > > > > > hangs.
> > > > >
> > > > > Hello, Paul.
> > > > >
> > > > > I changed slab.c to use synchronize_sched() for full memory barrier. First
> > > > > call happens on kmem_cache_init_late() and it would not be a problem
> > > > > because, at this time, num_online_cpus() <= 1 and synchronize_sched()
> > > > > would return immediately. Second call site would be shmem_init()
> > > > > and it seems that system hangs on it. Since smp is already initialized
> > > > > at that time, there would be some effect of synchronize_sched() but I
> > > > > can't imagine what's wrong here. Is it invalid moment to call
> > > > > synchronize_sched()?
> > > > >
> > > > > Note that my x86 virtual machine works fine even if
> > > > > synchronize_sched() is called in shmem_init() but Geert's some ARM
> > > > > machines (not all ARM machine) don't work well with it.
> > > >
> > > > Color me confused.
> > > >
> > > > Is Geert's ARM system somehow adding the second CPU before
> > > > rcu_spawn_gp_kthread() is called, that is, before or during
> > > > early_initcall() time?
> > >
> > > Hang would happen on shmem_init() which is called in do_basic_setup().
> > > do_basic_setup() is called after early_initcall().
> >
> > Thank you for the info!
> >
> > That should be lat enough that the RCU kthreads are alive and well.
> >
> > Can you get sysalt-t output?
> >
> > > Hmm... Is it okay to call synchronize_sched() by kernel thread?
> >
> > Yes, it can, in fact, rcutorture does this all the time. As do any
> > number of other kthreads.
>
> Paul, thanks for confirmation.
>
> Geert, we need to try more debugging.
>
> Could you try below patch to check who causes the hang?
Nice! That might be quite valuable!
> And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> used it before but Paul could find some culprit on it. :)
And the other thing to do is to read the last portion of
Documentation/RCU/stallwarn.txt, the part starting with
"What Causes RCU CPU Stall Warnings?". I would expect any
of these things to also result in an RCU CPU stall warning,
but perhaps something is preventing them from being printed.
Short summary: If a CPU gets stuck badly enough, RCU grace
periods won't end and therefore synchronize_sched() won't
ever return.
Thanx, Paul
> Thanks.
>
>
> ----->8-----
> diff --git a/mm/slab.c b/mm/slab.c
> index 763096a..9652d38 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> * guaranteed to be valid until irq is re-enabled, because it will be
> * freed after synchronize_sched().
> */
> - if (force_change)
> + if (force_change) {
> + if (num_online_cpus() > 1)
> + dump_stack();
> synchronize_sched();
> + if (num_online_cpus() > 1)
> + dump_stack();
> + }
>
> fail:
> kfree(old_shared);
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-22 0:52 ` Joonsoo Kim
2016-06-22 3:15 ` Paul E. McKenney
@ 2016-06-22 15:01 ` Geert Uytterhoeven
2016-06-22 19:08 ` Paul E. McKenney
1 sibling, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-22 15:01 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Paul E. McKenney, Andrew Morton, Christoph Lameter, Pekka Enberg,
David Rientjes, Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> Could you try below patch to check who causes the hang?
>
> And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> used it before but Paul could find some culprit on it. :)
>
> Thanks.
>
>
> ----->8-----
> diff --git a/mm/slab.c b/mm/slab.c
> index 763096a..9652d38 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> * guaranteed to be valid until irq is re-enabled, because it will be
> * freed after synchronize_sched().
> */
> - if (force_change)
> + if (force_change) {
> + if (num_online_cpus() > 1)
> + dump_stack();
> synchronize_sched();
> + if (num_online_cpus() > 1)
> + dump_stack();
> + }
I've only added the first one, as I would never see the second one. All of
this happens before the serial console is activated, earlycon is not supported,
and I only have remote access.
Brought up 2 CPUs
SMP: Total of 2 processors activated (2132.00 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted
4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
[<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
[<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
[<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
[<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
(__do_tune_cpucache+0xf4/0x114)
[<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
[<c01cff54>] (enable_cpucache) from [<c01d0190>]
(__kmem_cache_create+0x1a8/0x1d0)
[<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
(kmem_cache_create+0xbc/0x190)
[<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
[<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
[<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
[<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
devtmpfs: initialized
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-22 15:01 ` Geert Uytterhoeven
@ 2016-06-22 19:08 ` Paul E. McKenney
2016-06-23 0:49 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-22 19:08 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Joonsoo Kim, Andrew Morton, Christoph Lameter, Pekka Enberg,
David Rientjes, Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
> On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > Could you try below patch to check who causes the hang?
> >
> > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> > used it before but Paul could find some culprit on it. :)
> >
> > Thanks.
> >
> >
> > ----->8-----
> > diff --git a/mm/slab.c b/mm/slab.c
> > index 763096a..9652d38 100644
> > --- a/mm/slab.c
> > +++ b/mm/slab.c
> > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > * guaranteed to be valid until irq is re-enabled, because it will be
> > * freed after synchronize_sched().
> > */
> > - if (force_change)
> > + if (force_change) {
> > + if (num_online_cpus() > 1)
> > + dump_stack();
> > synchronize_sched();
> > + if (num_online_cpus() > 1)
> > + dump_stack();
> > + }
>
> I've only added the first one, as I would never see the second one. All of
> this happens before the serial console is activated, earlycon is not supported,
> and I only have remote access.
>
> Brought up 2 CPUs
> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> CPU: All CPU(s) started in SVC mode.
> CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
> Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
> [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
> [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
> [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
> (__do_tune_cpucache+0xf4/0x114)
> [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
> [<c01cff54>] (enable_cpucache) from [<c01d0190>]
> (__kmem_cache_create+0x1a8/0x1d0)
> [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
> (kmem_cache_create+0xbc/0x190)
> [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
> [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
> [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
> [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
> devtmpfs: initialized
I don't see anything here that would prevent grace periods from completing.
The CPUs are using the normal hotplug sequence to come online, correct?
Thanx, Paul
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-22 19:08 ` Paul E. McKenney
@ 2016-06-23 0:49 ` Paul E. McKenney
2016-06-23 2:37 ` Joonsoo Kim
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-23 0:49 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Joonsoo Kim, Andrew Morton, Christoph Lameter, Pekka Enberg,
David Rientjes, Jesper Dangaard Brouer, Linux MM, linux-kernel,
linux-renesas-soc, linux-arm-kernel
On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote:
> On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
> > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > Could you try below patch to check who causes the hang?
> > >
> > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> > > used it before but Paul could find some culprit on it. :)
> > >
> > > Thanks.
> > >
> > >
> > > ----->8-----
> > > diff --git a/mm/slab.c b/mm/slab.c
> > > index 763096a..9652d38 100644
> > > --- a/mm/slab.c
> > > +++ b/mm/slab.c
> > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > * freed after synchronize_sched().
> > > */
> > > - if (force_change)
> > > + if (force_change) {
> > > + if (num_online_cpus() > 1)
> > > + dump_stack();
> > > synchronize_sched();
> > > + if (num_online_cpus() > 1)
> > > + dump_stack();
> > > + }
> >
> > I've only added the first one, as I would never see the second one. All of
> > this happens before the serial console is activated, earlycon is not supported,
> > and I only have remote access.
> >
> > Brought up 2 CPUs
> > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > CPU: All CPU(s) started in SVC mode.
> > CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
> > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
> > [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
> > [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
> > [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
> > (__do_tune_cpucache+0xf4/0x114)
> > [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
> > [<c01cff54>] (enable_cpucache) from [<c01d0190>]
> > (__kmem_cache_create+0x1a8/0x1d0)
> > [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
> > (kmem_cache_create+0xbc/0x190)
> > [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
> > [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
> > [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
> > [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
> > devtmpfs: initialized
>
> I don't see anything here that would prevent grace periods from completing.
>
> The CPUs are using the normal hotplug sequence to come online, correct?
And either way, could you please apply the patch below and then
invoke rcu_dump_rcu_sched_tree() just before the offending call to
synchronize_sched()? That will tell me what CPUs RCU believes exist,
and perhaps also which CPU is holding it up.
Thanx, Paul
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-23 0:49 ` Paul E. McKenney
@ 2016-06-23 2:37 ` Joonsoo Kim
2016-06-23 2:47 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-23 2:37 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Wed, Jun 22, 2016 at 05:49:35PM -0700, Paul E. McKenney wrote:
> On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote:
> > On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
> > > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > Could you try below patch to check who causes the hang?
> > > >
> > > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> > > > used it before but Paul could find some culprit on it. :)
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > ----->8-----
> > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > index 763096a..9652d38 100644
> > > > --- a/mm/slab.c
> > > > +++ b/mm/slab.c
> > > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > * freed after synchronize_sched().
> > > > */
> > > > - if (force_change)
> > > > + if (force_change) {
> > > > + if (num_online_cpus() > 1)
> > > > + dump_stack();
> > > > synchronize_sched();
> > > > + if (num_online_cpus() > 1)
> > > > + dump_stack();
> > > > + }
> > >
> > > I've only added the first one, as I would never see the second one. All of
> > > this happens before the serial console is activated, earlycon is not supported,
> > > and I only have remote access.
> > >
> > > Brought up 2 CPUs
> > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > CPU: All CPU(s) started in SVC mode.
> > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> > > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
> > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
> > > [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
> > > [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
> > > [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
> > > (__do_tune_cpucache+0xf4/0x114)
> > > [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
> > > [<c01cff54>] (enable_cpucache) from [<c01d0190>]
> > > (__kmem_cache_create+0x1a8/0x1d0)
> > > [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
> > > (kmem_cache_create+0xbc/0x190)
> > > [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
> > > [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
> > > [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
> > > [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
> > > devtmpfs: initialized
> >
> > I don't see anything here that would prevent grace periods from completing.
> >
> > The CPUs are using the normal hotplug sequence to come online, correct?
>
> And either way, could you please apply the patch below and then
> invoke rcu_dump_rcu_sched_tree() just before the offending call to
> synchronize_sched()? That will tell me what CPUs RCU believes exist,
> and perhaps also which CPU is holding it up.
I can't find rcu_dump_rcu_sched_tree(). Do you mean
rcu_dump_rcu_node_tree()? Anyway, there is no patch below so I attach
one which does what Paul want, maybe.
Thanks.
------->8---------
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 88d3f95..6b650f0 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4171,7 +4171,7 @@ static void __init rcu_init_geometry(void)
* Dump out the structure of the rcu_node combining tree associated
* with the rcu_state structure referenced by rsp.
*/
-static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
+static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
{
int level = 0;
struct rcu_node *rnp;
@@ -4189,6 +4189,11 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
pr_cont("\n");
}
+void rcu_dump_rcu_sched_tree(void)
+{
+ rcu_dump_rcu_node_tree(&rcu_sched_state);
+}
+
void __init rcu_init(void)
{
int cpu;
diff --git a/mm/slab.c b/mm/slab.c
index 763096a..d88976c 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -909,6 +909,8 @@ static int init_cache_node_node(int node)
return 0;
}
+extern void rcu_dump_rcu_sched_tree(void);
+
static int setup_kmem_cache_node(struct kmem_cache *cachep,
int node, gfp_t gfp, bool force_change)
{
@@ -964,8 +966,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
* guaranteed to be valid until irq is re-enabled, because it will be
* freed after synchronize_sched().
*/
- if (force_change)
+ if (force_change) {
+ rcu_dump_rcu_sched_tree();
synchronize_sched();
+ }
fail:
kfree(old_shared);
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-23 2:37 ` Joonsoo Kim
@ 2016-06-23 2:47 ` Paul E. McKenney
2016-06-23 2:53 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-23 2:47 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Thu, Jun 23, 2016 at 11:37:56AM +0900, Joonsoo Kim wrote:
> On Wed, Jun 22, 2016 at 05:49:35PM -0700, Paul E. McKenney wrote:
> > On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote:
> > > On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
> > > > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > Could you try below patch to check who causes the hang?
> > > > >
> > > > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> > > > > used it before but Paul could find some culprit on it. :)
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > > ----->8-----
> > > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > > index 763096a..9652d38 100644
> > > > > --- a/mm/slab.c
> > > > > +++ b/mm/slab.c
> > > > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > > * freed after synchronize_sched().
> > > > > */
> > > > > - if (force_change)
> > > > > + if (force_change) {
> > > > > + if (num_online_cpus() > 1)
> > > > > + dump_stack();
> > > > > synchronize_sched();
> > > > > + if (num_online_cpus() > 1)
> > > > > + dump_stack();
> > > > > + }
> > > >
> > > > I've only added the first one, as I would never see the second one. All of
> > > > this happens before the serial console is activated, earlycon is not supported,
> > > > and I only have remote access.
> > > >
> > > > Brought up 2 CPUs
> > > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > > CPU: All CPU(s) started in SVC mode.
> > > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> > > > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
> > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > > [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
> > > > [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
> > > > [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
> > > > [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
> > > > (__do_tune_cpucache+0xf4/0x114)
> > > > [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
> > > > [<c01cff54>] (enable_cpucache) from [<c01d0190>]
> > > > (__kmem_cache_create+0x1a8/0x1d0)
> > > > [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
> > > > (kmem_cache_create+0xbc/0x190)
> > > > [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
> > > > [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
> > > > [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
> > > > [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
> > > > devtmpfs: initialized
> > >
> > > I don't see anything here that would prevent grace periods from completing.
> > >
> > > The CPUs are using the normal hotplug sequence to come online, correct?
> >
> > And either way, could you please apply the patch below and then
> > invoke rcu_dump_rcu_sched_tree() just before the offending call to
> > synchronize_sched()? That will tell me what CPUs RCU believes exist,
> > and perhaps also which CPU is holding it up.
>
> I can't find rcu_dump_rcu_sched_tree(). Do you mean
> rcu_dump_rcu_node_tree()? Anyway, there is no patch below so I attach
> one which does what Paul want, maybe.
One of those days, I guess! :-/
Your patch is exactly what I intended to send, thank you!
Thanx, Paul
> Thanks.
>
> ------->8---------
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 88d3f95..6b650f0 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4171,7 +4171,7 @@ static void __init rcu_init_geometry(void)
> * Dump out the structure of the rcu_node combining tree associated
> * with the rcu_state structure referenced by rsp.
> */
> -static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> +static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> {
> int level = 0;
> struct rcu_node *rnp;
> @@ -4189,6 +4189,11 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> pr_cont("\n");
> }
>
> +void rcu_dump_rcu_sched_tree(void)
> +{
> + rcu_dump_rcu_node_tree(&rcu_sched_state);
> +}
> +
> void __init rcu_init(void)
> {
> int cpu;
> diff --git a/mm/slab.c b/mm/slab.c
> index 763096a..d88976c 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -909,6 +909,8 @@ static int init_cache_node_node(int node)
> return 0;
> }
>
> +extern void rcu_dump_rcu_sched_tree(void);
> +
> static int setup_kmem_cache_node(struct kmem_cache *cachep,
> int node, gfp_t gfp, bool force_change)
> {
> @@ -964,8 +966,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> * guaranteed to be valid until irq is re-enabled, because it will be
> * freed after synchronize_sched().
> */
> - if (force_change)
> + if (force_change) {
> + rcu_dump_rcu_sched_tree();
> synchronize_sched();
> + }
>
> fail:
> kfree(old_shared);
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-23 2:47 ` Paul E. McKenney
@ 2016-06-23 2:53 ` Paul E. McKenney
2016-06-28 0:12 ` Paul E. McKenney
2016-06-29 14:54 ` Geert Uytterhoeven
0 siblings, 2 replies; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-23 2:53 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> On Thu, Jun 23, 2016 at 11:37:56AM +0900, Joonsoo Kim wrote:
> > On Wed, Jun 22, 2016 at 05:49:35PM -0700, Paul E. McKenney wrote:
> > > On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote:
> > > > On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
> > > > > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > > Could you try below patch to check who causes the hang?
> > > > > >
> > > > > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> > > > > > used it before but Paul could find some culprit on it. :)
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > > ----->8-----
> > > > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > > > index 763096a..9652d38 100644
> > > > > > --- a/mm/slab.c
> > > > > > +++ b/mm/slab.c
> > > > > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > > > * freed after synchronize_sched().
> > > > > > */
> > > > > > - if (force_change)
> > > > > > + if (force_change) {
> > > > > > + if (num_online_cpus() > 1)
> > > > > > + dump_stack();
> > > > > > synchronize_sched();
> > > > > > + if (num_online_cpus() > 1)
> > > > > > + dump_stack();
> > > > > > + }
> > > > >
> > > > > I've only added the first one, as I would never see the second one. All of
> > > > > this happens before the serial console is activated, earlycon is not supported,
> > > > > and I only have remote access.
> > > > >
> > > > > Brought up 2 CPUs
> > > > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > > > CPU: All CPU(s) started in SVC mode.
> > > > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> > > > > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
> > > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > > > [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
> > > > > [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
> > > > > [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
> > > > > [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
> > > > > (__do_tune_cpucache+0xf4/0x114)
> > > > > [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
> > > > > [<c01cff54>] (enable_cpucache) from [<c01d0190>]
> > > > > (__kmem_cache_create+0x1a8/0x1d0)
> > > > > [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
> > > > > (kmem_cache_create+0xbc/0x190)
> > > > > [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
> > > > > [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
> > > > > [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
> > > > > [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
> > > > > devtmpfs: initialized
> > > >
> > > > I don't see anything here that would prevent grace periods from completing.
> > > >
> > > > The CPUs are using the normal hotplug sequence to come online, correct?
> > >
> > > And either way, could you please apply the patch below and then
> > > invoke rcu_dump_rcu_sched_tree() just before the offending call to
> > > synchronize_sched()? That will tell me what CPUs RCU believes exist,
> > > and perhaps also which CPU is holding it up.
> >
> > I can't find rcu_dump_rcu_sched_tree(). Do you mean
> > rcu_dump_rcu_node_tree()? Anyway, there is no patch below so I attach
> > one which does what Paul want, maybe.
>
> One of those days, I guess! :-/
>
> Your patch is exactly what I intended to send, thank you!
Ah, but your telepathy was not sufficient to intuit the additional
information I need. Please see the patch at the end. Your hunk
in mm/slab.c is needed on top of my patch.
So I am clearly having difficulties reading as well as including patches
today...
Thanx, Paul
> > Thanks.
> >
> > ------->8---------
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 88d3f95..6b650f0 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -4171,7 +4171,7 @@ static void __init rcu_init_geometry(void)
> > * Dump out the structure of the rcu_node combining tree associated
> > * with the rcu_state structure referenced by rsp.
> > */
> > -static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > +static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > {
> > int level = 0;
> > struct rcu_node *rnp;
> > @@ -4189,6 +4189,11 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > pr_cont("\n");
> > }
> >
> > +void rcu_dump_rcu_sched_tree(void)
> > +{
> > + rcu_dump_rcu_node_tree(&rcu_sched_state);
> > +}
> > +
> > void __init rcu_init(void)
> > {
> > int cpu;
> > diff --git a/mm/slab.c b/mm/slab.c
> > index 763096a..d88976c 100644
> > --- a/mm/slab.c
> > +++ b/mm/slab.c
> > @@ -909,6 +909,8 @@ static int init_cache_node_node(int node)
> > return 0;
> > }
> >
> > +extern void rcu_dump_rcu_sched_tree(void);
> > +
> > static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > int node, gfp_t gfp, bool force_change)
> > {
> > @@ -964,8 +966,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > * guaranteed to be valid until irq is re-enabled, because it will be
> > * freed after synchronize_sched().
> > */
> > - if (force_change)
> > + if (force_change) {
> > + rcu_dump_rcu_sched_tree();
> > synchronize_sched();
> > + }
> >
> > fail:
> > kfree(old_shared);
> >
------------------------------------------------------------------------
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index c7f1bc4f817c..2eda7bece401 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4707,7 +4707,7 @@ static void __init rcu_init_geometry(void)
* Dump out the structure of the rcu_node combining tree associated
* with the rcu_state structure referenced by rsp.
*/
-static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
+static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
{
int level = 0;
struct rcu_node *rnp;
@@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
pr_info(" ");
level = rnp->level;
}
- pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
+ pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
+ rnp->qsmask,
+ rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
}
pr_cont("\n");
}
+void rcu_dump_rcu_sched_tree(void)
+{
+ rcu_dump_rcu_node_tree(&rcu_sched_state);
+}
+
void __init rcu_init(void)
{
int cpu;
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-23 2:53 ` Paul E. McKenney
@ 2016-06-28 0:12 ` Paul E. McKenney
2016-06-28 8:33 ` Joonsoo Kim
2016-06-29 14:54 ` Geert Uytterhoeven
1 sibling, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-28 0:12 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Wed, Jun 22, 2016 at 07:53:29PM -0700, Paul E. McKenney wrote:
> On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> > On Thu, Jun 23, 2016 at 11:37:56AM +0900, Joonsoo Kim wrote:
> > > On Wed, Jun 22, 2016 at 05:49:35PM -0700, Paul E. McKenney wrote:
> > > > On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote:
> > > > > On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
> > > > > > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > > > Could you try below patch to check who causes the hang?
> > > > > > >
> > > > > > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> > > > > > > used it before but Paul could find some culprit on it. :)
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > > ----->8-----
> > > > > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > > > > index 763096a..9652d38 100644
> > > > > > > --- a/mm/slab.c
> > > > > > > +++ b/mm/slab.c
> > > > > > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > > > > * freed after synchronize_sched().
> > > > > > > */
> > > > > > > - if (force_change)
> > > > > > > + if (force_change) {
> > > > > > > + if (num_online_cpus() > 1)
> > > > > > > + dump_stack();
> > > > > > > synchronize_sched();
> > > > > > > + if (num_online_cpus() > 1)
> > > > > > > + dump_stack();
> > > > > > > + }
> > > > > >
> > > > > > I've only added the first one, as I would never see the second one. All of
> > > > > > this happens before the serial console is activated, earlycon is not supported,
> > > > > > and I only have remote access.
> > > > > >
> > > > > > Brought up 2 CPUs
> > > > > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > > > > CPU: All CPU(s) started in SVC mode.
> > > > > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> > > > > > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
> > > > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > > > > [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
> > > > > > [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
> > > > > > [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
> > > > > > [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
> > > > > > (__do_tune_cpucache+0xf4/0x114)
> > > > > > [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
> > > > > > [<c01cff54>] (enable_cpucache) from [<c01d0190>]
> > > > > > (__kmem_cache_create+0x1a8/0x1d0)
> > > > > > [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
> > > > > > (kmem_cache_create+0xbc/0x190)
> > > > > > [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
> > > > > > [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
> > > > > > [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
> > > > > > [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
> > > > > > devtmpfs: initialized
> > > > >
> > > > > I don't see anything here that would prevent grace periods from completing.
> > > > >
> > > > > The CPUs are using the normal hotplug sequence to come online, correct?
> > > >
> > > > And either way, could you please apply the patch below and then
> > > > invoke rcu_dump_rcu_sched_tree() just before the offending call to
> > > > synchronize_sched()? That will tell me what CPUs RCU believes exist,
> > > > and perhaps also which CPU is holding it up.
> > >
> > > I can't find rcu_dump_rcu_sched_tree(). Do you mean
> > > rcu_dump_rcu_node_tree()? Anyway, there is no patch below so I attach
> > > one which does what Paul want, maybe.
> >
> > One of those days, I guess! :-/
> >
> > Your patch is exactly what I intended to send, thank you!
>
> Ah, but your telepathy was not sufficient to intuit the additional
> information I need. Please see the patch at the end. Your hunk
> in mm/slab.c is needed on top of my patch.
>
> So I am clearly having difficulties reading as well as including patches
> today...
Just following up, any news using my diagnostic patch?
Thanx, Paul
> > > Thanks.
> > >
> > > ------->8---------
> > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > index 88d3f95..6b650f0 100644
> > > --- a/kernel/rcu/tree.c
> > > +++ b/kernel/rcu/tree.c
> > > @@ -4171,7 +4171,7 @@ static void __init rcu_init_geometry(void)
> > > * Dump out the structure of the rcu_node combining tree associated
> > > * with the rcu_state structure referenced by rsp.
> > > */
> > > -static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > > +static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > > {
> > > int level = 0;
> > > struct rcu_node *rnp;
> > > @@ -4189,6 +4189,11 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > > pr_cont("\n");
> > > }
> > >
> > > +void rcu_dump_rcu_sched_tree(void)
> > > +{
> > > + rcu_dump_rcu_node_tree(&rcu_sched_state);
> > > +}
> > > +
> > > void __init rcu_init(void)
> > > {
> > > int cpu;
> > > diff --git a/mm/slab.c b/mm/slab.c
> > > index 763096a..d88976c 100644
> > > --- a/mm/slab.c
> > > +++ b/mm/slab.c
> > > @@ -909,6 +909,8 @@ static int init_cache_node_node(int node)
> > > return 0;
> > > }
> > >
> > > +extern void rcu_dump_rcu_sched_tree(void);
> > > +
> > > static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > int node, gfp_t gfp, bool force_change)
> > > {
> > > @@ -964,8 +966,10 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > * freed after synchronize_sched().
> > > */
> > > - if (force_change)
> > > + if (force_change) {
> > > + rcu_dump_rcu_sched_tree();
> > > synchronize_sched();
> > > + }
> > >
> > > fail:
> > > kfree(old_shared);
> > >
>
> ------------------------------------------------------------------------
>
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index c7f1bc4f817c..2eda7bece401 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4707,7 +4707,7 @@ static void __init rcu_init_geometry(void)
> * Dump out the structure of the rcu_node combining tree associated
> * with the rcu_state structure referenced by rsp.
> */
> -static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> +static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> {
> int level = 0;
> struct rcu_node *rnp;
> @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> pr_info(" ");
> level = rnp->level;
> }
> - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> + rnp->qsmask,
> + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> }
> pr_cont("\n");
> }
>
> +void rcu_dump_rcu_sched_tree(void)
> +{
> + rcu_dump_rcu_node_tree(&rcu_sched_state);
> +}
> +
> void __init rcu_init(void)
> {
> int cpu;
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-28 0:12 ` Paul E. McKenney
@ 2016-06-28 8:33 ` Joonsoo Kim
0 siblings, 0 replies; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-28 8:33 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Geert Uytterhoeven, Andrew Morton, Christoph Lameter,
Pekka Enberg, David Rientjes, Jesper Dangaard Brouer, Linux MM,
linux-kernel, linux-renesas-soc, linux-arm-kernel
On Mon, Jun 27, 2016 at 05:12:43PM -0700, Paul E. McKenney wrote:
> On Wed, Jun 22, 2016 at 07:53:29PM -0700, Paul E. McKenney wrote:
> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> > > On Thu, Jun 23, 2016 at 11:37:56AM +0900, Joonsoo Kim wrote:
> > > > On Wed, Jun 22, 2016 at 05:49:35PM -0700, Paul E. McKenney wrote:
> > > > > On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote:
> > > > > > On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
> > > > > > > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > > > > > > Could you try below patch to check who causes the hang?
> > > > > > > >
> > > > > > > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
> > > > > > > > used it before but Paul could find some culprit on it. :)
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > > ----->8-----
> > > > > > > > diff --git a/mm/slab.c b/mm/slab.c
> > > > > > > > index 763096a..9652d38 100644
> > > > > > > > --- a/mm/slab.c
> > > > > > > > +++ b/mm/slab.c
> > > > > > > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> > > > > > > > * guaranteed to be valid until irq is re-enabled, because it will be
> > > > > > > > * freed after synchronize_sched().
> > > > > > > > */
> > > > > > > > - if (force_change)
> > > > > > > > + if (force_change) {
> > > > > > > > + if (num_online_cpus() > 1)
> > > > > > > > + dump_stack();
> > > > > > > > synchronize_sched();
> > > > > > > > + if (num_online_cpus() > 1)
> > > > > > > > + dump_stack();
> > > > > > > > + }
> > > > > > >
> > > > > > > I've only added the first one, as I would never see the second one. All of
> > > > > > > this happens before the serial console is activated, earlycon is not supported,
> > > > > > > and I only have remote access.
> > > > > > >
> > > > > > > Brought up 2 CPUs
> > > > > > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > > > > > CPU: All CPU(s) started in SVC mode.
> > > > > > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> > > > > > > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
> > > > > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
> > > > > > > [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
> > > > > > > [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
> > > > > > > [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
> > > > > > > [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
> > > > > > > (__do_tune_cpucache+0xf4/0x114)
> > > > > > > [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
> > > > > > > [<c01cff54>] (enable_cpucache) from [<c01d0190>]
> > > > > > > (__kmem_cache_create+0x1a8/0x1d0)
> > > > > > > [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
> > > > > > > (kmem_cache_create+0xbc/0x190)
> > > > > > > [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
> > > > > > > [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
> > > > > > > [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
> > > > > > > [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
> > > > > > > devtmpfs: initialized
> > > > > >
> > > > > > I don't see anything here that would prevent grace periods from completing.
> > > > > >
> > > > > > The CPUs are using the normal hotplug sequence to come online, correct?
> > > > >
> > > > > And either way, could you please apply the patch below and then
> > > > > invoke rcu_dump_rcu_sched_tree() just before the offending call to
> > > > > synchronize_sched()? That will tell me what CPUs RCU believes exist,
> > > > > and perhaps also which CPU is holding it up.
> > > >
> > > > I can't find rcu_dump_rcu_sched_tree(). Do you mean
> > > > rcu_dump_rcu_node_tree()? Anyway, there is no patch below so I attach
> > > > one which does what Paul want, maybe.
> > >
> > > One of those days, I guess! :-/
> > >
> > > Your patch is exactly what I intended to send, thank you!
> >
> > Ah, but your telepathy was not sufficient to intuit the additional
> > information I need. Please see the patch at the end. Your hunk
> > in mm/slab.c is needed on top of my patch.
> >
> > So I am clearly having difficulties reading as well as including patches
> > today...
>
> Just following up, any news using my diagnostic patch?
Hello, Paul.
Unfortunately, I have no hardware to re-generate it, so we need to wait Geert's
feedback.
Thanks.
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-23 2:53 ` Paul E. McKenney
2016-06-28 0:12 ` Paul E. McKenney
@ 2016-06-29 14:54 ` Geert Uytterhoeven
2016-06-29 16:44 ` Paul E. McKenney
1 sibling, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-29 14:54 UTC (permalink / raw)
To: Paul McKenney
Cc: Joonsoo Kim, Andrew Morton, Christoph Lameter, Pekka Enberg,
David Rientjes, Jesper Dangaard Brouer, Linux MM, linux-kernel,
Linux-Renesas, linux-arm-kernel
Hi Paul,
On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
>> On Thu, Jun 23, 2016 at 11:37:56AM +0900, Joonsoo Kim wrote:
>> > On Wed, Jun 22, 2016 at 05:49:35PM -0700, Paul E. McKenney wrote:
>> > > On Wed, Jun 22, 2016 at 12:08:59PM -0700, Paul E. McKenney wrote:
>> > > > On Wed, Jun 22, 2016 at 05:01:35PM +0200, Geert Uytterhoeven wrote:
>> > > > > On Wed, Jun 22, 2016 at 2:52 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
>> > > > > > Could you try below patch to check who causes the hang?
>> > > > > >
>> > > > > > And, if sysalt-t works when hang, could you get sysalt-t output? I haven't
>> > > > > > used it before but Paul could find some culprit on it. :)
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > >
>> > > > > > ----->8-----
>> > > > > > diff --git a/mm/slab.c b/mm/slab.c
>> > > > > > index 763096a..9652d38 100644
>> > > > > > --- a/mm/slab.c
>> > > > > > +++ b/mm/slab.c
>> > > > > > @@ -964,8 +964,13 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
>> > > > > > * guaranteed to be valid until irq is re-enabled, because it will be
>> > > > > > * freed after synchronize_sched().
>> > > > > > */
>> > > > > > - if (force_change)
>> > > > > > + if (force_change) {
>> > > > > > + if (num_online_cpus() > 1)
>> > > > > > + dump_stack();
>> > > > > > synchronize_sched();
>> > > > > > + if (num_online_cpus() > 1)
>> > > > > > + dump_stack();
>> > > > > > + }
>> > > > >
>> > > > > I've only added the first one, as I would never see the second one. All of
>> > > > > this happens before the serial console is activated, earlycon is not supported,
>> > > > > and I only have remote access.
>> > > > >
>> > > > > Brought up 2 CPUs
>> > > > > SMP: Total of 2 processors activated (2132.00 BogoMIPS).
>> > > > > CPU: All CPU(s) started in SVC mode.
>> > > > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted
>> > > > > 4.7.0-rc4-kzm9d-00404-g4a235e6dde4404dd-dirty #89
>> > > > > Hardware name: Generic Emma Mobile EV2 (Flattened Device Tree)
>> > > > > [<c010de68>] (unwind_backtrace) from [<c010a658>] (show_stack+0x10/0x14)
>> > > > > [<c010a658>] (show_stack) from [<c02b5cf8>] (dump_stack+0x7c/0x9c)
>> > > > > [<c02b5cf8>] (dump_stack) from [<c01cfa4c>] (setup_kmem_cache_node+0x140/0x170)
>> > > > > [<c01cfa4c>] (setup_kmem_cache_node) from [<c01cfe3c>]
>> > > > > (__do_tune_cpucache+0xf4/0x114)
>> > > > > [<c01cfe3c>] (__do_tune_cpucache) from [<c01cff54>] (enable_cpucache+0xf8/0x148)
>> > > > > [<c01cff54>] (enable_cpucache) from [<c01d0190>]
>> > > > > (__kmem_cache_create+0x1a8/0x1d0)
>> > > > > [<c01d0190>] (__kmem_cache_create) from [<c01b32d0>]
>> > > > > (kmem_cache_create+0xbc/0x190)
>> > > > > [<c01b32d0>] (kmem_cache_create) from [<c070d968>] (shmem_init+0x34/0xb0)
>> > > > > [<c070d968>] (shmem_init) from [<c0700cc8>] (kernel_init_freeable+0x98/0x1ec)
>> > > > > [<c0700cc8>] (kernel_init_freeable) from [<c049fdbc>] (kernel_init+0x8/0x110)
>> > > > > [<c049fdbc>] (kernel_init) from [<c0106cb8>] (ret_from_fork+0x14/0x3c)
>> > > > > devtmpfs: initialized
>> > > >
>> > > > I don't see anything here that would prevent grace periods from completing.
>> > > >
>> > > > The CPUs are using the normal hotplug sequence to come online, correct?
>> > >
>> > > And either way, could you please apply the patch below and then
>> > > invoke rcu_dump_rcu_sched_tree() just before the offending call to
>> > > synchronize_sched()? That will tell me what CPUs RCU believes exist,
>> > > and perhaps also which CPU is holding it up.
>> >
>> > I can't find rcu_dump_rcu_sched_tree(). Do you mean
>> > rcu_dump_rcu_node_tree()? Anyway, there is no patch below so I attach
>> > one which does what Paul want, maybe.
>>
>> One of those days, I guess! :-/
>>
>> Your patch is exactly what I intended to send, thank you!
>
> Ah, but your telepathy was not sufficient to intuit the additional
> information I need. Please see the patch at the end. Your hunk
> in mm/slab.c is needed on top of my patch.
> @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> pr_info(" ");
> level = rnp->level;
> }
> - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> + rnp->qsmask,
> + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> }
> pr_cont("\n");
> }
For me it always crashes during the 37th call of synchronize_sched() in
setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
With your and my debug code, I get:
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x40100000 - 0x40100058
cnt = 36, sync
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Brought up 2 CPUs
SMP: Total of 2 processors activated (2132.00 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
rcu_node tree layout dump
0:1/0x0/0x3 ^0
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
max_idle_ns: 19112604462750000 ns
I hope it helps. Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-29 14:54 ` Geert Uytterhoeven
@ 2016-06-29 16:44 ` Paul E. McKenney
2016-06-29 17:52 ` Geert Uytterhoeven
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-29 16:44 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Joonsoo Kim, Andrew Morton, Christoph Lameter, Pekka Enberg,
David Rientjes, Jesper Dangaard Brouer, Linux MM, linux-kernel,
Linux-Renesas, linux-arm-kernel
On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
> Hi Paul,
>
> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
[ . . . ]
> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > pr_info(" ");
> > level = rnp->level;
> > }
> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> > + rnp->qsmask,
> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> > }
> > pr_cont("\n");
> > }
>
> For me it always crashes during the 37th call of synchronize_sched() in
> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
> With your and my debug code, I get:
>
> CPU: Testing write buffer coherency: ok
> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> Setting up static identity map for 0x40100000 - 0x40100058
> cnt = 36, sync
> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> Brought up 2 CPUs
> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> CPU: All CPU(s) started in SVC mode.
> rcu_node tree layout dump
> 0:1/0x0/0x3 ^0
Thank you for running this!
OK, so RCU knows about both CPUs (the "0x3"), and the previous
grace period has seen quiescent states from both of them (the "0x0").
That would indicate that your synchronize_sched() showed up when RCU was
idle, so it had to start a new grace period. It also rules out failure
modes where RCU thinks that there are more CPUs than really exist.
(Don't laugh, such things have really happened.)
> devtmpfs: initialized
> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> max_idle_ns: 19112604462750000 ns
>
> I hope it helps. Thanks!
I am going to guess that this was the first grace period since the second
CPU came online. When there only on CPU online, synchronize_sched()
is a no-op.
OK, this showed some things that aren't a problem. What might the
problem be?
o The grace-period kthread has not yet started. It -should- start
at early_initcall() time, but who knows? Adding code to print
out that kthread's task_struct address.
o The grace-period kthread might not be responding to wakeups.
Checking this requires that a grace period be in progress,
so please put a call_rcu_sched() just before the call to
rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
to my patch to print out more GP-kthread state as well.
o One of the CPUs might not be responding to RCU. That -should-
result in an RCU CPU stall warning, so I will ignore this
possibility for the moment.
That said, do you have some way to determine whether scheduling
clock interrupts are really happening? Without these interrupts,
no RCU CPU stall warnings.
OK, that should be enough for the next phase, please see the end for the
patch. This patch applies on top of my previous one.
Could you please set this up as follows?
struct rcu_head rh;
rcu_dump_rcu_node_tree(&rcu_sched_state); /* Initial state. */
call_rcu(&rh, do_nothing_cb);
schedule_timeout_uninterruptible(5 * HZ); /* Or whatever delay. */
rcu_dump_rcu_node_tree(&rcu_sched_state); /* GP state. */
synchronize_sched(); /* Probably hangs. */
rcu_barrier(); /* Drop RCU's references to rh before return. */
Thanx, Paul
------------------------------------------------------------------------
commit 82829ec76c2c0de18874a60ebd7ff8ee80f244d1
Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Date: Wed Jun 29 09:42:13 2016 -0700
rcu: More diagnostics
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 2eda7bece401..ff55c569473c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4712,6 +4712,11 @@ static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
int level = 0;
struct rcu_node *rnp;
+ pr_info("RCU: %s GP kthread: %p state: %d flags: %#x g:%ld c:%ld\n",
+ rsp->name, rsp->gp_kthread, rsp->gp_state, rsp->gp_flags,
+ (long)rsp->gpnum, (long)rsp->completed);
+ pr_info(" jiffies: %#lx GP start: %#lx Last GP activity: %#lx\n",
+ jiffies, rsp->gp_start, rsp->gp_activity);
pr_info("rcu_node tree layout dump\n");
pr_info(" ");
rcu_for_each_node_breadth_first(rsp, rnp) {
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-29 16:44 ` Paul E. McKenney
@ 2016-06-29 17:52 ` Geert Uytterhoeven
2016-06-29 18:12 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-29 17:52 UTC (permalink / raw)
To: Paul McKenney
Cc: Linux-Renesas, Andrew Morton, David Rientjes, linux-kernel,
Pekka Enberg, Linux MM, Jesper Dangaard Brouer, Joonsoo Kim,
Christoph Lameter, linux-arm-kernel
Hi Paul,
On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
>> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
>
> [ . . . ]
>
>> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
>> > pr_info(" ");
>> > level = rnp->level;
>> > }
>> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
>> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
>> > + rnp->qsmask,
>> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
>> > }
>> > pr_cont("\n");
>> > }
>>
>> For me it always crashes during the 37th call of synchronize_sched() in
>> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
>> With your and my debug code, I get:
>>
>> CPU: Testing write buffer coherency: ok
>> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
>> Setting up static identity map for 0x40100000 - 0x40100058
>> cnt = 36, sync
>> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
>> Brought up 2 CPUs
>> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
>> CPU: All CPU(s) started in SVC mode.
>> rcu_node tree layout dump
>> 0:1/0x0/0x3 ^0
>
> Thank you for running this!
>
> OK, so RCU knows about both CPUs (the "0x3"), and the previous
> grace period has seen quiescent states from both of them (the "0x0").
> That would indicate that your synchronize_sched() showed up when RCU was
> idle, so it had to start a new grace period. It also rules out failure
> modes where RCU thinks that there are more CPUs than really exist.
> (Don't laugh, such things have really happened.)
>
>> devtmpfs: initialized
>> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
>> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
>> max_idle_ns: 19112604462750000 ns
>>
>> I hope it helps. Thanks!
>
> I am going to guess that this was the first grace period since the second
> CPU came online. When there only on CPU online, synchronize_sched()
> is a no-op.
>
> OK, this showed some things that aren't a problem. What might the
> problem be?
>
> o The grace-period kthread has not yet started. It -should- start
> at early_initcall() time, but who knows? Adding code to print
> out that kthread's task_struct address.
>
> o The grace-period kthread might not be responding to wakeups.
> Checking this requires that a grace period be in progress,
> so please put a call_rcu_sched() just before the call to
> rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
> to my patch to print out more GP-kthread state as well.
>
> o One of the CPUs might not be responding to RCU. That -should-
> result in an RCU CPU stall warning, so I will ignore this
> possibility for the moment.
>
> That said, do you have some way to determine whether scheduling
> clock interrupts are really happening? Without these interrupts,
> no RCU CPU stall warnings.
I believe there are no clocksources yet. The jiffies clocksource is the first
clocksource found, and that happens after the first call to
synchronize_sched(), cfr. my dmesg snippet above.
In a working boot:
# cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
e0180000.timer jiffies
# cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
e0180000.timer
> OK, that should be enough for the next phase, please see the end for the
> patch. This patch applies on top of my previous one.
>
> Could you please set this up as follows?
>
> struct rcu_head rh;
>
> rcu_dump_rcu_node_tree(&rcu_sched_state); /* Initial state. */
> call_rcu(&rh, do_nothing_cb);
I added an empty do_nothing_cb() for this:
static void do_nothing_cb(struct rcu_head *rcu_head)
{
}
According to the debugging technique "comment everything out until it boots",
it now hangs in call_rcu().
> schedule_timeout_uninterruptible(5 * HZ); /* Or whatever delay. */
> rcu_dump_rcu_node_tree(&rcu_sched_state); /* GP state. */
> synchronize_sched(); /* Probably hangs. */
> rcu_barrier(); /* Drop RCU's references to rh before return. */
Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-29 17:52 ` Geert Uytterhoeven
@ 2016-06-29 18:12 ` Paul E. McKenney
2016-06-30 7:47 ` Joonsoo Kim
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-29 18:12 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Linux-Renesas, Andrew Morton, David Rientjes, linux-kernel,
Pekka Enberg, Linux MM, Jesper Dangaard Brouer, Joonsoo Kim,
Christoph Lameter, linux-arm-kernel
On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
> Hi Paul,
>
> On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
> >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> >
> > [ . . . ]
> >
> >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> >> > pr_info(" ");
> >> > level = rnp->level;
> >> > }
> >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> >> > + rnp->qsmask,
> >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> >> > }
> >> > pr_cont("\n");
> >> > }
> >>
> >> For me it always crashes during the 37th call of synchronize_sched() in
> >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
> >> With your and my debug code, I get:
> >>
> >> CPU: Testing write buffer coherency: ok
> >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> >> Setting up static identity map for 0x40100000 - 0x40100058
> >> cnt = 36, sync
> >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> >> Brought up 2 CPUs
> >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> >> CPU: All CPU(s) started in SVC mode.
> >> rcu_node tree layout dump
> >> 0:1/0x0/0x3 ^0
> >
> > Thank you for running this!
> >
> > OK, so RCU knows about both CPUs (the "0x3"), and the previous
> > grace period has seen quiescent states from both of them (the "0x0").
> > That would indicate that your synchronize_sched() showed up when RCU was
> > idle, so it had to start a new grace period. It also rules out failure
> > modes where RCU thinks that there are more CPUs than really exist.
> > (Don't laugh, such things have really happened.)
> >
> >> devtmpfs: initialized
> >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> >> max_idle_ns: 19112604462750000 ns
> >>
> >> I hope it helps. Thanks!
> >
> > I am going to guess that this was the first grace period since the second
> > CPU came online. When there only on CPU online, synchronize_sched()
> > is a no-op.
> >
> > OK, this showed some things that aren't a problem. What might the
> > problem be?
> >
> > o The grace-period kthread has not yet started. It -should- start
> > at early_initcall() time, but who knows? Adding code to print
> > out that kthread's task_struct address.
> >
> > o The grace-period kthread might not be responding to wakeups.
> > Checking this requires that a grace period be in progress,
> > so please put a call_rcu_sched() just before the call to
> > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
> > to my patch to print out more GP-kthread state as well.
> >
> > o One of the CPUs might not be responding to RCU. That -should-
> > result in an RCU CPU stall warning, so I will ignore this
> > possibility for the moment.
> >
> > That said, do you have some way to determine whether scheduling
> > clock interrupts are really happening? Without these interrupts,
> > no RCU CPU stall warnings.
>
> I believe there are no clocksources yet. The jiffies clocksource is the first
> clocksource found, and that happens after the first call to
> synchronize_sched(), cfr. my dmesg snippet above.
>
> In a working boot:
> # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
> e0180000.timer jiffies
> # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
> e0180000.timer
Ah! But if there is no jiffies clocksource, then schedule_timeout()
and friends will never return, correct? If so, I guarantee you that
synchronize_sched() will unconditionally hang.
So if I understand correctly, the fix is to get the jiffies clocksource
running before the first call to synchronize_sched().
Thanx, Paul
> > OK, that should be enough for the next phase, please see the end for the
> > patch. This patch applies on top of my previous one.
> >
> > Could you please set this up as follows?
> >
> > struct rcu_head rh;
> >
> > rcu_dump_rcu_node_tree(&rcu_sched_state); /* Initial state. */
> > call_rcu(&rh, do_nothing_cb);
>
> I added an empty do_nothing_cb() for this:
>
> static void do_nothing_cb(struct rcu_head *rcu_head)
> {
> }
>
> According to the debugging technique "comment everything out until it boots",
> it now hangs in call_rcu().
>
> > schedule_timeout_uninterruptible(5 * HZ); /* Or whatever delay. */
> > rcu_dump_rcu_node_tree(&rcu_sched_state); /* GP state. */
> > synchronize_sched(); /* Probably hangs. */
> > rcu_barrier(); /* Drop RCU's references to rh before return. */
>
> Thanks!
>
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> -- Linus Torvalds
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-29 18:12 ` Paul E. McKenney
@ 2016-06-30 7:47 ` Joonsoo Kim
2016-06-30 7:58 ` Geert Uytterhoeven
0 siblings, 1 reply; 34+ messages in thread
From: Joonsoo Kim @ 2016-06-30 7:47 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Geert Uytterhoeven, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
On Wed, Jun 29, 2016 at 11:12:08AM -0700, Paul E. McKenney wrote:
> On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
> > Hi Paul,
> >
> > On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
> > <paulmck@linux.vnet.ibm.com> wrote:
> > > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
> > >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
> > >> <paulmck@linux.vnet.ibm.com> wrote:
> > >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> > >
> > > [ . . . ]
> > >
> > >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > >> > pr_info(" ");
> > >> > level = rnp->level;
> > >> > }
> > >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> > >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> > >> > + rnp->qsmask,
> > >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> > >> > }
> > >> > pr_cont("\n");
> > >> > }
> > >>
> > >> For me it always crashes during the 37th call of synchronize_sched() in
> > >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
> > >> With your and my debug code, I get:
> > >>
> > >> CPU: Testing write buffer coherency: ok
> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > >> Setting up static identity map for 0x40100000 - 0x40100058
> > >> cnt = 36, sync
> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > >> Brought up 2 CPUs
> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > >> CPU: All CPU(s) started in SVC mode.
> > >> rcu_node tree layout dump
> > >> 0:1/0x0/0x3 ^0
> > >
> > > Thank you for running this!
> > >
> > > OK, so RCU knows about both CPUs (the "0x3"), and the previous
> > > grace period has seen quiescent states from both of them (the "0x0").
> > > That would indicate that your synchronize_sched() showed up when RCU was
> > > idle, so it had to start a new grace period. It also rules out failure
> > > modes where RCU thinks that there are more CPUs than really exist.
> > > (Don't laugh, such things have really happened.)
> > >
> > >> devtmpfs: initialized
> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > >> max_idle_ns: 19112604462750000 ns
> > >>
> > >> I hope it helps. Thanks!
> > >
> > > I am going to guess that this was the first grace period since the second
> > > CPU came online. When there only on CPU online, synchronize_sched()
> > > is a no-op.
> > >
> > > OK, this showed some things that aren't a problem. What might the
> > > problem be?
> > >
> > > o The grace-period kthread has not yet started. It -should- start
> > > at early_initcall() time, but who knows? Adding code to print
> > > out that kthread's task_struct address.
> > >
> > > o The grace-period kthread might not be responding to wakeups.
> > > Checking this requires that a grace period be in progress,
> > > so please put a call_rcu_sched() just before the call to
> > > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
> > > to my patch to print out more GP-kthread state as well.
> > >
> > > o One of the CPUs might not be responding to RCU. That -should-
> > > result in an RCU CPU stall warning, so I will ignore this
> > > possibility for the moment.
> > >
> > > That said, do you have some way to determine whether scheduling
> > > clock interrupts are really happening? Without these interrupts,
> > > no RCU CPU stall warnings.
> >
> > I believe there are no clocksources yet. The jiffies clocksource is the first
> > clocksource found, and that happens after the first call to
> > synchronize_sched(), cfr. my dmesg snippet above.
> >
> > In a working boot:
> > # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
> > e0180000.timer jiffies
> > # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
> > e0180000.timer
>
> Ah! But if there is no jiffies clocksource, then schedule_timeout()
> and friends will never return, correct? If so, I guarantee you that
> synchronize_sched() will unconditionally hang.
>
> So if I understand correctly, the fix is to get the jiffies clocksource
> running before the first call to synchronize_sched().
If so, following change would be sufficient.
Thanks.
------>8-------
diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
index 555e21f..4f6471f 100644
--- a/kernel/time/jiffies.c
+++ b/kernel/time/jiffies.c
@@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
return __clocksource_register(&clocksource_jiffies);
}
-core_initcall(init_jiffies_clocksource);
+early_initcall(init_jiffies_clocksource);
struct clocksource * __init __weak clocksource_default_clock(void)
{
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-30 7:47 ` Joonsoo Kim
@ 2016-06-30 7:58 ` Geert Uytterhoeven
2016-06-30 13:24 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-30 7:58 UTC (permalink / raw)
To: Joonsoo Kim
Cc: Paul E. McKenney, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
Hi Joonsoo,
On Thu, Jun 30, 2016 at 9:47 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> On Wed, Jun 29, 2016 at 11:12:08AM -0700, Paul E. McKenney wrote:
>> On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
>> > On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
>> > <paulmck@linux.vnet.ibm.com> wrote:
>> > > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
>> > >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
>> > >> <paulmck@linux.vnet.ibm.com> wrote:
>> > >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
>> > >
>> > > [ . . . ]
>> > >
>> > >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
>> > >> > pr_info(" ");
>> > >> > level = rnp->level;
>> > >> > }
>> > >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
>> > >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
>> > >> > + rnp->qsmask,
>> > >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
>> > >> > }
>> > >> > pr_cont("\n");
>> > >> > }
>> > >>
>> > >> For me it always crashes during the 37th call of synchronize_sched() in
>> > >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
>> > >> With your and my debug code, I get:
>> > >>
>> > >> CPU: Testing write buffer coherency: ok
>> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
>> > >> Setting up static identity map for 0x40100000 - 0x40100058
>> > >> cnt = 36, sync
>> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
>> > >> Brought up 2 CPUs
>> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
>> > >> CPU: All CPU(s) started in SVC mode.
>> > >> rcu_node tree layout dump
>> > >> 0:1/0x0/0x3 ^0
>> > >
>> > > Thank you for running this!
>> > >
>> > > OK, so RCU knows about both CPUs (the "0x3"), and the previous
>> > > grace period has seen quiescent states from both of them (the "0x0").
>> > > That would indicate that your synchronize_sched() showed up when RCU was
>> > > idle, so it had to start a new grace period. It also rules out failure
>> > > modes where RCU thinks that there are more CPUs than really exist.
>> > > (Don't laugh, such things have really happened.)
>> > >
>> > >> devtmpfs: initialized
>> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
>> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
>> > >> max_idle_ns: 19112604462750000 ns
>> > >>
>> > >> I hope it helps. Thanks!
>> > >
>> > > I am going to guess that this was the first grace period since the second
>> > > CPU came online. When there only on CPU online, synchronize_sched()
>> > > is a no-op.
>> > >
>> > > OK, this showed some things that aren't a problem. What might the
>> > > problem be?
>> > >
>> > > o The grace-period kthread has not yet started. It -should- start
>> > > at early_initcall() time, but who knows? Adding code to print
>> > > out that kthread's task_struct address.
>> > >
>> > > o The grace-period kthread might not be responding to wakeups.
>> > > Checking this requires that a grace period be in progress,
>> > > so please put a call_rcu_sched() just before the call to
>> > > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
>> > > to my patch to print out more GP-kthread state as well.
>> > >
>> > > o One of the CPUs might not be responding to RCU. That -should-
>> > > result in an RCU CPU stall warning, so I will ignore this
>> > > possibility for the moment.
>> > >
>> > > That said, do you have some way to determine whether scheduling
>> > > clock interrupts are really happening? Without these interrupts,
>> > > no RCU CPU stall warnings.
>> >
>> > I believe there are no clocksources yet. The jiffies clocksource is the first
>> > clocksource found, and that happens after the first call to
>> > synchronize_sched(), cfr. my dmesg snippet above.
>> >
>> > In a working boot:
>> > # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
>> > e0180000.timer jiffies
>> > # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
>> > e0180000.timer
>>
>> Ah! But if there is no jiffies clocksource, then schedule_timeout()
>> and friends will never return, correct? If so, I guarantee you that
>> synchronize_sched() will unconditionally hang.
>>
>> So if I understand correctly, the fix is to get the jiffies clocksource
>> running before the first call to synchronize_sched().
>
> If so, following change would be sufficient.
>
> Thanks.
>
> ------>8-------
> diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
> index 555e21f..4f6471f 100644
> --- a/kernel/time/jiffies.c
> +++ b/kernel/time/jiffies.c
> @@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
> return __clocksource_register(&clocksource_jiffies);
> }
>
> -core_initcall(init_jiffies_clocksource);
> +early_initcall(init_jiffies_clocksource);
>
> struct clocksource * __init __weak clocksource_default_clock(void)
> {
Thanks for your patch!
While this does move jiffies clocksource initialization before secondary CPU
bringup, it still hangs when calling call_rcu() or synchronize_sched():
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x40100000 - 0x40100058
cnt = 36, sync
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
max_idle_ns: 19112604462750000 ns
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Brought up 2 CPUs
SMP: Total of 2 processors activated (2132.00 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
rcu_node tree layout dump
0:1/0x0/0x3 ^0
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-30 7:58 ` Geert Uytterhoeven
@ 2016-06-30 13:24 ` Paul E. McKenney
2016-06-30 13:31 ` Geert Uytterhoeven
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-30 13:24 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Joonsoo Kim, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
On Thu, Jun 30, 2016 at 09:58:51AM +0200, Geert Uytterhoeven wrote:
> Hi Joonsoo,
>
> On Thu, Jun 30, 2016 at 9:47 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > On Wed, Jun 29, 2016 at 11:12:08AM -0700, Paul E. McKenney wrote:
> >> On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
> >> > On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> > > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
> >> > >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
> >> > >> <paulmck@linux.vnet.ibm.com> wrote:
> >> > >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> >> > >
> >> > > [ . . . ]
> >> > >
> >> > >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> >> > >> > pr_info(" ");
> >> > >> > level = rnp->level;
> >> > >> > }
> >> > >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> >> > >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> >> > >> > + rnp->qsmask,
> >> > >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> >> > >> > }
> >> > >> > pr_cont("\n");
> >> > >> > }
> >> > >>
> >> > >> For me it always crashes during the 37th call of synchronize_sched() in
> >> > >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
> >> > >> With your and my debug code, I get:
> >> > >>
> >> > >> CPU: Testing write buffer coherency: ok
> >> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> >> > >> Setting up static identity map for 0x40100000 - 0x40100058
> >> > >> cnt = 36, sync
> >> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> >> > >> Brought up 2 CPUs
> >> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> >> > >> CPU: All CPU(s) started in SVC mode.
> >> > >> rcu_node tree layout dump
> >> > >> 0:1/0x0/0x3 ^0
> >> > >
> >> > > Thank you for running this!
> >> > >
> >> > > OK, so RCU knows about both CPUs (the "0x3"), and the previous
> >> > > grace period has seen quiescent states from both of them (the "0x0").
> >> > > That would indicate that your synchronize_sched() showed up when RCU was
> >> > > idle, so it had to start a new grace period. It also rules out failure
> >> > > modes where RCU thinks that there are more CPUs than really exist.
> >> > > (Don't laugh, such things have really happened.)
> >> > >
> >> > >> devtmpfs: initialized
> >> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> >> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> >> > >> max_idle_ns: 19112604462750000 ns
> >> > >>
> >> > >> I hope it helps. Thanks!
> >> > >
> >> > > I am going to guess that this was the first grace period since the second
> >> > > CPU came online. When there only on CPU online, synchronize_sched()
> >> > > is a no-op.
> >> > >
> >> > > OK, this showed some things that aren't a problem. What might the
> >> > > problem be?
> >> > >
> >> > > o The grace-period kthread has not yet started. It -should- start
> >> > > at early_initcall() time, but who knows? Adding code to print
> >> > > out that kthread's task_struct address.
> >> > >
> >> > > o The grace-period kthread might not be responding to wakeups.
> >> > > Checking this requires that a grace period be in progress,
> >> > > so please put a call_rcu_sched() just before the call to
> >> > > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
> >> > > to my patch to print out more GP-kthread state as well.
> >> > >
> >> > > o One of the CPUs might not be responding to RCU. That -should-
> >> > > result in an RCU CPU stall warning, so I will ignore this
> >> > > possibility for the moment.
> >> > >
> >> > > That said, do you have some way to determine whether scheduling
> >> > > clock interrupts are really happening? Without these interrupts,
> >> > > no RCU CPU stall warnings.
> >> >
> >> > I believe there are no clocksources yet. The jiffies clocksource is the first
> >> > clocksource found, and that happens after the first call to
> >> > synchronize_sched(), cfr. my dmesg snippet above.
> >> >
> >> > In a working boot:
> >> > # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
> >> > e0180000.timer jiffies
> >> > # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
> >> > e0180000.timer
> >>
> >> Ah! But if there is no jiffies clocksource, then schedule_timeout()
> >> and friends will never return, correct? If so, I guarantee you that
> >> synchronize_sched() will unconditionally hang.
> >>
> >> So if I understand correctly, the fix is to get the jiffies clocksource
> >> running before the first call to synchronize_sched().
> >
> > If so, following change would be sufficient.
> >
> > Thanks.
> >
> > ------>8-------
> > diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
> > index 555e21f..4f6471f 100644
> > --- a/kernel/time/jiffies.c
> > +++ b/kernel/time/jiffies.c
> > @@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
> > return __clocksource_register(&clocksource_jiffies);
> > }
> >
> > -core_initcall(init_jiffies_clocksource);
> > +early_initcall(init_jiffies_clocksource);
> >
> > struct clocksource * __init __weak clocksource_default_clock(void)
> > {
>
> Thanks for your patch!
>
> While this does move jiffies clocksource initialization before secondary CPU
> bringup, it still hangs when calling call_rcu() or synchronize_sched():
>
> CPU: Testing write buffer coherency: ok
> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> Setting up static identity map for 0x40100000 - 0x40100058
> cnt = 36, sync
> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> max_idle_ns: 19112604462750000 ns
> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> Brought up 2 CPUs
> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> CPU: All CPU(s) started in SVC mode.
> RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
> jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
> rcu_node tree layout dump
> 0:1/0x0/0x3 ^0
This is in fact the initial state for RCU grace periods. In other words,
all the earlier calls to synchronize_sched() likely happened while there
was only one CPU online.
> devtmpfs: initialized
> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
Could you please add the call_rcu() and timed delay as described in my
earlier email? That would hopefully help me see the state of the stalled
grace period.
Thanx, Paul
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> -- Linus Torvalds
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-30 13:24 ` Paul E. McKenney
@ 2016-06-30 13:31 ` Geert Uytterhoeven
2016-06-30 15:18 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-30 13:31 UTC (permalink / raw)
To: Paul McKenney
Cc: Joonsoo Kim, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
Hi Paul,
On Thu, Jun 30, 2016 at 3:24 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Thu, Jun 30, 2016 at 09:58:51AM +0200, Geert Uytterhoeven wrote:
>> On Thu, Jun 30, 2016 at 9:47 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
>> > On Wed, Jun 29, 2016 at 11:12:08AM -0700, Paul E. McKenney wrote:
>> >> On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
>> >> > On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
>> >> > <paulmck@linux.vnet.ibm.com> wrote:
>> >> > > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
>> >> > >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
>> >> > >> <paulmck@linux.vnet.ibm.com> wrote:
>> >> > >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
>> >> > >
>> >> > > [ . . . ]
>> >> > >
>> >> > >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
>> >> > >> > pr_info(" ");
>> >> > >> > level = rnp->level;
>> >> > >> > }
>> >> > >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
>> >> > >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
>> >> > >> > + rnp->qsmask,
>> >> > >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
>> >> > >> > }
>> >> > >> > pr_cont("\n");
>> >> > >> > }
>> >> > >>
>> >> > >> For me it always crashes during the 37th call of synchronize_sched() in
>> >> > >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
>> >> > >> With your and my debug code, I get:
>> >> > >>
>> >> > >> CPU: Testing write buffer coherency: ok
>> >> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
>> >> > >> Setting up static identity map for 0x40100000 - 0x40100058
>> >> > >> cnt = 36, sync
>> >> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
>> >> > >> Brought up 2 CPUs
>> >> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
>> >> > >> CPU: All CPU(s) started in SVC mode.
>> >> > >> rcu_node tree layout dump
>> >> > >> 0:1/0x0/0x3 ^0
>> >> > >
>> >> > > Thank you for running this!
>> >> > >
>> >> > > OK, so RCU knows about both CPUs (the "0x3"), and the previous
>> >> > > grace period has seen quiescent states from both of them (the "0x0").
>> >> > > That would indicate that your synchronize_sched() showed up when RCU was
>> >> > > idle, so it had to start a new grace period. It also rules out failure
>> >> > > modes where RCU thinks that there are more CPUs than really exist.
>> >> > > (Don't laugh, such things have really happened.)
>> >> > >
>> >> > >> devtmpfs: initialized
>> >> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
>> >> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
>> >> > >> max_idle_ns: 19112604462750000 ns
>> >> > >>
>> >> > >> I hope it helps. Thanks!
>> >> > >
>> >> > > I am going to guess that this was the first grace period since the second
>> >> > > CPU came online. When there only on CPU online, synchronize_sched()
>> >> > > is a no-op.
>> >> > >
>> >> > > OK, this showed some things that aren't a problem. What might the
>> >> > > problem be?
>> >> > >
>> >> > > o The grace-period kthread has not yet started. It -should- start
>> >> > > at early_initcall() time, but who knows? Adding code to print
>> >> > > out that kthread's task_struct address.
>> >> > >
>> >> > > o The grace-period kthread might not be responding to wakeups.
>> >> > > Checking this requires that a grace period be in progress,
>> >> > > so please put a call_rcu_sched() just before the call to
>> >> > > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
>> >> > > to my patch to print out more GP-kthread state as well.
>> >> > >
>> >> > > o One of the CPUs might not be responding to RCU. That -should-
>> >> > > result in an RCU CPU stall warning, so I will ignore this
>> >> > > possibility for the moment.
>> >> > >
>> >> > > That said, do you have some way to determine whether scheduling
>> >> > > clock interrupts are really happening? Without these interrupts,
>> >> > > no RCU CPU stall warnings.
>> >> >
>> >> > I believe there are no clocksources yet. The jiffies clocksource is the first
>> >> > clocksource found, and that happens after the first call to
>> >> > synchronize_sched(), cfr. my dmesg snippet above.
>> >> >
>> >> > In a working boot:
>> >> > # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
>> >> > e0180000.timer jiffies
>> >> > # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
>> >> > e0180000.timer
>> >>
>> >> Ah! But if there is no jiffies clocksource, then schedule_timeout()
>> >> and friends will never return, correct? If so, I guarantee you that
>> >> synchronize_sched() will unconditionally hang.
>> >>
>> >> So if I understand correctly, the fix is to get the jiffies clocksource
>> >> running before the first call to synchronize_sched().
>> >
>> > If so, following change would be sufficient.
>> >
>> > Thanks.
>> >
>> > ------>8-------
>> > diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
>> > index 555e21f..4f6471f 100644
>> > --- a/kernel/time/jiffies.c
>> > +++ b/kernel/time/jiffies.c
>> > @@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
>> > return __clocksource_register(&clocksource_jiffies);
>> > }
>> >
>> > -core_initcall(init_jiffies_clocksource);
>> > +early_initcall(init_jiffies_clocksource);
>> >
>> > struct clocksource * __init __weak clocksource_default_clock(void)
>> > {
>>
>> Thanks for your patch!
>>
>> While this does move jiffies clocksource initialization before secondary CPU
>> bringup, it still hangs when calling call_rcu() or synchronize_sched():
>>
>> CPU: Testing write buffer coherency: ok
>> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
>> Setting up static identity map for 0x40100000 - 0x40100058
>> cnt = 36, sync
>> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
>> max_idle_ns: 19112604462750000 ns
>> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
>> Brought up 2 CPUs
>> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
>> CPU: All CPU(s) started in SVC mode.
>> RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
>> jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
>> rcu_node tree layout dump
>> 0:1/0x0/0x3 ^0
>
> This is in fact the initial state for RCU grace periods. In other words,
> all the earlier calls to synchronize_sched() likely happened while there
> was only one CPU online.
>
>> devtmpfs: initialized
>> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
>
> Could you please add the call_rcu() and timed delay as described in my
> earlier email? That would hopefully help me see the state of the stalled
> grace period.
I already did, cfr. "it still hangs when calling call_rcu() or
synchronize_sched()".
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-30 13:31 ` Geert Uytterhoeven
@ 2016-06-30 15:18 ` Paul E. McKenney
2016-06-30 15:53 ` Geert Uytterhoeven
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-30 15:18 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Joonsoo Kim, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
On Thu, Jun 30, 2016 at 03:31:57PM +0200, Geert Uytterhoeven wrote:
> Hi Paul,
>
> On Thu, Jun 30, 2016 at 3:24 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Thu, Jun 30, 2016 at 09:58:51AM +0200, Geert Uytterhoeven wrote:
> >> On Thu, Jun 30, 2016 at 9:47 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> >> > On Wed, Jun 29, 2016 at 11:12:08AM -0700, Paul E. McKenney wrote:
> >> >> On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
> >> >> > On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
> >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> >> >> > > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
> >> >> > >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
> >> >> > >> <paulmck@linux.vnet.ibm.com> wrote:
> >> >> > >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> >> >> > >
> >> >> > > [ . . . ]
> >> >> > >
> >> >> > >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> >> >> > >> > pr_info(" ");
> >> >> > >> > level = rnp->level;
> >> >> > >> > }
> >> >> > >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> >> >> > >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> >> >> > >> > + rnp->qsmask,
> >> >> > >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> >> >> > >> > }
> >> >> > >> > pr_cont("\n");
> >> >> > >> > }
> >> >> > >>
> >> >> > >> For me it always crashes during the 37th call of synchronize_sched() in
> >> >> > >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
> >> >> > >> With your and my debug code, I get:
> >> >> > >>
> >> >> > >> CPU: Testing write buffer coherency: ok
> >> >> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> >> >> > >> Setting up static identity map for 0x40100000 - 0x40100058
> >> >> > >> cnt = 36, sync
> >> >> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> >> >> > >> Brought up 2 CPUs
> >> >> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> >> >> > >> CPU: All CPU(s) started in SVC mode.
> >> >> > >> rcu_node tree layout dump
> >> >> > >> 0:1/0x0/0x3 ^0
> >> >> > >
> >> >> > > Thank you for running this!
> >> >> > >
> >> >> > > OK, so RCU knows about both CPUs (the "0x3"), and the previous
> >> >> > > grace period has seen quiescent states from both of them (the "0x0").
> >> >> > > That would indicate that your synchronize_sched() showed up when RCU was
> >> >> > > idle, so it had to start a new grace period. It also rules out failure
> >> >> > > modes where RCU thinks that there are more CPUs than really exist.
> >> >> > > (Don't laugh, such things have really happened.)
> >> >> > >
> >> >> > >> devtmpfs: initialized
> >> >> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> >> >> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> >> >> > >> max_idle_ns: 19112604462750000 ns
> >> >> > >>
> >> >> > >> I hope it helps. Thanks!
> >> >> > >
> >> >> > > I am going to guess that this was the first grace period since the second
> >> >> > > CPU came online. When there only on CPU online, synchronize_sched()
> >> >> > > is a no-op.
> >> >> > >
> >> >> > > OK, this showed some things that aren't a problem. What might the
> >> >> > > problem be?
> >> >> > >
> >> >> > > o The grace-period kthread has not yet started. It -should- start
> >> >> > > at early_initcall() time, but who knows? Adding code to print
> >> >> > > out that kthread's task_struct address.
> >> >> > >
> >> >> > > o The grace-period kthread might not be responding to wakeups.
> >> >> > > Checking this requires that a grace period be in progress,
> >> >> > > so please put a call_rcu_sched() just before the call to
> >> >> > > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
> >> >> > > to my patch to print out more GP-kthread state as well.
> >> >> > >
> >> >> > > o One of the CPUs might not be responding to RCU. That -should-
> >> >> > > result in an RCU CPU stall warning, so I will ignore this
> >> >> > > possibility for the moment.
> >> >> > >
> >> >> > > That said, do you have some way to determine whether scheduling
> >> >> > > clock interrupts are really happening? Without these interrupts,
> >> >> > > no RCU CPU stall warnings.
> >> >> >
> >> >> > I believe there are no clocksources yet. The jiffies clocksource is the first
> >> >> > clocksource found, and that happens after the first call to
> >> >> > synchronize_sched(), cfr. my dmesg snippet above.
> >> >> >
> >> >> > In a working boot:
> >> >> > # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
> >> >> > e0180000.timer jiffies
> >> >> > # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
> >> >> > e0180000.timer
> >> >>
> >> >> Ah! But if there is no jiffies clocksource, then schedule_timeout()
> >> >> and friends will never return, correct? If so, I guarantee you that
> >> >> synchronize_sched() will unconditionally hang.
> >> >>
> >> >> So if I understand correctly, the fix is to get the jiffies clocksource
> >> >> running before the first call to synchronize_sched().
> >> >
> >> > If so, following change would be sufficient.
> >> >
> >> > Thanks.
> >> >
> >> > ------>8-------
> >> > diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
> >> > index 555e21f..4f6471f 100644
> >> > --- a/kernel/time/jiffies.c
> >> > +++ b/kernel/time/jiffies.c
> >> > @@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
> >> > return __clocksource_register(&clocksource_jiffies);
> >> > }
> >> >
> >> > -core_initcall(init_jiffies_clocksource);
> >> > +early_initcall(init_jiffies_clocksource);
> >> >
> >> > struct clocksource * __init __weak clocksource_default_clock(void)
> >> > {
> >>
> >> Thanks for your patch!
> >>
> >> While this does move jiffies clocksource initialization before secondary CPU
> >> bringup, it still hangs when calling call_rcu() or synchronize_sched():
> >>
> >> CPU: Testing write buffer coherency: ok
> >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> >> Setting up static identity map for 0x40100000 - 0x40100058
> >> cnt = 36, sync
> >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> >> max_idle_ns: 19112604462750000 ns
> >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> >> Brought up 2 CPUs
> >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> >> CPU: All CPU(s) started in SVC mode.
> >> RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
> >> jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
> >> rcu_node tree layout dump
> >> 0:1/0x0/0x3 ^0
> >
> > This is in fact the initial state for RCU grace periods. In other words,
> > all the earlier calls to synchronize_sched() likely happened while there
> > was only one CPU online.
> >
> >> devtmpfs: initialized
> >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> >
> > Could you please add the call_rcu() and timed delay as described in my
> > earlier email? That would hopefully help me see the state of the stalled
> > grace period.
>
> I already did, cfr. "it still hangs when calling call_rcu() or
> synchronize_sched()".
Ah, sorry for my inattention.
I am a bit surprised that it could hang when calling call_rcu(), given
that call_rcu() is callable from atomic contexts. Could you please show
me the current test code you have?
If the hang is in call_rcu(), could you please try disabling irqs across
the call to call_rcu()?
Thanx, Paul
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-30 15:18 ` Paul E. McKenney
@ 2016-06-30 15:53 ` Geert Uytterhoeven
2016-06-30 16:52 ` Paul E. McKenney
0 siblings, 1 reply; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-30 15:53 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Joonsoo Kim, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
On Thu, 30 Jun 2016, Paul E. McKenney wrote:
> On Thu, Jun 30, 2016 at 03:31:57PM +0200, Geert Uytterhoeven wrote:
> > On Thu, Jun 30, 2016 at 3:24 PM, Paul E. McKenney
> > <paulmck@linux.vnet.ibm.com> wrote:
> > > On Thu, Jun 30, 2016 at 09:58:51AM +0200, Geert Uytterhoeven wrote:
> > >> On Thu, Jun 30, 2016 at 9:47 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > >> > On Wed, Jun 29, 2016 at 11:12:08AM -0700, Paul E. McKenney wrote:
> > >> >> On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
> > >> >> > On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
> > >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> > >> >> > > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
> > >> >> > >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
> > >> >> > >> <paulmck@linux.vnet.ibm.com> wrote:
> > >> >> > >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> > >> >> > >
> > >> >> > > [ . . . ]
> > >> >> > >
> > >> >> > >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > >> >> > >> > pr_info(" ");
> > >> >> > >> > level = rnp->level;
> > >> >> > >> > }
> > >> >> > >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> > >> >> > >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> > >> >> > >> > + rnp->qsmask,
> > >> >> > >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> > >> >> > >> > }
> > >> >> > >> > pr_cont("\n");
> > >> >> > >> > }
> > >> >> > >>
> > >> >> > >> For me it always crashes during the 37th call of synchronize_sched() in
> > >> >> > >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
> > >> >> > >> With your and my debug code, I get:
> > >> >> > >>
> > >> >> > >> CPU: Testing write buffer coherency: ok
> > >> >> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > >> >> > >> Setting up static identity map for 0x40100000 - 0x40100058
> > >> >> > >> cnt = 36, sync
> > >> >> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > >> >> > >> Brought up 2 CPUs
> > >> >> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > >> >> > >> CPU: All CPU(s) started in SVC mode.
> > >> >> > >> rcu_node tree layout dump
> > >> >> > >> 0:1/0x0/0x3 ^0
> > >> >> > >
> > >> >> > > Thank you for running this!
> > >> >> > >
> > >> >> > > OK, so RCU knows about both CPUs (the "0x3"), and the previous
> > >> >> > > grace period has seen quiescent states from both of them (the "0x0").
> > >> >> > > That would indicate that your synchronize_sched() showed up when RCU was
> > >> >> > > idle, so it had to start a new grace period. It also rules out failure
> > >> >> > > modes where RCU thinks that there are more CPUs than really exist.
> > >> >> > > (Don't laugh, such things have really happened.)
> > >> >> > >
> > >> >> > >> devtmpfs: initialized
> > >> >> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > >> >> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > >> >> > >> max_idle_ns: 19112604462750000 ns
> > >> >> > >>
> > >> >> > >> I hope it helps. Thanks!
> > >> >> > >
> > >> >> > > I am going to guess that this was the first grace period since the second
> > >> >> > > CPU came online. When there only on CPU online, synchronize_sched()
> > >> >> > > is a no-op.
> > >> >> > >
> > >> >> > > OK, this showed some things that aren't a problem. What might the
> > >> >> > > problem be?
> > >> >> > >
> > >> >> > > o The grace-period kthread has not yet started. It -should- start
> > >> >> > > at early_initcall() time, but who knows? Adding code to print
> > >> >> > > out that kthread's task_struct address.
> > >> >> > >
> > >> >> > > o The grace-period kthread might not be responding to wakeups.
> > >> >> > > Checking this requires that a grace period be in progress,
> > >> >> > > so please put a call_rcu_sched() just before the call to
> > >> >> > > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
> > >> >> > > to my patch to print out more GP-kthread state as well.
> > >> >> > >
> > >> >> > > o One of the CPUs might not be responding to RCU. That -should-
> > >> >> > > result in an RCU CPU stall warning, so I will ignore this
> > >> >> > > possibility for the moment.
> > >> >> > >
> > >> >> > > That said, do you have some way to determine whether scheduling
> > >> >> > > clock interrupts are really happening? Without these interrupts,
> > >> >> > > no RCU CPU stall warnings.
> > >> >> >
> > >> >> > I believe there are no clocksources yet. The jiffies clocksource is the first
> > >> >> > clocksource found, and that happens after the first call to
> > >> >> > synchronize_sched(), cfr. my dmesg snippet above.
> > >> >> >
> > >> >> > In a working boot:
> > >> >> > # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
> > >> >> > e0180000.timer jiffies
> > >> >> > # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
> > >> >> > e0180000.timer
> > >> >>
> > >> >> Ah! But if there is no jiffies clocksource, then schedule_timeout()
> > >> >> and friends will never return, correct? If so, I guarantee you that
> > >> >> synchronize_sched() will unconditionally hang.
> > >> >>
> > >> >> So if I understand correctly, the fix is to get the jiffies clocksource
> > >> >> running before the first call to synchronize_sched().
> > >> >
> > >> > If so, following change would be sufficient.
> > >> >
> > >> > Thanks.
> > >> >
> > >> > ------>8-------
> > >> > diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
> > >> > index 555e21f..4f6471f 100644
> > >> > --- a/kernel/time/jiffies.c
> > >> > +++ b/kernel/time/jiffies.c
> > >> > @@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
> > >> > return __clocksource_register(&clocksource_jiffies);
> > >> > }
> > >> >
> > >> > -core_initcall(init_jiffies_clocksource);
> > >> > +early_initcall(init_jiffies_clocksource);
> > >> >
> > >> > struct clocksource * __init __weak clocksource_default_clock(void)
> > >> > {
> > >>
> > >> Thanks for your patch!
> > >>
> > >> While this does move jiffies clocksource initialization before secondary CPU
> > >> bringup, it still hangs when calling call_rcu() or synchronize_sched():
> > >>
> > >> CPU: Testing write buffer coherency: ok
> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > >> Setting up static identity map for 0x40100000 - 0x40100058
> > >> cnt = 36, sync
> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > >> max_idle_ns: 19112604462750000 ns
> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > >> Brought up 2 CPUs
> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > >> CPU: All CPU(s) started in SVC mode.
> > >> RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
> > >> jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
> > >> rcu_node tree layout dump
> > >> 0:1/0x0/0x3 ^0
> > >
> > > This is in fact the initial state for RCU grace periods. In other words,
> > > all the earlier calls to synchronize_sched() likely happened while there
> > > was only one CPU online.
> > >
> > >> devtmpfs: initialized
> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > >
> > > Could you please add the call_rcu() and timed delay as described in my
> > > earlier email? That would hopefully help me see the state of the stalled
> > > grace period.
> >
> > I already did, cfr. "it still hangs when calling call_rcu() or
> > synchronize_sched()".
>
> Ah, sorry for my inattention.
>
> I am a bit surprised that it could hang when calling call_rcu(), given
> that call_rcu() is callable from atomic contexts. Could you please show
> me the current test code you have?
>
> If the hang is in call_rcu(), could you please try disabling irqs across
> the call to call_rcu()?
These are my local changes:
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index c7f1bc4f817c4a34..50bea263e510006f 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4707,11 +4707,16 @@ static void __init rcu_init_geometry(void)
* Dump out the structure of the rcu_node combining tree associated
* with the rcu_state structure referenced by rsp.
*/
-static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
+static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
{
int level = 0;
struct rcu_node *rnp;
+ pr_info("RCU: %s GP kthread: %p state: %d flags: %#x g:%ld c:%ld\n",
+ rsp->name, rsp->gp_kthread, rsp->gp_state, rsp->gp_flags,
+ (long)rsp->gpnum, (long)rsp->completed);
+ pr_info(" jiffies: %#lx GP start: %#lx Last GP activity: %#lx\n",
+ jiffies, rsp->gp_start, rsp->gp_activity);
pr_info("rcu_node tree layout dump\n");
pr_info(" ");
rcu_for_each_node_breadth_first(rsp, rnp) {
@@ -4720,11 +4725,32 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
pr_info(" ");
level = rnp->level;
}
- pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
+ pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
+ rnp->qsmask,
+ rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
}
pr_cont("\n");
}
+static void do_nothing_cb(struct rcu_head *rcu_head)
+{
+}
+
+void rcu_dump_rcu_sched_tree(void)
+{
+ struct rcu_head rh;
+ unsigned long flags;
+
+ rcu_dump_rcu_node_tree(&rcu_sched_state); /* Initial state. */
+ local_irq_save(flags);
+ // call_rcu(&rh, do_nothing_cb);
+ local_irq_restore(flags);
+ // schedule_timeout_uninterruptible(5 * HZ); /* Or whatever delay. */
+ rcu_dump_rcu_node_tree(&rcu_sched_state); /* GP state. */
+ //synchronize_sched(); /* Probably hangs. */
+ //rcu_barrier(); /* Drop RCU's references to rh before return. */
+}
+
void __init rcu_init(void)
{
int cpu;
diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
index 555e21f7b966c789..4f6471f54f69a6fe 100644
--- a/kernel/time/jiffies.c
+++ b/kernel/time/jiffies.c
@@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
return __clocksource_register(&clocksource_jiffies);
}
-core_initcall(init_jiffies_clocksource);
+early_initcall(init_jiffies_clocksource);
struct clocksource * __init __weak clocksource_default_clock(void)
{
diff --git a/mm/slab.c b/mm/slab.c
index cc8bbc1e6bc9b6fe..f9b2f50adc705173 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -909,6 +909,8 @@ static int init_cache_node_node(int node)
return 0;
}
+extern void rcu_dump_rcu_sched_tree(void);
+
static int setup_kmem_cache_node(struct kmem_cache *cachep,
int node, gfp_t gfp, bool force_change)
{
@@ -964,8 +966,19 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
* guaranteed to be valid until irq is re-enabled, because it will be
* freed after synchronize_sched().
*/
- if (force_change)
- synchronize_sched();
+ if (force_change) {
+ static int cnt;
+
+ if (++cnt < 37) {
+printk("cnt = %d, sync\n", cnt);
+ synchronize_sched();
+ } else if (cnt == 37) {
+printk("cnt = %d, dump\n", cnt);
+ rcu_dump_rcu_sched_tree();
+ } else {
+printk("cnt = %d\n", cnt);
+ }
+ }
fail:
kfree(old_shared);
With this it boots fine:
...
cnt = 35, sync
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x40100000 - 0x40100058
cnt = 36, sync
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Brought up 2 CPUs
SMP: Total of 2 processors activated (2132.00 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
cnt = 37, dump
RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
rcu_node tree layout dump
0:1/0x0/0x3 ^0
RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
rcu_node tree layout dump
0:1/0x0/0x3 ^0
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
cnt = 38
cnt = 39
...
When enabling any of the 4 commented-out lines in rcu_dump_rcu_sched_tree(),
it will lock up.
Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-30 15:53 ` Geert Uytterhoeven
@ 2016-06-30 16:52 ` Paul E. McKenney
2016-06-30 17:54 ` Geert Uytterhoeven
0 siblings, 1 reply; 34+ messages in thread
From: Paul E. McKenney @ 2016-06-30 16:52 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Joonsoo Kim, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
On Thu, Jun 30, 2016 at 05:53:42PM +0200, Geert Uytterhoeven wrote:
> On Thu, 30 Jun 2016, Paul E. McKenney wrote:
> > On Thu, Jun 30, 2016 at 03:31:57PM +0200, Geert Uytterhoeven wrote:
> > > On Thu, Jun 30, 2016 at 3:24 PM, Paul E. McKenney
> > > <paulmck@linux.vnet.ibm.com> wrote:
> > > > On Thu, Jun 30, 2016 at 09:58:51AM +0200, Geert Uytterhoeven wrote:
> > > >> On Thu, Jun 30, 2016 at 9:47 AM, Joonsoo Kim <iamjoonsoo.kim@lge.com> wrote:
> > > >> > On Wed, Jun 29, 2016 at 11:12:08AM -0700, Paul E. McKenney wrote:
> > > >> >> On Wed, Jun 29, 2016 at 07:52:06PM +0200, Geert Uytterhoeven wrote:
> > > >> >> > On Wed, Jun 29, 2016 at 6:44 PM, Paul E. McKenney
> > > >> >> > <paulmck@linux.vnet.ibm.com> wrote:
> > > >> >> > > On Wed, Jun 29, 2016 at 04:54:44PM +0200, Geert Uytterhoeven wrote:
> > > >> >> > >> On Thu, Jun 23, 2016 at 4:53 AM, Paul E. McKenney
> > > >> >> > >> <paulmck@linux.vnet.ibm.com> wrote:
> > > >> >> > >> > On Wed, Jun 22, 2016 at 07:47:42PM -0700, Paul E. McKenney wrote:
> > > >> >> > >
> > > >> >> > > [ . . . ]
> > > >> >> > >
> > > >> >> > >> > @@ -4720,11 +4720,18 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> > > >> >> > >> > pr_info(" ");
> > > >> >> > >> > level = rnp->level;
> > > >> >> > >> > }
> > > >> >> > >> > - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> > > >> >> > >> > + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> > > >> >> > >> > + rnp->qsmask,
> > > >> >> > >> > + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> > > >> >> > >> > }
> > > >> >> > >> > pr_cont("\n");
> > > >> >> > >> > }
> > > >> >> > >>
> > > >> >> > >> For me it always crashes during the 37th call of synchronize_sched() in
> > > >> >> > >> setup_kmem_cache_node(), which is the first call after secondary CPU bring up.
> > > >> >> > >> With your and my debug code, I get:
> > > >> >> > >>
> > > >> >> > >> CPU: Testing write buffer coherency: ok
> > > >> >> > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > > >> >> > >> Setting up static identity map for 0x40100000 - 0x40100058
> > > >> >> > >> cnt = 36, sync
> > > >> >> > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > > >> >> > >> Brought up 2 CPUs
> > > >> >> > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > >> >> > >> CPU: All CPU(s) started in SVC mode.
> > > >> >> > >> rcu_node tree layout dump
> > > >> >> > >> 0:1/0x0/0x3 ^0
> > > >> >> > >
> > > >> >> > > Thank you for running this!
> > > >> >> > >
> > > >> >> > > OK, so RCU knows about both CPUs (the "0x3"), and the previous
> > > >> >> > > grace period has seen quiescent states from both of them (the "0x0").
> > > >> >> > > That would indicate that your synchronize_sched() showed up when RCU was
> > > >> >> > > idle, so it had to start a new grace period. It also rules out failure
> > > >> >> > > modes where RCU thinks that there are more CPUs than really exist.
> > > >> >> > > (Don't laugh, such things have really happened.)
> > > >> >> > >
> > > >> >> > >> devtmpfs: initialized
> > > >> >> > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > > >> >> > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > > >> >> > >> max_idle_ns: 19112604462750000 ns
> > > >> >> > >>
> > > >> >> > >> I hope it helps. Thanks!
> > > >> >> > >
> > > >> >> > > I am going to guess that this was the first grace period since the second
> > > >> >> > > CPU came online. When there only on CPU online, synchronize_sched()
> > > >> >> > > is a no-op.
> > > >> >> > >
> > > >> >> > > OK, this showed some things that aren't a problem. What might the
> > > >> >> > > problem be?
> > > >> >> > >
> > > >> >> > > o The grace-period kthread has not yet started. It -should- start
> > > >> >> > > at early_initcall() time, but who knows? Adding code to print
> > > >> >> > > out that kthread's task_struct address.
> > > >> >> > >
> > > >> >> > > o The grace-period kthread might not be responding to wakeups.
> > > >> >> > > Checking this requires that a grace period be in progress,
> > > >> >> > > so please put a call_rcu_sched() just before the call to
> > > >> >> > > rcu_dump_rcu_node_tree(). (Sample code below.) Adding code
> > > >> >> > > to my patch to print out more GP-kthread state as well.
> > > >> >> > >
> > > >> >> > > o One of the CPUs might not be responding to RCU. That -should-
> > > >> >> > > result in an RCU CPU stall warning, so I will ignore this
> > > >> >> > > possibility for the moment.
> > > >> >> > >
> > > >> >> > > That said, do you have some way to determine whether scheduling
> > > >> >> > > clock interrupts are really happening? Without these interrupts,
> > > >> >> > > no RCU CPU stall warnings.
> > > >> >> >
> > > >> >> > I believe there are no clocksources yet. The jiffies clocksource is the first
> > > >> >> > clocksource found, and that happens after the first call to
> > > >> >> > synchronize_sched(), cfr. my dmesg snippet above.
> > > >> >> >
> > > >> >> > In a working boot:
> > > >> >> > # cat /sys/bus/clocksource/devices/clocksource0/available_clocksource
> > > >> >> > e0180000.timer jiffies
> > > >> >> > # cat /sys/bus/clocksource/devices/clocksource0/current_clocksource
> > > >> >> > e0180000.timer
> > > >> >>
> > > >> >> Ah! But if there is no jiffies clocksource, then schedule_timeout()
> > > >> >> and friends will never return, correct? If so, I guarantee you that
> > > >> >> synchronize_sched() will unconditionally hang.
> > > >> >>
> > > >> >> So if I understand correctly, the fix is to get the jiffies clocksource
> > > >> >> running before the first call to synchronize_sched().
> > > >> >
> > > >> > If so, following change would be sufficient.
> > > >> >
> > > >> > Thanks.
> > > >> >
> > > >> > ------>8-------
> > > >> > diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
> > > >> > index 555e21f..4f6471f 100644
> > > >> > --- a/kernel/time/jiffies.c
> > > >> > +++ b/kernel/time/jiffies.c
> > > >> > @@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
> > > >> > return __clocksource_register(&clocksource_jiffies);
> > > >> > }
> > > >> >
> > > >> > -core_initcall(init_jiffies_clocksource);
> > > >> > +early_initcall(init_jiffies_clocksource);
> > > >> >
> > > >> > struct clocksource * __init __weak clocksource_default_clock(void)
> > > >> > {
> > > >>
> > > >> Thanks for your patch!
> > > >>
> > > >> While this does move jiffies clocksource initialization before secondary CPU
> > > >> bringup, it still hangs when calling call_rcu() or synchronize_sched():
> > > >>
> > > >> CPU: Testing write buffer coherency: ok
> > > >> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> > > >> Setting up static identity map for 0x40100000 - 0x40100058
> > > >> cnt = 36, sync
> > > >> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff,
> > > >> max_idle_ns: 19112604462750000 ns
> > > >> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> > > >> Brought up 2 CPUs
> > > >> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> > > >> CPU: All CPU(s) started in SVC mode.
> > > >> RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
> > > >> jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
> > > >> rcu_node tree layout dump
> > > >> 0:1/0x0/0x3 ^0
> > > >
> > > > This is in fact the initial state for RCU grace periods. In other words,
> > > > all the earlier calls to synchronize_sched() likely happened while there
> > > > was only one CPU online.
> > > >
> > > >> devtmpfs: initialized
> > > >> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> > > >
> > > > Could you please add the call_rcu() and timed delay as described in my
> > > > earlier email? That would hopefully help me see the state of the stalled
> > > > grace period.
> > >
> > > I already did, cfr. "it still hangs when calling call_rcu() or
> > > synchronize_sched()".
> >
> > Ah, sorry for my inattention.
> >
> > I am a bit surprised that it could hang when calling call_rcu(), given
> > that call_rcu() is callable from atomic contexts. Could you please show
> > me the current test code you have?
> >
> > If the hang is in call_rcu(), could you please try disabling irqs across
> > the call to call_rcu()?
>
> These are my local changes:
>
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index c7f1bc4f817c4a34..50bea263e510006f 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4707,11 +4707,16 @@ static void __init rcu_init_geometry(void)
> * Dump out the structure of the rcu_node combining tree associated
> * with the rcu_state structure referenced by rsp.
> */
> -static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> +static void rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> {
> int level = 0;
> struct rcu_node *rnp;
>
> + pr_info("RCU: %s GP kthread: %p state: %d flags: %#x g:%ld c:%ld\n",
> + rsp->name, rsp->gp_kthread, rsp->gp_state, rsp->gp_flags,
> + (long)rsp->gpnum, (long)rsp->completed);
> + pr_info(" jiffies: %#lx GP start: %#lx Last GP activity: %#lx\n",
> + jiffies, rsp->gp_start, rsp->gp_activity);
> pr_info("rcu_node tree layout dump\n");
> pr_info(" ");
> rcu_for_each_node_breadth_first(rsp, rnp) {
> @@ -4720,11 +4725,32 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)
> pr_info(" ");
> level = rnp->level;
> }
> - pr_cont("%d:%d ^%d ", rnp->grplo, rnp->grphi, rnp->grpnum);
> + pr_cont("%d:%d/%#lx/%#lx ^%d ", rnp->grplo, rnp->grphi,
> + rnp->qsmask,
> + rnp->qsmaskinit | rnp->qsmaskinitnext, rnp->grpnum);
> }
> pr_cont("\n");
> }
>
> +static void do_nothing_cb(struct rcu_head *rcu_head)
> +{
> +}
> +
> +void rcu_dump_rcu_sched_tree(void)
> +{
> + struct rcu_head rh;
> + unsigned long flags;
> +
> + rcu_dump_rcu_node_tree(&rcu_sched_state); /* Initial state. */
> + local_irq_save(flags);
> + // call_rcu(&rh, do_nothing_cb);
> + local_irq_restore(flags);
> + // schedule_timeout_uninterruptible(5 * HZ); /* Or whatever delay. */
> + rcu_dump_rcu_node_tree(&rcu_sched_state); /* GP state. */
> + //synchronize_sched(); /* Probably hangs. */
> + //rcu_barrier(); /* Drop RCU's references to rh before return. */
> +}
> +
> void __init rcu_init(void)
> {
> int cpu;
> diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c
> index 555e21f7b966c789..4f6471f54f69a6fe 100644
> --- a/kernel/time/jiffies.c
> +++ b/kernel/time/jiffies.c
> @@ -98,7 +98,7 @@ static int __init init_jiffies_clocksource(void)
> return __clocksource_register(&clocksource_jiffies);
> }
>
> -core_initcall(init_jiffies_clocksource);
> +early_initcall(init_jiffies_clocksource);
>
> struct clocksource * __init __weak clocksource_default_clock(void)
> {
> diff --git a/mm/slab.c b/mm/slab.c
> index cc8bbc1e6bc9b6fe..f9b2f50adc705173 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -909,6 +909,8 @@ static int init_cache_node_node(int node)
> return 0;
> }
>
> +extern void rcu_dump_rcu_sched_tree(void);
> +
> static int setup_kmem_cache_node(struct kmem_cache *cachep,
> int node, gfp_t gfp, bool force_change)
> {
> @@ -964,8 +966,19 @@ static int setup_kmem_cache_node(struct kmem_cache *cachep,
> * guaranteed to be valid until irq is re-enabled, because it will be
> * freed after synchronize_sched().
> */
> - if (force_change)
> - synchronize_sched();
> + if (force_change) {
> + static int cnt;
> +
> + if (++cnt < 37) {
> +printk("cnt = %d, sync\n", cnt);
> + synchronize_sched();
> + } else if (cnt == 37) {
> +printk("cnt = %d, dump\n", cnt);
> + rcu_dump_rcu_sched_tree();
> + } else {
> +printk("cnt = %d\n", cnt);
> + }
> + }
>
> fail:
> kfree(old_shared);
>
>
> With this it boots fine:
>
> ...
> cnt = 35, sync
> CPU: Testing write buffer coherency: ok
> CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
> Setting up static identity map for 0x40100000 - 0x40100058
> cnt = 36, sync
> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
> CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
> Brought up 2 CPUs
> SMP: Total of 2 processors activated (2132.00 BogoMIPS).
> CPU: All CPU(s) started in SVC mode.
> cnt = 37, dump
> RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
> jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
> rcu_node tree layout dump
> 0:1/0x0/0x3 ^0
> RCU: rcu_sched GP kthread: c784e1c0 state: 1 flags: 0x0 g:-300 c:-300
> jiffies: 0xffff8ad0 GP start: 0x0 Last GP activity: 0x0
> rcu_node tree layout dump
> 0:1/0x0/0x3 ^0
> devtmpfs: initialized
> VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 1
> cnt = 38
> cnt = 39
> ...
>
> When enabling any of the 4 commented-out lines in rcu_dump_rcu_sched_tree(),
> it will lock up.
OK, but that includes schedule_timeout_uninterruptible(5 * HZ), right?
If schedule_timeout_uninterruptible() hangs, RCU grace periods have no
chance of completing.
I am still surprised that call_rcu() hangs, but timed wait not working
will definitely cause RCU grace-period hangs. Perhaps fixing timed
waits will also prevent the call_rcu() hang.
Thanx, Paul
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache)
2016-06-30 16:52 ` Paul E. McKenney
@ 2016-06-30 17:54 ` Geert Uytterhoeven
0 siblings, 0 replies; 34+ messages in thread
From: Geert Uytterhoeven @ 2016-06-30 17:54 UTC (permalink / raw)
To: Paul McKenney
Cc: Joonsoo Kim, Linux-Renesas, Andrew Morton, David Rientjes,
linux-kernel, Pekka Enberg, Linux MM, Jesper Dangaard Brouer,
Christoph Lameter, linux-arm-kernel
Hi Paul,
On Thu, Jun 30, 2016 at 6:52 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Thu, Jun 30, 2016 at 05:53:42PM +0200, Geert Uytterhoeven wrote:
>> +void rcu_dump_rcu_sched_tree(void)
>> +{
>> + struct rcu_head rh;
>> + unsigned long flags;
>> +
>> + rcu_dump_rcu_node_tree(&rcu_sched_state); /* Initial state. */
>> + local_irq_save(flags);
>> + // call_rcu(&rh, do_nothing_cb);
>> + local_irq_restore(flags);
>> + // schedule_timeout_uninterruptible(5 * HZ); /* Or whatever delay. */
>> + rcu_dump_rcu_node_tree(&rcu_sched_state); /* GP state. */
>> + //synchronize_sched(); /* Probably hangs. */
>> + //rcu_barrier(); /* Drop RCU's references to rh before return. */
>> +}
>>
>> When enabling any of the 4 commented-out lines in rcu_dump_rcu_sched_tree(),
>> it will lock up.
>
> OK, but that includes schedule_timeout_uninterruptible(5 * HZ), right?
Yes it does.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2016-06-30 18:13 UTC | newest]
Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-13 19:43 Boot failure on emev2/kzm9d (was: Re: [PATCH v2 11/11] mm/slab: lockless decision to grow cache) Geert Uytterhoeven
2016-06-14 6:24 ` Joonsoo Kim
2016-06-14 7:31 ` Geert Uytterhoeven
2016-06-14 8:11 ` Joonsoo Kim
2016-06-14 10:45 ` Geert Uytterhoeven
2016-06-15 2:23 ` Joonsoo Kim
2016-06-15 8:39 ` Geert Uytterhoeven
2016-06-20 6:39 ` Joonsoo Kim
2016-06-20 13:12 ` Paul E. McKenney
2016-06-21 6:43 ` Joonsoo Kim
2016-06-21 12:54 ` Paul E. McKenney
2016-06-22 0:52 ` Joonsoo Kim
2016-06-22 3:15 ` Paul E. McKenney
2016-06-22 15:01 ` Geert Uytterhoeven
2016-06-22 19:08 ` Paul E. McKenney
2016-06-23 0:49 ` Paul E. McKenney
2016-06-23 2:37 ` Joonsoo Kim
2016-06-23 2:47 ` Paul E. McKenney
2016-06-23 2:53 ` Paul E. McKenney
2016-06-28 0:12 ` Paul E. McKenney
2016-06-28 8:33 ` Joonsoo Kim
2016-06-29 14:54 ` Geert Uytterhoeven
2016-06-29 16:44 ` Paul E. McKenney
2016-06-29 17:52 ` Geert Uytterhoeven
2016-06-29 18:12 ` Paul E. McKenney
2016-06-30 7:47 ` Joonsoo Kim
2016-06-30 7:58 ` Geert Uytterhoeven
2016-06-30 13:24 ` Paul E. McKenney
2016-06-30 13:31 ` Geert Uytterhoeven
2016-06-30 15:18 ` Paul E. McKenney
2016-06-30 15:53 ` Geert Uytterhoeven
2016-06-30 16:52 ` Paul E. McKenney
2016-06-30 17:54 ` Geert Uytterhoeven
2016-06-14 13:10 ` Geert Uytterhoeven
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).