All of lore.kernel.org
 help / color / mirror / Atom feed
* [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-06-30 11:41 Zdenek Kabelac
  2012-06-30 13:20 ` Zdenek Kabelac
  2012-06-30 19:55   ` [linux-lvm] " Hugh Dickins
  0 siblings, 2 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-06-30 11:41 UTC (permalink / raw)
  To: LVM general discussion and development, Hugh Dickins, amwang,
	Alasdair G Kergon

[-- Attachment #1: Type: text/plain, Size: 6223 bytes --]

Hi

When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
Ooops log included. After some experimenting - reliable way to hit this oops 
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not 
included anything related to this oops I've went for bisect.

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

My HW - t61, 4GB

Zdenek

BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
PGD 0
Oops: 0000 [#1] PREEMPT SMP
CPU 1
Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq 
async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror 
dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables 
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state 
nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables 
x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 
iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel 
snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e 
snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc 
loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd 
sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core 
yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight

Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 
6464CTO/6464CTO
RIP: 0010:[<ffffffffa031204d>]  [<ffffffffa031204d>] 
dm_rh_region_context+0xd/0x20 [dm_region_hash]
RSP: 0018:ffff880126b77c68  EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607
RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40
RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400
R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task 
ffff8801344ca310)
Stack:
  ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0
  ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000
  0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0
Call Trace:
  [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror]
  [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror]
  [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod]
  [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod]
  [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod]
  [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod]
  [<ffffffff81060065>] process_one_work+0x1a5/0x6b0
  [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0
  [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80
  [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod]
  [<ffffffff810608ee>] worker_thread+0x12e/0x2d0
  [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210
  [<ffffffff81069cf2>] kthread+0xa2/0xb0
  [<ffffffff81533334>] kernel_thread_helper+0x4/0x10
  [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120
  [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60
  [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe
  [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0
  [<ffffffff81533330>] ? gs_change+0xb/0xb
Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 
c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 
00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
RIP  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
  RSP <ffff880126b77c68>
CR2: 0000000000000178


# bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' 
of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic 
strnlen_user() function
git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 
'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' 
of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of 
git://git.linaro.org/people/rmk/linux-arm
git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump 
PG_compound_lock page flag
git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should 
not be user visible
git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug 
fs entries on failure of file creation and made extfrag_debug_root dentry local
git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove 
redundant offset check when finally freeing bootmem
git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if 
mapping excludes its zone
git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range
git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing 
when writing
git bisect good ec9516fbc5fa814014991e1ae7f8860127122105

[-- Attachment #2: CONFIG --]
[-- Type: text/plain, Size: 33663 bytes --]

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_LZMA=y
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_TREE_RCU_TRACE=y
CONFIG_IKCONFIG=m
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_UID16=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y

CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_THROTTLING=y

CONFIG_PARTITION_ADVANCED=y
CONFIG_AMIGA_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_LDM_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_BLOCK_COMPAT=y

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=m
CONFIG_DEFAULT_DEADLINE=y
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_FREEZER=y

CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MEMTEST=y
CONFIG_MCORE2=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_NR_CPUS=2
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=999999
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_EFI=y
CONFIG_HZ_300=y
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE=""
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_WAKELOCKS_LIMIT=100
CONFIG_PM_WAKELOCKS_GC=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_CAN_PM_TRACE=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_CUSTOM_METHOD=m

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_P4_CLOCKMOD=m

CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y


CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_STUB=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y

CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_X86_X32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y

CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_MROUTE=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_TUNNEL=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_DEFAULT_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_IPV6=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=m

CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m

CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TIME=m

CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_FTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_NONE=y
CONFIG_STP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_LLC=m
CONFIG_DNS_RESOLVER=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_NETPRIO_CGROUP=m
CONFIG_BQL=y

CONFIG_NET_PKTGEN=m
CONFIG_IRDA=m




CONFIG_IRTTY_SIR=m


CONFIG_BT=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_HIDP=m

CONFIG_BT_HCIBTUSB=m
CONFIG_WIRELESS=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_CFG80211=m
CONFIG_CFG80211_DEVELOPER_WARNINGS=y
CONFIG_CFG80211_DEFAULT_PS=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_CFG80211_WEXT=y
CONFIG_WIRELESS_EXT_SYSFS=y
CONFIG_LIB80211=m
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_RFKILL=y
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_RFKILL_REGULATOR=m
CONFIG_HAVE_BPF_JIT=y


CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_DMA_SHARED_BUFFER=y
CONFIG_CONNECTOR=m
CONFIG_MTD=m
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1

CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_SM_FTL=m

CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y


CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m

CONFIG_MTD_NAND_ECC=m
CONFIG_MTD_NAND_ECC_SMC=y

CONFIG_PNP=y

CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
CONFIG_VIRTIO_BLK=y




CONFIG_INTEL_MEI=m
CONFIG_HAVE_IDE=y

CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y

CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_VIRTIO=m
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

CONFIG_SATA_AHCI=y
CONFIG_ATA_SFF=y

CONFIG_ATA_BMDMA=y

CONFIG_ATA_PIIX=y


CONFIG_PATA_MPIIX=y

CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MULTICORE_RAID456=y
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
CONFIG_DM_PERSISTENT_DATA=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
CONFIG_DM_MIRROR=m
CONFIG_DM_RAID=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m

CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_DUMMY=m
CONFIG_MII=m
CONFIG_NETCONSOLE=m
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_VIRTIO_NET=m

CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E1000E=m
CONFIG_IXGBEVF=m
CONFIG_NET_VENDOR_REALTEK=y
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_ASYNC=m
CONFIG_SLHC=m

CONFIG_WLAN=y
CONFIG_IWLWIFI=m

CONFIG_IWLEGACY=m
CONFIG_IWL3945=m

CONFIG_IWLEGACY_DEBUGFS=y


CONFIG_INPUT=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=m

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_EVDEV=m

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_SERIAL=m
CONFIG_INPUT_MISC=y
CONFIG_INPUT_UINPUT=m

CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m

CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVKMEM=y

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=16
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_RSA=y

CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_HVC_DRIVER=y
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=m

CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m


CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m



CONFIG_I2C_TINY_USB=m

CONFIG_I2C_DEBUG_ALGO=y
CONFIG_HSI=m
CONFIG_HSI_BOARDINFO=y

CONFIG_HSI_CHAR=m




CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_POWER_SUPPLY=y
CONFIG_HWMON=y


CONFIG_SENSORS_ACPI_POWER=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y

CONFIG_SOFT_WATCHDOG=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y


CONFIG_SSB_POSSIBLE=y

CONFIG_BCMA_POSSIBLE=y


CONFIG_MFD_CORE=m
CONFIG_LPC_SCH=m
CONFIG_LPC_ICH=m
CONFIG_REGULATOR=y
CONFIG_REGULATOR_DEBUG=y
CONFIG_REGULATOR_DUMMY=y
CONFIG_REGULATOR_VIRTUAL_CONSUMER=m
CONFIG_MEDIA_SUPPORT=m

CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m

CONFIG_RC_CORE=m
CONFIG_LIRC=m
CONFIG_RC_MAP=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_RC_LOOPBACK=m
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_CUSTOMISE=y

CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_MEDIA_TUNER_TDA18212=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_V4L2_MEM2MEM_DEV=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
CONFIG_VIDEO_ADV_DEBUG=y
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m

CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m

CONFIG_VIDEO_SAA6588=m

CONFIG_VIDEO_SAA711X=m

CONFIG_VIDEO_CX25840=m

CONFIG_VIDEO_CX2341X=m





CONFIG_VIDEO_VIVI=m
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
CONFIG_V4L_PCI_DRIVERS=y
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_MEM2MEM_TESTDEV=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y


CONFIG_DVB_USB=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_TTUSB_BUDGET=m









CONFIG_DVB_FE_CUSTOMISE=y




CONFIG_DVB_TDA8261=m
CONFIG_DVB_TDA10071=m

CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_AF9013=m



CONFIG_DVB_DIB8000=m

CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m


CONFIG_DVB_DUMMY_FE=m

CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m

CONFIG_DRM_I915=m
CONFIG_DRM_MGA=m
CONFIG_DRM_CIRRUS_QEMU=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_PLATFORM=m
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_BACKLIGHT_GENERIC=m

CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_DUMMY=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_PCI=y
CONFIG_SND_EMU10K1=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_PREALLOC_SIZE=64
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=500
CONFIG_SND_INTEL8X0=m
CONFIG_AC97_BUS=m

CONFIG_HID=m
CONFIG_HID_GENERIC=m

CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_KYE=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m

CONFIG_USB_HID=m
CONFIG_USB_HIDDEV=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB_ARCH_HAS_XHCI=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=m
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=m
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

CONFIG_USB_SUSPEND=y
CONFIG_USB_MON=m

CONFIG_USB_XHCI_HCD=m
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m

CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m


CONFIG_USB_STORAGE=m
CONFIG_USB_UAS=m


CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m

CONFIG_USB_TEST=m


CONFIG_MMC=m

CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y

CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_SDRICOH_CS=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

CONFIG_LEDS_TRIGGERS=y

CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m

CONFIG_EDAC=y

CONFIG_EDAC_DEBUG=y
CONFIG_EDAC_DECODE_MCE=y
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_E752X=m
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"

CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y



CONFIG_RTC_DRV_CMOS=y

CONFIG_DMADEVICES=y

CONFIG_INTEL_IOATDMA=m
CONFIG_DMA_ENGINE=y

CONFIG_NET_DMA=y
CONFIG_DCA=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y

CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=m
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y

CONFIG_STAGING=y
CONFIG_ZRAM=m
CONFIG_ZRAM_DEBUG=y
CONFIG_ZCACHE=y
CONFIG_ZSMALLOC=y


CONFIG_USB_WPAN_HCD=m
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
CONFIG_THINKPAD_ACPI_DEBUG=y
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_SENSORS_HDAPS=y
CONFIG_ACPI_WMI=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_INTEL_IPS=m

CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
CONFIG_IOMMU_SUPPORT=y


CONFIG_VIRT_DRIVERS=y
CONFIG_PM_DEVFREQ=y

CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
CONFIG_DEVFREQ_GOV_PERFORMANCE=y
CONFIG_DEVFREQ_GOV_POWERSAVE=y
CONFIG_DEVFREQ_GOV_USERSPACE=y


CONFIG_EDD=m
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=m
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=m

CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=m
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y

CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
CONFIG_FSCACHE_HISTOGRAM=y
CONFIG_FSCACHE_DEBUG=y
CONFIG_CACHEFILES=m
CONFIG_CACHEFILES_DEBUG=y

CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

CONFIG_FAT_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=852
CONFIG_FAT_DEFAULT_IOCHARSET="cp1250"
CONFIG_NTFS_FS=m
CONFIG_NTFS_RW=y

CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
CONFIG_JFFS2_COMPRESSION_OPTIONS=y
CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_LZO=y
CONFIG_JFFS2_RTIME=y
CONFIG_JFFS2_RUBIN=y
CONFIG_JFFS2_CMODE_PRIORITY=y
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_CIFS=m
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m

CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
CONFIG_SLUB_STATS=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=500
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RCU=y
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
CONFIG_RCU_TRACE=y
CONFIG_LKDTM=m
CONFIG_LATENCYTOP=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_PREEMPT_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BRANCH_PROFILE_NONE=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_UPROBE_EVENT=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_MMIOTRACE=y
CONFIG_MMIOTRACE_TEST=m
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DMA_API_DEBUG=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_RODATA=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_OPTIMIZE_INLINING=y

CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_LSM_MMAP_MIN_ADDR=65535
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
CONFIG_INTEGRITY=y
CONFIG_EVM=y
CONFIG_DEFAULT_SECURITY_SELINUX=y
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m


CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_XTS=m

CONFIG_CRYPTO_HMAC=y

CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=y
CONFIG_CRYPTO_SHA256=y

CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m

CONFIG_CRYPTO_LZO=y

CONFIG_CRYPTO_HW=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
CONFIG_BINARY_PRINTF=y

CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC32_SLICEBY8=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
CONFIG_DDR=y

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-06-30 11:41 [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel Zdenek Kabelac
@ 2012-06-30 13:20 ` Zdenek Kabelac
  2012-06-30 19:55   ` [linux-lvm] " Hugh Dickins
  1 sibling, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-06-30 13:20 UTC (permalink / raw)
  To: LKML

[-- Attachment #1: Type: text/plain, Size: 0 bytes --]



[-- Attachment #2: Regression with FALLOC_FL_PUNCH_HOLE in 3_5-rc kernel.eml --]
[-- Type: message/rfc822, Size: 40837 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 6223 bytes --]

Hi

When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
Ooops log included. After some experimenting - reliable way to hit this oops 
is to run lvm test suite for 10 minutes. Since 3.5 merge window does not 
included anything related to this oops I've went for bisect.

Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17

mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

My HW - t61, 4GB

Zdenek

BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
PGD 0
Oops: 0000 [#1] PREEMPT SMP
CPU 1
Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq 
async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror 
dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables 
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state 
nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables 
x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 
iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel 
snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e 
snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc 
loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd 
sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core 
yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight

Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 
6464CTO/6464CTO
RIP: 0010:[<ffffffffa031204d>]  [<ffffffffa031204d>] 
dm_rh_region_context+0xd/0x20 [dm_region_hash]
RSP: 0018:ffff880126b77c68  EFLAGS: 00010282
RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607
RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40
RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400
R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task 
ffff8801344ca310)
Stack:
  ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0
  ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000
  0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0
Call Trace:
  [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror]
  [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror]
  [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod]
  [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod]
  [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod]
  [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod]
  [<ffffffff81060065>] process_one_work+0x1a5/0x6b0
  [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0
  [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80
  [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod]
  [<ffffffff810608ee>] worker_thread+0x12e/0x2d0
  [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210
  [<ffffffff81069cf2>] kthread+0xa2/0xb0
  [<ffffffff81533334>] kernel_thread_helper+0x4/0x10
  [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120
  [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60
  [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe
  [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0
  [<ffffffff81533330>] ? gs_change+0xb/0xb
Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 
c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 
00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
RIP  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
  RSP <ffff880126b77c68>
CR2: 0000000000000178


# bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
# good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
git bisect start 'v3.5-rc2' 'v3.4'
# good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' 
of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
# good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic 
strnlen_user() function
git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
# bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 
'for-3.5/drivers' of git://git.kernel.dk/linux-block
git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
# good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' 
of git://git.linux-nfs.org/projects/trondmy/linux-nfs
git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
# bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of 
git://git.linaro.org/people/rmk/linux-arm
git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
# bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump 
PG_compound_lock page flag
git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
# good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should 
not be user visible
git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
# good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug 
fs entries on failure of file creation and made extfrag_debug_root dentry local
git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
# good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove 
redundant offset check when finally freeing bootmem
git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
# good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if 
mapping excludes its zone
git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
# bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range
git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
# good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing 
when writing
git bisect good ec9516fbc5fa814014991e1ae7f8860127122105

[-- Attachment #2.1.2: CONFIG --]
[-- Type: text/plain, Size: 33663 bytes --]

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_LZMA=y
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
CONFIG_RCU_FAST_NO_HZ=y
CONFIG_TREE_RCU_TRACE=y
CONFIG_IKCONFIG=m
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_UID16=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y

CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_THROTTLING=y

CONFIG_PARTITION_ADVANCED=y
CONFIG_AMIGA_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_LDM_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_BLOCK_COMPAT=y

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=m
CONFIG_DEFAULT_DEADLINE=y
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_FREEZER=y

CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MEMTEST=y
CONFIG_MCORE2=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_NR_CPUS=2
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=999999
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_EFI=y
CONFIG_HZ_300=y
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE=""
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_WAKELOCKS=y
CONFIG_PM_WAKELOCKS_LIMIT=100
CONFIG_PM_WAKELOCKS_GC=y
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_CAN_PM_TRACE=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_CUSTOM_METHOD=m

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_P4_CLOCKMOD=m

CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y


CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_STUB=y
CONFIG_HT_IRQ=y
CONFIG_PCI_ATS=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y

CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_X86_X32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y

CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_MROUTE=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_TUNNEL=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_DEFAULT_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_IPV6=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=m

CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m

CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TIME=m

CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_FTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_NONE=y
CONFIG_STP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_LLC=m
CONFIG_DNS_RESOLVER=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_NETPRIO_CGROUP=m
CONFIG_BQL=y

CONFIG_NET_PKTGEN=m
CONFIG_IRDA=m




CONFIG_IRTTY_SIR=m


CONFIG_BT=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_HIDP=m

CONFIG_BT_HCIBTUSB=m
CONFIG_WIRELESS=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_CFG80211=m
CONFIG_CFG80211_DEVELOPER_WARNINGS=y
CONFIG_CFG80211_DEFAULT_PS=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_CFG80211_WEXT=y
CONFIG_WIRELESS_EXT_SYSFS=y
CONFIG_LIB80211=m
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_RFKILL=y
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_RFKILL_REGULATOR=m
CONFIG_HAVE_BPF_JIT=y


CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_DMA_SHARED_BUFFER=y
CONFIG_CONNECTOR=m
CONFIG_MTD=m
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1

CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_SM_FTL=m

CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y


CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m

CONFIG_MTD_NAND_ECC=m
CONFIG_MTD_NAND_ECC_SMC=y

CONFIG_PNP=y

CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
CONFIG_VIRTIO_BLK=y




CONFIG_INTEL_MEI=m
CONFIG_HAVE_IDE=y

CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y

CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_VIRTIO=m
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

CONFIG_SATA_AHCI=y
CONFIG_ATA_SFF=y

CONFIG_ATA_BMDMA=y

CONFIG_ATA_PIIX=y


CONFIG_PATA_MPIIX=y

CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MULTICORE_RAID456=y
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
CONFIG_DM_PERSISTENT_DATA=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
CONFIG_DM_MIRROR=m
CONFIG_DM_RAID=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m

CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_DUMMY=m
CONFIG_MII=m
CONFIG_NETCONSOLE=m
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_VIRTIO_NET=m

CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E1000E=m
CONFIG_IXGBEVF=m
CONFIG_NET_VENDOR_REALTEK=y
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_ASYNC=m
CONFIG_SLHC=m

CONFIG_WLAN=y
CONFIG_IWLWIFI=m

CONFIG_IWLEGACY=m
CONFIG_IWL3945=m

CONFIG_IWLEGACY_DEBUGFS=y


CONFIG_INPUT=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=m

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_EVDEV=m

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_SERIAL=m
CONFIG_INPUT_MISC=y
CONFIG_INPUT_UINPUT=m

CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m

CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVKMEM=y

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=16
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_RSA=y

CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_HVC_DRIVER=y
CONFIG_VIRTIO_CONSOLE=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=m

CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m


CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m



CONFIG_I2C_TINY_USB=m

CONFIG_I2C_DEBUG_ALGO=y
CONFIG_HSI=m
CONFIG_HSI_BOARDINFO=y

CONFIG_HSI_CHAR=m




CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_POWER_SUPPLY=y
CONFIG_HWMON=y


CONFIG_SENSORS_ACPI_POWER=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y

CONFIG_SOFT_WATCHDOG=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y


CONFIG_SSB_POSSIBLE=y

CONFIG_BCMA_POSSIBLE=y


CONFIG_MFD_CORE=m
CONFIG_LPC_SCH=m
CONFIG_LPC_ICH=m
CONFIG_REGULATOR=y
CONFIG_REGULATOR_DEBUG=y
CONFIG_REGULATOR_DUMMY=y
CONFIG_REGULATOR_VIRTUAL_CONSUMER=m
CONFIG_MEDIA_SUPPORT=m

CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m

CONFIG_RC_CORE=m
CONFIG_LIRC=m
CONFIG_RC_MAP=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_RC_LOOPBACK=m
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_CUSTOMISE=y

CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_MEDIA_TUNER_TDA18212=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_V4L2_MEM2MEM_DEV=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
CONFIG_VIDEO_ADV_DEBUG=y
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m

CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m

CONFIG_VIDEO_SAA6588=m

CONFIG_VIDEO_SAA711X=m

CONFIG_VIDEO_CX25840=m

CONFIG_VIDEO_CX2341X=m





CONFIG_VIDEO_VIVI=m
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
CONFIG_V4L_PCI_DRIVERS=y
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_V4L_MEM2MEM_DRIVERS=y
CONFIG_VIDEO_MEM2MEM_TESTDEV=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y


CONFIG_DVB_USB=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_TTUSB_BUDGET=m









CONFIG_DVB_FE_CUSTOMISE=y




CONFIG_DVB_TDA8261=m
CONFIG_DVB_TDA10071=m

CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_AF9013=m



CONFIG_DVB_DIB8000=m

CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m


CONFIG_DVB_DUMMY_FE=m

CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m

CONFIG_DRM_I915=m
CONFIG_DRM_MGA=m
CONFIG_DRM_CIRRUS_QEMU=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_PLATFORM=m
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_BACKLIGHT_GENERIC=m

CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VMASTER=y
CONFIG_SND_KCTL_JACK=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_DUMMY=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_PCI=y
CONFIG_SND_EMU10K1=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_PREALLOC_SIZE=64
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=500
CONFIG_SND_INTEL8X0=m
CONFIG_AC97_BUS=m

CONFIG_HID=m
CONFIG_HID_GENERIC=m

CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_KYE=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m

CONFIG_USB_HID=m
CONFIG_USB_HIDDEV=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB_ARCH_HAS_XHCI=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=m
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=m
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

CONFIG_USB_SUSPEND=y
CONFIG_USB_MON=m

CONFIG_USB_XHCI_HCD=m
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m

CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m


CONFIG_USB_STORAGE=m
CONFIG_USB_UAS=m


CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m

CONFIG_USB_TEST=m


CONFIG_MMC=m

CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y

CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_SDRICOH_CS=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

CONFIG_LEDS_TRIGGERS=y

CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m

CONFIG_EDAC=y

CONFIG_EDAC_DEBUG=y
CONFIG_EDAC_DECODE_MCE=y
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_E752X=m
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"

CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y



CONFIG_RTC_DRV_CMOS=y

CONFIG_DMADEVICES=y

CONFIG_INTEL_IOATDMA=m
CONFIG_DMA_ENGINE=y

CONFIG_NET_DMA=y
CONFIG_DCA=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y

CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=m
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y

CONFIG_STAGING=y
CONFIG_ZRAM=m
CONFIG_ZRAM_DEBUG=y
CONFIG_ZCACHE=y
CONFIG_ZSMALLOC=y


CONFIG_USB_WPAN_HCD=m
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
CONFIG_THINKPAD_ACPI_DEBUG=y
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_SENSORS_HDAPS=y
CONFIG_ACPI_WMI=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_INTEL_IPS=m

CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
CONFIG_IOMMU_SUPPORT=y


CONFIG_VIRT_DRIVERS=y
CONFIG_PM_DEVFREQ=y

CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
CONFIG_DEVFREQ_GOV_PERFORMANCE=y
CONFIG_DEVFREQ_GOV_POWERSAVE=y
CONFIG_DEVFREQ_GOV_USERSPACE=y


CONFIG_EDD=m
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=m
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=m

CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=m
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y

CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
CONFIG_FSCACHE_HISTOGRAM=y
CONFIG_FSCACHE_DEBUG=y
CONFIG_CACHEFILES=m
CONFIG_CACHEFILES_DEBUG=y

CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

CONFIG_FAT_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=852
CONFIG_FAT_DEFAULT_IOCHARSET="cp1250"
CONFIG_NTFS_FS=m
CONFIG_NTFS_RW=y

CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
CONFIG_JFFS2_COMPRESSION_OPTIONS=y
CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_LZO=y
CONFIG_JFFS2_RTIME=y
CONFIG_JFFS2_RUBIN=y
CONFIG_JFFS2_CMODE_PRIORITY=y
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_CIFS=m
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m

CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
CONFIG_SLUB_STATS=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=500
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_PROVE_RCU=y
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
CONFIG_RCU_CPU_STALL_INFO=y
CONFIG_RCU_TRACE=y
CONFIG_LKDTM=m
CONFIG_LATENCYTOP=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_PREEMPT_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BRANCH_PROFILE_NONE=y
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_UPROBE_EVENT=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_MMIOTRACE=y
CONFIG_MMIOTRACE_TEST=m
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DMA_API_DEBUG=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_RODATA=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_OPTIMIZE_INLINING=y

CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_LSM_MMAP_MIN_ADDR=65535
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
CONFIG_INTEGRITY=y
CONFIG_EVM=y
CONFIG_DEFAULT_SECURITY_SELINUX=y
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m


CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_XTS=m

CONFIG_CRYPTO_HMAC=y

CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=y
CONFIG_CRYPTO_SHA256=y

CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m

CONFIG_CRYPTO_LZO=y

CONFIG_CRYPTO_HW=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
CONFIG_BINARY_PRINTF=y

CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_IO=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC32_SLICEBY8=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
CONFIG_DDR=y

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-06-30 11:41 [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel Zdenek Kabelac
@ 2012-06-30 19:55   ` Hugh Dickins
  2012-06-30 19:55   ` [linux-lvm] " Hugh Dickins
  1 sibling, 0 replies; 26+ messages in thread
From: Hugh Dickins @ 2012-06-30 19:55 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel

On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> 
> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> Ooops log included. After some experimenting - reliable way to hit this oops
> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
> included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find
a reproducible testcase that you could bisect on.

> 
> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> 
> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
under tests/ ?

I see no mention of madvise or MADV_REMOVE or fallocate or anything
related in that git tree.

If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

I'll be surprised if that is the case, but it's something that you can
easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.

I'm not an LVM user, so I doubt I'll be able to reproduce your setup.

Any ideas from the DM guys?  Has anyone else seen anything like this?

Do all your oopses look like this one?

Thanks,
Hugh

> 
> My HW - t61, 4GB
> 
> Zdenek
> 
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
> IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
> PGD 0
> Oops: 0000 [#1] PREEMPT SMP
> CPU 1
> Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq
> async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror
> dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables
> ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables
> x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945
> iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel
> snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e
> snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc
> loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd
> sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core
> yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight
> 
> Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO
> 6464CTO/6464CTO
> RIP: 0010:[<ffffffffa031204d>]  [<ffffffffa031204d>]
> dm_rh_region_context+0xd/0x20 [dm_region_hash]
> RSP: 0018:ffff880126b77c68  EFLAGS: 00010282
> RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607
> RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40
> RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000
> R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400
> R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task
> ffff8801344ca310)
> Stack:
>  ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0
>  ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000
>  0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0
> Call Trace:
>  [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror]
>  [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror]
>  [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod]
>  [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod]
>  [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod]
>  [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod]
>  [<ffffffff81060065>] process_one_work+0x1a5/0x6b0
>  [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0
>  [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80
>  [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod]
>  [<ffffffff810608ee>] worker_thread+0x12e/0x2d0
>  [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210
>  [<ffffffff81069cf2>] kthread+0xa2/0xb0
>  [<ffffffff81533334>] kernel_thread_helper+0x4/0x10
>  [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120
>  [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60
>  [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe
>  [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0
>  [<ffffffff81533330>] ? gs_change+0xb/0xb
> Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8
> c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00
> 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
> RIP  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
>  RSP <ffff880126b77c68>
> CR2: 0000000000000178
> 
> 
> # bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
> # good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
> git bisect start 'v3.5-rc2' 'v3.4'
> # good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge'
> of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
> git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
> # good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic
> strnlen_user() function
> git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
> # bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch
> 'for-3.5/drivers' of git://git.kernel.dk/linux-block
> git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
> # good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1'
> of git://git.linux-nfs.org/projects/trondmy/linux-nfs
> git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
> # bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of
> git://git.linaro.org/people/rmk/linux-arm
> git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
> # bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump
> PG_compound_lock page flag
> git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
> # good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should
> not be user visible
> git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
> # good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug
> fs entries on failure of file creation and made extfrag_debug_root dentry
> local
> git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
> # good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove
> redundant offset check when finally freeing bootmem
> git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
> # good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if
> mapping excludes its zone
> git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
> # bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove
> truncate_range
> git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
> # good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing
> when writing
> git bisect good ec9516fbc5fa814014991e1ae7f8860127122105

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-06-30 19:55   ` Hugh Dickins
  0 siblings, 0 replies; 26+ messages in thread
From: Hugh Dickins @ 2012-06-30 19:55 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: linux-kernel, amwang, Alasdair G Kergon,
	LVM general discussion and development

On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> 
> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> Ooops log included. After some experimenting - reliable way to hit this oops
> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
> included anything related to this oops I've went for bisect.

Thanks a lot for reporting, and going to such effort to find
a reproducible testcase that you could bisect on.

> 
> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> 
> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE

But this leaves me very puzzled.

Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
under tests/ ?

I see no mention of madvise or MADV_REMOVE or fallocate or anything
related in that git tree.

If you have something else running at the same time, which happens to use
madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
it on (I guess ext4 from the =y in your config), then I suppose we should
start searching for improper memory freeing or scribbling in its holepunch
support: something that might be corrupting the dm_region in your oops.

I'll be surprised if that is the case, but it's something that you can
easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
that should tell us what process is using it.

I'm not an LVM user, so I doubt I'll be able to reproduce your setup.

Any ideas from the DM guys?  Has anyone else seen anything like this?

Do all your oopses look like this one?

Thanks,
Hugh

> 
> My HW - t61, 4GB
> 
> Zdenek
> 
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
> IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
> PGD 0
> Oops: 0000 [#1] PREEMPT SMP
> CPU 1
> Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq
> async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror
> dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables
> ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables
> x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945
> iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel
> snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e
> snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc
> loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd
> sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core
> yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight
> 
> Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO
> 6464CTO/6464CTO
> RIP: 0010:[<ffffffffa031204d>]  [<ffffffffa031204d>]
> dm_rh_region_context+0xd/0x20 [dm_region_hash]
> RSP: 0018:ffff880126b77c68  EFLAGS: 00010282
> RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607
> RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40
> RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000
> R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400
> R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task
> ffff8801344ca310)
> Stack:
>  ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0
>  ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000
>  0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0
> Call Trace:
>  [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror]
>  [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror]
>  [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod]
>  [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod]
>  [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod]
>  [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod]
>  [<ffffffff81060065>] process_one_work+0x1a5/0x6b0
>  [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0
>  [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80
>  [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod]
>  [<ffffffff810608ee>] worker_thread+0x12e/0x2d0
>  [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210
>  [<ffffffff81069cf2>] kthread+0xa2/0xb0
>  [<ffffffff81533334>] kernel_thread_helper+0x4/0x10
>  [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120
>  [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60
>  [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe
>  [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0
>  [<ffffffff81533330>] ? gs_change+0xb/0xb
> Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8
> c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00
> 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
> RIP  [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash]
>  RSP <ffff880126b77c68>
> CR2: 0000000000000178
> 
> 
> # bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2
> # good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4
> git bisect start 'v3.5-rc2' 'v3.4'
> # good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge'
> of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux
> git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232
> # good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic
> strnlen_user() function
> git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879
> # bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch
> 'for-3.5/drivers' of git://git.kernel.dk/linux-block
> git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6
> # good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1'
> of git://git.linux-nfs.org/projects/trondmy/linux-nfs
> git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e
> # bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of
> git://git.linaro.org/people/rmk/linux-arm
> git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9
> # bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump
> PG_compound_lock page flag
> git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26
> # good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should
> not be user visible
> git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca
> # good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug
> fs entries on failure of file creation and made extfrag_debug_root dentry
> local
> git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729
> # good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove
> redundant offset check when finally freeing bootmem
> git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a
> # good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if
> mapping excludes its zone
> git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605
> # bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove
> truncate_range
> git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811
> # good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing
> when writing
> git bisect good ec9516fbc5fa814014991e1ae7f8860127122105

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-06-30 19:55   ` [linux-lvm] " Hugh Dickins
@ 2012-06-30 20:51     ` Zdenek Kabelac
  -1 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-06-30 20:51 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>
>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>> Ooops log included. After some experimenting - reliable way to hit this oops
>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
>> included anything related to this oops I've went for bisect.
>
> Thanks a lot for reporting, and going to such effort to find
> a reproducible testcase that you could bisect on.
>
>>
>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>
>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>
> But this leaves me very puzzled.
>
> Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
> under tests/ ?

Yes - that's it -

  make
as root:
  cd test
  make check_local

(inside test subdirectory should be enough, if not - just report any problem)

>
> I see no mention of madvise or MADV_REMOVE or fallocate or anything
> related in that git tree.
>
> If you have something else running at the same time, which happens to use
> madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
> it on (I guess ext4 from the =y in your config), then I suppose we should
> start searching for improper memory freeing or scribbling in its holepunch
> support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in  LVM_TEST_DIR (default is /tmp)
and using loop device to simulate device (small size - it should fit bellow 200MB)

Within this file second layer through virtual DM devices is created and 
simulates various numbers of PV devices to play with.

So since everything now support TRIM - such operations should be passed
down to the backend file - which probably triggers the path.

> I'll be surprised if that is the case, but it's something that you can
> easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
> that should tell us what process is using it.

I could try that if that will help.

> I'm not an LVM user, so I doubt I'll be able to reproduce your setup.

Shouldn't be hard to run - unsure if every config setup is influnenced
or just mine config.

>
> Any ideas from the DM guys?  Has anyone else seen anything like this?
>
> Do all your oopses look like this one?

I think I've get yet another one - but also within  dm_rh_region

It could be that your patch exposed problem of some different part of stack - 
not really sure - it's just now with 3.5  this crash will not allow to pass 
whole test suite  -  I've tried also in kvm machine and it's been reproducible 
(so in the worst case I could eventually send you 2GB image)

The problem is - there is not a 'single test case' to trigger the oops (at 
least I've not figured out one)  - it's the combination of multiple tests 
running after each other - but for simplication this should be enough:

make check_local T=shell/lvconvert

Which usually dies on shell/lvconvert-repair-transient.sh

Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-06-30 20:51     ` Zdenek Kabelac
  0 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-06-30 20:51 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: linux-kernel, amwang, Alasdair G Kergon,
	LVM general discussion and development

Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>
>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>> Ooops log included. After some experimenting - reliable way to hit this oops
>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
>> included anything related to this oops I've went for bisect.
>
> Thanks a lot for reporting, and going to such effort to find
> a reproducible testcase that you could bisect on.
>
>>
>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>
>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>
> But this leaves me very puzzled.
>
> Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
> under tests/ ?

Yes - that's it -

  make
as root:
  cd test
  make check_local

(inside test subdirectory should be enough, if not - just report any problem)

>
> I see no mention of madvise or MADV_REMOVE or fallocate or anything
> related in that git tree.
>
> If you have something else running at the same time, which happens to use
> madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
> it on (I guess ext4 from the =y in your config), then I suppose we should
> start searching for improper memory freeing or scribbling in its holepunch
> support: something that might be corrupting the dm_region in your oops.

What the test is doing - it creates file in  LVM_TEST_DIR (default is /tmp)
and using loop device to simulate device (small size - it should fit bellow 200MB)

Within this file second layer through virtual DM devices is created and 
simulates various numbers of PV devices to play with.

So since everything now support TRIM - such operations should be passed
down to the backend file - which probably triggers the path.

> I'll be surprised if that is the case, but it's something that you can
> easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
> that should tell us what process is using it.

I could try that if that will help.

> I'm not an LVM user, so I doubt I'll be able to reproduce your setup.

Shouldn't be hard to run - unsure if every config setup is influnenced
or just mine config.

>
> Any ideas from the DM guys?  Has anyone else seen anything like this?
>
> Do all your oopses look like this one?

I think I've get yet another one - but also within  dm_rh_region

It could be that your patch exposed problem of some different part of stack - 
not really sure - it's just now with 3.5  this crash will not allow to pass 
whole test suite  -  I've tried also in kvm machine and it's been reproducible 
(so in the worst case I could eventually send you 2GB image)

The problem is - there is not a 'single test case' to trigger the oops (at 
least I've not figured out one)  - it's the combination of multiple tests 
running after each other - but for simplication this should be enough:

make check_local T=shell/lvconvert

Which usually dies on shell/lvconvert-repair-transient.sh

Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-06-30 20:51     ` [linux-lvm] " Zdenek Kabelac
@ 2012-06-30 23:10       ` Hugh Dickins
  -1 siblings, 0 replies; 26+ messages in thread
From: Hugh Dickins @ 2012-06-30 23:10 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel

On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > 
> > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > Ooops log included. After some experimenting - reliable way to hit this
> > > oops
> > > is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
> > > included anything related to this oops I've went for bisect.
> > 
> > Thanks a lot for reporting, and going to such effort to find
> > a reproducible testcase that you could bisect on.
> > 
> > > 
> > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > 
> > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > 
> > But this leaves me very puzzled.
> > 
> > Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
> > under tests/ ?
> 
> Yes - that's it -
> 
>  make
> as root:
>  cd test
>  make check_local
> 
> (inside test subdirectory should be enough, if not - just report any problem)
> 
> > 
> > I see no mention of madvise or MADV_REMOVE or fallocate or anything
> > related in that git tree.
> > 
> > If you have something else running at the same time, which happens to use
> > madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
> > it on (I guess ext4 from the =y in your config), then I suppose we should
> > start searching for improper memory freeing or scribbling in its holepunch
> > support: something that might be corrupting the dm_region in your oops.
> 
> What the test is doing - it creates file in  LVM_TEST_DIR (default is /tmp)
> and using loop device to simulate device (small size - it should fit bellow
> 200MB)
> 
> Within this file second layer through virtual DM devices is created and
> simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

> 
> So since everything now support TRIM - such operations should be passed
> down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

If tmpfs, then it will make much more sense if we assume your bisection
endpoint was off by one.  Your bisection log was not quite complete;
and even if it did appear to converge on the commit you cite, you might
have got (un)lucky when testing the commit before it, and concluded
"good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
"tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by
fallocate - that indeed played a part in my own testing.

Whereas if your /tmp is on ext4, loop has been passing TRIM down
with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
should be completely irrelevant.

> 
> > I'll be surprised if that is the case, but it's something that you can
> > easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
> > that should tell us what process is using it.
> 
> I could try that if that will help.

That would help, if you're very sure of your bisection endpoint;
but if your /tmp is on tmpfs, then I do think it's more likely
that you've actually found a bug in the commit before.

> 
> > I'm not an LVM user, so I doubt I'll be able to reproduce your setup.
> 
> Shouldn't be hard to run - unsure if every config setup is influnenced
> or just mine config.

I'll start from your config.

> 
> > 
> > Any ideas from the DM guys?  Has anyone else seen anything like this?
> > 
> > Do all your oopses look like this one?
> 
> I think I've get yet another one - but also within  dm_rh_region
> 
> It could be that your patch exposed problem of some different part of stack -
> not really sure - it's just now with 3.5  this crash will not allow to pass
> whole test suite  -  I've tried also in kvm machine and it's been
> reproducible (so in the worst case I could eventually send you 2GB image)
> 
> The problem is - there is not a 'single test case' to trigger the oops (at
> least I've not figured out one)  - it's the combination of multiple tests
> running after each other - but for simplication this should be enough:
> 
> make check_local T=shell/lvconvert
> 
> Which usually dies on shell/lvconvert-repair-transient.sh

Thanks again, I'll report back later.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-06-30 23:10       ` Hugh Dickins
  0 siblings, 0 replies; 26+ messages in thread
From: Hugh Dickins @ 2012-06-30 23:10 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: linux-kernel, amwang, Alasdair G Kergon,
	LVM general discussion and development

On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > 
> > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > Ooops log included. After some experimenting - reliable way to hit this
> > > oops
> > > is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
> > > included anything related to this oops I've went for bisect.
> > 
> > Thanks a lot for reporting, and going to such effort to find
> > a reproducible testcase that you could bisect on.
> > 
> > > 
> > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > 
> > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > 
> > But this leaves me very puzzled.
> > 
> > Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
> > under tests/ ?
> 
> Yes - that's it -
> 
>  make
> as root:
>  cd test
>  make check_local
> 
> (inside test subdirectory should be enough, if not - just report any problem)
> 
> > 
> > I see no mention of madvise or MADV_REMOVE or fallocate or anything
> > related in that git tree.
> > 
> > If you have something else running at the same time, which happens to use
> > madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
> > it on (I guess ext4 from the =y in your config), then I suppose we should
> > start searching for improper memory freeing or scribbling in its holepunch
> > support: something that might be corrupting the dm_region in your oops.
> 
> What the test is doing - it creates file in  LVM_TEST_DIR (default is /tmp)
> and using loop device to simulate device (small size - it should fit bellow
> 200MB)
> 
> Within this file second layer through virtual DM devices is created and
> simulates various numbers of PV devices to play with.

This sounds much easier to set up than I was expecting:
thanks for the info, I'll try it later on today.

> 
> So since everything now support TRIM - such operations should be passed
> down to the backend file - which probably triggers the path.

What filesystem do you have for /tmp?

If tmpfs, then it will make much more sense if we assume your bisection
endpoint was off by one.  Your bisection log was not quite complete;
and even if it did appear to converge on the commit you cite, you might
have got (un)lucky when testing the commit before it, and concluded
"good" when more attempts would have said "bad".

The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
"tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
much more likely first bad commit if your /tmp is on tmpfs:
that does indeed wire up loop to pass TRIM down to tmpfs by
fallocate - that indeed played a part in my own testing.

Whereas if your /tmp is on ext4, loop has been passing TRIM down
with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
should be completely irrelevant.

> 
> > I'll be surprised if that is the case, but it's something that you can
> > easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
> > that should tell us what process is using it.
> 
> I could try that if that will help.

That would help, if you're very sure of your bisection endpoint;
but if your /tmp is on tmpfs, then I do think it's more likely
that you've actually found a bug in the commit before.

> 
> > I'm not an LVM user, so I doubt I'll be able to reproduce your setup.
> 
> Shouldn't be hard to run - unsure if every config setup is influnenced
> or just mine config.

I'll start from your config.

> 
> > 
> > Any ideas from the DM guys?  Has anyone else seen anything like this?
> > 
> > Do all your oopses look like this one?
> 
> I think I've get yet another one - but also within  dm_rh_region
> 
> It could be that your patch exposed problem of some different part of stack -
> not really sure - it's just now with 3.5  this crash will not allow to pass
> whole test suite  -  I've tried also in kvm machine and it's been
> reproducible (so in the worst case I could eventually send you 2GB image)
> 
> The problem is - there is not a 'single test case' to trigger the oops (at
> least I've not figured out one)  - it's the combination of multiple tests
> running after each other - but for simplication this should be enough:
> 
> make check_local T=shell/lvconvert
> 
> Which usually dies on shell/lvconvert-repair-transient.sh

Thanks again, I'll report back later.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-06-30 23:10       ` [linux-lvm] " Hugh Dickins
@ 2012-07-01 12:57         ` Zdenek Kabelac
  -1 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-01 12:57 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel, Lukas Czerner

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>
>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>>>> Ooops log included. After some experimenting - reliable way to hit this
>>>> oops
>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
>>>> included anything related to this oops I've went for bisect.
>>>
>>> Thanks a lot for reporting, and going to such effort to find
>>> a reproducible testcase that you could bisect on.
>>>
>>>>
>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>>>
>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>>>
>>> But this leaves me very puzzled.
>>>
>>> Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
>>> under tests/ ?
>>
>> Yes - that's it -
>>
>>   make
>> as root:
>>   cd test
>>   make check_local
>>
>> (inside test subdirectory should be enough, if not - just report any problem)
>>
>>> If you have something else running at the same time, which happens to use
>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
>>> it on (I guess ext4 from the =y in your config), then I suppose we should
>>> start searching for improper memory freeing or scribbling in its holepunch
>>> support: something that might be corrupting the dm_region in your oops.
>>
>> What the test is doing - it creates file in  LVM_TEST_DIR (default is /tmp)
>> and using loop device to simulate device (small size - it should fit bellow
>> 200MB)
>>
>> Within this file second layer through virtual DM devices is created and
>> simulates various numbers of PV devices to play with.
>
> This sounds much easier to set up than I was expecting:
> thanks for the info, I'll try it later on today.
>
>>
>> So since everything now support TRIM - such operations should be passed
>> down to the backend file - which probably triggers the path.
>
> What filesystem do you have for /tmp?
>
> If tmpfs, then it will make much more sense if we assume your bisection
> endpoint was off by one.  Your bisection log was not quite complete;
> and even if it did appear to converge on the commit you cite, you might
> have got (un)lucky when testing the commit before it, and concluded
> "good" when more attempts would have said "bad".
>
> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> much more likely first bad commit if your /tmp is on tmpfs:
> that does indeed wire up loop to pass TRIM down to tmpfs by
> fallocate - that indeed played a part in my own testing.
>
> Whereas if your /tmp is on ext4, loop has been passing TRIM down
> with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> should be completely irrelevant.

While I've been aware of the fact that tmpfs was enhanced with trim support - 
I've not tried to run on real ext4 filesystem since for my tests I'm using 
tmpfs for quite some time to safe rewrites of SSD :)

So now I've checked with real ext4 - and the bug is there as well
so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

And you are correct, the first commit which causes crash really is
83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've 
missed to properly identify this commit in my bisect is that crash usually 
happens on the second pass of the lvm test suite 'make check_local' execution 
- and I've been running test just once. To be sure I've run 5 run passes on 
3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing 
usually on second run, with commit 3f31d07571e  the crash always happens in 
the first pass.

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
and it's crashing as well - so it's probably not uniqueness of my config.

So is there any primary suspect in 3.2 which is worth to check - or I need 
another day to play another bisect game ?

>
>>
>>> I'll be surprised if that is the case, but it's something that you can
>>> easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
>>> that should tell us what process is using it.
>>
>> I could try that if that will help.
>
> That would help, if you're very sure of your bisection endpoint;
> but if your /tmp is on tmpfs, then I do think it's more likely
> that you've actually found a bug in the commit before.

the only thing which could be tricky is  udev support
(by default it's not enabled ---enable-udev_sync)
However Debian based distros are distributing their own rules, which are not 
100% compatible with upstream and create some unpredictable issues,
where the slowness is the least problem.)

If you have Fedora Rawhide with latest lvm2 installed - you should get pretty 
well configured system for running test dir   (unfortunately there is no way 
to virtualize udev...)

Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-01 12:57         ` Zdenek Kabelac
  0 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-01 12:57 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: linux-kernel, Lukas Czerner, amwang, Alasdair G Kergon,
	LVM general discussion and development

Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>
>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>>>> Ooops log included. After some experimenting - reliable way to hit this
>>>> oops
>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not
>>>> included anything related to this oops I've went for bisect.
>>>
>>> Thanks a lot for reporting, and going to such effort to find
>>> a reproducible testcase that you could bisect on.
>>>
>>>>
>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>>>
>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>>>
>>> But this leaves me very puzzled.
>>>
>>> Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git
>>> under tests/ ?
>>
>> Yes - that's it -
>>
>>   make
>> as root:
>>   cd test
>>   make check_local
>>
>> (inside test subdirectory should be enough, if not - just report any problem)
>>
>>> If you have something else running at the same time, which happens to use
>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables
>>> it on (I guess ext4 from the =y in your config), then I suppose we should
>>> start searching for improper memory freeing or scribbling in its holepunch
>>> support: something that might be corrupting the dm_region in your oops.
>>
>> What the test is doing - it creates file in  LVM_TEST_DIR (default is /tmp)
>> and using loop device to simulate device (small size - it should fit bellow
>> 200MB)
>>
>> Within this file second layer through virtual DM devices is created and
>> simulates various numbers of PV devices to play with.
>
> This sounds much easier to set up than I was expecting:
> thanks for the info, I'll try it later on today.
>
>>
>> So since everything now support TRIM - such operations should be passed
>> down to the backend file - which probably triggers the path.
>
> What filesystem do you have for /tmp?
>
> If tmpfs, then it will make much more sense if we assume your bisection
> endpoint was off by one.  Your bisection log was not quite complete;
> and even if it did appear to converge on the commit you cite, you might
> have got (un)lucky when testing the commit before it, and concluded
> "good" when more attempts would have said "bad".
>
> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> much more likely first bad commit if your /tmp is on tmpfs:
> that does indeed wire up loop to pass TRIM down to tmpfs by
> fallocate - that indeed played a part in my own testing.
>
> Whereas if your /tmp is on ext4, loop has been passing TRIM down
> with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> should be completely irrelevant.

While I've been aware of the fact that tmpfs was enhanced with trim support - 
I've not tried to run on real ext4 filesystem since for my tests I'm using 
tmpfs for quite some time to safe rewrites of SSD :)

So now I've checked with real ext4 - and the bug is there as well
so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.

3.1 is the first kernel which does survive (checked 5 repeated runs)

And you are correct, the first commit which causes crash really is
83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've 
missed to properly identify this commit in my bisect is that crash usually 
happens on the second pass of the lvm test suite 'make check_local' execution 
- and I've been running test just once. To be sure I've run 5 run passes on 
3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing 
usually on second run, with commit 3f31d07571e  the crash always happens in 
the first pass.

I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
and it's crashing as well - so it's probably not uniqueness of my config.

So is there any primary suspect in 3.2 which is worth to check - or I need 
another day to play another bisect game ?

>
>>
>>> I'll be surprised if that is the case, but it's something that you can
>>> easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove():
>>> that should tell us what process is using it.
>>
>> I could try that if that will help.
>
> That would help, if you're very sure of your bisection endpoint;
> but if your /tmp is on tmpfs, then I do think it's more likely
> that you've actually found a bug in the commit before.

the only thing which could be tricky is  udev support
(by default it's not enabled ---enable-udev_sync)
However Debian based distros are distributing their own rules, which are not 
100% compatible with upstream and create some unpredictable issues,
where the slowness is the least problem.)

If you have Fedora Rawhide with latest lvm2 installed - you should get pretty 
well configured system for running test dir   (unfortunately there is no way 
to virtualize udev...)

Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-01 12:57         ` [linux-lvm] " Zdenek Kabelac
@ 2012-07-01 18:45           ` Hugh Dickins
  -1 siblings, 0 replies; 26+ messages in thread
From: Hugh Dickins @ 2012-07-01 18:45 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: Mike Snitzer, Mikulas Patocka, Joe Thornber,
	LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel, Lukas Czerner

On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
> Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > 
> > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > > > Ooops log included. After some experimenting - reliable way to hit
> > > > > this
> > > > > oops
> > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window does
> > > > > not
> > > > > included anything related to this oops I've went for bisect.
> > > > 
> > > > Thanks a lot for reporting, and going to such effort to find
> > > > a reproducible testcase that you could bisect on.
> > > > 
> > > > > 
> > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > > > 
> > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > > > 
> > > > But this leaves me very puzzled.
> > > > 
> > > > Is the "lvm test suite" what I find at
> > > > git.fedorahosted.org/git/lvm2.git
> > > > under tests/ ?
> > > 
> > > Yes - that's it -
> > > 
> > >   make
> > > as root:
> > >   cd test
> > >   make check_local
> > > 
> > > (inside test subdirectory should be enough, if not - just report any
> > > problem)
> > > 
> > > > If you have something else running at the same time, which happens to
> > > > use
> > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now
> > > > enables
> > > > it on (I guess ext4 from the =y in your config), then I suppose we
> > > > should
> > > > start searching for improper memory freeing or scribbling in its
> > > > holepunch
> > > > support: something that might be corrupting the dm_region in your oops.
> > > 
> > > What the test is doing - it creates file in  LVM_TEST_DIR (default is
> > > /tmp)

I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.

And being a year or two out of date in my userspace, and unfamiliar with
the syntax and whereabouts of lvm.conf, it was easiest for me to hack
lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
after I spotted a warning message about issue_discards.

> > > and using loop device to simulate device (small size - it should fit
> > > bellow
> > > 200MB)
> > > 
> > > Within this file second layer through virtual DM devices is created and
> > > simulates various numbers of PV devices to play with.
> > 
> > This sounds much easier to set up than I was expecting:
> > thanks for the info, I'll try it later on today.

Sorry, I never reached it yesterday, but arrived there this morning.

> > 
> > > 
> > > So since everything now support TRIM - such operations should be passed
> > > down to the backend file - which probably triggers the path.
> > 
> > What filesystem do you have for /tmp?

>From your later remarks, I inferred tmpfs.

> > 
> > If tmpfs, then it will make much more sense if we assume your bisection
> > endpoint was off by one.  Your bisection log was not quite complete;
> > and even if it did appear to converge on the commit you cite, you might
> > have got (un)lucky when testing the commit before it, and concluded
> > "good" when more attempts would have said "bad".
> > 
> > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> > much more likely first bad commit if your /tmp is on tmpfs:
> > that does indeed wire up loop to pass TRIM down to tmpfs by
> > fallocate - that indeed played a part in my own testing.
> > 
> > Whereas if your /tmp is on ext4, loop has been passing TRIM down
> > with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> > should be completely irrelevant.
> 
> While I've been aware of the fact that tmpfs was enhanced with trim support -
> I've not tried to run on real ext4 filesystem since for my tests I'm using
> tmpfs for quite some time to safe rewrites of SSD :)
> 
> So now I've checked with real ext4 - and the bug is there as well
> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
> 
> 3.1 is the first kernel which does survive (checked 5 repeated runs)

Very useful research, thank you.

> 
> And you are correct, the first commit which causes crash really is
> 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've
> missed to properly identify this commit in my bisect is that crash usually
> happens on the second pass of the lvm test suite 'make check_local' execution
> - and I've been running test just once. To be sure I've run 5 run passes on
> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
> usually on second run, with commit 3f31d07571e  the crash always happens in
> the first pass.
> 
> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
> and it's crashing as well - so it's probably not uniqueness of my config.
> 
> So is there any primary suspect in 3.2 which is worth to check - or I need
> another day to play another bisect game ?

No need for a further bisect: 3.2 is when the disard/trim/fallocate
support went into drivers/block/loop.c, so these tests would be unable
to show if DM was right or wrong before then.

> 
> > 
> > > 
> > > > I'll be surprised if that is the case, but it's something that you can
> > > > easily check by inserting a WARN_ON(1) in mm/madvise.c
> > > > madvise_remove():
> > > > that should tell us what process is using it.
> > > 
> > > I could try that if that will help.
> > 
> > That would help, if you're very sure of your bisection endpoint;
> > but if your /tmp is on tmpfs, then I do think it's more likely
> > that you've actually found a bug in the commit before.
> 
> the only thing which could be tricky is  udev support
> (by default it's not enabled ---enable-udev_sync)
> However Debian based distros are distributing their own rules, which are not
> 100% compatible with upstream and create some unpredictable issues,
> where the slowness is the least problem.)
> 
> If you have Fedora Rawhide with latest lvm2 installed - you should get pretty
> well configured system for running test dir   (unfortunately there is no way
> to virtualize udev...)

I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
I did quickly crash around where you expected; though in my case
it was in dm_rh_dec() called from mirror_end_io().

But I've not taken it any further than that.  You've shown that it's
as much a problem with ext4 as with tmpfs, and has been a problem
ever since these tests' use of discard reached DM.

I think it's fair to assume that there's something wrong with DM's
handling of REQ_DISCARD.  (Perhaps it was all correct when written,
but I think there has been a series of modifications to REQ_DISCARD
handling in the block layer, it's been rather troublesome.)

I've added a few people to the Cc, innocents fingered by git blame
on some REQ_DISCARD lines in drivers/md.  But I'll back out myself:
I'm not short of other things to look at, and have little to offer
in drivers/md.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-01 18:45           ` Hugh Dickins
  0 siblings, 0 replies; 26+ messages in thread
From: Hugh Dickins @ 2012-07-01 18:45 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: amwang, Mike Snitzer, linux-kernel, Joe Thornber,
	LVM general discussion and development, Lukas Czerner,
	Alasdair G Kergon

On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
> Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > 
> > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > > > Ooops log included. After some experimenting - reliable way to hit
> > > > > this
> > > > > oops
> > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window does
> > > > > not
> > > > > included anything related to this oops I've went for bisect.
> > > > 
> > > > Thanks a lot for reporting, and going to such effort to find
> > > > a reproducible testcase that you could bisect on.
> > > > 
> > > > > 
> > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > > > 
> > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > > > 
> > > > But this leaves me very puzzled.
> > > > 
> > > > Is the "lvm test suite" what I find at
> > > > git.fedorahosted.org/git/lvm2.git
> > > > under tests/ ?
> > > 
> > > Yes - that's it -
> > > 
> > >   make
> > > as root:
> > >   cd test
> > >   make check_local
> > > 
> > > (inside test subdirectory should be enough, if not - just report any
> > > problem)
> > > 
> > > > If you have something else running at the same time, which happens to
> > > > use
> > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now
> > > > enables
> > > > it on (I guess ext4 from the =y in your config), then I suppose we
> > > > should
> > > > start searching for improper memory freeing or scribbling in its
> > > > holepunch
> > > > support: something that might be corrupting the dm_region in your oops.
> > > 
> > > What the test is doing - it creates file in  LVM_TEST_DIR (default is
> > > /tmp)

I ran "LVM_TEST_DIR=/tmp make check_local":
without that it appeared to be using a subdirectory made under test/.

And being a year or two out of date in my userspace, and unfamiliar with
the syntax and whereabouts of lvm.conf, it was easiest for me to hack
lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
after I spotted a warning message about issue_discards.

> > > and using loop device to simulate device (small size - it should fit
> > > bellow
> > > 200MB)
> > > 
> > > Within this file second layer through virtual DM devices is created and
> > > simulates various numbers of PV devices to play with.
> > 
> > This sounds much easier to set up than I was expecting:
> > thanks for the info, I'll try it later on today.

Sorry, I never reached it yesterday, but arrived there this morning.

> > 
> > > 
> > > So since everything now support TRIM - such operations should be passed
> > > down to the backend file - which probably triggers the path.
> > 
> > What filesystem do you have for /tmp?

From your later remarks, I inferred tmpfs.

> > 
> > If tmpfs, then it will make much more sense if we assume your bisection
> > endpoint was off by one.  Your bisection log was not quite complete;
> > and even if it did appear to converge on the commit you cite, you might
> > have got (un)lucky when testing the commit before it, and concluded
> > "good" when more attempts would have said "bad".
> > 
> > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> > much more likely first bad commit if your /tmp is on tmpfs:
> > that does indeed wire up loop to pass TRIM down to tmpfs by
> > fallocate - that indeed played a part in my own testing.
> > 
> > Whereas if your /tmp is on ext4, loop has been passing TRIM down
> > with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> > should be completely irrelevant.
> 
> While I've been aware of the fact that tmpfs was enhanced with trim support -
> I've not tried to run on real ext4 filesystem since for my tests I'm using
> tmpfs for quite some time to safe rewrites of SSD :)
> 
> So now I've checked with real ext4 - and the bug is there as well
> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
> 
> 3.1 is the first kernel which does survive (checked 5 repeated runs)

Very useful research, thank you.

> 
> And you are correct, the first commit which causes crash really is
> 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've
> missed to properly identify this commit in my bisect is that crash usually
> happens on the second pass of the lvm test suite 'make check_local' execution
> - and I've been running test just once. To be sure I've run 5 run passes on
> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
> usually on second run, with commit 3f31d07571e  the crash always happens in
> the first pass.
> 
> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
> and it's crashing as well - so it's probably not uniqueness of my config.
> 
> So is there any primary suspect in 3.2 which is worth to check - or I need
> another day to play another bisect game ?

No need for a further bisect: 3.2 is when the disard/trim/fallocate
support went into drivers/block/loop.c, so these tests would be unable
to show if DM was right or wrong before then.

> 
> > 
> > > 
> > > > I'll be surprised if that is the case, but it's something that you can
> > > > easily check by inserting a WARN_ON(1) in mm/madvise.c
> > > > madvise_remove():
> > > > that should tell us what process is using it.
> > > 
> > > I could try that if that will help.
> > 
> > That would help, if you're very sure of your bisection endpoint;
> > but if your /tmp is on tmpfs, then I do think it's more likely
> > that you've actually found a bug in the commit before.
> 
> the only thing which could be tricky is  udev support
> (by default it's not enabled ---enable-udev_sync)
> However Debian based distros are distributing their own rules, which are not
> 100% compatible with upstream and create some unpredictable issues,
> where the slowness is the least problem.)
> 
> If you have Fedora Rawhide with latest lvm2 installed - you should get pretty
> well configured system for running test dir   (unfortunately there is no way
> to virtualize udev...)

I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
I did quickly crash around where you expected; though in my case
it was in dm_rh_dec() called from mirror_end_io().

But I've not taken it any further than that.  You've shown that it's
as much a problem with ext4 as with tmpfs, and has been a problem
ever since these tests' use of discard reached DM.

I think it's fair to assume that there's something wrong with DM's
handling of REQ_DISCARD.  (Perhaps it was all correct when written,
but I think there has been a series of modifications to REQ_DISCARD
handling in the block layer, it's been rather troublesome.)

I've added a few people to the Cc, innocents fingered by git blame
on some REQ_DISCARD lines in drivers/md.  But I'll back out myself:
I'm not short of other things to look at, and have little to offer
in drivers/md.

Hugh

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-01 18:45           ` [linux-lvm] " Hugh Dickins
@ 2012-07-01 20:10             ` Zdenek Kabelac
  -1 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-01 20:10 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Mike Snitzer, Mikulas Patocka, Joe Thornber,
	LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel, Lukas Czerner

Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
> On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
>> Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
>>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>>>
>>>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>>>>>> Ooops log included. After some experimenting - reliable way to hit
>>>>>> this
>>>>>> oops
>>>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does
>>>>>> not
>>>>>> included anything related to this oops I've went for bisect.
>>>>>
>>>>> Thanks a lot for reporting, and going to such effort to find
>>>>> a reproducible testcase that you could bisect on.
>>>>>
>>>>>>
>>>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>>>>>
>>>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>>>>>
>>>>> But this leaves me very puzzled.
>>>>>
>>>>> Is the "lvm test suite" what I find at
>>>>> git.fedorahosted.org/git/lvm2.git
>>>>> under tests/ ?
>>>>
>>>> Yes - that's it -
>>>>
>>>>    make
>>>> as root:
>>>>    cd test
>>>>    make check_local
>>>>
>>>> (inside test subdirectory should be enough, if not - just report any
>>>> problem)
>>>>
>>>>> If you have something else running at the same time, which happens to
>>>>> use
>>>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now
>>>>> enables
>>>>> it on (I guess ext4 from the =y in your config), then I suppose we
>>>>> should
>>>>> start searching for improper memory freeing or scribbling in its
>>>>> holepunch
>>>>> support: something that might be corrupting the dm_region in your oops.
>>>>
>>>> What the test is doing - it creates file in  LVM_TEST_DIR (default is
>>>> /tmp)
>
> I ran "LVM_TEST_DIR=/tmp make check_local":
> without that it appeared to be using a subdirectory made under test/.
>
> And being a year or two out of date in my userspace, and unfamiliar with
> the syntax and whereabouts of lvm.conf, it was easiest for me to hack
> lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
> after I spotted a warning message about issue_discards.
>
>>>> and using loop device to simulate device (small size - it should fit
>>>> bellow
>>>> 200MB)
>>>>
>>>> Within this file second layer through virtual DM devices is created and
>>>> simulates various numbers of PV devices to play with.
>>>
>>> This sounds much easier to set up than I was expecting:
>>> thanks for the info, I'll try it later on today.
>
> Sorry, I never reached it yesterday, but arrived there this morning.
>
>>>
>>>>
>>>> So since everything now support TRIM - such operations should be passed
>>>> down to the backend file - which probably triggers the path.
>>>
>>> What filesystem do you have for /tmp?
>
>  From your later remarks, I inferred tmpfs.
>
>>>
>>> If tmpfs, then it will make much more sense if we assume your bisection
>>> endpoint was off by one.  Your bisection log was not quite complete;
>>> and even if it did appear to converge on the commit you cite, you might
>>> have got (un)lucky when testing the commit before it, and concluded
>>> "good" when more attempts would have said "bad".
>>>
>>> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
>>> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
>>> much more likely first bad commit if your /tmp is on tmpfs:
>>> that does indeed wire up loop to pass TRIM down to tmpfs by
>>> fallocate - that indeed played a part in my own testing.
>>>
>>> Whereas if your /tmp is on ext4, loop has been passing TRIM down
>>> with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
>>> should be completely irrelevant.
>>
>> While I've been aware of the fact that tmpfs was enhanced with trim support -
>> I've not tried to run on real ext4 filesystem since for my tests I'm using
>> tmpfs for quite some time to safe rewrites of SSD :)
>>
>> So now I've checked with real ext4 - and the bug is there as well
>> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
>>
>> 3.1 is the first kernel which does survive (checked 5 repeated runs)
>
> Very useful research, thank you.
>
>>
>> And you are correct, the first commit which causes crash really is
>> 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've
>> missed to properly identify this commit in my bisect is that crash usually
>> happens on the second pass of the lvm test suite 'make check_local' execution
>> - and I've been running test just once. To be sure I've run 5 run passes on
>> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
>> usually on second run, with commit 3f31d07571e  the crash always happens in
>> the first pass.
>>
>> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
>> and it's crashing as well - so it's probably not uniqueness of my config.
>>
>> So is there any primary suspect in 3.2 which is worth to check - or I need
>> another day to play another bisect game ?
>
> No need for a further bisect: 3.2 is when the disard/trim/fallocate
> support went into drivers/block/loop.c, so these tests would be unable
> to show if DM was right or wrong before then.

Well I've played meanwhile the game with minimized kernel config
and the outcome is:

last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef

dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devices

> I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
> I did quickly crash around where you expected; though in my case
> it was in dm_rh_dec() called from mirror_end_io().

Change of Issue Discards option in lvm.conf is not needed.
I'm able to get these oopses with this setting turned off.

> But I've not taken it any further than that.  You've shown that it's
> as much a problem with ext4 as with tmpfs, and has been a problem
> ever since these tests' use of discard reached DM.
>
> I think it's fair to assume that there's something wrong with DM's
> handling of REQ_DISCARD.  (Perhaps it was all correct when written,
> but I think there has been a series of modifications to REQ_DISCARD
> handling in the block layer, it's been rather troublesome.)
>

So does anyone has some idea what should be checked next ?

Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-01 20:10             ` Zdenek Kabelac
  0 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-01 20:10 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: amwang, Mike Snitzer, linux-kernel, Joe Thornber,
	LVM general discussion and development, Lukas Czerner,
	Alasdair G Kergon

Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
> On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
>> Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
>>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>>>
>>>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>>>>>> Ooops log included. After some experimenting - reliable way to hit
>>>>>> this
>>>>>> oops
>>>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does
>>>>>> not
>>>>>> included anything related to this oops I've went for bisect.
>>>>>
>>>>> Thanks a lot for reporting, and going to such effort to find
>>>>> a reproducible testcase that you could bisect on.
>>>>>
>>>>>>
>>>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>>>>>
>>>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>>>>>
>>>>> But this leaves me very puzzled.
>>>>>
>>>>> Is the "lvm test suite" what I find at
>>>>> git.fedorahosted.org/git/lvm2.git
>>>>> under tests/ ?
>>>>
>>>> Yes - that's it -
>>>>
>>>>    make
>>>> as root:
>>>>    cd test
>>>>    make check_local
>>>>
>>>> (inside test subdirectory should be enough, if not - just report any
>>>> problem)
>>>>
>>>>> If you have something else running at the same time, which happens to
>>>>> use
>>>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now
>>>>> enables
>>>>> it on (I guess ext4 from the =y in your config), then I suppose we
>>>>> should
>>>>> start searching for improper memory freeing or scribbling in its
>>>>> holepunch
>>>>> support: something that might be corrupting the dm_region in your oops.
>>>>
>>>> What the test is doing - it creates file in  LVM_TEST_DIR (default is
>>>> /tmp)
>
> I ran "LVM_TEST_DIR=/tmp make check_local":
> without that it appeared to be using a subdirectory made under test/.
>
> And being a year or two out of date in my userspace, and unfamiliar with
> the syntax and whereabouts of lvm.conf, it was easiest for me to hack
> lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
> after I spotted a warning message about issue_discards.
>
>>>> and using loop device to simulate device (small size - it should fit
>>>> bellow
>>>> 200MB)
>>>>
>>>> Within this file second layer through virtual DM devices is created and
>>>> simulates various numbers of PV devices to play with.
>>>
>>> This sounds much easier to set up than I was expecting:
>>> thanks for the info, I'll try it later on today.
>
> Sorry, I never reached it yesterday, but arrived there this morning.
>
>>>
>>>>
>>>> So since everything now support TRIM - such operations should be passed
>>>> down to the backend file - which probably triggers the path.
>>>
>>> What filesystem do you have for /tmp?
>
>  From your later remarks, I inferred tmpfs.
>
>>>
>>> If tmpfs, then it will make much more sense if we assume your bisection
>>> endpoint was off by one.  Your bisection log was not quite complete;
>>> and even if it did appear to converge on the commit you cite, you might
>>> have got (un)lucky when testing the commit before it, and concluded
>>> "good" when more attempts would have said "bad".
>>>
>>> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
>>> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
>>> much more likely first bad commit if your /tmp is on tmpfs:
>>> that does indeed wire up loop to pass TRIM down to tmpfs by
>>> fallocate - that indeed played a part in my own testing.
>>>
>>> Whereas if your /tmp is on ext4, loop has been passing TRIM down
>>> with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
>>> should be completely irrelevant.
>>
>> While I've been aware of the fact that tmpfs was enhanced with trim support -
>> I've not tried to run on real ext4 filesystem since for my tests I'm using
>> tmpfs for quite some time to safe rewrites of SSD :)
>>
>> So now I've checked with real ext4 - and the bug is there as well
>> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
>>
>> 3.1 is the first kernel which does survive (checked 5 repeated runs)
>
> Very useful research, thank you.
>
>>
>> And you are correct, the first commit which causes crash really is
>> 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've
>> missed to properly identify this commit in my bisect is that crash usually
>> happens on the second pass of the lvm test suite 'make check_local' execution
>> - and I've been running test just once. To be sure I've run 5 run passes on
>> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
>> usually on second run, with commit 3f31d07571e  the crash always happens in
>> the first pass.
>>
>> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
>> and it's crashing as well - so it's probably not uniqueness of my config.
>>
>> So is there any primary suspect in 3.2 which is worth to check - or I need
>> another day to play another bisect game ?
>
> No need for a further bisect: 3.2 is when the disard/trim/fallocate
> support went into drivers/block/loop.c, so these tests would be unable
> to show if DM was right or wrong before then.

Well I've played meanwhile the game with minimized kernel config
and the outcome is:

last working kernel is: 3.1.0-rc1-00008-g548ef6c
first broken: 3.1.0-rc1-00009-gdfaa2ef

dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
loop: add discard support for loop devices

> I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
> I did quickly crash around where you expected; though in my case
> it was in dm_rh_dec() called from mirror_end_io().

Change of Issue Discards option in lvm.conf is not needed.
I'm able to get these oopses with this setting turned off.

> But I've not taken it any further than that.  You've shown that it's
> as much a problem with ext4 as with tmpfs, and has been a problem
> ever since these tests' use of discard reached DM.
>
> I think it's fair to assume that there's something wrong with DM's
> handling of REQ_DISCARD.  (Perhaps it was all correct when written,
> but I think there has been a series of modifications to REQ_DISCARD
> handling in the block layer, it's been rather troublesome.)
>

So does anyone has some idea what should be checked next ?

Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-01 20:10             ` [linux-lvm] " Zdenek Kabelac
@ 2012-07-01 22:03               ` Zdenek Kabelac
  -1 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-01 22:03 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Mike Snitzer, Mikulas Patocka, Joe Thornber,
	LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel, Lukas Czerner

Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
> Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
>> On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
>>> Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
>>>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>>>>
>>>>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>>>>>>> Ooops log included. After some experimenting - reliable way to hit
>>>>>>> this
>>>>>>> oops
>>>>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does
>>>>>>> not
>>>>>>> included anything related to this oops I've went for bisect.
>>>>>>
>>>>>> Thanks a lot for reporting, and going to such effort to find
>>>>>> a reproducible testcase that you could bisect on.
>>>>>>
>>>>>>>
>>>>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>>>>>>
>>>>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>>>>>>
>>>>>> But this leaves me very puzzled.
>>>>>>
>>>>>> Is the "lvm test suite" what I find at
>>>>>> git.fedorahosted.org/git/lvm2.git
>>>>>> under tests/ ?
>>>>>
>>>>> Yes - that's it -
>>>>>
>>>>>    make
>>>>> as root:
>>>>>    cd test
>>>>>    make check_local
>>>>>
>>>>> (inside test subdirectory should be enough, if not - just report any
>>>>> problem)
>>>>>
>>>>>> If you have something else running at the same time, which happens to
>>>>>> use
>>>>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now
>>>>>> enables
>>>>>> it on (I guess ext4 from the =y in your config), then I suppose we
>>>>>> should
>>>>>> start searching for improper memory freeing or scribbling in its
>>>>>> holepunch
>>>>>> support: something that might be corrupting the dm_region in your oops.
>>>>>
>>>>> What the test is doing - it creates file in  LVM_TEST_DIR (default is
>>>>> /tmp)
>>
>> I ran "LVM_TEST_DIR=/tmp make check_local":
>> without that it appeared to be using a subdirectory made under test/.
>>
>> And being a year or two out of date in my userspace, and unfamiliar with
>> the syntax and whereabouts of lvm.conf, it was easiest for me to hack
>> lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
>> after I spotted a warning message about issue_discards.
>>
>>>>> and using loop device to simulate device (small size - it should fit
>>>>> bellow
>>>>> 200MB)
>>>>>
>>>>> Within this file second layer through virtual DM devices is created and
>>>>> simulates various numbers of PV devices to play with.
>>>>
>>>> This sounds much easier to set up than I was expecting:
>>>> thanks for the info, I'll try it later on today.
>>
>> Sorry, I never reached it yesterday, but arrived there this morning.
>>
>>>>
>>>>>
>>>>> So since everything now support TRIM - such operations should be passed
>>>>> down to the backend file - which probably triggers the path.
>>>>
>>>> What filesystem do you have for /tmp?
>>
>>  From your later remarks, I inferred tmpfs.
>>
>>>>
>>>> If tmpfs, then it will make much more sense if we assume your bisection
>>>> endpoint was off by one.  Your bisection log was not quite complete;
>>>> and even if it did appear to converge on the commit you cite, you might
>>>> have got (un)lucky when testing the commit before it, and concluded
>>>> "good" when more attempts would have said "bad".
>>>>
>>>> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
>>>> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
>>>> much more likely first bad commit if your /tmp is on tmpfs:
>>>> that does indeed wire up loop to pass TRIM down to tmpfs by
>>>> fallocate - that indeed played a part in my own testing.
>>>>
>>>> Whereas if your /tmp is on ext4, loop has been passing TRIM down
>>>> with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
>>>> should be completely irrelevant.
>>>
>>> While I've been aware of the fact that tmpfs was enhanced with trim support -
>>> I've not tried to run on real ext4 filesystem since for my tests I'm using
>>> tmpfs for quite some time to safe rewrites of SSD :)
>>>
>>> So now I've checked with real ext4 - and the bug is there as well
>>> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
>>>
>>> 3.1 is the first kernel which does survive (checked 5 repeated runs)
>>
>> Very useful research, thank you.
>>
>>>
>>> And you are correct, the first commit which causes crash really is
>>> 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've
>>> missed to properly identify this commit in my bisect is that crash usually
>>> happens on the second pass of the lvm test suite 'make check_local' execution
>>> - and I've been running test just once. To be sure I've run 5 run passes on
>>> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
>>> usually on second run, with commit 3f31d07571e  the crash always happens in
>>> the first pass.
>>>
>>> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
>>> and it's crashing as well - so it's probably not uniqueness of my config.
>>>
>>> So is there any primary suspect in 3.2 which is worth to check - or I need
>>> another day to play another bisect game ?
>>
>> No need for a further bisect: 3.2 is when the disard/trim/fallocate
>> support went into drivers/block/loop.c, so these tests would be unable
>> to show if DM was right or wrong before then.
>
> Well I've played meanwhile the game with minimized kernel config
> and the outcome is:
>
> last working kernel is: 3.1.0-rc1-00008-g548ef6c
> first broken: 3.1.0-rc1-00009-gdfaa2ef
>
> dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
> loop: add discard support for loop devices
>
>> I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
>> I did quickly crash around where you expected; though in my case
>> it was in dm_rh_dec() called from mirror_end_io().
>
> Change of Issue Discards option in lvm.conf is not needed.
> I'm able to get these oopses with this setting turned off.
>
>> But I've not taken it any further than that.  You've shown that it's
>> as much a problem with ext4 as with tmpfs, and has been a problem
>> ever since these tests' use of discard reached DM.
>>
>> I think it's fair to assume that there's something wrong with DM's
>> handling of REQ_DISCARD.  (Perhaps it was all correct when written,
>> but I think there has been a series of modifications to REQ_DISCARD
>> handling in the block layer, it's been rather troublesome.)
>>
>
> So does anyone has some idea what should be checked next ?


I've enabled more kernel debugging tools - and this came out when running test 
suite:

Unsure if this could be source of troubles?

kernel BUG at arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU 1
Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp 
llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 
iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4

Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO 
6464CTO/6464CTO
RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090)
Stack:
  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
Call Trace:
  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
  [<ffffffff81050177>] process_one_work+0x197/0x460
  [<ffffffff81050116>] ? process_one_work+0x136/0x460
  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
  [<ffffffff81050986>] worker_thread+0x126/0x2d0
  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
  [<ffffffff81056066>] kthread+0x96/0xa0
  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
  [<ffffffff8133b610>] ? gs_change+0xb/0xb
Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00 
78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 
0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
  RSP <ffff880136e41aa0>
---[ end trace 040dfd9d68fccabc ]---



Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-01 22:03               ` Zdenek Kabelac
  0 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-01 22:03 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: amwang, Mike Snitzer, linux-kernel, Joe Thornber,
	LVM general discussion and development, Lukas Czerner,
	Alasdair G Kergon

Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
> Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
>> On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
>>> Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
>>>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
>>>>>>>
>>>>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
>>>>>>> Ooops log included. After some experimenting - reliable way to hit
>>>>>>> this
>>>>>>> oops
>>>>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does
>>>>>>> not
>>>>>>> included anything related to this oops I've went for bisect.
>>>>>>
>>>>>> Thanks a lot for reporting, and going to such effort to find
>>>>>> a reproducible testcase that you could bisect on.
>>>>>>
>>>>>>>
>>>>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
>>>>>>>
>>>>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
>>>>>>
>>>>>> But this leaves me very puzzled.
>>>>>>
>>>>>> Is the "lvm test suite" what I find at
>>>>>> git.fedorahosted.org/git/lvm2.git
>>>>>> under tests/ ?
>>>>>
>>>>> Yes - that's it -
>>>>>
>>>>>    make
>>>>> as root:
>>>>>    cd test
>>>>>    make check_local
>>>>>
>>>>> (inside test subdirectory should be enough, if not - just report any
>>>>> problem)
>>>>>
>>>>>> If you have something else running at the same time, which happens to
>>>>>> use
>>>>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now
>>>>>> enables
>>>>>> it on (I guess ext4 from the =y in your config), then I suppose we
>>>>>> should
>>>>>> start searching for improper memory freeing or scribbling in its
>>>>>> holepunch
>>>>>> support: something that might be corrupting the dm_region in your oops.
>>>>>
>>>>> What the test is doing - it creates file in  LVM_TEST_DIR (default is
>>>>> /tmp)
>>
>> I ran "LVM_TEST_DIR=/tmp make check_local":
>> without that it appeared to be using a subdirectory made under test/.
>>
>> And being a year or two out of date in my userspace, and unfamiliar with
>> the syntax and whereabouts of lvm.conf, it was easiest for me to hack
>> lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
>> after I spotted a warning message about issue_discards.
>>
>>>>> and using loop device to simulate device (small size - it should fit
>>>>> bellow
>>>>> 200MB)
>>>>>
>>>>> Within this file second layer through virtual DM devices is created and
>>>>> simulates various numbers of PV devices to play with.
>>>>
>>>> This sounds much easier to set up than I was expecting:
>>>> thanks for the info, I'll try it later on today.
>>
>> Sorry, I never reached it yesterday, but arrived there this morning.
>>
>>>>
>>>>>
>>>>> So since everything now support TRIM - such operations should be passed
>>>>> down to the backend file - which probably triggers the path.
>>>>
>>>> What filesystem do you have for /tmp?
>>
>>  From your later remarks, I inferred tmpfs.
>>
>>>>
>>>> If tmpfs, then it will make much more sense if we assume your bisection
>>>> endpoint was off by one.  Your bisection log was not quite complete;
>>>> and even if it did appear to converge on the commit you cite, you might
>>>> have got (un)lucky when testing the commit before it, and concluded
>>>> "good" when more attempts would have said "bad".
>>>>
>>>> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
>>>> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
>>>> much more likely first bad commit if your /tmp is on tmpfs:
>>>> that does indeed wire up loop to pass TRIM down to tmpfs by
>>>> fallocate - that indeed played a part in my own testing.
>>>>
>>>> Whereas if your /tmp is on ext4, loop has been passing TRIM down
>>>> with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
>>>> should be completely irrelevant.
>>>
>>> While I've been aware of the fact that tmpfs was enhanced with trim support -
>>> I've not tried to run on real ext4 filesystem since for my tests I'm using
>>> tmpfs for quite some time to safe rewrites of SSD :)
>>>
>>> So now I've checked with real ext4 - and the bug is there as well
>>> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
>>>
>>> 3.1 is the first kernel which does survive (checked 5 repeated runs)
>>
>> Very useful research, thank you.
>>
>>>
>>> And you are correct, the first commit which causes crash really is
>>> 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why I've
>>> missed to properly identify this commit in my bisect is that crash usually
>>> happens on the second pass of the lvm test suite 'make check_local' execution
>>> - and I've been running test just once. To be sure I've run 5 run passes on
>>> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
>>> usually on second run, with commit 3f31d07571e  the crash always happens in
>>> the first pass.
>>>
>>> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
>>> and it's crashing as well - so it's probably not uniqueness of my config.
>>>
>>> So is there any primary suspect in 3.2 which is worth to check - or I need
>>> another day to play another bisect game ?
>>
>> No need for a further bisect: 3.2 is when the disard/trim/fallocate
>> support went into drivers/block/loop.c, so these tests would be unable
>> to show if DM was right or wrong before then.
>
> Well I've played meanwhile the game with minimized kernel config
> and the outcome is:
>
> last working kernel is: 3.1.0-rc1-00008-g548ef6c
> first broken: 3.1.0-rc1-00009-gdfaa2ef
>
> dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
> loop: add discard support for loop devices
>
>> I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
>> I did quickly crash around where you expected; though in my case
>> it was in dm_rh_dec() called from mirror_end_io().
>
> Change of Issue Discards option in lvm.conf is not needed.
> I'm able to get these oopses with this setting turned off.
>
>> But I've not taken it any further than that.  You've shown that it's
>> as much a problem with ext4 as with tmpfs, and has been a problem
>> ever since these tests' use of discard reached DM.
>>
>> I think it's fair to assume that there's something wrong with DM's
>> handling of REQ_DISCARD.  (Perhaps it was all correct when written,
>> but I think there has been a series of modifications to REQ_DISCARD
>> handling in the block layer, it's been rather troublesome.)
>>
>
> So does anyone has some idea what should be checked next ?


I've enabled more kernel debugging tools - and this came out when running test 
suite:

Unsure if this could be source of troubles?

kernel BUG at arch/x86/mm/physaddr.c:18!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU 1
Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp 
llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 
iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4

Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO 
6464CTO/6464CTO
RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090)
Stack:
  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
Call Trace:
  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
  [<ffffffff81050177>] process_one_work+0x197/0x460
  [<ffffffff81050116>] ? process_one_work+0x136/0x460
  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
  [<ffffffff81050986>] worker_thread+0x126/0x2d0
  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
  [<ffffffff81056066>] kthread+0x96/0xa0
  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
  [<ffffffff8133b610>] ? gs_change+0xb/0xb
Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00 
78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 
0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
  RSP <ffff880136e41aa0>
---[ end trace 040dfd9d68fccabc ]---



Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-01 22:03               ` [linux-lvm] " Zdenek Kabelac
@ 2012-07-02  9:00                 ` Lukáš Czerner
  -1 siblings, 0 replies; 26+ messages in thread
From: Lukáš Czerner @ 2012-07-02  9:00 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: Hugh Dickins, Mike Snitzer, Mikulas Patocka, Joe Thornber,
	LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel, Lukas Czerner

On Mon, 2 Jul 2012, Zdenek Kabelac wrote:

> Date: Mon, 02 Jul 2012 00:03:07 +0200
> From: Zdenek Kabelac <zkabelac@redhat.com>
> To: Hugh Dickins <hughd@google.com>
> Cc: Mike Snitzer <snitzer@redhat.com>, Mikulas Patocka <mpatocka@redhat.com>,
>     Joe Thornber <ejt@redhat.com>,
>     LVM general discussion and development <linux-lvm@redhat.com>,
>     amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>,
>     linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com>
> Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
> 
> Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
> > Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
> > > On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
> > > > Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > > > 
> > > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > > > > > > Ooops log included. After some experimenting - reliable way to
> > > > > > > > hit
> > > > > > > > this
> > > > > > > > oops
> > > > > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window
> > > > > > > > does
> > > > > > > > not
> > > > > > > > included anything related to this oops I've went for bisect.
> > > > > > > 
> > > > > > > Thanks a lot for reporting, and going to such effort to find
> > > > > > > a reproducible testcase that you could bisect on.
> > > > > > > 
> > > > > > > > 
> > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > > > > > > 
> > > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > > > > > > 
> > > > > > > But this leaves me very puzzled.
> > > > > > > 
> > > > > > > Is the "lvm test suite" what I find at
> > > > > > > git.fedorahosted.org/git/lvm2.git
> > > > > > > under tests/ ?
> > > > > > 
> > > > > > Yes - that's it -
> > > > > > 
> > > > > >    make
> > > > > > as root:
> > > > > >    cd test
> > > > > >    make check_local
> > > > > > 
> > > > > > (inside test subdirectory should be enough, if not - just report any
> > > > > > problem)
> > > > > > 
> > > > > > > If you have something else running at the same time, which happens
> > > > > > > to
> > > > > > > use
> > > > > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now
> > > > > > > enables
> > > > > > > it on (I guess ext4 from the =y in your config), then I suppose we
> > > > > > > should
> > > > > > > start searching for improper memory freeing or scribbling in its
> > > > > > > holepunch
> > > > > > > support: something that might be corrupting the dm_region in your
> > > > > > > oops.
> > > > > > 
> > > > > > What the test is doing - it creates file in  LVM_TEST_DIR (default
> > > > > > is
> > > > > > /tmp)
> > > 
> > > I ran "LVM_TEST_DIR=/tmp make check_local":
> > > without that it appeared to be using a subdirectory made under test/.
> > > 
> > > And being a year or two out of date in my userspace, and unfamiliar with
> > > the syntax and whereabouts of lvm.conf, it was easiest for me to hack
> > > lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
> > > after I spotted a warning message about issue_discards.
> > > 
> > > > > > and using loop device to simulate device (small size - it should fit
> > > > > > bellow
> > > > > > 200MB)
> > > > > > 
> > > > > > Within this file second layer through virtual DM devices is created
> > > > > > and
> > > > > > simulates various numbers of PV devices to play with.
> > > > > 
> > > > > This sounds much easier to set up than I was expecting:
> > > > > thanks for the info, I'll try it later on today.
> > > 
> > > Sorry, I never reached it yesterday, but arrived there this morning.
> > > 
> > > > > 
> > > > > > 
> > > > > > So since everything now support TRIM - such operations should be
> > > > > > passed
> > > > > > down to the backend file - which probably triggers the path.
> > > > > 
> > > > > What filesystem do you have for /tmp?
> > > 
> > >  From your later remarks, I inferred tmpfs.
> > > 
> > > > > 
> > > > > If tmpfs, then it will make much more sense if we assume your
> > > > > bisection
> > > > > endpoint was off by one.  Your bisection log was not quite complete;
> > > > > and even if it did appear to converge on the commit you cite, you
> > > > > might
> > > > > have got (un)lucky when testing the commit before it, and concluded
> > > > > "good" when more attempts would have said "bad".
> > > > > 
> > > > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> > > > > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> > > > > much more likely first bad commit if your /tmp is on tmpfs:
> > > > > that does indeed wire up loop to pass TRIM down to tmpfs by
> > > > > fallocate - that indeed played a part in my own testing.
> > > > > 
> > > > > Whereas if your /tmp is on ext4, loop has been passing TRIM down
> > > > > with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> > > > > should be completely irrelevant.
> > > > 
> > > > While I've been aware of the fact that tmpfs was enhanced with trim
> > > > support -
> > > > I've not tried to run on real ext4 filesystem since for my tests I'm
> > > > using
> > > > tmpfs for quite some time to safe rewrites of SSD :)
> > > > 
> > > > So now I've checked with real ext4 - and the bug is there as well
> > > > so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
> > > > 
> > > > 3.1 is the first kernel which does survive (checked 5 repeated runs)
> > > 
> > > Very useful research, thank you.
> > > 
> > > > 
> > > > And you are correct, the first commit which causes crash really is
> > > > 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why
> > > > I've
> > > > missed to properly identify this commit in my bisect is that crash
> > > > usually
> > > > happens on the second pass of the lvm test suite 'make check_local'
> > > > execution
> > > > - and I've been running test just once. To be sure I've run 5 run passes
> > > > on
> > > > 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
> > > > usually on second run, with commit 3f31d07571e  the crash always happens
> > > > in
> > > > the first pass.
> > > > 
> > > > I've also checked some rawhide kernel
> > > > vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
> > > > and it's crashing as well - so it's probably not uniqueness of my
> > > > config.
> > > > 
> > > > So is there any primary suspect in 3.2 which is worth to check - or I
> > > > need
> > > > another day to play another bisect game ?
> > > 
> > > No need for a further bisect: 3.2 is when the disard/trim/fallocate
> > > support went into drivers/block/loop.c, so these tests would be unable
> > > to show if DM was right or wrong before then.
> > 
> > Well I've played meanwhile the game with minimized kernel config
> > and the outcome is:
> > 
> > last working kernel is: 3.1.0-rc1-00008-g548ef6c
> > first broken: 3.1.0-rc1-00009-gdfaa2ef
> > 
> > dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
> > loop: add discard support for loop devices
> > 
> > > I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
> > > I did quickly crash around where you expected; though in my case
> > > it was in dm_rh_dec() called from mirror_end_io().
> > 
> > Change of Issue Discards option in lvm.conf is not needed.
> > I'm able to get these oopses with this setting turned off.
> > 
> > > But I've not taken it any further than that.  You've shown that it's
> > > as much a problem with ext4 as with tmpfs, and has been a problem
> > > ever since these tests' use of discard reached DM.
> > > 
> > > I think it's fair to assume that there's something wrong with DM's
> > > handling of REQ_DISCARD.  (Perhaps it was all correct when written,
> > > but I think there has been a series of modifications to REQ_DISCARD
> > > handling in the block layer, it's been rather troublesome.)
> > > 

Hi Zdenek,

> > 
> > So does anyone has some idea what should be checked next ?

I've missed the information in the thread, but since you're the most
familiar with the lvm test suite, could you distil the reproducer
which does not require to run lvm test suit ?


> 
> 
> I've enabled more kernel debugging tools - and this came out when running test
> suite:
> 
> Unsure if this could be source of troubles?
> 
> kernel BUG at arch/x86/mm/physaddr.c:18!
> invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> CPU 1
> Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp
> llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945
> iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4
> 
> Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO

So you're testing rather old kernel so you might be missing some
fixes there. Could you rerun the test with the recent kernel ?

Also it appears that the bug here happens because dm requested a
destination page which is within the kernel space. It seems that
this has been initiated by the write request from the mirror target.
So I do not immediately see how punch hole (discard) is involved at
all. You might have been lucky enough to hit a different bug
probably ?

Looking at git log, this commit has been brought to my attention:

0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support

seems related to this crash.

Please retest with recent kernel.

Thanks!
-Lukas

> 6464CTO/6464CTO
> RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
> RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
> RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
> RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
> R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
> FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task
> ffff880136e3e090)
> Stack:
>  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
>  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
>  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
> Call Trace:
>  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
>  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
>  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
>  [<ffffffff81050177>] process_one_work+0x197/0x460
>  [<ffffffff81050116>] ? process_one_work+0x136/0x460
>  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
>  [<ffffffff81050986>] worker_thread+0x126/0x2d0
>  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
>  [<ffffffff81056066>] kthread+0x96/0xa0
>  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
>  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
>  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
>  [<ffffffff8133b610>] ? gs_change+0xb/0xb
> Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00
> 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e
> 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
> RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
>  RSP <ffff880136e41aa0>
> ---[ end trace 040dfd9d68fccabc ]---
> 
> 
> 
> Zdenek
> 

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-02  9:00                 ` Lukáš Czerner
  0 siblings, 0 replies; 26+ messages in thread
From: Lukáš Czerner @ 2012-07-02  9:00 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: amwang, Mike Snitzer, Hugh Dickins, linux-kernel, Joe Thornber,
	LVM general discussion and development, Lukas Czerner,
	Alasdair G Kergon

On Mon, 2 Jul 2012, Zdenek Kabelac wrote:

> Date: Mon, 02 Jul 2012 00:03:07 +0200
> From: Zdenek Kabelac <zkabelac@redhat.com>
> To: Hugh Dickins <hughd@google.com>
> Cc: Mike Snitzer <snitzer@redhat.com>, Mikulas Patocka <mpatocka@redhat.com>,
>     Joe Thornber <ejt@redhat.com>,
>     LVM general discussion and development <linux-lvm@redhat.com>,
>     amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>,
>     linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com>
> Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
> 
> Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
> > Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
> > > On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
> > > > Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > > > 
> > > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > > > > > > Ooops log included. After some experimenting - reliable way to
> > > > > > > > hit
> > > > > > > > this
> > > > > > > > oops
> > > > > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window
> > > > > > > > does
> > > > > > > > not
> > > > > > > > included anything related to this oops I've went for bisect.
> > > > > > > 
> > > > > > > Thanks a lot for reporting, and going to such effort to find
> > > > > > > a reproducible testcase that you could bisect on.
> > > > > > > 
> > > > > > > > 
> > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > > > > > > 
> > > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > > > > > > 
> > > > > > > But this leaves me very puzzled.
> > > > > > > 
> > > > > > > Is the "lvm test suite" what I find at
> > > > > > > git.fedorahosted.org/git/lvm2.git
> > > > > > > under tests/ ?
> > > > > > 
> > > > > > Yes - that's it -
> > > > > > 
> > > > > >    make
> > > > > > as root:
> > > > > >    cd test
> > > > > >    make check_local
> > > > > > 
> > > > > > (inside test subdirectory should be enough, if not - just report any
> > > > > > problem)
> > > > > > 
> > > > > > > If you have something else running at the same time, which happens
> > > > > > > to
> > > > > > > use
> > > > > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now
> > > > > > > enables
> > > > > > > it on (I guess ext4 from the =y in your config), then I suppose we
> > > > > > > should
> > > > > > > start searching for improper memory freeing or scribbling in its
> > > > > > > holepunch
> > > > > > > support: something that might be corrupting the dm_region in your
> > > > > > > oops.
> > > > > > 
> > > > > > What the test is doing - it creates file in  LVM_TEST_DIR (default
> > > > > > is
> > > > > > /tmp)
> > > 
> > > I ran "LVM_TEST_DIR=/tmp make check_local":
> > > without that it appeared to be using a subdirectory made under test/.
> > > 
> > > And being a year or two out of date in my userspace, and unfamiliar with
> > > the syntax and whereabouts of lvm.conf, it was easiest for me to hack
> > > lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
> > > after I spotted a warning message about issue_discards.
> > > 
> > > > > > and using loop device to simulate device (small size - it should fit
> > > > > > bellow
> > > > > > 200MB)
> > > > > > 
> > > > > > Within this file second layer through virtual DM devices is created
> > > > > > and
> > > > > > simulates various numbers of PV devices to play with.
> > > > > 
> > > > > This sounds much easier to set up than I was expecting:
> > > > > thanks for the info, I'll try it later on today.
> > > 
> > > Sorry, I never reached it yesterday, but arrived there this morning.
> > > 
> > > > > 
> > > > > > 
> > > > > > So since everything now support TRIM - such operations should be
> > > > > > passed
> > > > > > down to the backend file - which probably triggers the path.
> > > > > 
> > > > > What filesystem do you have for /tmp?
> > > 
> > >  From your later remarks, I inferred tmpfs.
> > > 
> > > > > 
> > > > > If tmpfs, then it will make much more sense if we assume your
> > > > > bisection
> > > > > endpoint was off by one.  Your bisection log was not quite complete;
> > > > > and even if it did appear to converge on the commit you cite, you
> > > > > might
> > > > > have got (un)lucky when testing the commit before it, and concluded
> > > > > "good" when more attempts would have said "bad".
> > > > > 
> > > > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> > > > > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> > > > > much more likely first bad commit if your /tmp is on tmpfs:
> > > > > that does indeed wire up loop to pass TRIM down to tmpfs by
> > > > > fallocate - that indeed played a part in my own testing.
> > > > > 
> > > > > Whereas if your /tmp is on ext4, loop has been passing TRIM down
> > > > > with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> > > > > should be completely irrelevant.
> > > > 
> > > > While I've been aware of the fact that tmpfs was enhanced with trim
> > > > support -
> > > > I've not tried to run on real ext4 filesystem since for my tests I'm
> > > > using
> > > > tmpfs for quite some time to safe rewrites of SSD :)
> > > > 
> > > > So now I've checked with real ext4 - and the bug is there as well
> > > > so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
> > > > 
> > > > 3.1 is the first kernel which does survive (checked 5 repeated runs)
> > > 
> > > Very useful research, thank you.
> > > 
> > > > 
> > > > And you are correct, the first commit which causes crash really is
> > > > 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why
> > > > I've
> > > > missed to properly identify this commit in my bisect is that crash
> > > > usually
> > > > happens on the second pass of the lvm test suite 'make check_local'
> > > > execution
> > > > - and I've been running test just once. To be sure I've run 5 run passes
> > > > on
> > > > 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
> > > > usually on second run, with commit 3f31d07571e  the crash always happens
> > > > in
> > > > the first pass.
> > > > 
> > > > I've also checked some rawhide kernel
> > > > vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
> > > > and it's crashing as well - so it's probably not uniqueness of my
> > > > config.
> > > > 
> > > > So is there any primary suspect in 3.2 which is worth to check - or I
> > > > need
> > > > another day to play another bisect game ?
> > > 
> > > No need for a further bisect: 3.2 is when the disard/trim/fallocate
> > > support went into drivers/block/loop.c, so these tests would be unable
> > > to show if DM was right or wrong before then.
> > 
> > Well I've played meanwhile the game with minimized kernel config
> > and the outcome is:
> > 
> > last working kernel is: 3.1.0-rc1-00008-g548ef6c
> > first broken: 3.1.0-rc1-00009-gdfaa2ef
> > 
> > dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
> > loop: add discard support for loop devices
> > 
> > > I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
> > > I did quickly crash around where you expected; though in my case
> > > it was in dm_rh_dec() called from mirror_end_io().
> > 
> > Change of Issue Discards option in lvm.conf is not needed.
> > I'm able to get these oopses with this setting turned off.
> > 
> > > But I've not taken it any further than that.  You've shown that it's
> > > as much a problem with ext4 as with tmpfs, and has been a problem
> > > ever since these tests' use of discard reached DM.
> > > 
> > > I think it's fair to assume that there's something wrong with DM's
> > > handling of REQ_DISCARD.  (Perhaps it was all correct when written,
> > > but I think there has been a series of modifications to REQ_DISCARD
> > > handling in the block layer, it's been rather troublesome.)
> > > 

Hi Zdenek,

> > 
> > So does anyone has some idea what should be checked next ?

I've missed the information in the thread, but since you're the most
familiar with the lvm test suite, could you distil the reproducer
which does not require to run lvm test suit ?


> 
> 
> I've enabled more kernel debugging tools - and this came out when running test
> suite:
> 
> Unsure if this could be source of troubles?
> 
> kernel BUG at arch/x86/mm/physaddr.c:18!
> invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> CPU 1
> Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp
> llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945
> iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4
> 
> Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO

So you're testing rather old kernel so you might be missing some
fixes there. Could you rerun the test with the recent kernel ?

Also it appears that the bug here happens because dm requested a
destination page which is within the kernel space. It seems that
this has been initiated by the write request from the mirror target.
So I do not immediately see how punch hole (discard) is involved at
all. You might have been lucky enough to hit a different bug
probably ?

Looking at git log, this commit has been brought to my attention:

0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support

seems related to this crash.

Please retest with recent kernel.

Thanks!
-Lukas

> 6464CTO/6464CTO
> RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
> RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
> RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
> RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
> R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
> FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task
> ffff880136e3e090)
> Stack:
>  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
>  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
>  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
> Call Trace:
>  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
>  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
>  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
>  [<ffffffff81050177>] process_one_work+0x197/0x460
>  [<ffffffff81050116>] ? process_one_work+0x136/0x460
>  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
>  [<ffffffff81050986>] worker_thread+0x126/0x2d0
>  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
>  [<ffffffff81056066>] kthread+0x96/0xa0
>  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
>  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
>  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
>  [<ffffffff8133b610>] ? gs_change+0xb/0xb
> Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00
> 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e
> 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
> RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
>  RSP <ffff880136e41aa0>
> ---[ end trace 040dfd9d68fccabc ]---
> 
> 
> 
> Zdenek
> 

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-02  9:00                 ` [linux-lvm] " Lukáš Czerner
@ 2012-07-02 10:35                   ` Lukáš Czerner
  -1 siblings, 0 replies; 26+ messages in thread
From: Lukáš Czerner @ 2012-07-02 10:35 UTC (permalink / raw)
  To: Lukáš Czerner
  Cc: Zdenek Kabelac, Hugh Dickins, Mike Snitzer, Mikulas Patocka,
	Joe Thornber, LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel

[-- Attachment #1: Type: TEXT/PLAIN, Size: 14535 bytes --]

On Mon, 2 Jul 2012, Lukáš Czerner wrote:

> Date: Mon, 2 Jul 2012 11:00:06 +0200 (CEST)
> From: Lukáš Czerner <lczerner@redhat.com>
> To: Zdenek Kabelac <zkabelac@redhat.com>
> Cc: Hugh Dickins <hughd@google.com>, Mike Snitzer <snitzer@redhat.com>,
>     Mikulas Patocka <mpatocka@redhat.com>, Joe Thornber <ejt@redhat.com>,
>     LVM general discussion and development <linux-lvm@redhat.com>,
>     amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>,
>     linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com>
> Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
> 
> On Mon, 2 Jul 2012, Zdenek Kabelac wrote:
> 
> > Date: Mon, 02 Jul 2012 00:03:07 +0200
> > From: Zdenek Kabelac <zkabelac@redhat.com>
> > To: Hugh Dickins <hughd@google.com>
> > Cc: Mike Snitzer <snitzer@redhat.com>, Mikulas Patocka <mpatocka@redhat.com>,
> >     Joe Thornber <ejt@redhat.com>,
> >     LVM general discussion and development <linux-lvm@redhat.com>,
> >     amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>,
> >     linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com>
> > Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
> > 
> > Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
> > > Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
> > > > On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
> > > > > Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > > > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > > > > 
> > > > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > > > > > > > Ooops log included. After some experimenting - reliable way to
> > > > > > > > > hit
> > > > > > > > > this
> > > > > > > > > oops
> > > > > > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window
> > > > > > > > > does
> > > > > > > > > not
> > > > > > > > > included anything related to this oops I've went for bisect.
> > > > > > > > 
> > > > > > > > Thanks a lot for reporting, and going to such effort to find
> > > > > > > > a reproducible testcase that you could bisect on.
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > > > > > > > 
> > > > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > > > > > > > 
> > > > > > > > But this leaves me very puzzled.
> > > > > > > > 
> > > > > > > > Is the "lvm test suite" what I find at
> > > > > > > > git.fedorahosted.org/git/lvm2.git
> > > > > > > > under tests/ ?
> > > > > > > 
> > > > > > > Yes - that's it -
> > > > > > > 
> > > > > > >    make
> > > > > > > as root:
> > > > > > >    cd test
> > > > > > >    make check_local
> > > > > > > 
> > > > > > > (inside test subdirectory should be enough, if not - just report any
> > > > > > > problem)
> > > > > > > 
> > > > > > > > If you have something else running at the same time, which happens
> > > > > > > > to
> > > > > > > > use
> > > > > > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now
> > > > > > > > enables
> > > > > > > > it on (I guess ext4 from the =y in your config), then I suppose we
> > > > > > > > should
> > > > > > > > start searching for improper memory freeing or scribbling in its
> > > > > > > > holepunch
> > > > > > > > support: something that might be corrupting the dm_region in your
> > > > > > > > oops.
> > > > > > > 
> > > > > > > What the test is doing - it creates file in  LVM_TEST_DIR (default
> > > > > > > is
> > > > > > > /tmp)
> > > > 
> > > > I ran "LVM_TEST_DIR=/tmp make check_local":
> > > > without that it appeared to be using a subdirectory made under test/.
> > > > 
> > > > And being a year or two out of date in my userspace, and unfamiliar with
> > > > the syntax and whereabouts of lvm.conf, it was easiest for me to hack
> > > > lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
> > > > after I spotted a warning message about issue_discards.
> > > > 
> > > > > > > and using loop device to simulate device (small size - it should fit
> > > > > > > bellow
> > > > > > > 200MB)
> > > > > > > 
> > > > > > > Within this file second layer through virtual DM devices is created
> > > > > > > and
> > > > > > > simulates various numbers of PV devices to play with.
> > > > > > 
> > > > > > This sounds much easier to set up than I was expecting:
> > > > > > thanks for the info, I'll try it later on today.
> > > > 
> > > > Sorry, I never reached it yesterday, but arrived there this morning.
> > > > 
> > > > > > 
> > > > > > > 
> > > > > > > So since everything now support TRIM - such operations should be
> > > > > > > passed
> > > > > > > down to the backend file - which probably triggers the path.
> > > > > > 
> > > > > > What filesystem do you have for /tmp?
> > > > 
> > > >  From your later remarks, I inferred tmpfs.
> > > > 
> > > > > > 
> > > > > > If tmpfs, then it will make much more sense if we assume your
> > > > > > bisection
> > > > > > endpoint was off by one.  Your bisection log was not quite complete;
> > > > > > and even if it did appear to converge on the commit you cite, you
> > > > > > might
> > > > > > have got (un)lucky when testing the commit before it, and concluded
> > > > > > "good" when more attempts would have said "bad".
> > > > > > 
> > > > > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> > > > > > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> > > > > > much more likely first bad commit if your /tmp is on tmpfs:
> > > > > > that does indeed wire up loop to pass TRIM down to tmpfs by
> > > > > > fallocate - that indeed played a part in my own testing.
> > > > > > 
> > > > > > Whereas if your /tmp is on ext4, loop has been passing TRIM down
> > > > > > with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> > > > > > should be completely irrelevant.
> > > > > 
> > > > > While I've been aware of the fact that tmpfs was enhanced with trim
> > > > > support -
> > > > > I've not tried to run on real ext4 filesystem since for my tests I'm
> > > > > using
> > > > > tmpfs for quite some time to safe rewrites of SSD :)
> > > > > 
> > > > > So now I've checked with real ext4 - and the bug is there as well
> > > > > so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
> > > > > 
> > > > > 3.1 is the first kernel which does survive (checked 5 repeated runs)
> > > > 
> > > > Very useful research, thank you.
> > > > 
> > > > > 
> > > > > And you are correct, the first commit which causes crash really is
> > > > > 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why
> > > > > I've
> > > > > missed to properly identify this commit in my bisect is that crash
> > > > > usually
> > > > > happens on the second pass of the lvm test suite 'make check_local'
> > > > > execution
> > > > > - and I've been running test just once. To be sure I've run 5 run passes
> > > > > on
> > > > > 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
> > > > > usually on second run, with commit 3f31d07571e  the crash always happens
> > > > > in
> > > > > the first pass.
> > > > > 
> > > > > I've also checked some rawhide kernel
> > > > > vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
> > > > > and it's crashing as well - so it's probably not uniqueness of my
> > > > > config.
> > > > > 
> > > > > So is there any primary suspect in 3.2 which is worth to check - or I
> > > > > need
> > > > > another day to play another bisect game ?
> > > > 
> > > > No need for a further bisect: 3.2 is when the disard/trim/fallocate
> > > > support went into drivers/block/loop.c, so these tests would be unable
> > > > to show if DM was right or wrong before then.
> > > 
> > > Well I've played meanwhile the game with minimized kernel config
> > > and the outcome is:
> > > 
> > > last working kernel is: 3.1.0-rc1-00008-g548ef6c
> > > first broken: 3.1.0-rc1-00009-gdfaa2ef
> > > 
> > > dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
> > > loop: add discard support for loop devices
> > > 
> > > > I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
> > > > I did quickly crash around where you expected; though in my case
> > > > it was in dm_rh_dec() called from mirror_end_io().
> > > 
> > > Change of Issue Discards option in lvm.conf is not needed.
> > > I'm able to get these oopses with this setting turned off.
> > > 
> > > > But I've not taken it any further than that.  You've shown that it's
> > > > as much a problem with ext4 as with tmpfs, and has been a problem
> > > > ever since these tests' use of discard reached DM.
> > > > 
> > > > I think it's fair to assume that there's something wrong with DM's
> > > > handling of REQ_DISCARD.  (Perhaps it was all correct when written,
> > > > but I think there has been a series of modifications to REQ_DISCARD
> > > > handling in the block layer, it's been rather troublesome.)
> > > > 
> 
> Hi Zdenek,
> 
> > > 
> > > So does anyone has some idea what should be checked next ?
> 
> I've missed the information in the thread, but since you're the most
> familiar with the lvm test suite, could you distil the reproducer
> which does not require to run lvm test suit ?
> 
> 
> > 
> > 
> > I've enabled more kernel debugging tools - and this came out when running test
> > suite:
> > 
> > Unsure if this could be source of troubles?
> > 
> > kernel BUG at arch/x86/mm/physaddr.c:18!
> > invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> > CPU 1
> > Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp
> > llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945
> > iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4
> > 
> > Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO
> 
> So you're testing rather old kernel so you might be missing some
> fixes there. Could you rerun the test with the recent kernel ?
> 
> Also it appears that the bug here happens because dm requested a
> destination page which is within the kernel space. It seems that
> this has been initiated by the write request from the mirror target.
> So I do not immediately see how punch hole (discard) is involved at
> all. You might have been lucky enough to hit a different bug
> probably ?
> 
> Looking at git log, this commit has been brought to my attention:
> 
> 0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support
> 
> seems related to this crash.
> 
> Please retest with recent kernel.
> 
> Thanks!
> -Lukas
> 
> > 6464CTO/6464CTO
> > RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> > RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
> > RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
> > RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
> > RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
> > R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
> > FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task
> > ffff880136e3e090)
> > Stack:
> >  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
> >  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
> >  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
> > Call Trace:
> >  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
> >  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
> >  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
> >  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
> >  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
> >  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
> >  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
> >  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
> >  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
> >  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
> >  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
> >  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
> >  [<ffffffff81050177>] process_one_work+0x197/0x460
> >  [<ffffffff81050116>] ? process_one_work+0x136/0x460
> >  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
> >  [<ffffffff81050986>] worker_thread+0x126/0x2d0
> >  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
> >  [<ffffffff81056066>] kthread+0x96/0xa0
> >  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
> >  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
> >  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
> >  [<ffffffff8133b610>] ? gs_change+0xb/0xb
> > Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00
> > 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e
> > 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
> > RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> >  RSP <ffff880136e41aa0>
> > ---[ end trace 040dfd9d68fccabc ]---
> > 
> > 
> > 
> > Zdenek

So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
(https://lkml.org/lkml/2012/6/30/98) I think that this is
problem in the device mapper itself. I do not think it has anything
to do with tmpfs or mm. According to bisects from Zdenek it clearly
shows that the problem appear when the discard support for the loop
device is added, so it is most likely related to the dm discard support.

Anyway, the backtrace points to the NULL pointed dereference in
dm_rh_region_context() which is simple function:

void *dm_rh_region_context(struct dm_region *reg)
{
       return reg->rh->context;
}

so either reg, or reg-rh is NULL. Now the only place this is used is
from recovery_complete() in dm-raid1.c. So this is somewhat related
to raid recovery. I am not familiar with the dm code, but can
someone from the dm team look at this ?

But just to be sure to rule out the punch hole thing Zdenek can you
run your tests on the "real" discard capable device ? Or at least on
the device which does not convert discard requests into punch hole ?
You can use scsi_debug to create such device:

modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1

Thanks!
-Lukas

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-02 10:35                   ` Lukáš Czerner
  0 siblings, 0 replies; 26+ messages in thread
From: Lukáš Czerner @ 2012-07-02 10:35 UTC (permalink / raw)
  To: Lukáš Czerner
  Cc: amwang, Mike Snitzer, Zdenek Kabelac, Hugh Dickins, linux-kernel,
	Joe Thornber, LVM general discussion and development,
	Alasdair G Kergon

[-- Attachment #1: Type: TEXT/PLAIN, Size: 14864 bytes --]

On Mon, 2 Jul 2012, Lukᅵ Czerner wrote:

> Date: Mon, 2 Jul 2012 11:00:06 +0200 (CEST)
> From: Lukᅵ Czerner <lczerner@redhat.com>
> To: Zdenek Kabelac <zkabelac@redhat.com>
> Cc: Hugh Dickins <hughd@google.com>, Mike Snitzer <snitzer@redhat.com>,
>     Mikulas Patocka <mpatocka@redhat.com>, Joe Thornber <ejt@redhat.com>,
>     LVM general discussion and development <linux-lvm@redhat.com>,
>     amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>,
>     linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com>
> Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
> 
> On Mon, 2 Jul 2012, Zdenek Kabelac wrote:
> 
> > Date: Mon, 02 Jul 2012 00:03:07 +0200
> > From: Zdenek Kabelac <zkabelac@redhat.com>
> > To: Hugh Dickins <hughd@google.com>
> > Cc: Mike Snitzer <snitzer@redhat.com>, Mikulas Patocka <mpatocka@redhat.com>,
> >     Joe Thornber <ejt@redhat.com>,
> >     LVM general discussion and development <linux-lvm@redhat.com>,
> >     amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>,
> >     linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com>
> > Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
> > 
> > Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a):
> > > Dne 1.7.2012 20:45, Hugh Dickins napsal(a):
> > > > On Sun, 1 Jul 2012, Zdenek Kabelac wrote:
> > > > > Dne 1.7.2012 01:10, Hugh Dickins napsal(a):
> > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a):
> > > > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote:
> > > > > > > > > 
> > > > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks.
> > > > > > > > > Ooops log included. After some experimenting - reliable way to
> > > > > > > > > hit
> > > > > > > > > this
> > > > > > > > > oops
> > > > > > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window
> > > > > > > > > does
> > > > > > > > > not
> > > > > > > > > included anything related to this oops I've went for bisect.
> > > > > > > > 
> > > > > > > > Thanks a lot for reporting, and going to such effort to find
> > > > > > > > a reproducible testcase that you could bisect on.
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17
> > > > > > > > > 
> > > > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE
> > > > > > > > 
> > > > > > > > But this leaves me very puzzled.
> > > > > > > > 
> > > > > > > > Is the "lvm test suite" what I find at
> > > > > > > > git.fedorahosted.org/git/lvm2.git
> > > > > > > > under tests/ ?
> > > > > > > 
> > > > > > > Yes - that's it -
> > > > > > > 
> > > > > > >    make
> > > > > > > as root:
> > > > > > >    cd test
> > > > > > >    make check_local
> > > > > > > 
> > > > > > > (inside test subdirectory should be enough, if not - just report any
> > > > > > > problem)
> > > > > > > 
> > > > > > > > If you have something else running at the same time, which happens
> > > > > > > > to
> > > > > > > > use
> > > > > > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now
> > > > > > > > enables
> > > > > > > > it on (I guess ext4 from the =y in your config), then I suppose we
> > > > > > > > should
> > > > > > > > start searching for improper memory freeing or scribbling in its
> > > > > > > > holepunch
> > > > > > > > support: something that might be corrupting the dm_region in your
> > > > > > > > oops.
> > > > > > > 
> > > > > > > What the test is doing - it creates file in  LVM_TEST_DIR (default
> > > > > > > is
> > > > > > > /tmp)
> > > > 
> > > > I ran "LVM_TEST_DIR=/tmp make check_local":
> > > > without that it appeared to be using a subdirectory made under test/.
> > > > 
> > > > And being a year or two out of date in my userspace, and unfamiliar with
> > > > the syntax and whereabouts of lvm.conf, it was easiest for me to hack
> > > > lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1
> > > > after I spotted a warning message about issue_discards.
> > > > 
> > > > > > > and using loop device to simulate device (small size - it should fit
> > > > > > > bellow
> > > > > > > 200MB)
> > > > > > > 
> > > > > > > Within this file second layer through virtual DM devices is created
> > > > > > > and
> > > > > > > simulates various numbers of PV devices to play with.
> > > > > > 
> > > > > > This sounds much easier to set up than I was expecting:
> > > > > > thanks for the info, I'll try it later on today.
> > > > 
> > > > Sorry, I never reached it yesterday, but arrived there this morning.
> > > > 
> > > > > > 
> > > > > > > 
> > > > > > > So since everything now support TRIM - such operations should be
> > > > > > > passed
> > > > > > > down to the backend file - which probably triggers the path.
> > > > > > 
> > > > > > What filesystem do you have for /tmp?
> > > > 
> > > >  From your later remarks, I inferred tmpfs.
> > > > 
> > > > > > 
> > > > > > If tmpfs, then it will make much more sense if we assume your
> > > > > > bisection
> > > > > > endpoint was off by one.  Your bisection log was not quite complete;
> > > > > > and even if it did appear to converge on the commit you cite, you
> > > > > > might
> > > > > > have got (un)lucky when testing the commit before it, and concluded
> > > > > > "good" when more attempts would have said "bad".
> > > > > > 
> > > > > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe
> > > > > > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a
> > > > > > much more likely first bad commit if your /tmp is on tmpfs:
> > > > > > that does indeed wire up loop to pass TRIM down to tmpfs by
> > > > > > fallocate - that indeed played a part in my own testing.
> > > > > > 
> > > > > > Whereas if your /tmp is on ext4, loop has been passing TRIM down
> > > > > > with fallocate since v3.0.  And whichever, madvise(,,MADV_REMOVE)
> > > > > > should be completely irrelevant.
> > > > > 
> > > > > While I've been aware of the fact that tmpfs was enhanced with trim
> > > > > support -
> > > > > I've not tried to run on real ext4 filesystem since for my tests I'm
> > > > > using
> > > > > tmpfs for quite some time to safe rewrites of SSD :)
> > > > > 
> > > > > So now I've checked with real ext4 - and the bug is there as well
> > > > > so I've went back - it crashes on 3.4, 3.3 and 3.2 as well.
> > > > > 
> > > > > 3.1 is the first kernel which does survive (checked 5 repeated runs)
> > > > 
> > > > Very useful research, thank you.
> > > > 
> > > > > 
> > > > > And you are correct, the first commit which causes crash really is
> > > > > 83e4fa9c16e4af  when I use  tmpfs as backend storage - the problem why
> > > > > I've
> > > > > missed to properly identify this commit in my bisect is that crash
> > > > > usually
> > > > > happens on the second pass of the lvm test suite 'make check_local'
> > > > > execution
> > > > > - and I've been running test just once. To be sure I've run 5 run passes
> > > > > on
> > > > > 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing
> > > > > usually on second run, with commit 3f31d07571e  the crash always happens
> > > > > in
> > > > > the first pass.
> > > > > 
> > > > > I've also checked some rawhide kernel
> > > > > vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64
> > > > > and it's crashing as well - so it's probably not uniqueness of my
> > > > > config.
> > > > > 
> > > > > So is there any primary suspect in 3.2 which is worth to check - or I
> > > > > need
> > > > > another day to play another bisect game ?
> > > > 
> > > > No need for a further bisect: 3.2 is when the disard/trim/fallocate
> > > > support went into drivers/block/loop.c, so these tests would be unable
> > > > to show if DM was right or wrong before then.
> > > 
> > > Well I've played meanwhile the game with minimized kernel config
> > > and the outcome is:
> > > 
> > > last working kernel is: 3.1.0-rc1-00008-g548ef6c
> > > first broken: 3.1.0-rc1-00009-gdfaa2ef
> > > 
> > > dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef
> > > loop: add discard support for loop devices
> > > 
> > > > I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS
> > > > I did quickly crash around where you expected; though in my case
> > > > it was in dm_rh_dec() called from mirror_end_io().
> > > 
> > > Change of Issue Discards option in lvm.conf is not needed.
> > > I'm able to get these oopses with this setting turned off.
> > > 
> > > > But I've not taken it any further than that.  You've shown that it's
> > > > as much a problem with ext4 as with tmpfs, and has been a problem
> > > > ever since these tests' use of discard reached DM.
> > > > 
> > > > I think it's fair to assume that there's something wrong with DM's
> > > > handling of REQ_DISCARD.  (Perhaps it was all correct when written,
> > > > but I think there has been a series of modifications to REQ_DISCARD
> > > > handling in the block layer, it's been rather troublesome.)
> > > > 
> 
> Hi Zdenek,
> 
> > > 
> > > So does anyone has some idea what should be checked next ?
> 
> I've missed the information in the thread, but since you're the most
> familiar with the lvm test suite, could you distil the reproducer
> which does not require to run lvm test suit ?
> 
> 
> > 
> > 
> > I've enabled more kernel debugging tools - and this came out when running test
> > suite:
> > 
> > Unsure if this could be source of troubles?
> > 
> > kernel BUG at arch/x86/mm/physaddr.c:18!
> > invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> > CPU 1
> > Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp
> > llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945
> > iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4
> > 
> > Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO
> 
> So you're testing rather old kernel so you might be missing some
> fixes there. Could you rerun the test with the recent kernel ?
> 
> Also it appears that the bug here happens because dm requested a
> destination page which is within the kernel space. It seems that
> this has been initiated by the write request from the mirror target.
> So I do not immediately see how punch hole (discard) is involved at
> all. You might have been lucky enough to hit a different bug
> probably ?
> 
> Looking at git log, this commit has been brought to my attention:
> 
> 0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support
> 
> seems related to this crash.
> 
> Please retest with recent kernel.
> 
> Thanks!
> -Lukas
> 
> > 6464CTO/6464CTO
> > RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> > RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
> > RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
> > RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
> > RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
> > R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
> > FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task
> > ffff880136e3e090)
> > Stack:
> >  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
> >  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
> >  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
> > Call Trace:
> >  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
> >  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
> >  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
> >  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
> >  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
> >  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
> >  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
> >  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
> >  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
> >  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
> >  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
> >  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
> >  [<ffffffff81050177>] process_one_work+0x197/0x460
> >  [<ffffffff81050116>] ? process_one_work+0x136/0x460
> >  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
> >  [<ffffffff81050986>] worker_thread+0x126/0x2d0
> >  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
> >  [<ffffffff81056066>] kthread+0x96/0xa0
> >  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
> >  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
> >  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
> >  [<ffffffff8133b610>] ? gs_change+0xb/0xb
> > Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00
> > 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e
> > 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
> > RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> >  RSP <ffff880136e41aa0>
> > ---[ end trace 040dfd9d68fccabc ]---
> > 
> > 
> > 
> > Zdenek

So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
(https://lkml.org/lkml/2012/6/30/98) I think that this is
problem in the device mapper itself. I do not think it has anything
to do with tmpfs or mm. According to bisects from Zdenek it clearly
shows that the problem appear when the discard support for the loop
device is added, so it is most likely related to the dm discard support.

Anyway, the backtrace points to the NULL pointed dereference in
dm_rh_region_context() which is simple function:

void *dm_rh_region_context(struct dm_region *reg)
{
       return reg->rh->context;
}

so either reg, or reg-rh is NULL. Now the only place this is used is
from recovery_complete() in dm-raid1.c. So this is somewhat related
to raid recovery. I am not familiar with the dm code, but can
someone from the dm team look at this ?

But just to be sure to rule out the punch hole thing Zdenek can you
run your tests on the "real" discard capable device ? Or at least on
the device which does not convert discard requests into punch hole ?
You can use scsi_debug to create such device:

modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1

Thanks!
-Lukas

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-01 22:03               ` [linux-lvm] " Zdenek Kabelac
@ 2012-07-02 13:33                 ` Mike Snitzer
  -1 siblings, 0 replies; 26+ messages in thread
From: Mike Snitzer @ 2012-07-02 13:33 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: Hugh Dickins, Mikulas Patocka, Joe Thornber,
	LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel, Lukas Czerner

Not sure why you're testing such old kernels without regard for fixes
(likely just a hopeful bisect?).

In any case, you definitely need this commit (was also marked for
stable): 0c535e0 v3.3-rc7 dm io: fix discard support


On Sun, Jul 01 2012 at  6:03pm -0400,
Zdenek Kabelac <zkabelac@redhat.com> wrote:

> 
> I've enabled more kernel debugging tools - and this came out when
> running test suite:
> 
> Unsure if this could be source of troubles?
> 
> kernel BUG at arch/x86/mm/physaddr.c:18!
> invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> CPU 1
> Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun
> bridge stp llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi
> sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e
> evdev loop dm_mod autofs4
> 
> Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30
> LENOVO 6464CTO/6464CTO
> RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
> RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
> RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
> RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
> R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
> FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090)
> Stack:
>  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
>  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
>  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
> Call Trace:
>  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
>  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
>  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
>  [<ffffffff81050177>] process_one_work+0x197/0x460
>  [<ffffffff81050116>] ? process_one_work+0x136/0x460
>  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
>  [<ffffffff81050986>] worker_thread+0x126/0x2d0
>  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
>  [<ffffffff81056066>] kthread+0x96/0xa0
>  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
>  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
>  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
>  [<ffffffff8133b610>] ? gs_change+0xb/0xb
> Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00
> 00 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b
> <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
> RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
>  RSP <ffff880136e41aa0>
> ---[ end trace 040dfd9d68fccabc ]---
> 
> 
> 
> Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-02 13:33                 ` Mike Snitzer
  0 siblings, 0 replies; 26+ messages in thread
From: Mike Snitzer @ 2012-07-02 13:33 UTC (permalink / raw)
  To: Zdenek Kabelac
  Cc: amwang, Hugh Dickins, linux-kernel, Joe Thornber,
	LVM general discussion and development, Lukas Czerner,
	Alasdair G Kergon

Not sure why you're testing such old kernels without regard for fixes
(likely just a hopeful bisect?).

In any case, you definitely need this commit (was also marked for
stable): 0c535e0 v3.3-rc7 dm io: fix discard support


On Sun, Jul 01 2012 at  6:03pm -0400,
Zdenek Kabelac <zkabelac@redhat.com> wrote:

> 
> I've enabled more kernel debugging tools - and this came out when
> running test suite:
> 
> Unsure if this could be source of troubles?
> 
> kernel BUG at arch/x86/mm/physaddr.c:18!
> invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> CPU 1
> Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun
> bridge stp llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi
> sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e
> evdev loop dm_mod autofs4
> 
> Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30
> LENOVO 6464CTO/6464CTO
> RIP: 0010:[<ffffffff81021f00>]  [<ffffffff81021f00>] __phys_addr+0x60/0x70
> RSP: 0018:ffff880136e41aa0  EFLAGS: 00010213
> RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c
> RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000
> RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40
> R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40
> FS:  0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090)
> Stack:
>  ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0
>  ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e
>  00000000239a96d0 0000000000000000 0000000000000000 0000000000000000
> Call Trace:
>  [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod]
>  [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod]
>  [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod]
>  [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod]
>  [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror]
>  [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror]
>  [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror]
>  [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror]
>  [<ffffffff81050177>] process_one_work+0x197/0x460
>  [<ffffffff81050116>] ? process_one_work+0x136/0x460
>  [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror]
>  [<ffffffff81050986>] worker_thread+0x126/0x2d0
>  [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0
>  [<ffffffff81056066>] kthread+0x96/0xa0
>  [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10
>  [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe
>  [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70
>  [<ffffffff8133b610>] ? gs_change+0xb/0xb
> Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00
> 00 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b
> <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba
> RIP  [<ffffffff81021f00>] __phys_addr+0x60/0x70
>  RSP <ffff880136e41aa0>
> ---[ end trace 040dfd9d68fccabc ]---
> 
> 
> 
> Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-02 10:35                   ` [linux-lvm] " Lukáš Czerner
@ 2012-07-02 13:41                     ` Mike Snitzer
  -1 siblings, 0 replies; 26+ messages in thread
From: Mike Snitzer @ 2012-07-02 13:41 UTC (permalink / raw)
  To: Lukáš Czerner
  Cc: Zdenek Kabelac, Hugh Dickins, Mikulas Patocka, Joe Thornber,
	LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel

On Mon, Jul 02 2012 at  6:35am -0400,
Lukáš Czerner <lczerner@redhat.com> wrote:
> > 
> > So you're testing rather old kernel so you might be missing some
> > fixes there. Could you rerun the test with the recent kernel ?
> >
> > Also it appears that the bug here happens because dm requested a
> > destination page which is within the kernel space. It seems that
> > this has been initiated by the write request from the mirror target.
> > So I do not immediately see how punch hole (discard) is involved at
> > all. You might have been lucky enough to hit a different bug
> > probably ?
> > 
> > Looking at git log, this commit has been brought to my attention:
> > 
> > 0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support
> > 
> > seems related to this crash.
> > 
> > Please retest with recent kernel.

Ah, you beat me to recommending that fix ;)
 
> So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
> (https://lkml.org/lkml/2012/6/30/98) I think that this is
> problem in the device mapper itself. I do not think it has anything
> to do with tmpfs or mm. According to bisects from Zdenek it clearly
> shows that the problem appear when the discard support for the loop
> device is added, so it is most likely related to the dm discard support.

What about using scsi_debug with the dm-mirror target?

Never say never, DM-mirror and/or dm-io code could still have an issue,
but the commit referenced above did fix discard with the mirror target
back in 3.3.
 
> Anyway, the backtrace points to the NULL pointed dereference in
> dm_rh_region_context() which is simple function:
> 
> void *dm_rh_region_context(struct dm_region *reg)
> {
>        return reg->rh->context;
> }
> 
> so either reg, or reg-rh is NULL. Now the only place this is used is
> from recovery_complete() in dm-raid1.c. So this is somewhat related
> to raid recovery. I am not familiar with the dm code, but can
> someone from the dm team look at this ?

I'll coordiinate with Zdenek.

> But just to be sure to rule out the punch hole thing Zdenek can you
> run your tests on the "real" discard capable device ? Or at least on
> the device which does not convert discard requests into punch hole ?
> You can use scsi_debug to create such device:
> 
> modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1

Great minds think alike ;)

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-02 13:41                     ` Mike Snitzer
  0 siblings, 0 replies; 26+ messages in thread
From: Mike Snitzer @ 2012-07-02 13:41 UTC (permalink / raw)
  To: Lukáš Czerner
  Cc: amwang, Zdenek Kabelac, Hugh Dickins, linux-kernel, Joe Thornber,
	LVM general discussion and development, Alasdair G Kergon

On Mon, Jul 02 2012 at  6:35am -0400,
Lukáš Czerner <lczerner@redhat.com> wrote:
> > 
> > So you're testing rather old kernel so you might be missing some
> > fixes there. Could you rerun the test with the recent kernel ?
> >
> > Also it appears that the bug here happens because dm requested a
> > destination page which is within the kernel space. It seems that
> > this has been initiated by the write request from the mirror target.
> > So I do not immediately see how punch hole (discard) is involved at
> > all. You might have been lucky enough to hit a different bug
> > probably ?
> > 
> > Looking at git log, this commit has been brought to my attention:
> > 
> > 0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support
> > 
> > seems related to this crash.
> > 
> > Please retest with recent kernel.

Ah, you beat me to recommending that fix ;)
 
> So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4
> (https://lkml.org/lkml/2012/6/30/98) I think that this is
> problem in the device mapper itself. I do not think it has anything
> to do with tmpfs or mm. According to bisects from Zdenek it clearly
> shows that the problem appear when the discard support for the loop
> device is added, so it is most likely related to the dm discard support.

What about using scsi_debug with the dm-mirror target?

Never say never, DM-mirror and/or dm-io code could still have an issue,
but the commit referenced above did fix discard with the mirror target
back in 3.3.
 
> Anyway, the backtrace points to the NULL pointed dereference in
> dm_rh_region_context() which is simple function:
> 
> void *dm_rh_region_context(struct dm_region *reg)
> {
>        return reg->rh->context;
> }
> 
> so either reg, or reg-rh is NULL. Now the only place this is used is
> from recovery_complete() in dm-raid1.c. So this is somewhat related
> to raid recovery. I am not familiar with the dm code, but can
> someone from the dm team look at this ?

I'll coordiinate with Zdenek.

> But just to be sure to rule out the punch hole thing Zdenek can you
> run your tests on the "real" discard capable device ? Or at least on
> the device which does not convert discard requests into punch hole ?
> You can use scsi_debug to create such device:
> 
> modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1

Great minds think alike ;)

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
  2012-07-02 13:33                 ` [linux-lvm] " Mike Snitzer
@ 2012-07-04 11:51                   ` Zdenek Kabelac
  -1 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-04 11:51 UTC (permalink / raw)
  To: Mike Snitzer
  Cc: Hugh Dickins, Mikulas Patocka, Joe Thornber,
	LVM general discussion and development, amwang,
	Alasdair G Kergon, linux-kernel, Lukas Czerner

Dne 2.7.2012 15:33, Mike Snitzer napsal(a):
> Not sure why you're testing such old kernels without regard for fixes
> (likely just a hopeful bisect?).
>
> In any case, you definitely need this commit (was also marked for
> stable): 0c535e0 v3.3-rc7 dm io: fix discard support
>
>

After some more crashed filesystems here is the outcome with trial patch from 
Mikulas included:


https://bugzilla.redhat.com/show_bug.cgi?id=837607


Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel
@ 2012-07-04 11:51                   ` Zdenek Kabelac
  0 siblings, 0 replies; 26+ messages in thread
From: Zdenek Kabelac @ 2012-07-04 11:51 UTC (permalink / raw)
  To: Mike Snitzer
  Cc: amwang, Hugh Dickins, linux-kernel, Joe Thornber,
	LVM general discussion and development, Lukas Czerner,
	Alasdair G Kergon

Dne 2.7.2012 15:33, Mike Snitzer napsal(a):
> Not sure why you're testing such old kernels without regard for fixes
> (likely just a hopeful bisect?).
>
> In any case, you definitely need this commit (was also marked for
> stable): 0c535e0 v3.3-rc7 dm io: fix discard support
>
>

After some more crashed filesystems here is the outcome with trial patch from 
Mikulas included:


https://bugzilla.redhat.com/show_bug.cgi?id=837607


Zdenek

^ permalink raw reply	[flat|nested] 26+ messages in thread

end of thread, other threads:[~2012-07-04 11:51 UTC | newest]

Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-06-30 11:41 [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel Zdenek Kabelac
2012-06-30 13:20 ` Zdenek Kabelac
2012-06-30 19:55 ` Hugh Dickins
2012-06-30 19:55   ` [linux-lvm] " Hugh Dickins
2012-06-30 20:51   ` Zdenek Kabelac
2012-06-30 20:51     ` [linux-lvm] " Zdenek Kabelac
2012-06-30 23:10     ` Hugh Dickins
2012-06-30 23:10       ` [linux-lvm] " Hugh Dickins
2012-07-01 12:57       ` Zdenek Kabelac
2012-07-01 12:57         ` [linux-lvm] " Zdenek Kabelac
2012-07-01 18:45         ` Hugh Dickins
2012-07-01 18:45           ` [linux-lvm] " Hugh Dickins
2012-07-01 20:10           ` Zdenek Kabelac
2012-07-01 20:10             ` [linux-lvm] " Zdenek Kabelac
2012-07-01 22:03             ` Zdenek Kabelac
2012-07-01 22:03               ` [linux-lvm] " Zdenek Kabelac
2012-07-02  9:00               ` Lukáš Czerner
2012-07-02  9:00                 ` [linux-lvm] " Lukáš Czerner
2012-07-02 10:35                 ` Lukáš Czerner
2012-07-02 10:35                   ` [linux-lvm] " Lukáš Czerner
2012-07-02 13:41                   ` Mike Snitzer
2012-07-02 13:41                     ` [linux-lvm] " Mike Snitzer
2012-07-02 13:33               ` Mike Snitzer
2012-07-02 13:33                 ` [linux-lvm] " Mike Snitzer
2012-07-04 11:51                 ` Zdenek Kabelac
2012-07-04 11:51                   ` [linux-lvm] " Zdenek Kabelac

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.