* long hangs when deleting large directories (3.0-rc3)
@ 2011-06-18 14:19 Markus Trippelsdorf
2011-06-18 14:24 ` Christoph Hellwig
2011-06-19 22:24 ` Dave Chinner
0 siblings, 2 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-18 14:19 UTC (permalink / raw)
To: xfs
[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]
Running the latest git kernel (3.0-rc3) my machine hangs for long
periods (1-2 sec) whenever I delete a large directory recursively on my
xfs partition. During the hang I cannot move the mouse pointer or use
the keyboard (but the music keeps playing without stuttering). A quick
way to reproduce is to "rm -fr" a kernel tree.
This happens on a 4kb SATA hard drive:
xfs_info /var
meta-data=/dev/sda1 isize=256 agcount=4, agsize=12800000 blks
= sectsz=4096 attr=2
data = bsize=4096 blocks=51200000, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=25000, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
/dev/sda1 on /var type xfs (rw,noatime,attr2,delaylog,logbsize=256k,noquota)
My kernel config is attached.
Please let me know if you need additional information.
--
Markus
[-- Attachment #2: config --]
[-- Type: text/plain, Size: 52838 bytes --]
#
# Automatically generated make config: don't edit
# Linux/x86_64 3.0.0-rc3 Kernel Configuration
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
# CONFIG_KTIME_SCALAR is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
# CONFIG_KERNEL_GZIP is not set
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
CONFIG_KERNEL_LZO=y
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
# CONFIG_FHANDLE is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set
CONFIG_HAVE_GENERIC_HARDIRQS=y
#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
# CONFIG_SPARSE_IRQ is not set
#
# RCU Subsystem
#
CONFIG_TREE_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_RCU_BOOST is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=16
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_CGROUPS is not set
# CONFIG_NAMESPACES is not set
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_EXPERT=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
# CONFIG_ELF_CORE is not set
# CONFIG_PCSPKR_PLATFORM is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_EMBEDDED=y
CONFIG_HAVE_PERF_EVENTS=y
#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_PERF_COUNTERS is not set
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_SLUB_DEBUG is not set
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_HAVE_OPROFILE=y
CONFIG_JUMP_LABEL=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
# CONFIG_MODULES is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
# CONFIG_BLK_DEV_INTEGRITY is not set
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
# CONFIG_INLINE_SPIN_UNLOCK is not set
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
# CONFIG_INLINE_READ_UNLOCK is not set
# CONFIG_INLINE_READ_UNLOCK_BH is not set
# CONFIG_INLINE_READ_UNLOCK_IRQ is not set
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
# CONFIG_INLINE_WRITE_UNLOCK is not set
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
CONFIG_MUTEX_SPIN_ON_OWNER=y
# CONFIG_FREEZER is not set
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_X86_MPPARSE is not set
# CONFIG_X86_EXTENDED_PLATFORM is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_CMPXCHG_LOCAL=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PROCESSOR_SELECT=y
# CONFIG_CPU_SUP_INTEL is not set
CONFIG_CPU_SUP_AMD=y
# CONFIG_CPU_SUP_CENTAUR is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_GART_IOMMU is not set
# CONFIG_CALGARY_IOMMU is not set
# CONFIG_AMD_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_IOMMU_API is not set
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=4
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_INTEL is not set
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
# CONFIG_NUMA is not set
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_MEMORY_FAILURE is not set
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
# CONFIG_CLEANCACHE is not set
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=2
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x200000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
# CONFIG_HOTPLUG_CPU is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management and ACPI options
#
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
# CONFIG_PM_RUNTIME is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
# CONFIG_ACPI_EC_DEBUGFS is not set
# CONFIG_ACPI_PROC_EVENT is not set
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
# CONFIG_ACPI_VIDEO is not set
# CONFIG_ACPI_FAN is not set
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_PROCESSOR=y
# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set
# CONFIG_ACPI_SBS is not set
# CONFIG_ACPI_HED is not set
# CONFIG_ACPI_CUSTOM_METHOD is not set
# CONFIG_ACPI_APEI is not set
# CONFIG_SFI is not set
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set
#
# x86 CPU frequency scaling drivers
#
# CONFIG_X86_PCC_CPUFREQ is not set
# CONFIG_X86_ACPI_CPUFREQ is not set
CONFIG_X86_POWERNOW_K8=y
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set
#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MMCONFIG is not set
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
# CONFIG_DMAR is not set
# CONFIG_INTR_REMAP is not set
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
# CONFIG_PCI_IOV is not set
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set
# CONFIG_RAPIDIO is not set
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_HAVE_AOUT is not set
# CONFIG_BINFMT_MISC is not set
# CONFIG_IA32_EMULATION is not set
# CONFIG_COMPAT_FOR_U64_ALIGNMENT is not set
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_ADVANCED is not set
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_CONNTRACK_IRC=y
CONFIG_NF_CONNTRACK_SIP=y
CONFIG_NF_CT_NETLINK=y
CONFIG_NETFILTER_XTABLES=y
#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=y
#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_STATE=y
# CONFIG_IP_SET is not set
# CONFIG_IP_VS is not set
#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_NF_NAT_FTP=y
CONFIG_NF_NAT_IRC=y
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_NAT_AMANDA is not set
# CONFIG_NF_NAT_PPTP is not set
# CONFIG_NF_NAT_H323 is not set
CONFIG_NF_NAT_SIP=y
CONFIG_IP_NF_MANGLE=y
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
CONFIG_STP=y
CONFIG_BRIDGE=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
# CONFIG_BATMAN_ADV is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_HAVE_BPF_JIT=y
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE="radeon/R600_rlc.bin"
CONFIG_EXTRA_FIRMWARE_DIR="/lib/firmware"
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y
#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
#
# DRBD disabled because PROC_FS, INET or CONNECTOR not selected
#
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_VIRTIO_BLK is not set
# CONFIG_BLK_DEV_HD is not set
# CONFIG_BLK_DEV_RBD is not set
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
# CONFIG_SCSI_PROC_FS is not set
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
# CONFIG_SCSI_LOWLEVEL is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
# CONFIG_ATA_ACPI is not set
# CONFIG_SATA_PMP is not set
#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
# CONFIG_SATA_AHCI_PLATFORM is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
#
# SFF controllers with custom DMA interface
#
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_SX4 is not set
CONFIG_ATA_BMDMA=y
#
# SATA SFF controllers with BMDMA
#
# CONFIG_ATA_PIIX is not set
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_SVW is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
#
# PATA SFF controllers with BMDMA
#
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARASAN_CF is not set
# CONFIG_PATA_ARTOP is not set
CONFIG_PATA_ATIIXP=y
# CONFIG_PATA_ATP867X is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RDC is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SCH is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_TOSHIBA is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_PLATFORM is not set
# CONFIG_PATA_RZ1000 is not set
#
# Generic fallback / legacy drivers
#
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_LEGACY is not set
# CONFIG_MD is not set
# CONFIG_TARGET_CORE is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_FIREWIRE_NOSY is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=y
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=y
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
CONFIG_MII=y
# CONFIG_PHYLIB is not set
# CONFIG_NET_ETHERNET is not set
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_IGBVF is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_CNIC is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
CONFIG_ATL1E=y
# CONFIG_ATL1C is not set
# CONFIG_JME is not set
# CONFIG_STMMAC_ETH is not set
# CONFIG_PCH_GBE is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set
# CONFIG_WLAN is not set
#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_IPHETH is not set
# CONFIG_WAN is not set
#
# CAIF transport drivers
#
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=y
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_SYNC_TTY=y
# CONFIG_PPP_DEFLATE is not set
# CONFIG_PPP_BSDCOMP is not set
# CONFIG_PPP_MPPE is not set
# CONFIG_PPPOE is not set
# CONFIG_SLIP is not set
CONFIG_SLHC=y
# CONFIG_NET_FC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_VIRTIO_NET is not set
# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
# CONFIG_INPUT_SPARSEKMAP is not set
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
# CONFIG_MOUSE_PS2 is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
# CONFIG_DEVKMEM is not set
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
CONFIG_FIX_EARLYCON_MEM=y
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MFD_HSU is not set
# CONFIG_SERIAL_JSM is not set
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_PCH_UART is not set
# CONFIG_SERIAL_XILINX_PS_UART is not set
# CONFIG_TTY_PRINTK is not set
# CONFIG_VIRTIO_CONSOLE is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_RAMOOPS is not set
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y
#
# I2C Hardware Bus support
#
#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_ISCH is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
#
# ACPI drivers
#
# CONFIG_I2C_SCMI is not set
#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_INTEL_MID is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_PXA_PCI is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set
# CONFIG_I2C_EG20T is not set
#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set
#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_SPI is not set
#
# PPS support
#
# CONFIG_PPS is not set
#
# PPS generators support
#
#
# PTP clock support
#
#
# Enable Device Drivers -> PPS to see the PTP clock options.
#
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_BQ20Z75 is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_MAX8903 is not set
CONFIG_HWMON=y
# CONFIG_HWMON_VID is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
#
# Native drivers
#
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7411 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_ASC7621 is not set
# CONFIG_SENSORS_K8TEMP is not set
CONFIG_SENSORS_K10TEMP=y
# CONFIG_SENSORS_FAM15H_POWER is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS620 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_LINEAGE is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM73 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4151 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LTC4261 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_MAX16065 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX6639 is not set
# CONFIG_SENSORS_MAX6642 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_PMBUS is not set
# CONFIG_SENSORS_SHT21 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_EMC1403 is not set
# CONFIG_SENSORS_EMC2103 is not set
# CONFIG_SENSORS_EMC6W201 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_SCH5627 is not set
# CONFIG_SENSORS_ADS1015 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_AMC6821 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_TMP421 is not set
# CONFIG_SENSORS_VIA_CPUTEMP is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83795 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_APPLESMC is not set
#
# ACPI drivers
#
# CONFIG_SENSORS_ACPI_POWER is not set
CONFIG_SENSORS_ATK0110=y
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
#
# Broadcom specific AMBA
#
# CONFIG_BCMA is not set
# CONFIG_MFD_SUPPORT is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set
#
# Graphics support
#
# CONFIG_AGP is not set
# CONFIG_VGA_ARB is not set
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=y
CONFIG_DRM_KMS_HELPER=y
CONFIG_DRM_TTM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
CONFIG_DRM_RADEON=y
CONFIG_DRM_RADEON_KMS=y
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
# CONFIG_STUB_POULSBO is not set
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=y
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
# CONFIG_FB_DDC is not set
# CONFIG_FB_BOOT_VESA_SUPPORT is not set
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_WMT_GE_ROPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set
#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_VESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_UDL is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_LOGO is not set
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_SEQUENCER=y
CONFIG_SND_SEQ_DUMMY=y
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=y
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_SUPPORT_OLD_API=y
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=y
# CONFIG_SND_OPL3_LIB_SEQ is not set
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
# CONFIG_SND_EMU10K1_SEQ is not set
# CONFIG_SND_DRIVERS is not set
# CONFIG_SND_PCI is not set
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=y
# CONFIG_SND_USB_UA101 is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_US122L is not set
# CONFIG_SND_USB_6FIRE is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HIDRAW=y
#
# USB Input Devices
#
CONFIG_USB_HID=y
# CONFIG_HID_PID is not set
CONFIG_USB_HIDDEV=y
#
# Special HID drivers
#
# CONFIG_HID_A4TECH is not set
# CONFIG_HID_ACRUX is not set
# CONFIG_HID_APPLE is not set
# CONFIG_HID_BELKIN is not set
# CONFIG_HID_CHERRY is not set
# CONFIG_HID_CHICONY is not set
# CONFIG_HID_PRODIKEYS is not set
# CONFIG_HID_CYPRESS is not set
# CONFIG_HID_DRAGONRISE is not set
# CONFIG_HID_EMS_FF is not set
# CONFIG_HID_EZKEY is not set
# CONFIG_HID_KEYTOUCH is not set
# CONFIG_HID_KYE is not set
# CONFIG_HID_UCLOGIC is not set
# CONFIG_HID_WALTOP is not set
# CONFIG_HID_GYRATION is not set
# CONFIG_HID_TWINHAN is not set
# CONFIG_HID_KENSINGTON is not set
# CONFIG_HID_LCPOWER is not set
# CONFIG_HID_LOGITECH is not set
# CONFIG_HID_MICROSOFT is not set
# CONFIG_HID_MONTEREY is not set
# CONFIG_HID_MULTITOUCH is not set
# CONFIG_HID_NTRIG is not set
# CONFIG_HID_ORTEK is not set
# CONFIG_HID_PANTHERLORD is not set
# CONFIG_HID_PETALYNX is not set
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_QUANTA is not set
# CONFIG_HID_ROCCAT is not set
# CONFIG_HID_ROCCAT_ARVO is not set
# CONFIG_HID_ROCCAT_KONE is not set
# CONFIG_HID_ROCCAT_KONEPLUS is not set
# CONFIG_HID_ROCCAT_KOVAPLUS is not set
# CONFIG_HID_ROCCAT_PYRA is not set
# CONFIG_HID_SAMSUNG is not set
# CONFIG_HID_SONY is not set
# CONFIG_HID_SUNPLUS is not set
# CONFIG_HID_GREENASIA is not set
# CONFIG_HID_SMARTJOYPLUS is not set
# CONFIG_HID_TOPSEED is not set
# CONFIG_HID_THRUSTMASTER is not set
# CONFIG_HID_ZEROPLUS is not set
# CONFIG_HID_ZYDACRON is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_OTG_WHITELIST is not set
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
# CONFIG_USB_HWA_HCD is not set
#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=y
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set
#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#
#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_STORAGE_ENE_UB6250 is not set
# CONFIG_USB_UAS is not set
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
#
# USB port drivers
#
CONFIG_USB_SERIAL=y
# CONFIG_USB_SERIAL_CONSOLE is not set
# CONFIG_USB_EZUSB is not set
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP210X is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_FUNSOFT is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_MOTOROLA is not set
# CONFIG_USB_SERIAL_NAVMAN is not set
# CONFIG_USB_SERIAL_PL2303 is not set
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QCAUX is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
# CONFIG_USB_SERIAL_HP4X is not set
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIEMENS_MPI is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_SYMBOL is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
CONFIG_USB_SERIAL_WWAN=y
CONFIG_USB_SERIAL_OPTION=y
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_VIVOPAY_SERIAL is not set
# CONFIG_USB_SERIAL_ZIO is not set
# CONFIG_USB_SERIAL_SSU100 is not set
# CONFIG_USB_SERIAL_DEBUG is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_YUREX is not set
# CONFIG_USB_GADGET is not set
#
# OTG and related infrastructure
#
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_NFC_DEVICES is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y
#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=y
# CONFIG_EDAC_MCE_INJ is not set
CONFIG_EDAC_MM_EDAC=y
CONFIG_EDAC_AMD64=y
# CONFIG_EDAC_AMD64_ERROR_INJECTION is not set
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_I3200 is not set
# CONFIG_EDAC_X38 is not set
# CONFIG_EDAC_I5400 is not set
# CONFIG_EDAC_I7CORE is not set
# CONFIG_EDAC_I5000 is not set
# CONFIG_EDAC_I5100 is not set
# CONFIG_EDAC_I7300 is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_DS3232 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
# CONFIG_RTC_DRV_RS5C372 is not set
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_ISL12022 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_BQ32K is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set
# CONFIG_RTC_DRV_EM3027 is not set
# CONFIG_RTC_DRV_RV3029C2 is not set
#
# SPI RTC drivers
#
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set
# CONFIG_RTC_DRV_V3020 is not set
#
# on-CPU RTC drivers
#
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_STAGING is not set
# CONFIG_X86_PLATFORM_DEVICES is not set
#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
# CONFIG_DMIID is not set
# CONFIG_DMI_SYSFS is not set
# CONFIG_ISCSI_IBFT_FIND is not set
# CONFIG_SIGMA is not set
# CONFIG_GOOGLE_FIRMWARE is not set
#
# File systems
#
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
# CONFIG_EXT4_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_XFS_FS=y
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
# CONFIG_QUOTA is not set
# CONFIG_QUOTACTL is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
#
# Caches
#
# CONFIG_FSCACHE is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
# CONFIG_ZISOFS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
# CONFIG_CONFIGFS_FS is not set
# CONFIG_MISC_FILESYSTEMS is not set
# CONFIG_NETWORK_FILESYSTEMS is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=y
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_LOCKUP_DETECTOR is not set
# CONFIG_HARDLOCKUP_DETECTOR is not set
# CONFIG_DETECT_HUNG_TASK is not set
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
# CONFIG_DEBUG_MEMORY_INIT is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_CPU_STALL_VERBOSE=y
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_LKDTM is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
# CONFIG_SYSCTL_SYSCALL_CHECK is not set
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FTRACE_NMI_ENTER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_FTRACE_NMI_ENTER=y
CONFIG_EVENT_TRACING=y
# CONFIG_EVENT_POWER_TRACING_DEPRECATED is not set
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_PREEMPT_TRACER is not set
CONFIG_SCHED_TRACER=y
# CONFIG_FTRACE_SYSCALLS is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
CONFIG_DYNAMIC_FTRACE=y
# CONFIG_FUNCTION_PROFILER is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_TEST_KSTRTOX is not set
CONFIG_STRICT_DEVMEM=y
# CONFIG_X86_VERBOSE_BOOTUP is not set
# CONFIG_EARLY_PRINTK is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=y
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set
# CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
# CONFIG_CRYPTO is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
# CONFIG_KVM_INTEL is not set
CONFIG_KVM_AMD=y
# CONFIG_KVM_MMU_AUDIT is not set
CONFIG_VHOST_NET=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=y
# CONFIG_VIRTIO_BALLOON is not set
CONFIG_BINARY_PRINTF=y
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_CRC_CCITT=y
# CONFIG_CRC16 is not set
# CONFIG_CRC_T10DIF is not set
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
# CONFIG_XZ_DEC is not set
# CONFIG_XZ_DEC_BCJ is not set
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CPU_RMAP=y
CONFIG_NLATTR=y
# CONFIG_AVERAGE is not set
[-- Attachment #3: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-18 14:19 long hangs when deleting large directories (3.0-rc3) Markus Trippelsdorf
@ 2011-06-18 14:24 ` Christoph Hellwig
2011-06-18 14:37 ` Markus Trippelsdorf
2011-06-19 22:24 ` Dave Chinner
1 sibling, 1 reply; 33+ messages in thread
From: Christoph Hellwig @ 2011-06-18 14:24 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> Running the latest git kernel (3.0-rc3) my machine hangs for long
> periods (1-2 sec) whenever I delete a large directory recursively on my
> xfs partition. During the hang I cannot move the mouse pointer or use
> the keyboard (but the music keeps playing without stuttering). A quick
> way to reproduce is to "rm -fr" a kernel tree.
Does this also happen when using the deadline I/O schedule, that is
after a:
echo "deadline" > /sys/block/sda/queue/scheduler
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-18 14:24 ` Christoph Hellwig
@ 2011-06-18 14:37 ` Markus Trippelsdorf
2011-06-19 8:16 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-18 14:37 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: xfs
On 2011.06.18 at 10:24 -0400, Christoph Hellwig wrote:
> On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > periods (1-2 sec) whenever I delete a large directory recursively on my
> > xfs partition. During the hang I cannot move the mouse pointer or use
> > the keyboard (but the music keeps playing without stuttering). A quick
> > way to reproduce is to "rm -fr" a kernel tree.
>
> Does this also happen when using the deadline I/O schedule, that is
> after a:
>
> echo "deadline" > /sys/block/sda/queue/scheduler
Yes.
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-18 14:37 ` Markus Trippelsdorf
@ 2011-06-19 8:16 ` Markus Trippelsdorf
0 siblings, 0 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-19 8:16 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: xfs
On 2011.06.18 at 16:37 +0200, Markus Trippelsdorf wrote:
> On 2011.06.18 at 10:24 -0400, Christoph Hellwig wrote:
> > On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > > periods (1-2 sec) whenever I delete a large directory recursively on my
> > > xfs partition. During the hang I cannot move the mouse pointer or use
> > > the keyboard (but the music keeps playing without stuttering). A quick
> > > way to reproduce is to "rm -fr" a kernel tree.
> >
> > Does this also happen when using the deadline I/O schedule, that is
> > after a:
> >
> > echo "deadline" > /sys/block/sda/queue/scheduler
>
> Yes.
I've tested this a little further. The behavior is independent of the
kernel version used (tested back to 2.6.37). My SSD is also fine and a
freshly created xfs partition shows no problems, too.
Please note that the affected partition is used very heavily here
(several git trees, daily backups of / and the Gentoo build-dir reside
there).
So it appears that the observed "hangs" are the result of a strongly
aged file-system.
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-18 14:19 long hangs when deleting large directories (3.0-rc3) Markus Trippelsdorf
2011-06-18 14:24 ` Christoph Hellwig
@ 2011-06-19 22:24 ` Dave Chinner
2011-06-20 0:54 ` Markus Trippelsdorf
1 sibling, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2011-06-19 22:24 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> Running the latest git kernel (3.0-rc3) my machine hangs for long
> periods (1-2 sec) whenever I delete a large directory recursively on my
> xfs partition. During the hang I cannot move the mouse pointer or use
> the keyboard (but the music keeps playing without stuttering). A quick
> way to reproduce is to "rm -fr" a kernel tree.
So what is the system doing when it "hangs"? Is it CPU bound (e.g.
cpu scheduler issue)? Is the system running out of memory and
stalling everything in memory reclaim? What IO is occurring?
> This happens on a 4kb SATA hard drive:
How does this appear to the OS? as a 512/512, 512/4k or 4k/4k
logical/physical sector size drive?
> xfs_info /var
> meta-data=/dev/sda1 isize=256 agcount=4, agsize=12800000 blks
> = sectsz=4096 attr=2
> data = bsize=4096 blocks=51200000, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0
> log =internal bsize=4096 blocks=25000, version=2
> = sectsz=4096 sunit=1 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
>
> /dev/sda1 on /var type xfs (rw,noatime,attr2,delaylog,logbsize=256k,noquota)
Is your partition correctly sector aligned for however your drive
maps it's 4k sectors?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-19 22:24 ` Dave Chinner
@ 2011-06-20 0:54 ` Markus Trippelsdorf
2011-06-20 1:34 ` Dave Chinner
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-20 0:54 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.20 at 08:24 +1000, Dave Chinner wrote:
> On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > periods (1-2 sec) whenever I delete a large directory recursively on my
> > xfs partition. During the hang I cannot move the mouse pointer or use
> > the keyboard (but the music keeps playing without stuttering). A quick
> > way to reproduce is to "rm -fr" a kernel tree.
>
> So what is the system doing when it "hangs"? Is it CPU bound (e.g.
> cpu scheduler issue)? Is the system running out of memory and
> stalling everything in memory reclaim? What IO is occurring?
It's totally idle otherwise; just a desktop with a single xterm. The
machine has four cores (and also runs with "CONFIG_PREEMPT=y"), so I
don't think it is CPU bound at all. It has 8GB of memory (and the
"hangs" even occur after reboot when most of it is free). No other IO
activity is occurring.
> > This happens on a 4kb SATA hard drive:
>
> How does this appear to the OS? as a 512/512, 512/4k or 4k/4k
> logical/physical sector size drive?
It unfortunately appears as 512/512, but because I know it's a 4KB
drive, I formated it with "-s size=4096".
> > xfs_info /var
> > meta-data=/dev/sda1 isize=256 agcount=4, agsize=12800000 blks
> > = sectsz=4096 attr=2
> > data = bsize=4096 blocks=51200000, imaxpct=25
> > = sunit=0 swidth=0 blks
> > naming =version 2 bsize=4096 ascii-ci=0
> > log =internal bsize=4096 blocks=25000, version=2
> > = sectsz=4096 sunit=1 blks, lazy-count=1
> > realtime =none extsz=4096 blocks=0, rtextents=0
> >
> > /dev/sda1 on /var type xfs (rw,noatime,attr2,delaylog,logbsize=256k,noquota)
>
> Is your partition correctly sector aligned for however your drive
> maps it's 4k sectors?
Yes, it's a GPT partition that is aligned to 1MB.
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 0:54 ` Markus Trippelsdorf
@ 2011-06-20 1:34 ` Dave Chinner
2011-06-20 2:02 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2011-06-20 1:34 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Mon, Jun 20, 2011 at 02:54:15AM +0200, Markus Trippelsdorf wrote:
> On 2011.06.20 at 08:24 +1000, Dave Chinner wrote:
> > On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > > periods (1-2 sec) whenever I delete a large directory recursively on my
> > > xfs partition. During the hang I cannot move the mouse pointer or use
> > > the keyboard (but the music keeps playing without stuttering). A quick
> > > way to reproduce is to "rm -fr" a kernel tree.
> >
> > So what is the system doing when it "hangs"? Is it CPU bound (e.g.
> > cpu scheduler issue)? Is the system running out of memory and
> > stalling everything in memory reclaim? What IO is occurring?
>
> It's totally idle otherwise; just a desktop with a single xterm. The
> machine has four cores (and also runs with "CONFIG_PREEMPT=y"), so I
> don't think it is CPU bound at all. It has 8GB of memory (and the
> "hangs" even occur after reboot when most of it is free). No other IO
> activity is occurring.
Sure, the system might be otherwise idle, but what I was asking is
what load does the "rm -rf" cause. What IO does it cause? is it cpu
bound? etc.
> > > This happens on a 4kb SATA hard drive:
> >
> > How does this appear to the OS? as a 512/512, 512/4k or 4k/4k
> > logical/physical sector size drive?
>
> It unfortunately appears as 512/512, but because I know it's a 4KB
> drive, I formated it with "-s size=4096".
Oh, joy. Another user having strange performance problems on a 4k
sector drive that lies to the OS about it's geometry....
> > > xfs_info /var
> > > meta-data=/dev/sda1 isize=256 agcount=4, agsize=12800000 blks
> > > = sectsz=4096 attr=2
> > > data = bsize=4096 blocks=51200000, imaxpct=25
> > > = sunit=0 swidth=0 blks
> > > naming =version 2 bsize=4096 ascii-ci=0
> > > log =internal bsize=4096 blocks=25000, version=2
> > > = sectsz=4096 sunit=1 blks, lazy-count=1
> > > realtime =none extsz=4096 blocks=0, rtextents=0
> > >
> > > /dev/sda1 on /var type xfs (rw,noatime,attr2,delaylog,logbsize=256k,noquota)
> >
> > Is your partition correctly sector aligned for however your drive
> > maps it's 4k sectors?
>
> Yes, it's a GPT partition that is aligned to 1MB.
Ok, that is fine, but the big question now is how does the drive
align sector 0? Is that 4k aligned, or is it one of those drives
that aligns an odd 512 byte logical sector to the physical 4k sector
boundary (i.e. sector 63 is 4k aligned to work with msdos
partitions). FYI, some drives have jumpers on them to change this
odd/even sector alignment configuration.....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 1:34 ` Dave Chinner
@ 2011-06-20 2:02 ` Markus Trippelsdorf
2011-06-20 2:36 ` Dave Chinner
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-20 2:02 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.20 at 11:34 +1000, Dave Chinner wrote:
> On Mon, Jun 20, 2011 at 02:54:15AM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.20 at 08:24 +1000, Dave Chinner wrote:
> > > On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > > > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > > > periods (1-2 sec) whenever I delete a large directory recursively on my
> > > > xfs partition. During the hang I cannot move the mouse pointer or use
> > > > the keyboard (but the music keeps playing without stuttering). A quick
> > > > way to reproduce is to "rm -fr" a kernel tree.
> > >
> > > So what is the system doing when it "hangs"? Is it CPU bound (e.g.
> > > cpu scheduler issue)? Is the system running out of memory and
> > > stalling everything in memory reclaim? What IO is occurring?
> >
> > It's totally idle otherwise; just a desktop with a single xterm. The
> > machine has four cores (and also runs with "CONFIG_PREEMPT=y"), so I
> > don't think it is CPU bound at all. It has 8GB of memory (and the
> > "hangs" even occur after reboot when most of it is free). No other IO
> > activity is occurring.
>
> Sure, the system might be otherwise idle, but what I was asking is
> what load does the "rm -rf" cause. What IO does it cause? is it cpu
> bound? etc.
I have not measured this, so I cannot tell.
> > > > This happens on a 4kb SATA hard drive:
> > >
> > > How does this appear to the OS? as a 512/512, 512/4k or 4k/4k
> > > logical/physical sector size drive?
> >
> > It unfortunately appears as 512/512, but because I know it's a 4KB
> > drive, I formated it with "-s size=4096".
>
> Oh, joy. Another user having strange performance problems on a 4k
> sector drive that lies to the OS about it's geometry....
>
> > > > xfs_info /var
> > > > meta-data=/dev/sda1 isize=256 agcount=4, agsize=12800000 blks
> > > > = sectsz=4096 attr=2
> > > > data = bsize=4096 blocks=51200000, imaxpct=25
> > > > = sunit=0 swidth=0 blks
> > > > naming =version 2 bsize=4096 ascii-ci=0
> > > > log =internal bsize=4096 blocks=25000, version=2
> > > > = sectsz=4096 sunit=1 blks, lazy-count=1
> > > > realtime =none extsz=4096 blocks=0, rtextents=0
> > > >
> > > > /dev/sda1 on /var type xfs (rw,noatime,attr2,delaylog,logbsize=256k,noquota)
> > >
> > > Is your partition correctly sector aligned for however your drive
> > > maps it's 4k sectors?
> >
> > Yes, it's a GPT partition that is aligned to 1MB.
>
> Ok, that is fine, but the big question now is how does the drive
> align sector 0? Is that 4k aligned, or is it one of those drives
> that aligns an odd 512 byte logical sector to the physical 4k sector
> boundary (i.e. sector 63 is 4k aligned to work with msdos
> partitions). FYI, some drives have jumpers on them to change this
> odd/even sector alignment configuration.....
No, it's none of those (it's a Seagate Barracuda Green ST1500). Sector 0
is 4k aligned for sure. The odd 512 byte offset was present only on some
first generation drives.
But I think the whole alignment issue is a red herring, because I cannot
reproduce the "hangs" on the next partition on the same drive. This
partition is larger and contains my music and film collection (so mostly
static content and no traffic). And as I wrote in my other reply to this
thread: »it appears that the observed "hangs" are the result of a
strongly aged file-system.«
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 2:02 ` Markus Trippelsdorf
@ 2011-06-20 2:36 ` Dave Chinner
2011-06-20 6:03 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2011-06-20 2:36 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Mon, Jun 20, 2011 at 04:02:36AM +0200, Markus Trippelsdorf wrote:
> On 2011.06.20 at 11:34 +1000, Dave Chinner wrote:
> > On Mon, Jun 20, 2011 at 02:54:15AM +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.20 at 08:24 +1000, Dave Chinner wrote:
> > > > On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > > > > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > > > > periods (1-2 sec) whenever I delete a large directory recursively on my
> > > > > xfs partition. During the hang I cannot move the mouse pointer or use
> > > > > the keyboard (but the music keeps playing without stuttering). A quick
> > > > > way to reproduce is to "rm -fr" a kernel tree.
> > > >
> > > > So what is the system doing when it "hangs"? Is it CPU bound (e.g.
> > > > cpu scheduler issue)? Is the system running out of memory and
> > > > stalling everything in memory reclaim? What IO is occurring?
> > >
> > > It's totally idle otherwise; just a desktop with a single xterm. The
> > > machine has four cores (and also runs with "CONFIG_PREEMPT=y"), so I
> > > don't think it is CPU bound at all. It has 8GB of memory (and the
> > > "hangs" even occur after reboot when most of it is free). No other IO
> > > activity is occurring.
> >
> > Sure, the system might be otherwise idle, but what I was asking is
> > what load does the "rm -rf" cause. What IO does it cause? is it cpu
> > bound? etc.
>
> I have not measured this, so I cannot tell.
And so you are speculating as to the cause of the problem. What I'm
trying to do is work from the bottom up to ensure that the layers
below the fs are not the cause of the problem.
> > > > Is your partition correctly sector aligned for however your drive
> > > > maps it's 4k sectors?
> > >
> > > Yes, it's a GPT partition that is aligned to 1MB.
> >
> > Ok, that is fine, but the big question now is how does the drive
> > align sector 0? Is that 4k aligned, or is it one of those drives
> > that aligns an odd 512 byte logical sector to the physical 4k sector
> > boundary (i.e. sector 63 is 4k aligned to work with msdos
> > partitions). FYI, some drives have jumpers on them to change this
> > odd/even sector alignment configuration.....
>
> No, it's none of those (it's a Seagate Barracuda Green ST1500). Sector 0
> is 4k aligned for sure. The odd 512 byte offset was present only on some
> first generation drives.
> But I think the whole alignment issue is a red herring, because I cannot
> reproduce the "hangs" on the next partition on the same drive. This
> partition is larger and contains my music and film collection (so mostly
> static content and no traffic).
Which also means you might have one unaligned and one aligned
partition. i.e. the test results you have presented does not
necessarily point at a filesystem problem. We always ask for exact
details of your storage subsystem for these reasons - so we can
understand if there's something that you missed or didn't think was
important enough to tell us. You may have already checked those
things, but we don't know that if you don't tell us....
So, is the sector alignment of the second partition the same as the
first partition?
> And as I wrote in my other reply to this
> thread: »it appears that the observed "hangs" are the result of a
> strongly aged file-system.«
There is no evidence that points to any cause. Hell, I don't even
know what you consider a "strongly aged filesystem" looks like....
If the alignment is the cause of the problem, you should be able to
see a difference in performance when doing random 4k synchronous
writes to a large file on differently aligned partitions. Can you
run the same random 4k sync write test on both partitions (make sure
barriers are enabled) and determine if they perform the same?
If the filesystem layout is the cause of the problem, you should be
able to take a metadump of the problematic filesystem, restore it to
a normal 512 sector drive and reproduce the "rm -rf" problem. Can
you try this as well?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 2:36 ` Dave Chinner
@ 2011-06-20 6:03 ` Markus Trippelsdorf
2011-06-20 11:13 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-20 6:03 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.20 at 12:36 +1000, Dave Chinner wrote:
> On Mon, Jun 20, 2011 at 04:02:36AM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.20 at 11:34 +1000, Dave Chinner wrote:
> > > On Mon, Jun 20, 2011 at 02:54:15AM +0200, Markus Trippelsdorf wrote:
> > > > On 2011.06.20 at 08:24 +1000, Dave Chinner wrote:
> > > > > On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > > > > > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > > > > > periods (1-2 sec) whenever I delete a large directory recursively on my
> > > > > > xfs partition. During the hang I cannot move the mouse pointer or use
> > > > > > the keyboard (but the music keeps playing without stuttering). A quick
> > > > > > way to reproduce is to "rm -fr" a kernel tree.
> > > > >
> > > > > So what is the system doing when it "hangs"? Is it CPU bound (e.g.
> > > > > cpu scheduler issue)? Is the system running out of memory and
> > > > > stalling everything in memory reclaim? What IO is occurring?
> > > >
> > > > It's totally idle otherwise; just a desktop with a single xterm. The
> > > > machine has four cores (and also runs with "CONFIG_PREEMPT=y"), so I
> > > > don't think it is CPU bound at all. It has 8GB of memory (and the
> > > > "hangs" even occur after reboot when most of it is free). No other IO
> > > > activity is occurring.
> > >
> > > Sure, the system might be otherwise idle, but what I was asking is
> > > what load does the "rm -rf" cause. What IO does it cause? is it cpu
> > > bound? etc.
> >
> > I have not measured this, so I cannot tell.
>
> And so you are speculating as to the cause of the problem. What I'm
> trying to do is work from the bottom up to ensure that the layers
> below the fs are not the cause of the problem.
>
> > > > > Is your partition correctly sector aligned for however your drive
> > > > > maps it's 4k sectors?
> > > >
> > > > Yes, it's a GPT partition that is aligned to 1MB.
> > >
> > > Ok, that is fine, but the big question now is how does the drive
> > > align sector 0? Is that 4k aligned, or is it one of those drives
> > > that aligns an odd 512 byte logical sector to the physical 4k sector
> > > boundary (i.e. sector 63 is 4k aligned to work with msdos
> > > partitions). FYI, some drives have jumpers on them to change this
> > > odd/even sector alignment configuration.....
> >
> > No, it's none of those (it's a Seagate Barracuda Green ST1500). Sector 0
> > is 4k aligned for sure. The odd 512 byte offset was present only on some
> > first generation drives.
> > But I think the whole alignment issue is a red herring, because I cannot
> > reproduce the "hangs" on the next partition on the same drive. This
> > partition is larger and contains my music and film collection (so mostly
> > static content and no traffic).
>
> Which also means you might have one unaligned and one aligned
> partition. i.e. the test results you have presented does not
> necessarily point at a filesystem problem. We always ask for exact
> details of your storage subsystem for these reasons - so we can
> understand if there's something that you missed or didn't think was
> important enough to tell us. You may have already checked those
> things, but we don't know that if you don't tell us....
Understood.
> So, is the sector alignment of the second partition the same as the
> first partition?
Yes.
> > And as I wrote in my other reply to this
> > thread: »it appears that the observed "hangs" are the result of a
> > strongly aged file-system.«
>
> There is no evidence that points to any cause. Hell, I don't even
> know what you consider a "strongly aged filesystem" looks like....
>
> If the alignment is the cause of the problem, you should be able to
> see a difference in performance when doing random 4k synchronous
> writes to a large file on differently aligned partitions. Can you
> run the same random 4k sync write test on both partitions (make sure
> barriers are enabled) and determine if they perform the same?
>
> If the filesystem layout is the cause of the problem, you should be
> able to take a metadump of the problematic filesystem, restore it to
> a normal 512 sector drive and reproduce the "rm -rf" problem. Can
> you try this as well?
OK. I was able to reproduce the same hang on a conventional 512 sector drive.
The partition that I've used was the predecessor to the one on the 4k drive. So
it saw roughly the same usage pattern.
This is the output of "dstat -cdyl -C 0,1,2,3 -D sdc --disk-tps" during the
hang:
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
0 0 100 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 0 | 249 354 |0.33 0.58 0.38| 0 0
0 0 100 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 244 228 |0.33 0.58 0.38| 0 0
1 2 97 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 559 614 |0.33 0.58 0.38| 0 0
0 0 100 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0| 0 0 | 341 426 |0.33 0.58 0.38| 0 0
1 0 99 0 0 0: 1 4 95 0 0 0: 0 1 99 0 0 0: 1 16 83 0 0 0| 0 0 | 874 796 |0.33 0.58 0.38| 0 0
2 50 49 0 0 0: 1 9 90 0 0 0: 1 9 90 0 0 0: 1 23 76 0 0 0| 0 6400k|2803 2073 |0.46 0.60 0.39| 0 25
1 29 70 0 0 0: 1 1 98 0 0 0: 1 9 90 0 0 0: 1 53 46 0 0 0| 0 6400k|2047 1414 |0.46 0.60 0.39| 0 25
0 4 96 0 0 0: 0 0 100 0 0 0: 1 19 80 0 0 0: 0 80 20 0 0 0| 0 2048k|1425 685 |0.46 0.60 0.39| 0 8
2 1 97 0 0 0: 1 6 93 0 0 0: 0 5 95 0 0 0: 0 83 17 0 0 0| 0 4608k|1624 849 |0.46 0.60 0.39| 0 18
2 45 53 0 0 0: 1 16 83 0 0 0: 3 20 77 0 0 0: 1 15 84 0 0 0| 0 6400k|2420 1984 |0.46 0.60 0.39| 0 26
1 19 80 0 0 0: 2 8 90 0 0 0: 0 33 67 0 0 0: 0 33 67 0 0 0| 0 6400k|2694 2134 |0.59 0.63 0.40| 0 25
2 7 91 0 0 0: 2 1 97 0 0 0: 1 0 99 0 0 0: 0 49 10 41 0 0| 0 8269k|1865 1571 |0.59 0.63 0.40| 0 363
1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 0 99 0 0| 0 4778k|1509 1639 |0.59 0.63 0.40| 0 410
2 0 98 0 0 0: 2 1 97 0 0 0: 1 1 98 0 0 0: 2 0 0 98 0 0| 0 5318k|1663 1809 |0.59 0.63 0.40| 0 426
1 1 98 0 0 0: 2 7 91 0 0 0: 1 0 99 0 0 0: 1 0 0 99 0 0| 0 5446k|1659 1806 |0.59 0.63 0.40| 0 432
0 1 99 0 0 0: 1 0 99 0 0 0: 2 0 98 0 0 0: 0 1 17 82 0 0| 0 5472k|1572 1837 |0.62 0.63 0.40| 0 439
2 0 98 0 0 0: 2 2 96 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 397k|1058 1049 |0.62 0.63 0.40| 0 36
1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0| 0 0 | 617 689 |0.62 0.63 0.40| 0 0
9 4 87 0 0 0: 4 0 96 0 0 0: 1 1 98 0 0 0: 8 6 87 0 0 0| 0 0 |1234 1961 |0.62 0.63 0.40| 0 0
0 1 99 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 391 403 |0.62 0.63 0.40| 0 0
1 0 99 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 366 375 |0.57 0.62 0.40| 0 0
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 6:03 ` Markus Trippelsdorf
@ 2011-06-20 11:13 ` Markus Trippelsdorf
2011-06-20 11:45 ` Michael Monnerie
2011-06-21 4:25 ` Dave Chinner
0 siblings, 2 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-20 11:13 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.20 at 08:03 +0200, Markus Trippelsdorf wrote:
> On 2011.06.20 at 12:36 +1000, Dave Chinner wrote:
> > On Mon, Jun 20, 2011 at 04:02:36AM +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.20 at 11:34 +1000, Dave Chinner wrote:
> > > > On Mon, Jun 20, 2011 at 02:54:15AM +0200, Markus Trippelsdorf wrote:
> > > > > On 2011.06.20 at 08:24 +1000, Dave Chinner wrote:
> > > > > > On Sat, Jun 18, 2011 at 04:19:50PM +0200, Markus Trippelsdorf wrote:
> > > > > > > Running the latest git kernel (3.0-rc3) my machine hangs for long
> > > > > > > periods (1-2 sec) whenever I delete a large directory recursively on my
> > > > > > > xfs partition. During the hang I cannot move the mouse pointer or use
> > > > > > > the keyboard (but the music keeps playing without stuttering). A quick
> > > > > > > way to reproduce is to "rm -fr" a kernel tree.
> > > > > >
> > > > > > So what is the system doing when it "hangs"? Is it CPU bound (e.g.
> > > > > > cpu scheduler issue)? Is the system running out of memory and
> > > > > > stalling everything in memory reclaim? What IO is occurring?
> > > > >
> > > > > It's totally idle otherwise; just a desktop with a single xterm. The
> > > > > machine has four cores (and also runs with "CONFIG_PREEMPT=y"), so I
> > > > > don't think it is CPU bound at all. It has 8GB of memory (and the
> > > > > "hangs" even occur after reboot when most of it is free). No other IO
> > > > > activity is occurring.
> > > >
> > > > Sure, the system might be otherwise idle, but what I was asking is
> > > > what load does the "rm -rf" cause. What IO does it cause? is it cpu
> > > > bound? etc.
> > >
> > > I have not measured this, so I cannot tell.
> >
> > And so you are speculating as to the cause of the problem. What I'm
> > trying to do is work from the bottom up to ensure that the layers
> > below the fs are not the cause of the problem.
> >
> > > > > > Is your partition correctly sector aligned for however your drive
> > > > > > maps it's 4k sectors?
> > > > >
> > > > > Yes, it's a GPT partition that is aligned to 1MB.
> > > >
> > > > Ok, that is fine, but the big question now is how does the drive
> > > > align sector 0? Is that 4k aligned, or is it one of those drives
> > > > that aligns an odd 512 byte logical sector to the physical 4k sector
> > > > boundary (i.e. sector 63 is 4k aligned to work with msdos
> > > > partitions). FYI, some drives have jumpers on them to change this
> > > > odd/even sector alignment configuration.....
> > >
> > > No, it's none of those (it's a Seagate Barracuda Green ST1500). Sector 0
> > > is 4k aligned for sure. The odd 512 byte offset was present only on some
> > > first generation drives.
> > > But I think the whole alignment issue is a red herring, because I cannot
> > > reproduce the "hangs" on the next partition on the same drive. This
> > > partition is larger and contains my music and film collection (so mostly
> > > static content and no traffic).
> >
> > Which also means you might have one unaligned and one aligned
> > partition. i.e. the test results you have presented does not
> > necessarily point at a filesystem problem. We always ask for exact
> > details of your storage subsystem for these reasons - so we can
> > understand if there's something that you missed or didn't think was
> > important enough to tell us. You may have already checked those
> > things, but we don't know that if you don't tell us....
>
> Understood.
>
> > So, is the sector alignment of the second partition the same as the
> > first partition?
>
> Yes.
>
> > > And as I wrote in my other reply to this
> > > thread: »it appears that the observed "hangs" are the result of a
> > > strongly aged file-system.«
> >
> > There is no evidence that points to any cause. Hell, I don't even
> > know what you consider a "strongly aged filesystem" looks like....
> >
> > If the alignment is the cause of the problem, you should be able to
> > see a difference in performance when doing random 4k synchronous
> > writes to a large file on differently aligned partitions. Can you
> > run the same random 4k sync write test on both partitions (make sure
> > barriers are enabled) and determine if they perform the same?
> >
> > If the filesystem layout is the cause of the problem, you should be
> > able to take a metadump of the problematic filesystem, restore it to
> > a normal 512 sector drive and reproduce the "rm -rf" problem. Can
> > you try this as well?
>
> OK. I was able to reproduce the same hang on a conventional 512 sector drive.
> The partition that I've used was the predecessor to the one on the 4k drive. So
> it saw roughly the same usage pattern.
> This is the output of "dstat -cdyl -C 0,1,2,3 -D sdc --disk-tps" during the
> hang:
>
> -------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
> usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
>
> 0 0 100 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 0 | 249 354 |0.33 0.58 0.38| 0 0
> 0 0 100 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 244 228 |0.33 0.58 0.38| 0 0
> 1 2 97 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 559 614 |0.33 0.58 0.38| 0 0
> 0 0 100 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0| 0 0 | 341 426 |0.33 0.58 0.38| 0 0
> 1 0 99 0 0 0: 1 4 95 0 0 0: 0 1 99 0 0 0: 1 16 83 0 0 0| 0 0 | 874 796 |0.33 0.58 0.38| 0 0
> 2 50 49 0 0 0: 1 9 90 0 0 0: 1 9 90 0 0 0: 1 23 76 0 0 0| 0 6400k|2803 2073 |0.46 0.60 0.39| 0 25
> 1 29 70 0 0 0: 1 1 98 0 0 0: 1 9 90 0 0 0: 1 53 46 0 0 0| 0 6400k|2047 1414 |0.46 0.60 0.39| 0 25
> 0 4 96 0 0 0: 0 0 100 0 0 0: 1 19 80 0 0 0: 0 80 20 0 0 0| 0 2048k|1425 685 |0.46 0.60 0.39| 0 8
> 2 1 97 0 0 0: 1 6 93 0 0 0: 0 5 95 0 0 0: 0 83 17 0 0 0| 0 4608k|1624 849 |0.46 0.60 0.39| 0 18
> 2 45 53 0 0 0: 1 16 83 0 0 0: 3 20 77 0 0 0: 1 15 84 0 0 0| 0 6400k|2420 1984 |0.46 0.60 0.39| 0 26
> 1 19 80 0 0 0: 2 8 90 0 0 0: 0 33 67 0 0 0: 0 33 67 0 0 0| 0 6400k|2694 2134 |0.59 0.63 0.40| 0 25
> 2 7 91 0 0 0: 2 1 97 0 0 0: 1 0 99 0 0 0: 0 49 10 41 0 0| 0 8269k|1865 1571 |0.59 0.63 0.40| 0 363
> 1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 0 99 0 0| 0 4778k|1509 1639 |0.59 0.63 0.40| 0 410
> 2 0 98 0 0 0: 2 1 97 0 0 0: 1 1 98 0 0 0: 2 0 0 98 0 0| 0 5318k|1663 1809 |0.59 0.63 0.40| 0 426
> 1 1 98 0 0 0: 2 7 91 0 0 0: 1 0 99 0 0 0: 1 0 0 99 0 0| 0 5446k|1659 1806 |0.59 0.63 0.40| 0 432
> 0 1 99 0 0 0: 1 0 99 0 0 0: 2 0 98 0 0 0: 0 1 17 82 0 0| 0 5472k|1572 1837 |0.62 0.63 0.40| 0 439
> 2 0 98 0 0 0: 2 2 96 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 397k|1058 1049 |0.62 0.63 0.40| 0 36
> 1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0| 0 0 | 617 689 |0.62 0.63 0.40| 0 0
> 9 4 87 0 0 0: 4 0 96 0 0 0: 1 1 98 0 0 0: 8 6 87 0 0 0| 0 0 |1234 1961 |0.62 0.63 0.40| 0 0
> 0 1 99 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 391 403 |0.62 0.63 0.40| 0 0
> 1 0 99 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 366 375 |0.57 0.62 0.40| 0 0
>
Here are two more examples. The time when the hang occurs is marked with
"=>":
------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdb-- ---system-- ---load-avg--- --dsk/sdb--
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
0 0 100 0 0 0: 1 1 98 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0| 0 0 | 411 498 |1.14 0.58 0.27| 0 0
1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 0 100 0 0 0| 0 0 | 580 649 |1.05 0.57 0.27| 0 0
0 1 99 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0: 1 0 99 0 0 0| 0 0 | 297 353 |1.05 0.57 0.27| 0 0
1 0 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 322 393 |1.05 0.57 0.27| 0 0
0 0 100 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0| 0 0 | 301 382 |1.05 0.57 0.27| 0 0
1 16 83 0 0 0: 1 21 78 0 0 0: 0 13 87 0 0 0: 1 13 87 0 0 0| 0 2016k|2169 1495 |1.05 0.57 0.27| 0 57
1 10 89 0 0 0: 2 38 61 0 0 0: 1 8 81 10 0 0: 0 64 36 0 0 0| 0 4476k|2378 2010 |1.12 0.59 0.28| 0 146
=>0 0 100 0 0 0: 0 1 99 0 0 0: 0 1 0 99 0 0: 0 100 0 0 0 0| 0 0 |1237 283 |1.12 0.59 0.28| 0 0
0 27 73 0 0 0: 1 0 99 0 0 0: 0 0 47 53 0 0: 0 56 44 0 0 0| 0 5812k|1596 857 |1.12 0.59 0.28| 0 182
1 5 35 59 0 0: 2 0 85 13 0 0: 1 0 99 0 0 0: 0 7 28 65 0 0| 0 6263k|1789 1835 |1.12 0.59 0.28| 0 555
1 0 99 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0: 0 4 6 90 0 0| 0 2664k|1074 1155 |1.12 0.59 0.28| 0 286
4 1 95 0 0 0: 4 2 94 0 0 0: 2 1 97 0 0 0: 6 4 91 0 0 0| 0 1189k|1628 2106 |1.11 0.60 0.28| 0 137
1 0 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 1024B| 491 438 |1.11 0.60 0.28| 0 1
1 1 98 0 0 0: 2 0 98 0 0 0: 1 0 99 0 0 0: 0 0 100 0 0 0| 0 0 | 583 524 |1.11 0.60 0.28| 0 0
5 1 94 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 7 7 87 0 0 0| 0 0 |1148 1823 |1.11 0.60 0.28| 0 0
5 3 92 0 0 0: 4 1 95 0 0 0: 2 0 98 0 0 0: 5 2 93 0 0 0| 0 0 |1374 1678 |0.70 0.65 0.40| 0 0
2 2 96 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 1 0 99 0 0 0| 0 0 | 913 635 |0.70 0.65 0.40| 0 0
0 3 97 0 0 0: 1 0 99 0 0 0: 1 1 98 0 0 0: 1 5 94 0 0 0| 0 512B| 898 697 |0.70 0.65 0.40| 0 1
7 41 52 0 0 0: 2 11 87 0 0 0: 5 6 89 0 0 0: 3 1 96 0 0 0| 0 5856k|2771 2751 |0.70 0.65 0.40| 0 177
7 26 67 0 0 0: 4 1 95 0 0 0: 1 37 62 0 0 0: 3 38 59 0 0 0| 0 864k|2904 2806 |0.70 0.65 0.40| 0 33
=>2 1 97 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1725 639 |0.89 0.69 0.41| 0 0
1 1 98 0 0 0: 2 1 97 0 0 0: 2 1 97 0 0 0: 0 71 29 0 0 0| 0 3968k|1793 852 |0.89 0.69 0.41| 0 123
4 1 95 0 0 0: 3 48 49 0 0 0: 4 18 78 0 0 0: 5 6 89 0 0 0| 0 5120k|2920 3050 |0.89 0.69 0.41| 0 159
3 54 43 0 0 0: 1 8 91 0 0 0: 1 4 95 0 0 0: 1 5 94 0 0 0|4096B 4160k|2705 1936 |0.89 0.69 0.41| 1 132
1 40 59 0 0 0: 1 3 96 0 0 0: 1 13 86 0 0 0: 1 1 98 0 0 0| 0 5984k|2312 1568 |0.89 0.69 0.41| 0 187
1 23 76 0 0 0: 0 5 95 0 0 0: 1 16 83 0 0 0: 1 13 86 0 0 0| 0 6976k|2310 1751 |0.90 0.70 0.41| 0 213
0 1 31 68 0 0: 0 20 80 0 0 0: 0 19 36 45 0 0: 1 22 76 0 0 1| 0 8645k|2364 1773 |0.90 0.70 0.41| 0 578
1 0 98 1 0 0: 1 3 96 0 0 0: 1 1 28 70 0 0: 1 3 96 0 0 0| 0 4506k|1520 1358 |0.90 0.70 0.41| 0 374
2 2 96 0 0 0: 2 1 97 0 0 0: 0 0 100 0 0 0: 0 6 94 0 0 0| 0 504k| 877 726 |0.90 0.70 0.41| 0 92
1 1 98 0 0 0: 1 1 98 0 0 0: 1 0 99 0 0 0: 2 0 98 0 0 0| 0 78k| 907 753 |0.90 0.70 0.41| 0 17
1 3 96 0 0 0: 2 1 97 0 0 0: 0 1 99 0 0 0: 1 1 98 0 0 0| 0 0 | 894 710 |0.83 0.69 0.41| 0 0
2 0 98 0 0 0: 2 0 98 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0| 0 0 | 901 682 |0.83 0.69 0.41| 0 0
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 11:13 ` Markus Trippelsdorf
@ 2011-06-20 11:45 ` Michael Monnerie
2011-06-20 12:31 ` Markus Trippelsdorf
2011-06-21 4:25 ` Dave Chinner
1 sibling, 1 reply; 33+ messages in thread
From: Michael Monnerie @ 2011-06-20 11:45 UTC (permalink / raw)
To: xfs; +Cc: Markus Trippelsdorf
[-- Attachment #1.1: Type: Text/Plain, Size: 957 bytes --]
On Montag, 20. Juni 2011 Markus Trippelsdorf wrote:
> Here are two more examples. The time when the hang occurs is marked
Could it be that some sectors on the disk are not easy to read for the
drive, and that it simply retries several times until it works again?
SATA disks can show that behaviour. You could try with "dd" with
seek/skip parameters so you read 1gb at once, then skip 1gb and read 1gb
again etc, and compare the throughput over all 1gb areas. If there's one
slower, that might be the problem.
Maybe a check with "smartctl" could help, too.
Just an idea because during your hang, no CPU or I/O is done, so I guess
it wouldn't be the fault of Linux/XFS or other software, but more
hardware.
--
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc
it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531
// Haus zu verkaufen: http://zmi.at/langegg/
[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 11:45 ` Michael Monnerie
@ 2011-06-20 12:31 ` Markus Trippelsdorf
2011-06-20 21:16 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-20 12:31 UTC (permalink / raw)
To: Michael Monnerie; +Cc: xfs
On 2011.06.20 at 13:45 +0200, Michael Monnerie wrote:
> On Montag, 20. Juni 2011 Markus Trippelsdorf wrote:
> > Here are two more examples. The time when the hang occurs is marked
>
> Could it be that some sectors on the disk are not easy to read for the
> drive, and that it simply retries several times until it works again?
> SATA disks can show that behaviour. You could try with "dd" with
> seek/skip parameters so you read 1gb at once, then skip 1gb and read 1gb
> again etc, and compare the throughput over all 1gb areas. If there's one
> slower, that might be the problem.
>
> Maybe a check with "smartctl" could help, too.
Thanks for the hint, Michael. I've just checked the SMART status on
both disks and the 4kb drive looks indeed suspicious:
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8
The 512 byte drive appears to be fine. But I'm running the long
SMART self test on both of them right now and will report back
the result in a few hours.
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 12:31 ` Markus Trippelsdorf
@ 2011-06-20 21:16 ` Markus Trippelsdorf
0 siblings, 0 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-20 21:16 UTC (permalink / raw)
To: Michael Monnerie; +Cc: xfs
On 2011.06.20 at 14:31 +0200, Markus Trippelsdorf wrote:
> On 2011.06.20 at 13:45 +0200, Michael Monnerie wrote:
> > On Montag, 20. Juni 2011 Markus Trippelsdorf wrote:
> > > Here are two more examples. The time when the hang occurs is marked
> >
> > Could it be that some sectors on the disk are not easy to read for the
> > drive, and that it simply retries several times until it works again?
> > SATA disks can show that behaviour. You could try with "dd" with
> > seek/skip parameters so you read 1gb at once, then skip 1gb and read 1gb
> > again etc, and compare the throughput over all 1gb areas. If there's one
> > slower, that might be the problem.
> >
> > Maybe a check with "smartctl" could help, too.
>
> Thanks for the hint, Michael. I've just checked the SMART status on
> both disks and the 4kb drive looks indeed suspicious:
>
> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8
> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8
>
> The 512 byte drive appears to be fine. But I'm running the long
> SMART self test on both of them right now and will report back
> the result in a few hours.
Hmm, both tests ran fine without any errors. And the two SMART
attributes above are back to zero again (must have been a temporary
firmware hiccup).
As you can see in the data I've posted, the disk workload consists
almost only of writes. And I don't think a disk retries writes several
times. On the contrary a write to a bad sector should fix it, because
the drive can then remap it safely. (Current_Pending_Sector would
decrease and Reallocated_Sector_Ct would increase. But
Reallocated_Sector_Ct is still 0 on both affected drives)
And shouldn't I see these "hangs" in situations other than "rm -fr", if
the disk drive would be responsible?
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-20 11:13 ` Markus Trippelsdorf
2011-06-20 11:45 ` Michael Monnerie
@ 2011-06-21 4:25 ` Dave Chinner
2011-06-21 8:02 ` Markus Trippelsdorf
2011-06-21 10:18 ` Markus Trippelsdorf
1 sibling, 2 replies; 33+ messages in thread
From: Dave Chinner @ 2011-06-21 4:25 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Mon, Jun 20, 2011 at 01:13:59PM +0200, Markus Trippelsdorf wrote:
> On 2011.06.20 at 08:03 +0200, Markus Trippelsdorf wrote:
> > OK. I was able to reproduce the same hang on a conventional 512 sector drive.
> > The partition that I've used was the predecessor to the one on the 4k drive. So
> > it saw roughly the same usage pattern.
> > This is the output of "dstat -cdyl -C 0,1,2,3 -D sdc --disk-tps" during the
> > hang:
> >
> > -------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
> > usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
> >
> > 0 0 100 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 0 | 249 354 |0.33 0.58 0.38| 0 0
> > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 244 228 |0.33 0.58 0.38| 0 0
> > 1 2 97 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 559 614 |0.33 0.58 0.38| 0 0
> > 0 0 100 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0: 1 0 99 0 0 0| 0 0 | 341 426 |0.33 0.58 0.38| 0 0
> > 1 0 99 0 0 0: 1 4 95 0 0 0: 0 1 99 0 0 0: 1 16 83 0 0 0| 0 0 | 874 796 |0.33 0.58 0.38| 0 0
> > 2 50 49 0 0 0: 1 9 90 0 0 0: 1 9 90 0 0 0: 1 23 76 0 0 0| 0 6400k|2803 2073 |0.46 0.60 0.39| 0 25
> > 1 29 70 0 0 0: 1 1 98 0 0 0: 1 9 90 0 0 0: 1 53 46 0 0 0| 0 6400k|2047 1414 |0.46 0.60 0.39| 0 25
> > 0 4 96 0 0 0: 0 0 100 0 0 0: 1 19 80 0 0 0: 0 80 20 0 0 0| 0 2048k|1425 685 |0.46 0.60 0.39| 0 8
> > 2 1 97 0 0 0: 1 6 93 0 0 0: 0 5 95 0 0 0: 0 83 17 0 0 0| 0 4608k|1624 849 |0.46 0.60 0.39| 0 18
> > 2 45 53 0 0 0: 1 16 83 0 0 0: 3 20 77 0 0 0: 1 15 84 0 0 0| 0 6400k|2420 1984 |0.46 0.60 0.39| 0 26
> > 1 19 80 0 0 0: 2 8 90 0 0 0: 0 33 67 0 0 0: 0 33 67 0 0 0| 0 6400k|2694 2134 |0.59 0.63 0.40| 0 25
> > 2 7 91 0 0 0: 2 1 97 0 0 0: 1 0 99 0 0 0: 0 49 10 41 0 0| 0 8269k|1865 1571 |0.59 0.63 0.40| 0 363
> > 1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 0 99 0 0| 0 4778k|1509 1639 |0.59 0.63 0.40| 0 410
> > 2 0 98 0 0 0: 2 1 97 0 0 0: 1 1 98 0 0 0: 2 0 0 98 0 0| 0 5318k|1663 1809 |0.59 0.63 0.40| 0 426
> > 1 1 98 0 0 0: 2 7 91 0 0 0: 1 0 99 0 0 0: 1 0 0 99 0 0| 0 5446k|1659 1806 |0.59 0.63 0.40| 0 432
> > 0 1 99 0 0 0: 1 0 99 0 0 0: 2 0 98 0 0 0: 0 1 17 82 0 0| 0 5472k|1572 1837 |0.62 0.63 0.40| 0 439
> > 2 0 98 0 0 0: 2 2 96 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 397k|1058 1049 |0.62 0.63 0.40| 0 36
> > 1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0| 0 0 | 617 689 |0.62 0.63 0.40| 0 0
> > 9 4 87 0 0 0: 4 0 96 0 0 0: 1 1 98 0 0 0: 8 6 87 0 0 0| 0 0 |1234 1961 |0.62 0.63 0.40| 0 0
> > 0 1 99 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 391 403 |0.62 0.63 0.40| 0 0
> > 1 0 99 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 366 375 |0.57 0.62 0.40| 0 0
What is the resolution of the samples here? Where did the hang occur
during this output?
FWIW, Can you capture the hang while running 'iostat -x -d -m 1' so
we can see what is happening with queue depths, average Io sizes,
etc?
> Here are two more examples. The time when the hang occurs is marked with
> "=>":
> ------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdb-- ---system-- ---load-avg--- --dsk/sdb--
> usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
.....
> 1 10 89 0 0 0: 2 38 61 0 0 0: 1 8 81 10 0 0: 0 64 36 0 0 0| 0 4476k|2378 2010 |1.12 0.59 0.28| 0 146
> =>0 0 100 0 0 0: 0 1 99 0 0 0: 0 1 0 99 0 0: 0 100 0 0 0 0| 0 0 |1237 283 |1.12 0.59 0.28| 0 0
> 0 27 73 0 0 0: 1 0 99 0 0 0: 0 0 47 53 0 0: 0 56 44 0 0 0| 0 5812k|1596 857 |1.12 0.59 0.28| 0 182
.....
> 7 26 67 0 0 0: 4 1 95 0 0 0: 1 37 62 0 0 0: 3 38 59 0 0 0| 0 864k|2904 2806 |0.70 0.65 0.40| 0 33
> =>2 1 97 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1725 639 |0.89 0.69 0.41| 0 0
> 1 1 98 0 0 0: 2 1 97 0 0 0: 2 1 97 0 0 0: 0 71 29 0 0 0| 0 3968k|1793 852 |0.89 0.69 0.41| 0 123
.....
So in both cases here the hang occurs when there is -zero- IO
occurring, and a CPU has pegged at 100% in system time. That's CPU
bound doing <something>. It is possible that the CPU is getting
caught in a loop somewhere or has a -lot- of processing to do before
progress is made.
I know a CIL commit can take some time to process all the
objects in a checkpoint, but I haven't seen anything like this.
You've only got a relatively small log (100MB) and you're only
removing a kernel source tree, so there really shouldn't be an
excessive number of objects built up to process per checkpoint.
FWIW, I'm pretty sure a CPU getting stuck like this in the
filesystem code should not be causing problems with X or other
non-filesystem workloads. There's 3 idle CPU cores and you are
running a preemptible kernel, so I really can't see why system time
spent in XFS would cause mouse or keyboard updates to not be
processed in a timely manner. We really need to know what is
consuming all that CPU time.
That is, you really need to get a profile of the rm -rf process - or
whatever is consuming the CPU time - (e.g. via perf or ftrace)
across the hang to so we can narrow down the potential cause of the
latency. Speaking of which, latencytop might be helpful in
identifying where input is getting held up....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-21 4:25 ` Dave Chinner
@ 2011-06-21 8:02 ` Markus Trippelsdorf
2011-06-21 18:24 ` Markus Trippelsdorf
2011-06-21 10:18 ` Markus Trippelsdorf
1 sibling, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-21 8:02 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
[-- Attachment #1: Type: text/plain, Size: 35422 bytes --]
On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> On Mon, Jun 20, 2011 at 01:13:59PM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.20 at 08:03 +0200, Markus Trippelsdorf wrote:
> > > OK. I was able to reproduce the same hang on a conventional 512 sector drive.
> > > The partition that I've used was the predecessor to the one on the 4k drive. So
> > > it saw roughly the same usage pattern.
> > > 1 29 70 0 0 0: 1 1 98 0 0 0: 1 9 90 0 0 0: 1 53 46 0 0 0| 0 6400k|2047 1414 |0.46 0.60 0.39| 0 25
> > > 0 4 96 0 0 0: 0 0 100 0 0 0: 1 19 80 0 0 0: 0 80 20 0 0 0| 0 2048k|1425 685 |0.46 0.60 0.39| 0 8
> > > 2 1 97 0 0 0: 1 6 93 0 0 0: 0 5 95 0 0 0: 0 83 17 0 0 0| 0 4608k|1624 849 |0.46 0.60 0.39| 0 18
> > > 2 45 53 0 0 0: 1 16 83 0 0 0: 3 20 77 0 0 0: 1 15 84 0 0 0| 0 6400k|2420 1984 |0.46 0.60 0.39| 0 26
> > > 1 19 80 0 0 0: 2 8 90 0 0 0: 0 33 67 0 0 0: 0 33 67 0 0 0| 0 6400k|2694 2134 |0.59 0.63 0.40| 0 25
> > > 2 7 91 0 0 0: 2 1 97 0 0 0: 1 0 99 0 0 0: 0 49 10 41 0 0| 0 8269k|1865 1571 |0.59 0.63 0.40| 0 363
> > > 1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 0 99 0 0| 0 4778k|1509 1639 |0.59 0.63 0.40| 0 410
> > > 2 0 98 0 0 0: 2 1 97 0 0 0: 1 1 98 0 0 0: 2 0 0 98 0 0| 0 5318k|1663 1809 |0.59 0.63 0.40| 0 426
> > > 1 1 98 0 0 0: 2 7 91 0 0 0: 1 0 99 0 0 0: 1 0 0 99 0 0| 0 5446k|1659 1806 |0.59 0.63 0.40| 0 432
> > > 0 1 99 0 0 0: 1 0 99 0 0 0: 2 0 98 0 0 0: 0 1 17 82 0 0| 0 5472k|1572 1837 |0.62 0.63 0.40| 0 439
> > > 2 0 98 0 0 0: 2 2 96 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 397k|1058 1049 |0.62 0.63 0.40| 0 36
> > > 1 1 98 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0| 0 0 | 617 689 |0.62 0.63 0.40| 0 0
> > > 9 4 87 0 0 0: 4 0 96 0 0 0: 1 1 98 0 0 0: 8 6 87 0 0 0| 0 0 |1234 1961 |0.62 0.63 0.40| 0 0
> > > 0 1 99 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 391 403 |0.62 0.63 0.40| 0 0
> > > 1 0 99 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 366 375 |0.57 0.62 0.40| 0 0
>
> What is the resolution of the samples here? Where did the hang occur
> during this output?
Resolution is 1 sec. I think the hang occurred when cpu3 wait switched
from 41% to 99% (but I'm not sure).
> FWIW, Can you capture the hang while running 'iostat -x -d -m 1' so
> we can see what is happening with queue depths, average Io sizes,
> etc?
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc1 0.00 0.00 1.00 202.00 0.00 6.31 63.72 3.90 19.21 19.00 19.21 2.57 52.10
sdc1 0.00 0.00 0.00 8.00 0.00 0.25 64.00 0.15 18.50 0.00 18.50 2.50 2.00
sdc1 *hang* 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc1 0.00 0.00 0.00 116.00 0.00 3.78 66.76 2.25 18.66 0.00 18.66 2.45 28.40
sdc1 0.00 9.00 0.00 210.00 0.00 2.25 21.92 3.96 19.28 0.00 19.28 4.43 93.00
sdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> > Here are two more examples. The time when the hang occurs is marked with
> > "=>":
> > ------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdb-- ---system-- ---load-avg--- --dsk/sdb--
> > usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
> .....
> > 1 10 89 0 0 0: 2 38 61 0 0 0: 1 8 81 10 0 0: 0 64 36 0 0 0| 0 4476k|2378 2010 |1.12 0.59 0.28| 0 146
> > =>0 0 100 0 0 0: 0 1 99 0 0 0: 0 1 0 99 0 0: 0 100 0 0 0 0| 0 0 |1237 283 |1.12 0.59 0.28| 0 0
> > 0 27 73 0 0 0: 1 0 99 0 0 0: 0 0 47 53 0 0: 0 56 44 0 0 0| 0 5812k|1596 857 |1.12 0.59 0.28| 0 182
> .....
> > 7 26 67 0 0 0: 4 1 95 0 0 0: 1 37 62 0 0 0: 3 38 59 0 0 0| 0 864k|2904 2806 |0.70 0.65 0.40| 0 33
> > =>2 1 97 0 0 0: 1 0 99 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1725 639 |0.89 0.69 0.41| 0 0
> > 1 1 98 0 0 0: 2 1 97 0 0 0: 2 1 97 0 0 0: 0 71 29 0 0 0| 0 3968k|1793 852 |0.89 0.69 0.41| 0 123
> .....
>
> So in both cases here the hang occurs when there is -zero- IO
> occurring, and a CPU has pegged at 100% in system time. That's CPU
> bound doing <something>. It is possible that the CPU is getting
> caught in a loop somewhere or has a -lot- of processing to do before
> progress is made.
>
> I know a CIL commit can take some time to process all the
> objects in a checkpoint, but I haven't seen anything like this.
> You've only got a relatively small log (100MB) and you're only
> removing a kernel source tree, so there really shouldn't be an
> excessive number of objects built up to process per checkpoint.
>
> FWIW, I'm pretty sure a CPU getting stuck like this in the
> filesystem code should not be causing problems with X or other
> non-filesystem workloads. There's 3 idle CPU cores and you are
> running a preemptible kernel, so I really can't see why system time
> spent in XFS would cause mouse or keyboard updates to not be
> processed in a timely manner. We really need to know what is
> consuming all that CPU time.
>
> That is, you really need to get a profile of the rm -rf process - or
> whatever is consuming the CPU time - (e.g. via perf or ftrace)
> across the hang to so we can narrow down the potential cause of the
> latency. Speaking of which, latencytop might be helpful in
> identifying where input is getting held up....
I've recorded a profile with "perf record -g /home/markus/rm_sync"
~ % cat rm_sync
rm -fr /mnt/tmp/tmp/linux && sync
This is the output of "/perf report -g --stdio" (Just the first entries
the full report is attached).
# Events: 1K cycles
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ..................................
#
5.32% rm [kernel.kallsyms] [k] __memcpy
|
--- __memcpy
|
|--90.77%-- _xfs_trans_commit
| |
| |--62.41%-- xfs_itruncate_finish
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| |--19.50%-- xfs_remove
| | xfs_vn_unlink
| | vfs_unlink
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| |--16.70%-- xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --1.39%-- xfs_bmap_finish
| xfs_itruncate_finish
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--5.44%-- xlog_cil_push
| xfs_log_commit_cil
| _xfs_trans_commit
| xfs_remove
| xfs_vn_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--2.53%-- xfs_btree_insrec
| xfs_btree_insert
| xfs_free_ag_extent
| xfs_free_extent
| xfs_bmap_finish
| xfs_itruncate_finish
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--1.26%-- xfs_btree_update
xfs_inobt_update
xfs_difree
xfs_ifree
xfs_inactive
xfs_fs_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
3.85% sync [kernel.kallsyms] [k] xfs_inode_ag_walk.isra.7
|
--- xfs_inode_ag_walk.isra.7
xfs_inode_ag_iterator
xfs_sync_data
xfs_quiesce_data
xfs_fs_sync_fs
__sync_filesystem
sync_one_sb
iterate_supers
sync_filesystems
sys_sync
system_call_fastpath
__GI_sync
2.68% sync [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
mapping_tagged
xfs_sync_inode_data
xfs_inode_ag_walk.isra.7
xfs_inode_ag_iterator
xfs_sync_data
xfs_quiesce_data
xfs_fs_sync_fs
__sync_filesystem
sync_one_sb
iterate_supers
sync_filesystems
sys_sync
system_call_fastpath
__GI_sync
2.56% rm [kernel.kallsyms] [k] xfs_next_bit
|
--- xfs_next_bit
|
|--31.60%-- xfs_log_commit_cil
| _xfs_trans_commit
| |
| |--91.65%-- xfs_itruncate_finish
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --8.35%-- xfs_remove
| xfs_vn_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--28.93%-- xfs_buf_item_size
| xfs_trans_alloc_log_vecs
| _xfs_trans_commit
| |
| |--72.71%-- xfs_itruncate_finish
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| |--18.18%-- xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --9.11%-- xfs_remove
| xfs_vn_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--26.33%-- xfs_buf_item_format
| xfs_log_commit_cil
| _xfs_trans_commit
| |
| |--70.10%-- xfs_itruncate_finish
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| |--19.91%-- xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --9.99%-- xfs_remove
| xfs_vn_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--13.14%-- xfs_trans_alloc_log_vecs
_xfs_trans_commit
|
|--60.17%-- xfs_remove
| xfs_vn_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--39.83%-- xfs_inactive
xfs_fs_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
2.56% rm [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
|
|--39.47%-- xfs_trans_free_item_desc
| |
| |--79.98%-- xfs_trans_free_items
| | xfs_log_commit_cil
| | _xfs_trans_commit
| | |
| | |--50.02%-- xfs_inactive
| | | xfs_fs_evict_inode
| | | evict
| | | iput
| | | do_unlinkat
| | | sys_unlinkat
| | | system_call_fastpath
| | | unlinkat
| | |
| | |--33.35%-- xfs_itruncate_finish
| | | xfs_inactive
| | | xfs_fs_evict_inode
| | | evict
| | | iput
| | | do_unlinkat
| | | sys_unlinkat
| | | system_call_fastpath
| | | unlinkat
| | |
| | --16.63%-- xfs_bmap_finish
| | xfs_itruncate_finish
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --20.02%-- xfs_trans_del_item
| xfs_trans_brelse
| xfs_btree_del_cursor
| |
| |--66.61%-- xfs_free_ag_extent
| | xfs_free_extent
| | xfs_bmap_finish
| | |
| | |--50.15%-- xfs_remove
| | | xfs_vn_unlink
| | | vfs_unlink
| | | do_unlinkat
| | | sys_unlinkat
| | | system_call_fastpath
| | | unlinkat
| | |
| | --49.85%-- xfs_itruncate_finish
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --33.39%-- xfs_difree
| xfs_ifree
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--31.56%-- xfs_buf_get
| xfs_buf_read
| xfs_trans_read_buf
| |
| |--49.95%-- xfs_btree_read_buf_block.constprop.22
| | xfs_btree_lookup_get_block
| | xfs_btree_lookup
| | |
| | |--49.98%-- xfs_alloc_lookup_eq
| | | xfs_free_ag_extent
| | | xfs_free_extent
| | | xfs_bmap_finish
| | | xfs_itruncate_finish
| | | xfs_inactive
| | | xfs_fs_evict_inode
| | | evict
| | | iput
| | | do_unlinkat
| | | sys_unlinkat
| | | system_call_fastpath
| | | unlinkat
| | |
| | |--33.29%-- xfs_free_ag_extent
| | | xfs_free_extent
| | | xfs_bmap_finish
| | | xfs_itruncate_finish
| | | xfs_inactive
| | | xfs_fs_evict_inode
| | | evict
| | | iput
| | | do_unlinkat
| | | sys_unlinkat
| | | system_call_fastpath
| | | unlinkat
| | |
| | --16.74%-- xfs_inobt_lookup
| | xfs_difree
| | xfs_ifree
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| |--24.99%-- xfs_imap_to_bp.isra.9
| | xfs_itobp
| | xfs_iunlink_remove
| | xfs_ifree
| | xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | |
| | |--66.63%-- do_unlinkat
| | | sys_unlinkat
| | | system_call_fastpath
| | | unlinkat
| | |
| | --33.37%-- d_delete
| | vfs_rmdir
| | do_rmdir
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| |--16.70%-- xfs_read_agi
| | |
| | |--50.05%-- xfs_iunlink_remove
| | | xfs_ifree
| | | xfs_inactive
| | | xfs_fs_evict_inode
| | | evict
| | | iput
| | | do_unlinkat
| | | sys_unlinkat
| | | system_call_fastpath
| | | unlinkat
| | |
| | --49.95%-- xfs_iunlink
| | xfs_droplink
| | xfs_remove
| | xfs_vn_unlink
| | vfs_unlink
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --8.36%-- xfs_read_agf
| xfs_alloc_read_agf
| xfs_alloc_fix_freelist
| xfs_free_extent
| xfs_bmap_finish
| xfs_itruncate_finish
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--5.27%-- xfs_trans_free
| _xfs_trans_commit
| |
| |--50.08%-- xfs_inactive
| | xfs_fs_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --49.92%-- xfs_remove
| xfs_vn_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--5.27%-- xfs_trans_free_items
| xfs_log_commit_cil
| _xfs_trans_commit
| |
| |--50.13%-- xfs_remove
| | xfs_vn_unlink
| | vfs_unlink
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --49.87%-- xfs_itruncate_finish
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--2.64%-- xfs_trans_del_item
| xfs_trans_brelse
| xfs_iunlink_remove
| xfs_ifree
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--2.64%-- __rcu_process_callbacks
| rcu_process_callbacks
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| xfs_btree_lookup
| xfs_inobt_lookup
| xfs_difree
| xfs_ifree
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--2.64%-- free_buffer_head
| try_to_free_buffers
| xfs_vm_releasepage
| try_to_release_page
| block_invalidatepage
| xfs_vm_invalidatepage
| truncate_inode_page
| truncate_inode_pages_range
| truncate_inode_pages
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--2.64%-- xfs_bmap_del_free.constprop.18
| xfs_bmap_finish
| xfs_itruncate_finish
| xfs_inactive
| xfs_fs_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--2.63%-- xfs_log_done
| xfs_log_commit_cil
| _xfs_trans_commit
| xfs_remove
| xfs_vn_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--2.62%-- do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--2.62%-- xfs_difree
xfs_ifree
xfs_inactive
xfs_fs_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
--
Markus
[-- Attachment #2: report_hang.gz --]
[-- Type: application/octet-stream, Size: 23016 bytes --]
[-- Attachment #3: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-21 4:25 ` Dave Chinner
2011-06-21 8:02 ` Markus Trippelsdorf
@ 2011-06-21 10:18 ` Markus Trippelsdorf
2011-06-21 10:40 ` Markus Trippelsdorf
1 sibling, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-21 10:18 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> On Mon, Jun 20, 2011 at 01:13:59PM +0200, Markus Trippelsdorf wrote:
>
> That is, you really need to get a profile of the rm -rf process - or
> whatever is consuming the CPU time - (e.g. via perf or ftrace)
> across the hang to so we can narrow down the potential cause of the
> latency. Speaking of which, latencytop might be helpful in
> identifying where input is getting held up....
Just ran latencytop and this the result:
[xlog_state_get_iclog_space] 2104.4 msec 38.1 %
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-21 10:18 ` Markus Trippelsdorf
@ 2011-06-21 10:40 ` Markus Trippelsdorf
0 siblings, 0 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-21 10:40 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.21 at 12:18 +0200, Markus Trippelsdorf wrote:
> On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > On Mon, Jun 20, 2011 at 01:13:59PM +0200, Markus Trippelsdorf wrote:
> >
> > That is, you really need to get a profile of the rm -rf process - or
> > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > across the hang to so we can narrow down the potential cause of the
> > latency. Speaking of which, latencytop might be helpful in
> > identifying where input is getting held up....
>
> Just ran latencytop and this the result:
>
> [xlog_state_get_iclog_space] 2104.4 msec 38.1 %
And another example:
Cause Maximum Percentage
[xlog_state_get_iclog_space] 2098.7 msec 38.0 %
Waiting for TTY data 1953.4 msec 29.7 %
Syncing filesystem 723.0 msec 12.1 %
Unlinking file 25.8 msec 7.7 %
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-21 8:02 ` Markus Trippelsdorf
@ 2011-06-21 18:24 ` Markus Trippelsdorf
2011-06-21 18:57 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-21 18:24 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > That is, you really need to get a profile of the rm -rf process - or
> > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > across the hang to so we can narrow down the potential cause of the
> > latency. Speaking of which, latencytop might be helpful in
> > identifying where input is getting held up....
>
> I've recorded a profile with "perf record -g /home/markus/rm_sync"
> ~ % cat rm_sync
> rm -fr /mnt/tmp/tmp/linux && sync
FWIW here are two links to svg time-charts produced by:
perf timechart record /home/markus/rm_sync
http://trippelsdorf.de/timechart1.svg
http://trippelsdorf.de/timechart2.svg
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-21 18:24 ` Markus Trippelsdorf
@ 2011-06-21 18:57 ` Markus Trippelsdorf
2011-06-21 21:22 ` Markus Trippelsdorf
2011-06-22 0:04 ` Dave Chinner
0 siblings, 2 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-21 18:57 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.21 at 20:24 +0200, Markus Trippelsdorf wrote:
> On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> > On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > > That is, you really need to get a profile of the rm -rf process - or
> > > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > > across the hang to so we can narrow down the potential cause of the
> > > latency. Speaking of which, latencytop might be helpful in
> > > identifying where input is getting held up....
> >
> > I've recorded a profile with "perf record -g /home/markus/rm_sync"
> > ~ % cat rm_sync
> > rm -fr /mnt/tmp/tmp/linux && sync
>
> FWIW here are two links to svg time-charts produced by:
>
> perf timechart record /home/markus/rm_sync
>
> http://trippelsdorf.de/timechart1.svg
> http://trippelsdorf.de/timechart2.svg
>
And this is what the mysterious kworker is doing during the sync.
It's the one consuming most of the CPU time.
39.96% kworker/3:0 [kernel.kallsyms] 0xffffffff811da9da k [k] xfs_trans_ail_update_bulk
|
--- xfs_trans_ail_update_bulk
xfs_trans_committed_bulk
xlog_cil_committed
xlog_state_do_callback
xlog_state_done_syncing
xlog_iodone
xfs_buf_iodone_work
process_one_work
worker_thread
kthread
kernel_thread_helper
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-21 18:57 ` Markus Trippelsdorf
@ 2011-06-21 21:22 ` Markus Trippelsdorf
2011-06-22 0:04 ` Dave Chinner
1 sibling, 0 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-21 21:22 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.21 at 20:57 +0200, Markus Trippelsdorf wrote:
> On 2011.06.21 at 20:24 +0200, Markus Trippelsdorf wrote:
> > On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > > > That is, you really need to get a profile of the rm -rf process - or
> > > > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > > > across the hang to so we can narrow down the potential cause of the
> > > > latency. Speaking of which, latencytop might be helpful in
> > > > identifying where input is getting held up....
> > >
> > > I've recorded a profile with "perf record -g /home/markus/rm_sync"
> > > ~ % cat rm_sync
> > > rm -fr /mnt/tmp/tmp/linux && sync
> >
> > FWIW here are two links to svg time-charts produced by:
> >
> > perf timechart record /home/markus/rm_sync
> >
> > http://trippelsdorf.de/timechart1.svg
> > http://trippelsdorf.de/timechart2.svg
> >
>
> And this is what the mysterious kworker is doing during the sync.
> It's the one consuming most of the CPU time.
>
> 39.96% kworker/3:0 [kernel.kallsyms] 0xffffffff811da9da k [k] xfs_trans_ail_update_bulk
> |
> --- xfs_trans_ail_update_bulk
> xfs_trans_committed_bulk
> xlog_cil_committed
> xlog_state_do_callback
> xlog_state_done_syncing
> xlog_iodone
> xfs_buf_iodone_work
> process_one_work
> worker_thread
> kthread
> kernel_thread_helper
>
The following patch fixes the problem for me.
diff --git a/fs/xfs/linux-2.6/xfs_buf.c b/fs/xfs/linux-2.6/xfs_buf.c
index 5e68099..2f34efd 100644
--- a/fs/xfs/linux-2.6/xfs_buf.c
+++ b/fs/xfs/linux-2.6/xfs_buf.c
@@ -1856,7 +1856,7 @@ xfs_buf_init(void)
goto out;
xfslogd_workqueue = alloc_workqueue("xfslogd",
- WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
+ WQ_MEM_RECLAIM | WQ_UNBOUND, 1);
if (!xfslogd_workqueue)
goto out_free_buf_zone;
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-21 18:57 ` Markus Trippelsdorf
2011-06-21 21:22 ` Markus Trippelsdorf
@ 2011-06-22 0:04 ` Dave Chinner
2011-06-22 7:06 ` Markus Trippelsdorf
1 sibling, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2011-06-22 0:04 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Tue, Jun 21, 2011 at 08:57:01PM +0200, Markus Trippelsdorf wrote:
> On 2011.06.21 at 20:24 +0200, Markus Trippelsdorf wrote:
> > On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > > > That is, you really need to get a profile of the rm -rf process - or
> > > > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > > > across the hang to so we can narrow down the potential cause of the
> > > > latency. Speaking of which, latencytop might be helpful in
> > > > identifying where input is getting held up....
> > >
> > > I've recorded a profile with "perf record -g /home/markus/rm_sync"
> > > ~ % cat rm_sync
> > > rm -fr /mnt/tmp/tmp/linux && sync
> >
> > FWIW here are two links to svg time-charts produced by:
> >
> > perf timechart record /home/markus/rm_sync
> >
> > http://trippelsdorf.de/timechart1.svg
> > http://trippelsdorf.de/timechart2.svg
> >
>
> And this is what the mysterious kworker is doing during the sync.
> It's the one consuming most of the CPU time.
>
> 39.96% kworker/3:0 [kernel.kallsyms] 0xffffffff811da9da k [k] xfs_trans_ail_update_bulk
> |
> --- xfs_trans_ail_update_bulk
> xfs_trans_committed_bulk
> xlog_cil_committed
> xlog_state_do_callback
> xlog_state_done_syncing
> xlog_iodone
> xfs_buf_iodone_work
> process_one_work
> worker_thread
> kthread
> kernel_thread_helper
So that is inserting items into the AIL at transaction completion.
That can consume lots of CPU time if the CIL checkpoints are being
flushed quickly enough. Given you are doing a rm -rf at this point
in time, I'd expect to see this trace present in the profile, though
maybe not at that extent.
I have seen this before but have never been able to it reproduce
reliably. If checkpoints are completed "out of order" due to the
way the commit records are written into the iclogs. This can cause
extra CPU because the AIL insertion then has to skip over all the
items that were inserted out of order before splicing each group of
items into the AIL. I only rarely see this (maybe once every couple
of weeks of performance testing), so I'm not sure it's the problem
you are seeing.
Adding this debug to xfs_ail_splice() list walk will tell us if this is
happening and how many items it had to walk when you see a hang:
xfs_lsn_t lsn)
{
xfs_log_item_t *next_lip;
+ int walked = 0;
/* If the list is empty, just insert the item. */
if (list_empty(&ailp->xa_ail)) {
list_splice(list, &ailp->xa_ail);
return;
}
list_for_each_entry_reverse(next_lip, &ailp->xa_ail, li_ail) {
if (XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0)
break;
+ if (!walked++) {
+ xfs_warn(ailp->xa_mount,
+ "ail: ooo splice, tail 0x%llx, item 0x%llx\n",
+ next_lip->li_lsn, lsn);
+ }
}
+ if (walked > 10) {
+ xfs_warn(ailp->xa_mount,
+ "ail: ooo splice, walked %d items\n", walked);
+ }
ASSERT(&next_lip->li_ail == &ailp->xa_ail ||
XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0);
That will at least tell us if this is the cause of your problem. If
it is, I think I know how to avoid most of the list walk overhead
fairly easily and that should avoid the need to change workqueue
configurations at all.
> The following patch fixes the problem for me.
>
> diff --git a/fs/xfs/linux-2.6/xfs_buf.c b/fs/xfs/linux-2.6/xfs_buf.c
> index 5e68099..2f34efd 100644
> --- a/fs/xfs/linux-2.6/xfs_buf.c
> +++ b/fs/xfs/linux-2.6/xfs_buf.c
> @@ -1856,7 +1856,7 @@ xfs_buf_init(void)
> goto out;
>
> xfslogd_workqueue = alloc_workqueue("xfslogd",
> - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> + WQ_MEM_RECLAIM | WQ_UNBOUND, 1);
> if (!xfslogd_workqueue)
> goto out_free_buf_zone;
That implies the source of your X/ssh hangs is a workqueue priority
inversion.
We need the xfslogd to run with WQ_HIGHPRI because we can deadlock
in IO completion if it is blocked by xfsdatad/xfsconvertd wq
operations. Hence we need to process the logd workqueue ahead of
data IO completions and we do that by making the log IO completion
high priority.
This thing is, the CMWQ infrastructure only has one level of "high
priority" - we can't define any other sort of dependency between
various queues - and that queue is simply a FIFO. IOWs, CMWQ will
process anything that is high priority ahead of -all- other normal
priority work items. That means that if you get a high priority
workqueue with lots to do and consuming CPU (as we are seeing here)
it will starve all other work items that are queued on that CPU.
So what you end up seeing is interactive process related work being
queued on CPU 3 which is then being starved of processing by XFS IO
completion work on the same CPU. By setting the xfslogd workqueue to
unbound, it no longer holds precedence over any of the other work
and hence does not starve the queued interactive process related work.
That's why the patch "fixes" your 1-2s hangs while log IO is in
progress, but it will eventually deadlock and hang the filesystem
permanently.
There are two different things you can try with the wq
initialisation that might help prevent the problem. Firstly, try
this:
- WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
+ WQ_MEM_RECLAIM | WQ_HIGHPRI, 8);
To change the number of concurrent per-cpu work items that can be
processed on the work queue. I don't think that will fix the
inversion, but it may allow more concurrency which hides the
inversion.
The other thing you can try is:
- WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
+ WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_CPU_INTENSIVE, 1);
Which marks the workqueue as one that will consume a lot of CPU and
hence it is scheduled differently and hence should avoid other
pending work from being starved. We use this WQ_CPU_INTENSIVE flag
on other XFS workqueues that are known to consume lots of CPU, so I
suspect this is the right thing to do here.
Hopefully one of these things will point to a potential solution.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-22 0:04 ` Dave Chinner
@ 2011-06-22 7:06 ` Markus Trippelsdorf
2011-06-22 7:30 ` Dave Chinner
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-22 7:06 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
[-- Attachment #1: Type: text/plain, Size: 6613 bytes --]
On 2011.06.22 at 10:04 +1000, Dave Chinner wrote:
> On Tue, Jun 21, 2011 at 08:57:01PM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.21 at 20:24 +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> > > > On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > > > > That is, you really need to get a profile of the rm -rf process - or
> > > > > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > > > > across the hang to so we can narrow down the potential cause of the
> > > > > latency. Speaking of which, latencytop might be helpful in
> > > > > identifying where input is getting held up....
> > > >
> > > > I've recorded a profile with "perf record -g /home/markus/rm_sync"
> > > > ~ % cat rm_sync
> > > > rm -fr /mnt/tmp/tmp/linux && sync
> > >
> > > FWIW here are two links to svg time-charts produced by:
> > >
> > > perf timechart record /home/markus/rm_sync
> > >
> > > http://trippelsdorf.de/timechart1.svg
> > > http://trippelsdorf.de/timechart2.svg
> > >
> >
> > And this is what the mysterious kworker is doing during the sync.
> > It's the one consuming most of the CPU time.
> >
> > 39.96% kworker/3:0 [kernel.kallsyms] 0xffffffff811da9da k [k] xfs_trans_ail_update_bulk
> > |
> > --- xfs_trans_ail_update_bulk
> > xfs_trans_committed_bulk
> > xlog_cil_committed
> > xlog_state_do_callback
> > xlog_state_done_syncing
> > xlog_iodone
> > xfs_buf_iodone_work
> > process_one_work
> > worker_thread
> > kthread
> > kernel_thread_helper
>
> So that is inserting items into the AIL at transaction completion.
> That can consume lots of CPU time if the CIL checkpoints are being
> flushed quickly enough. Given you are doing a rm -rf at this point
> in time, I'd expect to see this trace present in the profile, though
> maybe not at that extent.
>
> I have seen this before but have never been able to it reproduce
> reliably. If checkpoints are completed "out of order" due to the
> way the commit records are written into the iclogs. This can cause
> extra CPU because the AIL insertion then has to skip over all the
> items that were inserted out of order before splicing each group of
> items into the AIL. I only rarely see this (maybe once every couple
> of weeks of performance testing), so I'm not sure it's the problem
> you are seeing.
>
> Adding this debug to xfs_ail_splice() list walk will tell us if this is
> happening and how many items it had to walk when you see a hang:
>
> xfs_lsn_t lsn)
> {
> xfs_log_item_t *next_lip;
> + int walked = 0;
>
> /* If the list is empty, just insert the item. */
> if (list_empty(&ailp->xa_ail)) {
> list_splice(list, &ailp->xa_ail);
> return;
> }
>
> list_for_each_entry_reverse(next_lip, &ailp->xa_ail, li_ail) {
> if (XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0)
> break;
> + if (!walked++) {
> + xfs_warn(ailp->xa_mount,
> + "ail: ooo splice, tail 0x%llx, item 0x%llx\n",
> + next_lip->li_lsn, lsn);
> + }
> }
> + if (walked > 10) {
> + xfs_warn(ailp->xa_mount,
> + "ail: ooo splice, walked %d items\n", walked);
> + }
>
> ASSERT(&next_lip->li_ail == &ailp->xa_ail ||
> XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0);
>
> That will at least tell us if this is the cause of your problem. If
> it is, I think I know how to avoid most of the list walk overhead
> fairly easily and that should avoid the need to change workqueue
> configurations at all.
The kernel log is attached.
> > The following patch fixes the problem for me.
> >
> > diff --git a/fs/xfs/linux-2.6/xfs_buf.c b/fs/xfs/linux-2.6/xfs_buf.c
> > index 5e68099..2f34efd 100644
> > --- a/fs/xfs/linux-2.6/xfs_buf.c
> > +++ b/fs/xfs/linux-2.6/xfs_buf.c
> > @@ -1856,7 +1856,7 @@ xfs_buf_init(void)
> > goto out;
> >
> > xfslogd_workqueue = alloc_workqueue("xfslogd",
> > - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> > + WQ_MEM_RECLAIM | WQ_UNBOUND, 1);
> > if (!xfslogd_workqueue)
> > goto out_free_buf_zone;
>
> That implies the source of your X/ssh hangs is a workqueue priority
> inversion.
>
> We need the xfslogd to run with WQ_HIGHPRI because we can deadlock
> in IO completion if it is blocked by xfsdatad/xfsconvertd wq
> operations. Hence we need to process the logd workqueue ahead of
> data IO completions and we do that by making the log IO completion
> high priority.
>
> This thing is, the CMWQ infrastructure only has one level of "high
> priority" - we can't define any other sort of dependency between
> various queues - and that queue is simply a FIFO. IOWs, CMWQ will
> process anything that is high priority ahead of -all- other normal
> priority work items. That means that if you get a high priority
> workqueue with lots to do and consuming CPU (as we are seeing here)
> it will starve all other work items that are queued on that CPU.
>
> So what you end up seeing is interactive process related work being
> queued on CPU 3 which is then being starved of processing by XFS IO
> completion work on the same CPU. By setting the xfslogd workqueue to
> unbound, it no longer holds precedence over any of the other work
> and hence does not starve the queued interactive process related work.
> That's why the patch "fixes" your 1-2s hangs while log IO is in
> progress, but it will eventually deadlock and hang the filesystem
> permanently.
>
> There are two different things you can try with the wq
> initialisation that might help prevent the problem. Firstly, try
> this:
>
> - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> + WQ_MEM_RECLAIM | WQ_HIGHPRI, 8);
>
> To change the number of concurrent per-cpu work items that can be
> processed on the work queue. I don't think that will fix the
> inversion, but it may allow more concurrency which hides the
> inversion.
>
> The other thing you can try is:
>
> - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> + WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_CPU_INTENSIVE, 1);
>
> Which marks the workqueue as one that will consume a lot of CPU and
> hence it is scheduled differently and hence should avoid other
> pending work from being starved. We use this WQ_CPU_INTENSIVE flag
> on other XFS workqueues that are known to consume lots of CPU, so I
> suspect this is the right thing to do here.
Yes, that was the next combination I've tested after WQ_UNBOUND and it
solves the issue, too.
--
Markus
[-- Attachment #2: kern.log.bz2 --]
[-- Type: application/x-bzip2, Size: 4328 bytes --]
[-- Attachment #3: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-22 7:06 ` Markus Trippelsdorf
@ 2011-06-22 7:30 ` Dave Chinner
2011-06-22 7:40 ` Markus Trippelsdorf
2011-06-29 4:31 ` Dave Chinner
0 siblings, 2 replies; 33+ messages in thread
From: Dave Chinner @ 2011-06-22 7:30 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Wed, Jun 22, 2011 at 09:06:47AM +0200, Markus Trippelsdorf wrote:
> On 2011.06.22 at 10:04 +1000, Dave Chinner wrote:
> > On Tue, Jun 21, 2011 at 08:57:01PM +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.21 at 20:24 +0200, Markus Trippelsdorf wrote:
> > > > On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> > > > > On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > > > > > That is, you really need to get a profile of the rm -rf process - or
> > > > > > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > > > > > across the hang to so we can narrow down the potential cause of the
> > > > > > latency. Speaking of which, latencytop might be helpful in
> > > > > > identifying where input is getting held up....
> > > > >
> > > > > I've recorded a profile with "perf record -g /home/markus/rm_sync"
> > > > > ~ % cat rm_sync
> > > > > rm -fr /mnt/tmp/tmp/linux && sync
> > > >
> > > > FWIW here are two links to svg time-charts produced by:
> > > >
> > > > perf timechart record /home/markus/rm_sync
> > > >
> > > > http://trippelsdorf.de/timechart1.svg
> > > > http://trippelsdorf.de/timechart2.svg
> > > >
> > >
> > > And this is what the mysterious kworker is doing during the sync.
> > > It's the one consuming most of the CPU time.
> > >
> > > 39.96% kworker/3:0 [kernel.kallsyms] 0xffffffff811da9da k [k] xfs_trans_ail_update_bulk
> > > |
> > > --- xfs_trans_ail_update_bulk
> > > xfs_trans_committed_bulk
> > > xlog_cil_committed
> > > xlog_state_do_callback
> > > xlog_state_done_syncing
> > > xlog_iodone
> > > xfs_buf_iodone_work
> > > process_one_work
> > > worker_thread
> > > kthread
> > > kernel_thread_helper
> >
> > So that is inserting items into the AIL at transaction completion.
> > That can consume lots of CPU time if the CIL checkpoints are being
> > flushed quickly enough. Given you are doing a rm -rf at this point
> > in time, I'd expect to see this trace present in the profile, though
> > maybe not at that extent.
> >
> > I have seen this before but have never been able to it reproduce
> > reliably. If checkpoints are completed "out of order" due to the
> > way the commit records are written into the iclogs. This can cause
> > extra CPU because the AIL insertion then has to skip over all the
> > items that were inserted out of order before splicing each group of
> > items into the AIL. I only rarely see this (maybe once every couple
> > of weeks of performance testing), so I'm not sure it's the problem
> > you are seeing.
> >
> > Adding this debug to xfs_ail_splice() list walk will tell us if this is
> > happening and how many items it had to walk when you see a hang:
> >
> > xfs_lsn_t lsn)
> > {
> > xfs_log_item_t *next_lip;
> > + int walked = 0;
> >
> > /* If the list is empty, just insert the item. */
> > if (list_empty(&ailp->xa_ail)) {
> > list_splice(list, &ailp->xa_ail);
> > return;
> > }
> >
> > list_for_each_entry_reverse(next_lip, &ailp->xa_ail, li_ail) {
> > if (XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0)
> > break;
> > + if (!walked++) {
> > + xfs_warn(ailp->xa_mount,
> > + "ail: ooo splice, tail 0x%llx, item 0x%llx\n",
> > + next_lip->li_lsn, lsn);
> > + }
> > }
> > + if (walked > 10) {
> > + xfs_warn(ailp->xa_mount,
> > + "ail: ooo splice, walked %d items\n", walked);
> > + }
> >
> > ASSERT(&next_lip->li_ail == &ailp->xa_ail ||
> > XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0);
> >
> > That will at least tell us if this is the cause of your problem. If
> > it is, I think I know how to avoid most of the list walk overhead
> > fairly easily and that should avoid the need to change workqueue
> > configurations at all.
>
> The kernel log is attached.
Ok, so that is the cause of the problem∵ THe 3 seconds of output
where it is nothing but:
Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
.....
Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
Interesting is the LSN of the tail - it's only one sector further on
than the items being inserted. That's what I'd expect from a commit
record write race between two checkpoints. I'll have a deeper look
into whether this can be avoided later tonight and also whether I
can easily implement a "last insert cursor" easily so subsequent
inserts at the same LSN avoid the walk....
> > > The following patch fixes the problem for me.
.....
> > There are two different things you can try with the wq
> > initialisation that might help prevent the problem. Firstly, try
> > this:
> >
> > - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> > + WQ_MEM_RECLAIM | WQ_HIGHPRI, 8);
> >
> > To change the number of concurrent per-cpu work items that can be
> > processed on the work queue. I don't think that will fix the
> > inversion, but it may allow more concurrency which hides the
> > inversion.
> >
> > The other thing you can try is:
> >
> > - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> > + WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_CPU_INTENSIVE, 1);
> >
> > Which marks the workqueue as one that will consume a lot of CPU and
> > hence it is scheduled differently and hence should avoid other
> > pending work from being starved. We use this WQ_CPU_INTENSIVE flag
> > on other XFS workqueues that are known to consume lots of CPU, so I
> > suspect this is the right thing to do here.
>
> Yes, that was the next combination I've tested after WQ_UNBOUND and it
> solves the issue, too.
Just to be clear, which combination did you test? The
WQ_CPU_INTENSIVE one?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-22 7:30 ` Dave Chinner
@ 2011-06-22 7:40 ` Markus Trippelsdorf
2011-06-29 4:31 ` Dave Chinner
1 sibling, 0 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-22 7:40 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.22 at 17:30 +1000, Dave Chinner wrote:
> On Wed, Jun 22, 2011 at 09:06:47AM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.22 at 10:04 +1000, Dave Chinner wrote:
> > > On Tue, Jun 21, 2011 at 08:57:01PM +0200, Markus Trippelsdorf wrote:
> > > > On 2011.06.21 at 20:24 +0200, Markus Trippelsdorf wrote:
> > > > > On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> > > > > > On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > > > > > > That is, you really need to get a profile of the rm -rf process - or
> > > > > > > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > > > > > > across the hang to so we can narrow down the potential cause of the
> > > > > > > latency. Speaking of which, latencytop might be helpful in
> > > > > > > identifying where input is getting held up....
> > > > > >
> > > > > > I've recorded a profile with "perf record -g /home/markus/rm_sync"
> > > > > > ~ % cat rm_sync
> > > > > > rm -fr /mnt/tmp/tmp/linux && sync
> > > > >
> > > > > FWIW here are two links to svg time-charts produced by:
> > > > >
> > > > > perf timechart record /home/markus/rm_sync
> > > > >
> > > > > http://trippelsdorf.de/timechart1.svg
> > > > > http://trippelsdorf.de/timechart2.svg
> > > > >
> > > >
> > > > And this is what the mysterious kworker is doing during the sync.
> > > > It's the one consuming most of the CPU time.
> > > >
> > > > 39.96% kworker/3:0 [kernel.kallsyms] 0xffffffff811da9da k [k] xfs_trans_ail_update_bulk
> > > > |
> > > > --- xfs_trans_ail_update_bulk
> > > > xfs_trans_committed_bulk
> > > > xlog_cil_committed
> > > > xlog_state_do_callback
> > > > xlog_state_done_syncing
> > > > xlog_iodone
> > > > xfs_buf_iodone_work
> > > > process_one_work
> > > > worker_thread
> > > > kthread
> > > > kernel_thread_helper
> > >
> > > So that is inserting items into the AIL at transaction completion.
> > > That can consume lots of CPU time if the CIL checkpoints are being
> > > flushed quickly enough. Given you are doing a rm -rf at this point
> > > in time, I'd expect to see this trace present in the profile, though
> > > maybe not at that extent.
> > >
> > > I have seen this before but have never been able to it reproduce
> > > reliably. If checkpoints are completed "out of order" due to the
> > > way the commit records are written into the iclogs. This can cause
> > > extra CPU because the AIL insertion then has to skip over all the
> > > items that were inserted out of order before splicing each group of
> > > items into the AIL. I only rarely see this (maybe once every couple
> > > of weeks of performance testing), so I'm not sure it's the problem
> > > you are seeing.
> > >
> > > Adding this debug to xfs_ail_splice() list walk will tell us if this is
> > > happening and how many items it had to walk when you see a hang:
> > >
> > > xfs_lsn_t lsn)
> > > {
> > > xfs_log_item_t *next_lip;
> > > + int walked = 0;
> > >
> > > /* If the list is empty, just insert the item. */
> > > if (list_empty(&ailp->xa_ail)) {
> > > list_splice(list, &ailp->xa_ail);
> > > return;
> > > }
> > >
> > > list_for_each_entry_reverse(next_lip, &ailp->xa_ail, li_ail) {
> > > if (XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0)
> > > break;
> > > + if (!walked++) {
> > > + xfs_warn(ailp->xa_mount,
> > > + "ail: ooo splice, tail 0x%llx, item 0x%llx\n",
> > > + next_lip->li_lsn, lsn);
> > > + }
> > > }
> > > + if (walked > 10) {
> > > + xfs_warn(ailp->xa_mount,
> > > + "ail: ooo splice, walked %d items\n", walked);
> > > + }
> > >
> > > ASSERT(&next_lip->li_ail == &ailp->xa_ail ||
> > > XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0);
> > >
> > > That will at least tell us if this is the cause of your problem. If
> > > it is, I think I know how to avoid most of the list walk overhead
> > > fairly easily and that should avoid the need to change workqueue
> > > configurations at all.
> >
> > The kernel log is attached.
>
> Ok, so that is the cause of the problem∵ THe 3 seconds of output
> where it is nothing but:
>
> Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> .....
> Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
>
> Interesting is the LSN of the tail - it's only one sector further on
> than the items being inserted. That's what I'd expect from a commit
> record write race between two checkpoints. I'll have a deeper look
> into whether this can be avoided later tonight and also whether I
> can easily implement a "last insert cursor" easily so subsequent
> inserts at the same LSN avoid the walk....
>
> > > > The following patch fixes the problem for me.
> .....
> > > There are two different things you can try with the wq
> > > initialisation that might help prevent the problem. Firstly, try
> > > this:
> > >
> > > - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> > > + WQ_MEM_RECLAIM | WQ_HIGHPRI, 8);
> > >
> > > To change the number of concurrent per-cpu work items that can be
> > > processed on the work queue. I don't think that will fix the
> > > inversion, but it may allow more concurrency which hides the
> > > inversion.
> > >
> > > The other thing you can try is:
> > >
> > > - WQ_MEM_RECLAIM | WQ_HIGHPRI, 1);
> > > + WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_CPU_INTENSIVE, 1);
> > >
> > > Which marks the workqueue as one that will consume a lot of CPU and
> > > hence it is scheduled differently and hence should avoid other
> > > pending work from being starved. We use this WQ_CPU_INTENSIVE flag
> > > on other XFS workqueues that are known to consume lots of CPU, so I
> > > suspect this is the right thing to do here.
> >
> > Yes, that was the next combination I've tested after WQ_UNBOUND and it
> > solves the issue, too.
>
> Just to be clear, which combination did you test? The
> WQ_CPU_INTENSIVE one?
Sorry, the "WQ_HIGHPRI | WQ_CPU_INTENSIVE" one.
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-22 7:30 ` Dave Chinner
2011-06-22 7:40 ` Markus Trippelsdorf
@ 2011-06-29 4:31 ` Dave Chinner
2011-06-29 6:19 ` Markus Trippelsdorf
1 sibling, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2011-06-29 4:31 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Wed, Jun 22, 2011 at 05:30:47PM +1000, Dave Chinner wrote:
> On Wed, Jun 22, 2011 at 09:06:47AM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.22 at 10:04 +1000, Dave Chinner wrote:
> > > On Tue, Jun 21, 2011 at 08:57:01PM +0200, Markus Trippelsdorf wrote:
> > > > On 2011.06.21 at 20:24 +0200, Markus Trippelsdorf wrote:
> > > > > On 2011.06.21 at 10:02 +0200, Markus Trippelsdorf wrote:
> > > > > > On 2011.06.21 at 14:25 +1000, Dave Chinner wrote:
> > > > > > > That is, you really need to get a profile of the rm -rf process - or
> > > > > > > whatever is consuming the CPU time - (e.g. via perf or ftrace)
> > > > > > > across the hang to so we can narrow down the potential cause of the
> > > > > > > latency. Speaking of which, latencytop might be helpful in
> > > > > > > identifying where input is getting held up....
> > > > > >
> > > > > > I've recorded a profile with "perf record -g /home/markus/rm_sync"
> > > > > > ~ % cat rm_sync
> > > > > > rm -fr /mnt/tmp/tmp/linux && sync
> > > > >
> > > > > FWIW here are two links to svg time-charts produced by:
> > > > >
> > > > > perf timechart record /home/markus/rm_sync
> > > > >
> > > > > http://trippelsdorf.de/timechart1.svg
> > > > > http://trippelsdorf.de/timechart2.svg
> > > > >
> > > >
> > > > And this is what the mysterious kworker is doing during the sync.
> > > > It's the one consuming most of the CPU time.
> > > >
> > > > 39.96% kworker/3:0 [kernel.kallsyms] 0xffffffff811da9da k [k] xfs_trans_ail_update_bulk
> > > > |
> > > > --- xfs_trans_ail_update_bulk
> > > > xfs_trans_committed_bulk
> > > > xlog_cil_committed
> > > > xlog_state_do_callback
> > > > xlog_state_done_syncing
> > > > xlog_iodone
> > > > xfs_buf_iodone_work
> > > > process_one_work
> > > > worker_thread
> > > > kthread
> > > > kernel_thread_helper
> > >
> > > So that is inserting items into the AIL at transaction completion.
> > > That can consume lots of CPU time if the CIL checkpoints are being
> > > flushed quickly enough. Given you are doing a rm -rf at this point
> > > in time, I'd expect to see this trace present in the profile, though
> > > maybe not at that extent.
> > >
> > > I have seen this before but have never been able to it reproduce
> > > reliably. If checkpoints are completed "out of order" due to the
> > > way the commit records are written into the iclogs. This can cause
> > > extra CPU because the AIL insertion then has to skip over all the
> > > items that were inserted out of order before splicing each group of
> > > items into the AIL. I only rarely see this (maybe once every couple
> > > of weeks of performance testing), so I'm not sure it's the problem
> > > you are seeing.
> > >
> > > Adding this debug to xfs_ail_splice() list walk will tell us if this is
> > > happening and how many items it had to walk when you see a hang:
> > >
> > > xfs_lsn_t lsn)
> > > {
> > > xfs_log_item_t *next_lip;
> > > + int walked = 0;
> > >
> > > /* If the list is empty, just insert the item. */
> > > if (list_empty(&ailp->xa_ail)) {
> > > list_splice(list, &ailp->xa_ail);
> > > return;
> > > }
> > >
> > > list_for_each_entry_reverse(next_lip, &ailp->xa_ail, li_ail) {
> > > if (XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0)
> > > break;
> > > + if (!walked++) {
> > > + xfs_warn(ailp->xa_mount,
> > > + "ail: ooo splice, tail 0x%llx, item 0x%llx\n",
> > > + next_lip->li_lsn, lsn);
> > > + }
> > > }
> > > + if (walked > 10) {
> > > + xfs_warn(ailp->xa_mount,
> > > + "ail: ooo splice, walked %d items\n", walked);
> > > + }
> > >
> > > ASSERT(&next_lip->li_ail == &ailp->xa_ail ||
> > > XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0);
> > >
> > > That will at least tell us if this is the cause of your problem. If
> > > it is, I think I know how to avoid most of the list walk overhead
> > > fairly easily and that should avoid the need to change workqueue
> > > configurations at all.
> >
> > The kernel log is attached.
>
> Ok, so that is the cause of the problem∵ THe 3 seconds of output
> where it is nothing but:
>
> Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> .....
> Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
>
> Interesting is the LSN of the tail - it's only one sector further on
> than the items being inserted. That's what I'd expect from a commit
> record write race between two checkpoints. I'll have a deeper look
> into whether this can be avoided later tonight and also whether I
> can easily implement a "last insert cursor" easily so subsequent
> inserts at the same LSN avoid the walk....
Ok, so here's a patch that does just this. I should probably also do
a little bit of cleanup on the cursor code as well, but this avoids
the repeated walks of the AIL to find the insert position.
Can you try it without the WQ changes you made, Marcus, and see if
the interactivity problems go away?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
xfs: use a cursor for bulk AIL insertion
From: Dave Chinner <dchinner@redhat.com>
Delayed logging can insert tens of thousands of log items into the
AIL at the same LSN. When the committing of log commit records
occur, we can get insertions occurring at an LSN that is not at the
end of the AIL. If there are thousands of items in the AIL on the
tail LSN, each insertion has to walk the AIL to find the correct
place to insert the new item into the AIL. This can consume large
amounts of CPU time and block other operations from occurring while
the traversals are in progress.
To avoid this repeated walk, use a AIL cursor to record
where we should be inserting the new items into the AIL without
having to repeat the walk. The cursor infrastructure already
provides this functionality for push walks, so is a simple extension
of existing code. While this will not avoid the initial walk, it
will avoid repeating it tens of thousands of times during a single
checkpoint commit.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
fs/xfs/xfs_trans.c | 27 +++++++++--
fs/xfs/xfs_trans_ail.c | 122 +++++++++++++++++++++++++++++++++++++++--------
fs/xfs/xfs_trans_priv.h | 10 +++-
3 files changed, 131 insertions(+), 28 deletions(-)
diff --git a/fs/xfs/xfs_trans.c b/fs/xfs/xfs_trans.c
index 7c7bc2b..7d60d7e 100644
--- a/fs/xfs/xfs_trans.c
+++ b/fs/xfs/xfs_trans.c
@@ -1426,6 +1426,7 @@ xfs_trans_committed(
static inline void
xfs_log_item_batch_insert(
struct xfs_ail *ailp,
+ struct xfs_ail_cursor *cur,
struct xfs_log_item **log_items,
int nr_items,
xfs_lsn_t commit_lsn)
@@ -1434,7 +1435,7 @@ xfs_log_item_batch_insert(
spin_lock(&ailp->xa_lock);
/* xfs_trans_ail_update_bulk drops ailp->xa_lock */
- xfs_trans_ail_update_bulk(ailp, log_items, nr_items, commit_lsn);
+ xfs_trans_ail_update_bulk(ailp, cur, log_items, nr_items, commit_lsn);
for (i = 0; i < nr_items; i++)
IOP_UNPIN(log_items[i], 0);
@@ -1452,6 +1453,13 @@ xfs_log_item_batch_insert(
* as an iclog write error even though we haven't started any IO yet. Hence in
* this case all we need to do is IOP_COMMITTED processing, followed by an
* IOP_UNPIN(aborted) call.
+ *
+ * The AIL cursor is used to optimise the insert process. If commit_lsn is not
+ * at the end of the AIL, the insert cursor avoids the need to walk
+ * the AIL to find the insertion point on every xfs_log_item_batch_insert()
+ * call. This saves a lot of needless list walking and is a net win, even
+ * though it slightly increases that amount of AIL lock traffic to set it up
+ * and tear it down.
*/
void
xfs_trans_committed_bulk(
@@ -1463,8 +1471,13 @@ xfs_trans_committed_bulk(
#define LOG_ITEM_BATCH_SIZE 32
struct xfs_log_item *log_items[LOG_ITEM_BATCH_SIZE];
struct xfs_log_vec *lv;
+ struct xfs_ail_cursor cur;
int i = 0;
+ spin_lock(&ailp->xa_lock);
+ xfs_trans_ail_cursor_last(ailp, &cur, commit_lsn);
+ spin_unlock(&ailp->xa_lock);
+
/* unpin all the log items */
for (lv = log_vector; lv; lv = lv->lv_next ) {
struct xfs_log_item *lip = lv->lv_item;
@@ -1493,7 +1506,9 @@ xfs_trans_committed_bulk(
/*
* Not a bulk update option due to unusual item_lsn.
* Push into AIL immediately, rechecking the lsn once
- * we have the ail lock. Then unpin the item.
+ * we have the ail lock. Then unpin the item. This does
+ * not affect the AIL cursor the bulk insert path is
+ * using.
*/
spin_lock(&ailp->xa_lock);
if (XFS_LSN_CMP(item_lsn, lip->li_lsn) > 0)
@@ -1507,7 +1522,7 @@ xfs_trans_committed_bulk(
/* Item is a candidate for bulk AIL insert. */
log_items[i++] = lv->lv_item;
if (i >= LOG_ITEM_BATCH_SIZE) {
- xfs_log_item_batch_insert(ailp, log_items,
+ xfs_log_item_batch_insert(ailp, &cur, log_items,
LOG_ITEM_BATCH_SIZE, commit_lsn);
i = 0;
}
@@ -1515,7 +1530,11 @@ xfs_trans_committed_bulk(
/* make sure we insert the remainder! */
if (i)
- xfs_log_item_batch_insert(ailp, log_items, i, commit_lsn);
+ xfs_log_item_batch_insert(ailp, &cur, log_items, i, commit_lsn);
+
+ spin_lock(&ailp->xa_lock);
+ xfs_trans_ail_cursor_done(ailp, &cur);
+ spin_unlock(&ailp->xa_lock);
}
/*
diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
index 5fc2380..272e7fa 100644
--- a/fs/xfs/xfs_trans_ail.c
+++ b/fs/xfs/xfs_trans_ail.c
@@ -272,9 +272,9 @@ xfs_trans_ail_cursor_clear(
}
/*
- * Return the item in the AIL with the current lsn.
- * Return the current tree generation number for use
- * in calls to xfs_trans_next_ail().
+ * Initialise the cursor to the first item in the AIL with the given @lsn.
+ * This searches the list from lowest LSN to highest. Pass a @lsn of zero
+ * to initialise the cursor to the first item in the AIL.
*/
xfs_log_item_t *
xfs_trans_ail_cursor_first(
@@ -300,31 +300,110 @@ out:
}
/*
- * splice the log item list into the AIL at the given LSN.
+ * Initialise the cursor to the last item in the AIL with the given @lsn.
+ * This searches the list from highest LSN to lowest.
*/
-static void
-xfs_ail_splice(
- struct xfs_ail *ailp,
- struct list_head *list,
- xfs_lsn_t lsn)
+static struct xfs_log_item *
+__xfs_trans_ail_cursor_last(
+ struct xfs_ail *ailp,
+ struct xfs_ail_cursor *cur,
+ xfs_lsn_t lsn,
+ bool do_init)
{
- xfs_log_item_t *next_lip;
+ xfs_log_item_t *lip = NULL;
- /* If the list is empty, just insert the item. */
- if (list_empty(&ailp->xa_ail)) {
- list_splice(list, &ailp->xa_ail);
- return;
- }
+ if (do_init)
+ xfs_trans_ail_cursor_init(ailp, cur);
+
+ if (list_empty(&ailp->xa_ail))
+ goto out;
- list_for_each_entry_reverse(next_lip, &ailp->xa_ail, li_ail) {
- if (XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0)
+ list_for_each_entry_reverse(lip, &ailp->xa_ail, li_ail) {
+ if (XFS_LSN_CMP(lip->li_lsn, lsn) <= 0)
break;
}
+out:
+ if (cur)
+ cur->item = lip;
+ return lip;
+}
- ASSERT(&next_lip->li_ail == &ailp->xa_ail ||
- XFS_LSN_CMP(next_lip->li_lsn, lsn) <= 0);
+/*
+ * Initialise the cursor to the last item in the AIL with the given @lsn.
+ * This searches the list from highest LSN to lowest.
+ */
+struct xfs_log_item *
+xfs_trans_ail_cursor_last(
+ struct xfs_ail *ailp,
+ struct xfs_ail_cursor *cur,
+ xfs_lsn_t lsn)
+{
+ return __xfs_trans_ail_cursor_last(ailp, cur, lsn, true);
+}
- list_splice_init(list, &next_lip->li_ail);
+/*
+ * splice the log item list into the AIL at the given LSN. We splice to the
+ * tail of the given LSN to maintain insert order for push traversals. The
+ * cursor is optional, allowing repeated updates to the same LSN to avoid
+ * repeated traversals.
+ */
+static void
+xfs_ail_splice(
+ struct xfs_ail *ailp,
+ struct xfs_ail_cursor *cur,
+ struct list_head *list,
+ xfs_lsn_t lsn)
+{
+ struct xfs_log_item *lip = cur ? cur->item : NULL;
+ struct xfs_log_item *next_lip;
+
+ do {
+ /* no placeholder, so get our insert location */
+ if (!lip)
+ lip = __xfs_trans_ail_cursor_last(ailp, cur,
+ lsn, false);
+
+ if (!lip) {
+ /*
+ * The list is empty, so just splice and return. Our
+ * cursor is already guaranteed to be up to date, so we
+ * don't need to touch it here.
+ */
+ list_splice(list, &ailp->xa_ail);
+ return;
+ }
+
+ /* The placeholder was invalidated, need to get a new cursor */
+ if ((__psint_t)lip & 1)
+ lip = NULL;
+
+ } while (lip == NULL);
+
+ /*
+ * Our cursor points to the item we want to insert _after_, so we have
+ * to update the cursor to point to the end of the list we are splicing
+ * in so that it points to the correct location for the next splice.
+ * i.e. before the splice
+ *
+ * lsn -> lsn -> lsn + x -> lsn + x ...
+ * ^
+ * | cursor points here
+ *
+ * After the splice we have:
+ *
+ * lsn -> lsn -> lsn -> lsn -> .... -> lsn -> lsn + x -> lsn + x ...
+ * ^ ^
+ * | cursor points here | needs to move here
+ *
+ * So we set the cursor to the last item in the list to be spliced
+ * before we execute the splice, resulting in the cursor pointing to
+ * the correct item after the splice occurs.
+ */
+ if (cur) {
+ next_lip = list_entry(list->prev, struct xfs_log_item, li_ail);
+ cur->item = next_lip;
+ }
+ list_splice_init(list, &lip->li_ail);
}
/*
@@ -645,6 +724,7 @@ xfs_trans_unlocked_item(
void
xfs_trans_ail_update_bulk(
struct xfs_ail *ailp,
+ struct xfs_ail_cursor *cur,
struct xfs_log_item **log_items,
int nr_items,
xfs_lsn_t lsn) __releases(ailp->xa_lock)
@@ -674,7 +754,7 @@ xfs_trans_ail_update_bulk(
list_add(&lip->li_ail, &tmp);
}
- xfs_ail_splice(ailp, &tmp, lsn);
+ xfs_ail_splice(ailp, cur, &tmp, lsn);
if (!mlip_changed) {
spin_unlock(&ailp->xa_lock);
diff --git a/fs/xfs/xfs_trans_priv.h b/fs/xfs/xfs_trans_priv.h
index 6b164e9..c0cb408 100644
--- a/fs/xfs/xfs_trans_priv.h
+++ b/fs/xfs/xfs_trans_priv.h
@@ -82,6 +82,7 @@ struct xfs_ail {
extern struct workqueue_struct *xfs_ail_wq; /* AIL workqueue */
void xfs_trans_ail_update_bulk(struct xfs_ail *ailp,
+ struct xfs_ail_cursor *cur,
struct xfs_log_item **log_items, int nr_items,
xfs_lsn_t lsn) __releases(ailp->xa_lock);
static inline void
@@ -90,7 +91,7 @@ xfs_trans_ail_update(
struct xfs_log_item *lip,
xfs_lsn_t lsn) __releases(ailp->xa_lock)
{
- xfs_trans_ail_update_bulk(ailp, &lip, 1, lsn);
+ xfs_trans_ail_update_bulk(ailp, NULL, &lip, 1, lsn);
}
void xfs_trans_ail_delete_bulk(struct xfs_ail *ailp,
@@ -111,10 +112,13 @@ xfs_lsn_t xfs_ail_min_lsn(struct xfs_ail *ailp);
void xfs_trans_unlocked_item(struct xfs_ail *,
xfs_log_item_t *);
-struct xfs_log_item *xfs_trans_ail_cursor_first(struct xfs_ail *ailp,
+struct xfs_log_item * xfs_trans_ail_cursor_first(struct xfs_ail *ailp,
struct xfs_ail_cursor *cur,
xfs_lsn_t lsn);
-struct xfs_log_item *xfs_trans_ail_cursor_next(struct xfs_ail *ailp,
+struct xfs_log_item * xfs_trans_ail_cursor_last(struct xfs_ail *ailp,
+ struct xfs_ail_cursor *cur,
+ xfs_lsn_t lsn);
+struct xfs_log_item * xfs_trans_ail_cursor_next(struct xfs_ail *ailp,
struct xfs_ail_cursor *cur);
void xfs_trans_ail_cursor_done(struct xfs_ail *ailp,
struct xfs_ail_cursor *cur);
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-29 4:31 ` Dave Chinner
@ 2011-06-29 6:19 ` Markus Trippelsdorf
2011-06-29 7:24 ` Dave Chinner
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-29 6:19 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.29 at 14:31 +1000, Dave Chinner wrote:
> On Wed, Jun 22, 2011 at 05:30:47PM +1000, Dave Chinner wrote:
> > On Wed, Jun 22, 2011 at 09:06:47AM +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.22 at 10:04 +1000, Dave Chinner wrote:
> > > > On Tue, Jun 21, 2011 at 08:57:01PM +0200, Markus Trippelsdorf wrote:
> > > >
> > > > That will at least tell us if this is the cause of your problem. If
> > > > it is, I think I know how to avoid most of the list walk overhead
> > > > fairly easily and that should avoid the need to change workqueue
> > > > configurations at all.
> > >
> > > The kernel log is attached.
> >
> > Ok, so that is the cause of the problem∵ THe 3 seconds of output
> > where it is nothing but:
> >
> > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> > .....
> > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
> >
> > Interesting is the LSN of the tail - it's only one sector further on
> > than the items being inserted. That's what I'd expect from a commit
> > record write race between two checkpoints. I'll have a deeper look
> > into whether this can be avoided later tonight and also whether I
> > can easily implement a "last insert cursor" easily so subsequent
> > inserts at the same LSN avoid the walk....
>
> Ok, so here's a patch that does just this. I should probably also do
> a little bit of cleanup on the cursor code as well, but this avoids
> the repeated walks of the AIL to find the insert position.
>
> Can you try it without the WQ changes you made, Marcus, and see if
> the interactivity problems go away?
Sorry to be the bringer of bad news, but this made things much worse:
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 603 380 |0.66 0.55 0.28| 0 0
1 0 99 0 0 0: 1 0 99 0 0 0: 1 19 80 0 0 0: 0 0 100 0 0 0| 0 0 | 719 383 |0.66 0.55 0.28| 0 0
3 1 96 0 0 0: 3 1 96 0 0 0: 1 52 47 0 0 0: 0 0 100 0 0 0| 0 6464k|1847 919 |0.66 0.55 0.28| 0 202
2 13 85 0 0 0: 2 2 96 0 0 0: 1 56 43 0 0 0: 1 31 69 0 0 0|4096B 256k|1910 1280 |0.68 0.56 0.28| 1 8
> 0 1 99 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1256 170 |0.68 0.56 0.28| 0 0
> 0 1 99 0 0 0: 1 1 98 0 0 0: 1 0 99 0 0 0: 0 99 0 0 0 1| 0 0 |1395 229 |0.68 0.56 0.28| 0 0
> 0 0 100 0 0 0: 0 0 100 0 0 0: 0 3 97 0 0 0: 0 100 0 0 0 0| 0 512B|1304 167 |0.68 0.56 0.28| 0 1
> 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 99 0 0 0 1| 0 0 |1211 146 |0.68 0.56 0.28| 0 0
> 0 0 100 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 97 0 0 0 3| 0 0 |1270 149 |0.87 0.60 0.30| 0 0
5 2 65 29 0 0: 2 3 95 0 0 0: 1 0 99 0 0 0: 2 24 72 0 0 1| 0 8866k|2654 2398 |0.87 0.60 0.30| 0 496
6 2 25 67 0 0: 3 1 59 37 0 0: 0 0 100 0 0 0: 4 4 92 0 0 0| 0 4554k|2224 2494 |0.87 0.60 0.30| 0 399
1 1 98 0 0 0: 0 0 83 17 0 0: 1 3 96 0 0 0: 0 1 99 0 0 0| 0 2270k|1079 1030 |0.87 0.60 0.30| 0 200
1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 9216B| 713 567 |0.87 0.60 0.30| 0 2
0 0 100 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0| 0 0 | 492 386 |0.80 0.59 0.30| 0 0
As you can see in the table above (resolution 1sec) the hang is now
5-6 seconds long, instead of the 1-3 seconds seen before.
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-29 6:19 ` Markus Trippelsdorf
@ 2011-06-29 7:24 ` Dave Chinner
2011-06-29 7:41 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2011-06-29 7:24 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Wed, Jun 29, 2011 at 08:19:54AM +0200, Markus Trippelsdorf wrote:
> On 2011.06.29 at 14:31 +1000, Dave Chinner wrote:
> > On Wed, Jun 22, 2011 at 05:30:47PM +1000, Dave Chinner wrote:
> > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> > > .....
> > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
> > >
> > > Interesting is the LSN of the tail - it's only one sector further on
> > > than the items being inserted. That's what I'd expect from a commit
> > > record write race between two checkpoints. I'll have a deeper look
> > > into whether this can be avoided later tonight and also whether I
> > > can easily implement a "last insert cursor" easily so subsequent
> > > inserts at the same LSN avoid the walk....
> >
> > Ok, so here's a patch that does just this. I should probably also do
> > a little bit of cleanup on the cursor code as well, but this avoids
> > the repeated walks of the AIL to find the insert position.
> >
> > Can you try it without the WQ changes you made, Marcus, and see if
> > the interactivity problems go away?
>
> Sorry to be the bringer of bad news, but this made things much worse:
>
> -------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
> usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
> 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 603 380 |0.66 0.55 0.28| 0 0
> 1 0 99 0 0 0: 1 0 99 0 0 0: 1 19 80 0 0 0: 0 0 100 0 0 0| 0 0 | 719 383 |0.66 0.55 0.28| 0 0
> 3 1 96 0 0 0: 3 1 96 0 0 0: 1 52 47 0 0 0: 0 0 100 0 0 0| 0 6464k|1847 919 |0.66 0.55 0.28| 0 202
> 2 13 85 0 0 0: 2 2 96 0 0 0: 1 56 43 0 0 0: 1 31 69 0 0 0|4096B 256k|1910 1280 |0.68 0.56 0.28| 1 8
> > 0 1 99 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1256 170 |0.68 0.56 0.28| 0 0
> > 0 1 99 0 0 0: 1 1 98 0 0 0: 1 0 99 0 0 0: 0 99 0 0 0 1| 0 0 |1395 229 |0.68 0.56 0.28| 0 0
> > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 3 97 0 0 0: 0 100 0 0 0 0| 0 512B|1304 167 |0.68 0.56 0.28| 0 1
> > 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 99 0 0 0 1| 0 0 |1211 146 |0.68 0.56 0.28| 0 0
> > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 97 0 0 0 3| 0 0 |1270 149 |0.87 0.60 0.30| 0 0
> 5 2 65 29 0 0: 2 3 95 0 0 0: 1 0 99 0 0 0: 2 24 72 0 0 1| 0 8866k|2654 2398 |0.87 0.60 0.30| 0 496
> 6 2 25 67 0 0: 3 1 59 37 0 0: 0 0 100 0 0 0: 4 4 92 0 0 0| 0 4554k|2224 2494 |0.87 0.60 0.30| 0 399
> 1 1 98 0 0 0: 0 0 83 17 0 0: 1 3 96 0 0 0: 0 1 99 0 0 0| 0 2270k|1079 1030 |0.87 0.60 0.30| 0 200
> 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 9216B| 713 567 |0.87 0.60 0.30| 0 2
> 0 0 100 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0| 0 0 | 492 386 |0.80 0.59 0.30| 0 0
>
> As you can see in the table above (resolution 1sec) the hang is now
> 5-6 seconds long, instead of the 1-3 seconds seen before.
Interesting. I checked that the ordering was correct in each case
adn that it was behaving correctly here.
Can you add the following patch and send me the dmesg output over a
hang? It will tell me where the cursor is being initialised and when
it is being dropped, so should indicate if a specific insert chain
is getting stuck or doing something stoopid.
Cheers,
Dave
--
Dave Chinner
david@fromorbit.com
---
fs/xfs/xfs_trans_ail.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
index 272e7fa..a087cbb 100644
--- a/fs/xfs/xfs_trans_ail.c
+++ b/fs/xfs/xfs_trans_ail.c
@@ -234,6 +234,8 @@ xfs_trans_ail_cursor_done(
struct xfs_ail_cursor *prev = NULL;
struct xfs_ail_cursor *cur;
+ printk(KERN_WARNING "done cur %p, lip %p/0x%llx\n",
+ done, done->item, done->item ? done->item->li_lsn : 0);
done->item = NULL;
if (done == &ailp->xa_cursors)
return;
@@ -323,6 +325,8 @@ __xfs_trans_ail_cursor_last(
break;
}
out:
+ printk(KERN_WARNING "last cur %p, init %d lsn 0x%llx, lip %p/0x%llx\n",
+ cur, do_init, lsn, lip, lip ? lip->li_lsn : 0);
if (cur)
cur->item = lip;
return lip;
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-29 7:24 ` Dave Chinner
@ 2011-06-29 7:41 ` Markus Trippelsdorf
2011-06-29 12:10 ` Dave Chinner
0 siblings, 1 reply; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-29 7:41 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
[-- Attachment #1: Type: text/plain, Size: 4505 bytes --]
On 2011.06.29 at 17:24 +1000, Dave Chinner wrote:
> On Wed, Jun 29, 2011 at 08:19:54AM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.29 at 14:31 +1000, Dave Chinner wrote:
> > > On Wed, Jun 22, 2011 at 05:30:47PM +1000, Dave Chinner wrote:
> > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> > > > .....
> > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
> > > >
> > > > Interesting is the LSN of the tail - it's only one sector further on
> > > > than the items being inserted. That's what I'd expect from a commit
> > > > record write race between two checkpoints. I'll have a deeper look
> > > > into whether this can be avoided later tonight and also whether I
> > > > can easily implement a "last insert cursor" easily so subsequent
> > > > inserts at the same LSN avoid the walk....
> > >
> > > Ok, so here's a patch that does just this. I should probably also do
> > > a little bit of cleanup on the cursor code as well, but this avoids
> > > the repeated walks of the AIL to find the insert position.
> > >
> > > Can you try it without the WQ changes you made, Marcus, and see if
> > > the interactivity problems go away?
> >
> > Sorry to be the bringer of bad news, but this made things much worse:
> >
> > -------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
> > usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
> > 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 603 380 |0.66 0.55 0.28| 0 0
> > 1 0 99 0 0 0: 1 0 99 0 0 0: 1 19 80 0 0 0: 0 0 100 0 0 0| 0 0 | 719 383 |0.66 0.55 0.28| 0 0
> > 3 1 96 0 0 0: 3 1 96 0 0 0: 1 52 47 0 0 0: 0 0 100 0 0 0| 0 6464k|1847 919 |0.66 0.55 0.28| 0 202
> > 2 13 85 0 0 0: 2 2 96 0 0 0: 1 56 43 0 0 0: 1 31 69 0 0 0|4096B 256k|1910 1280 |0.68 0.56 0.28| 1 8
> > > 0 1 99 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1256 170 |0.68 0.56 0.28| 0 0
> > > 0 1 99 0 0 0: 1 1 98 0 0 0: 1 0 99 0 0 0: 0 99 0 0 0 1| 0 0 |1395 229 |0.68 0.56 0.28| 0 0
> > > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 3 97 0 0 0: 0 100 0 0 0 0| 0 512B|1304 167 |0.68 0.56 0.28| 0 1
> > > 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 99 0 0 0 1| 0 0 |1211 146 |0.68 0.56 0.28| 0 0
> > > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 97 0 0 0 3| 0 0 |1270 149 |0.87 0.60 0.30| 0 0
> > 5 2 65 29 0 0: 2 3 95 0 0 0: 1 0 99 0 0 0: 2 24 72 0 0 1| 0 8866k|2654 2398 |0.87 0.60 0.30| 0 496
> > 6 2 25 67 0 0: 3 1 59 37 0 0: 0 0 100 0 0 0: 4 4 92 0 0 0| 0 4554k|2224 2494 |0.87 0.60 0.30| 0 399
> > 1 1 98 0 0 0: 0 0 83 17 0 0: 1 3 96 0 0 0: 0 1 99 0 0 0| 0 2270k|1079 1030 |0.87 0.60 0.30| 0 200
> > 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 9216B| 713 567 |0.87 0.60 0.30| 0 2
> > 0 0 100 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0| 0 0 | 492 386 |0.80 0.59 0.30| 0 0
> >
> > As you can see in the table above (resolution 1sec) the hang is now
> > 5-6 seconds long, instead of the 1-3 seconds seen before.
>
> Interesting. I checked that the ordering was correct in each case
> adn that it was behaving correctly here.
>
> Can you add the following patch and send me the dmesg output over a
> hang? It will tell me where the cursor is being initialised and when
> it is being dropped, so should indicate if a specific insert chain
> is getting stuck or doing something stoopid.
The kernel log is attached.
rm -fr && sync starts at Jun 29 09:32:24.
--
Markus
[-- Attachment #2: kern_.log.bz2 --]
[-- Type: application/x-bzip2, Size: 45937 bytes --]
[-- Attachment #3: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-29 7:41 ` Markus Trippelsdorf
@ 2011-06-29 12:10 ` Dave Chinner
2011-06-29 12:48 ` Markus Trippelsdorf
0 siblings, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2011-06-29 12:10 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Wed, Jun 29, 2011 at 09:41:27AM +0200, Markus Trippelsdorf wrote:
> On 2011.06.29 at 17:24 +1000, Dave Chinner wrote:
> > On Wed, Jun 29, 2011 at 08:19:54AM +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.29 at 14:31 +1000, Dave Chinner wrote:
> > > > On Wed, Jun 22, 2011 at 05:30:47PM +1000, Dave Chinner wrote:
> > > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> > > > > .....
> > > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
> > > > >
> > > > > Interesting is the LSN of the tail - it's only one sector further on
> > > > > than the items being inserted. That's what I'd expect from a commit
> > > > > record write race between two checkpoints. I'll have a deeper look
> > > > > into whether this can be avoided later tonight and also whether I
> > > > > can easily implement a "last insert cursor" easily so subsequent
> > > > > inserts at the same LSN avoid the walk....
> > > >
> > > > Ok, so here's a patch that does just this. I should probably also do
> > > > a little bit of cleanup on the cursor code as well, but this avoids
> > > > the repeated walks of the AIL to find the insert position.
> > > >
> > > > Can you try it without the WQ changes you made, Marcus, and see if
> > > > the interactivity problems go away?
> > >
> > > Sorry to be the bringer of bad news, but this made things much worse:
> > >
> > > -------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
> > > usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
> > > 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 603 380 |0.66 0.55 0.28| 0 0
> > > 1 0 99 0 0 0: 1 0 99 0 0 0: 1 19 80 0 0 0: 0 0 100 0 0 0| 0 0 | 719 383 |0.66 0.55 0.28| 0 0
> > > 3 1 96 0 0 0: 3 1 96 0 0 0: 1 52 47 0 0 0: 0 0 100 0 0 0| 0 6464k|1847 919 |0.66 0.55 0.28| 0 202
> > > 2 13 85 0 0 0: 2 2 96 0 0 0: 1 56 43 0 0 0: 1 31 69 0 0 0|4096B 256k|1910 1280 |0.68 0.56 0.28| 1 8
> > > > 0 1 99 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1256 170 |0.68 0.56 0.28| 0 0
> > > > 0 1 99 0 0 0: 1 1 98 0 0 0: 1 0 99 0 0 0: 0 99 0 0 0 1| 0 0 |1395 229 |0.68 0.56 0.28| 0 0
> > > > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 3 97 0 0 0: 0 100 0 0 0 0| 0 512B|1304 167 |0.68 0.56 0.28| 0 1
> > > > 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 99 0 0 0 1| 0 0 |1211 146 |0.68 0.56 0.28| 0 0
> > > > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 97 0 0 0 3| 0 0 |1270 149 |0.87 0.60 0.30| 0 0
> > > 5 2 65 29 0 0: 2 3 95 0 0 0: 1 0 99 0 0 0: 2 24 72 0 0 1| 0 8866k|2654 2398 |0.87 0.60 0.30| 0 496
> > > 6 2 25 67 0 0: 3 1 59 37 0 0: 0 0 100 0 0 0: 4 4 92 0 0 0| 0 4554k|2224 2494 |0.87 0.60 0.30| 0 399
> > > 1 1 98 0 0 0: 0 0 83 17 0 0: 1 3 96 0 0 0: 0 1 99 0 0 0| 0 2270k|1079 1030 |0.87 0.60 0.30| 0 200
> > > 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 9216B| 713 567 |0.87 0.60 0.30| 0 2
> > > 0 0 100 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0| 0 0 | 492 386 |0.80 0.59 0.30| 0 0
> > >
> > > As you can see in the table above (resolution 1sec) the hang is now
> > > 5-6 seconds long, instead of the 1-3 seconds seen before.
> >
> > Interesting. I checked that the ordering was correct in each case
> > adn that it was behaving correctly here.
> >
> > Can you add the following patch and send me the dmesg output over a
> > hang? It will tell me where the cursor is being initialised and when
> > it is being dropped, so should indicate if a specific insert chain
> > is getting stuck or doing something stoopid.
>
> The kernel log is attached.
> rm -fr && sync starts at Jun 29 09:32:24.
Add this patch on top of the first one I sent. If it doesn't fix the
problem, can you readd the debug patch and send the log again?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
xfs: unpin stale inodes directly in IOP_COMMITTED
From: Dave Chinner <dchinner@redhat.com>
When inodes are marked stale in a transaction, they are treated
specially when the iinode log item is being inserted into the AIL.
It trieѕ to avoid moving the log item forward in the AIL due to a
race condition with the writing the underlying buffer back to disk.
The was "fixed" in commit de25c18 ("xfs: avoid moving stale inodes in
the AIL").
To avoid moving the item forward, we return a LSN smaller than the
commit_lsn of the completing transaction, thereby trying to trick
the commit code into not moving the inode forward at all. I'm not
sure this ever worked as intended - it assumes the inode is already
in the AIL, but I don't think the returned LSN would have been small
enough to prevent moving the inode. It appears that the reason it
worked is that the lower LSN of the inodes meant they were inserted
into the AIL and flushed before the inode buffer (which was moved to
the commit_lsn of the transaction).
The big problem is that with delayed logging, the returning of the
different LSN means insertion takes the slow, non-bulk path. Worse
yet is that insertion is to a position -before- the commit_lsn so it
is doing a AIL traversal on every insertion, and has to walk over
all the items that have already been inserted into the AIL. It's
expensive.
To compound the matter further, with delayed logging inodes are
likely to go from clean to stale in a single checkpoint, which means
they aren't even in the AIL at all when we come across them at AIL
insertion time. Hence these were all getting inserted into the AIL
when they simply do not need to be as inodes marked XFS_ISTALE are
never written back.
Transactional/recovery integrity is maintained in this case by the
other items in the unlink transaction that were modified (e.g. the
AGI btree blocks) and committed in the same checkpoint.
So to fix this, simply unpin the stale inodes directly in
xfs_inode_item_committed() and return -1 to indicate that the AIL
insertion code does not need to do any further processing of these
inodes.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
fs/xfs/xfs_inode_item.c | 14 ++++++++------
fs/xfs/xfs_trans.c | 2 +-
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/fs/xfs/xfs_inode_item.c b/fs/xfs/xfs_inode_item.c
index 09983a3..b1e88d5 100644
--- a/fs/xfs/xfs_inode_item.c
+++ b/fs/xfs/xfs_inode_item.c
@@ -681,15 +681,15 @@ xfs_inode_item_unlock(
* where the cluster buffer may be unpinned before the inode is inserted into
* the AIL during transaction committed processing. If the buffer is unpinned
* before the inode item has been committed and inserted, then it is possible
- * for the buffer to be written and IO completions before the inode is inserted
+ * for the buffer to be written and IO completes before the inode is inserted
* into the AIL. In that case, we'd be inserting a clean, stale inode into the
* AIL which will never get removed. It will, however, get reclaimed which
* triggers an assert in xfs_inode_free() complaining about freein an inode
* still in the AIL.
*
- * To avoid this, return a lower LSN than the one passed in so that the
- * transaction committed code will not move the inode forward in the AIL but
- * will still unpin it properly.
+ * To avoid this, just unpin the inode directly and return a LSN of -1 so the
+ * transaction committed code knows that it does not need to do any further
+ * processing on the item.
*/
STATIC xfs_lsn_t
xfs_inode_item_committed(
@@ -699,8 +699,10 @@ xfs_inode_item_committed(
struct xfs_inode_log_item *iip = INODE_ITEM(lip);
struct xfs_inode *ip = iip->ili_inode;
- if (xfs_iflags_test(ip, XFS_ISTALE))
- return lsn - 1;
+ if (xfs_iflags_test(ip, XFS_ISTALE)) {
+ xfs_inode_item_unpin(lip, 0);
+ return -1;
+ }
return lsn;
}
diff --git a/fs/xfs/xfs_trans.c b/fs/xfs/xfs_trans.c
index 7d60d7e..d5d5708 100644
--- a/fs/xfs/xfs_trans.c
+++ b/fs/xfs/xfs_trans.c
@@ -1487,7 +1487,7 @@ xfs_trans_committed_bulk(
lip->li_flags |= XFS_LI_ABORTED;
item_lsn = IOP_COMMITTED(lip, commit_lsn);
- /* item_lsn of -1 means the item was freed */
+ /* item_lsn of -1 means the item needs no further processing */
if (XFS_LSN_CMP(item_lsn, (xfs_lsn_t)-1) == 0)
continue;
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-29 12:10 ` Dave Chinner
@ 2011-06-29 12:48 ` Markus Trippelsdorf
2011-06-29 15:08 ` Michael Weissenbacher
2011-06-29 23:53 ` Dave Chinner
0 siblings, 2 replies; 33+ messages in thread
From: Markus Trippelsdorf @ 2011-06-29 12:48 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 2011.06.29 at 22:10 +1000, Dave Chinner wrote:
> On Wed, Jun 29, 2011 at 09:41:27AM +0200, Markus Trippelsdorf wrote:
> > On 2011.06.29 at 17:24 +1000, Dave Chinner wrote:
> > > On Wed, Jun 29, 2011 at 08:19:54AM +0200, Markus Trippelsdorf wrote:
> > > > On 2011.06.29 at 14:31 +1000, Dave Chinner wrote:
> > > > > On Wed, Jun 22, 2011 at 05:30:47PM +1000, Dave Chinner wrote:
> > > > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> > > > > > .....
> > > > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
> > > > > >
> > > > > > Interesting is the LSN of the tail - it's only one sector further on
> > > > > > than the items being inserted. That's what I'd expect from a commit
> > > > > > record write race between two checkpoints. I'll have a deeper look
> > > > > > into whether this can be avoided later tonight and also whether I
> > > > > > can easily implement a "last insert cursor" easily so subsequent
> > > > > > inserts at the same LSN avoid the walk....
> > > > >
> > > > > Ok, so here's a patch that does just this. I should probably also do
> > > > > a little bit of cleanup on the cursor code as well, but this avoids
> > > > > the repeated walks of the AIL to find the insert position.
> > > > >
> > > > > Can you try it without the WQ changes you made, Marcus, and see if
> > > > > the interactivity problems go away?
> > > >
> > > > Sorry to be the bringer of bad news, but this made things much worse:
> > > >
> > > > -------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdc-- ---system-- ---load-avg--- --dsk/sdc--
> > > > usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
> > > > 1 1 98 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0: 0 1 99 0 0 0| 0 0 | 603 380 |0.66 0.55 0.28| 0 0
> > > > 1 0 99 0 0 0: 1 0 99 0 0 0: 1 19 80 0 0 0: 0 0 100 0 0 0| 0 0 | 719 383 |0.66 0.55 0.28| 0 0
> > > > 3 1 96 0 0 0: 3 1 96 0 0 0: 1 52 47 0 0 0: 0 0 100 0 0 0| 0 6464k|1847 919 |0.66 0.55 0.28| 0 202
> > > > 2 13 85 0 0 0: 2 2 96 0 0 0: 1 56 43 0 0 0: 1 31 69 0 0 0|4096B 256k|1910 1280 |0.68 0.56 0.28| 1 8
> > > > > 0 1 99 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 100 0 0 0 0| 0 0 |1256 170 |0.68 0.56 0.28| 0 0
> > > > > 0 1 99 0 0 0: 1 1 98 0 0 0: 1 0 99 0 0 0: 0 99 0 0 0 1| 0 0 |1395 229 |0.68 0.56 0.28| 0 0
> > > > > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 3 97 0 0 0: 0 100 0 0 0 0| 0 512B|1304 167 |0.68 0.56 0.28| 0 1
> > > > > 1 1 98 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 99 0 0 0 1| 0 0 |1211 146 |0.68 0.56 0.28| 0 0
> > > > > 0 0 100 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0: 0 97 0 0 0 3| 0 0 |1270 149 |0.87 0.60 0.30| 0 0
> > > > 5 2 65 29 0 0: 2 3 95 0 0 0: 1 0 99 0 0 0: 2 24 72 0 0 1| 0 8866k|2654 2398 |0.87 0.60 0.30| 0 496
> > > > 6 2 25 67 0 0: 3 1 59 37 0 0: 0 0 100 0 0 0: 4 4 92 0 0 0| 0 4554k|2224 2494 |0.87 0.60 0.30| 0 399
> > > > 1 1 98 0 0 0: 0 0 83 17 0 0: 1 3 96 0 0 0: 0 1 99 0 0 0| 0 2270k|1079 1030 |0.87 0.60 0.30| 0 200
> > > > 1 1 98 0 0 0: 1 1 98 0 0 0: 0 1 99 0 0 0: 1 0 99 0 0 0| 0 9216B| 713 567 |0.87 0.60 0.30| 0 2
> > > > 0 0 100 0 0 0: 1 1 98 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0| 0 0 | 492 386 |0.80 0.59 0.30| 0 0
> > > >
> > > > As you can see in the table above (resolution 1sec) the hang is now
> > > > 5-6 seconds long, instead of the 1-3 seconds seen before.
> > >
> > > Interesting. I checked that the ordering was correct in each case
> > > adn that it was behaving correctly here.
> > >
> > > Can you add the following patch and send me the dmesg output over a
> > > hang? It will tell me where the cursor is being initialised and when
> > > it is being dropped, so should indicate if a specific insert chain
> > > is getting stuck or doing something stoopid.
> >
> > The kernel log is attached.
> > rm -fr && sync starts at Jun 29 09:32:24.
>
> Add this patch on top of the first one I sent. If it doesn't fix the
> problem, can you readd the debug patch and send the log again?
This completely fixes the issue. As a bonus "rm -fr && sync" completes
much quicker now.
Here are three different examples of the "remove kernel tree" workload:
-------cpu0-usage--------------cpu1-usage--------------cpu2-usage--------------cpu3-usage------ --dsk/sdb-- ---system-- ---load-avg--- --dsk/sdb--
usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| int csw | 1m 5m 15m |reads writs
0 0 100 0 0 0: 1 0 99 0 0 0: 0 0 100 0 0 0: 1 20 79 0 0 0| 0 0 | 761 254 |0.86 0.29 0.10| 0 0
9 1 90 0 0 0: 3 8 89 0 0 0: 7 5 88 0 0 0: 7 51 41 0 0 0| 0 6464k|3250 2742 |0.86 0.29 0.10| 0 202
1 0 99 0 0 0: 2 51 47 0 0 0: 1 1 98 0 0 0: 0 3 97 0 0 0|4096B 5856k|1895 905 |0.86 0.29 0.10| 1 177
5 1 94 0 0 0: 1 0 99 0 0 0: 4 0 96 0 0 0: 8 9 83 0 0 0| 0 559k|1662 1470 |0.86 0.29 0.10| 0 148
5 1 94 0 0 0: 1 1 98 0 0 0: 1 1 98 0 0 0: 6 5 89 0 0 0| 0 0 |1697 1568 |0.86 0.29 0.10| 0 0
1 1 98 0 0 0: 0 0 100 0 0 0: 0 0 100 0 0 0: 0 1 99 0 0 0| 0 0 | 741 392 |1.04 0.43 0.16| 0 0
9 6 85 0 0 0: 6 1 93 0 0 0: 1 1 98 0 0 0: 2 61 37 0 0 0| 0 32k|2070 1383 |0.96 0.42 0.16| 0 0
6 3 91 0 0 0: 3 1 95 0 0 0: 2 44 54 0 0 0: 3 4 92 0 0 0| 0 6432k|2495 1663 |0.96 0.42 0.16| 0 202
2 6 92 0 0 0: 2 1 97 0 0 0: 2 7 91 0 0 0: 3 8 90 0 0 0|4096B 6140k|2394 1805 |0.96 0.42 0.16| 1 258
4 1 95 0 0 0: 3 3 94 0 0 0: 0 0 100 0 0 0: 3 2 95 0 0 0| 0 0 |1473 730 |0.96 0.42 0.16| 0 0
1 1 98 0 0 0: 0 1 99 0 0 0: 1 3 96 0 0 0: 1 1 98 0 0 0| 0 0 | 926 433 |0.60 0.42 0.19| 0 0
5 2 93 0 0 0: 3 0 97 0 0 0: 2 60 38 0 0 0: 7 3 90 0 0 0| 0 5312k|2328 1897 |0.60 0.42 0.19| 0 161
2 62 36 0 0 0: 6 2 92 0 0 0: 2 0 98 0 0 0: 8 7 85 0 0 0|4096B 4000k|2439 1827 |0.60 0.42 0.19| 1 129
8 1 91 0 0 0: 3 1 96 0 0 0: 0 0 100 0 0 0: 12 18 69 0 0 1| 0 3293k|2568 2223 |0.63 0.43 0.20| 0 170
4 1 95 0 0 0: 1 2 97 0 0 0: 2 1 97 0 0 0: 6 7 87 0 0 0| 0 0 |1606 1627 |0.63 0.43 0.20| 0 0
So many thanks Dave. Your prompt help is very much appreciated.
--
Markus
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-29 12:48 ` Markus Trippelsdorf
@ 2011-06-29 15:08 ` Michael Weissenbacher
2011-06-29 23:53 ` Dave Chinner
1 sibling, 0 replies; 33+ messages in thread
From: Michael Weissenbacher @ 2011-06-29 15:08 UTC (permalink / raw)
To: xfs
Hi Marcus / Dave!
>
> This completely fixes the issue. As a bonus "rm -fr && sync" completes
> much quicker now.
> ...
> So many thanks Dave. Your prompt help is very much appreciated.
>
I just wanna add that i've seen these kinds of hangs too when removing
directories with millions of files. My "solution" was to recreate the
filesystem - which was several years old at that time. Looking forward
to the fix. Thanks for the great work going on here - both for the
in-depth investigation and the fix.
cheers,
Michael
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: long hangs when deleting large directories (3.0-rc3)
2011-06-29 12:48 ` Markus Trippelsdorf
2011-06-29 15:08 ` Michael Weissenbacher
@ 2011-06-29 23:53 ` Dave Chinner
1 sibling, 0 replies; 33+ messages in thread
From: Dave Chinner @ 2011-06-29 23:53 UTC (permalink / raw)
To: Markus Trippelsdorf; +Cc: xfs
On Wed, Jun 29, 2011 at 02:48:14PM +0200, Markus Trippelsdorf wrote:
> On 2011.06.29 at 22:10 +1000, Dave Chinner wrote:
> > On Wed, Jun 29, 2011 at 09:41:27AM +0200, Markus Trippelsdorf wrote:
> > > On 2011.06.29 at 17:24 +1000, Dave Chinner wrote:
> > > > On Wed, Jun 29, 2011 at 08:19:54AM +0200, Markus Trippelsdorf wrote:
> > > > > On 2011.06.29 at 14:31 +1000, Dave Chinner wrote:
> > > > > > On Wed, Jun 22, 2011 at 05:30:47PM +1000, Dave Chinner wrote:
> > > > > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > > > > Jun 22 08:53:09 x4 kernel: XFS (sdb1): ail: ooo splice, walked 15503 items
> > > > > > > .....
> > > > > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, tail 0x12000156e7, item 0x12000156e6
> > > > > > > Jun 22 08:53:12 x4 kernel: XFS (sdb1): ail: ooo splice, walked 16945 items
> > > > > > >
> > > > > > > Interesting is the LSN of the tail - it's only one sector further on
> > > > > > > than the items being inserted. That's what I'd expect from a commit
> > > > > > > record write race between two checkpoints. I'll have a deeper look
> > > > > > > into whether this can be avoided later tonight and also whether I
> > > > > > > can easily implement a "last insert cursor" easily so subsequent
> > > > > > > inserts at the same LSN avoid the walk....
> > > > > >
> > > > > > Ok, so here's a patch that does just this. I should probably also do
> > > > > > a little bit of cleanup on the cursor code as well, but this avoids
> > > > > > the repeated walks of the AIL to find the insert position.
> > > > > >
> > > > > > Can you try it without the WQ changes you made, Marcus, and see if
> > > > > > the interactivity problems go away?
> > > > >
> > > > > Sorry to be the bringer of bad news, but this made things much worse:
....
> > > > > As you can see in the table above (resolution 1sec) the hang is now
> > > > > 5-6 seconds long, instead of the 1-3 seconds seen before.
> > > >
> > > > Interesting. I checked that the ordering was correct in each case
> > > > adn that it was behaving correctly here.
> > > >
> > > > Can you add the following patch and send me the dmesg output over a
> > > > hang? It will tell me where the cursor is being initialised and when
> > > > it is being dropped, so should indicate if a specific insert chain
> > > > is getting stuck or doing something stoopid.
> > >
> > > The kernel log is attached.
> > > rm -fr && sync starts at Jun 29 09:32:24.
> >
> > Add this patch on top of the first one I sent. If it doesn't fix the
> > problem, can you readd the debug patch and send the log again?
>
> This completely fixes the issue. As a bonus "rm -fr && sync" completes
> much quicker now.
Great to hear the hang has gone away.
I'm also seeing performance improvements on unlink workloads with
these two patches - quite significant, too. Cold cache parallel rm
-rf tests over tens of millions of inodes are finishing 15-20%
faster. Hot cache parallel rm -rf now go to being CPU bound on a
8p system with the unlink rate improving by about 50%....
As I always say, the hardest part of fixing a bug is getting a
reproducable test case to analyse and test. Thank's for providing
the test case and the testing, Markus!
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2011-06-29 23:53 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-06-18 14:19 long hangs when deleting large directories (3.0-rc3) Markus Trippelsdorf
2011-06-18 14:24 ` Christoph Hellwig
2011-06-18 14:37 ` Markus Trippelsdorf
2011-06-19 8:16 ` Markus Trippelsdorf
2011-06-19 22:24 ` Dave Chinner
2011-06-20 0:54 ` Markus Trippelsdorf
2011-06-20 1:34 ` Dave Chinner
2011-06-20 2:02 ` Markus Trippelsdorf
2011-06-20 2:36 ` Dave Chinner
2011-06-20 6:03 ` Markus Trippelsdorf
2011-06-20 11:13 ` Markus Trippelsdorf
2011-06-20 11:45 ` Michael Monnerie
2011-06-20 12:31 ` Markus Trippelsdorf
2011-06-20 21:16 ` Markus Trippelsdorf
2011-06-21 4:25 ` Dave Chinner
2011-06-21 8:02 ` Markus Trippelsdorf
2011-06-21 18:24 ` Markus Trippelsdorf
2011-06-21 18:57 ` Markus Trippelsdorf
2011-06-21 21:22 ` Markus Trippelsdorf
2011-06-22 0:04 ` Dave Chinner
2011-06-22 7:06 ` Markus Trippelsdorf
2011-06-22 7:30 ` Dave Chinner
2011-06-22 7:40 ` Markus Trippelsdorf
2011-06-29 4:31 ` Dave Chinner
2011-06-29 6:19 ` Markus Trippelsdorf
2011-06-29 7:24 ` Dave Chinner
2011-06-29 7:41 ` Markus Trippelsdorf
2011-06-29 12:10 ` Dave Chinner
2011-06-29 12:48 ` Markus Trippelsdorf
2011-06-29 15:08 ` Michael Weissenbacher
2011-06-29 23:53 ` Dave Chinner
2011-06-21 10:18 ` Markus Trippelsdorf
2011-06-21 10:40 ` Markus Trippelsdorf
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.