Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Mark_H_Johnson]

>i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
>downloaded from the usual place:
>
>            http://redhat.com/~mingo/realtime-preempt/

Just did a build with -V0.7.30-2 and was about to start testing when
the system locked up (no keyboard response, display frozen, etc.). No
response to Alt-SysRq keys. No messages on the serial console (other than
those showing a normal boot / telinit 5). Kernel was PREEMPT_RT plus a
patch to profile on NMI, not timer (been using this latter one for some
time). Basically same .config as previously provided but can send if
needed. Boot parameters included serial console, profile=2, nmi_watchdog.

Will retry shortly, but the steps leading to the failure were:
 - boot single user
 - telinit 5
 - su'd 3 times
 - created directories to log data / moved some files around
 - set IRQ threads, ksoftirqd/[01], events/[01] to RT fifo 99 priority
 - started two monitoring scripts (looking at latency & profile data)
 - cat /proc/sys/kernel/preempt_wakeup_timing (was 1)
 - echo 0 > /proc/sys/kernel/preempt_wakeup_timing [entered, but display
was frozen at this point and did not see newline nor any further output]

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

>  - echo 0 > /proc/sys/kernel/preempt_wakeup_timing [entered, but
> display was frozen at this point and did not see newline nor any
> further output]

managed to reproduce this on an SMP box but not on an UP box, so i think
this is SMP related. It definitely happens almost immediately after
preempt_wakeup_timing is reset - or after preempt_max_timing is reset. 
(Perhaps a dump_stack() from the wrong place, or something like that.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[K.R. Foley]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

Ingo Molnar wrote:
> * Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:
> 
> 
>> - echo 0 > /proc/sys/kernel/preempt_wakeup_timing [entered, but
>>display was frozen at this point and did not see newline nor any
>>further output]
> 
> 
> managed to reproduce this on an SMP box but not on an UP box, so i think
> this is SMP related. It definitely happens almost immediately after
> preempt_wakeup_timing is reset - or after preempt_max_timing is reset. 
> (Perhaps a dump_stack() from the wrong place, or something like that.)
> 
> 	Ingo
> 

I can't reproduce this on my SMP workstation here at the office now. As 
you may recall I did have something similar going on in the recent past 
when booting this system. I am attaching my config in hopes that maybe 
that will help.

kr


[-- Attachment #2: .config --]
[-- Type: text/plain, Size: 35658 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.10-rc2-mm2-V0.7.30-2
# Mon Nov 22 09:24:34 2004
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCK_KERNEL=y

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_LOG_BUF_SHIFT=14
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
# CONFIG_CPUSETS is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set

#
# Loadable module support
#
CONFIG_MODULES=y
# CONFIG_MODULE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
# CONFIG_HPET_TIMER is not set
CONFIG_SMP=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_NONFATAL is not set
# CONFIG_X86_MCE_P4THERMAL is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m

#
# Firmware Drivers
#
CONFIG_EDD=m
CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_IRQBALANCE=y
CONFIG_HAVE_DEC_LOCK=y

#
# Performance-monitoring counters support
#
# CONFIG_PERFCTR is not set
CONFIG_KERN_PHYS_OFFSET=1
# CONFIG_KEXEC is not set

#
# Power management options (ACPI, APM)
#
# CONFIG_PM is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
# CONFIG_ACPI is not set
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BLACKLIST_YEAR=0

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_LEGACY_PROC is not set
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_HOTPLUG_CPU is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PC-card bridges
#
CONFIG_PCMCIA_PROBE=y

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
CONFIG_ISAPNP=y
# CONFIG_PNPBIOS is not set

#
# Block devices
#
CONFIG_BLK_DEV_FD=m
# CONFIG_BLK_DEV_XD is not set
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_LBD is not set
# CONFIG_CDROM_PKTCDVD is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=m
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_CMD640=y
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
CONFIG_BLK_DEV_RZ1000=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
CONFIG_BLK_DEV_AEC62XX=y
CONFIG_BLK_DEV_ALI15X3=y
# CONFIG_WDC_ALI15X3 is not set
CONFIG_BLK_DEV_AMD74XX=y
# CONFIG_BLK_DEV_ATIIXP is not set
CONFIG_BLK_DEV_CMD64X=y
CONFIG_BLK_DEV_TRIFLEX=y
CONFIG_BLK_DEV_CY82C693=y
# CONFIG_BLK_DEV_CS5520 is not set
CONFIG_BLK_DEV_CS5530=y
CONFIG_BLK_DEV_HPT34X=y
# CONFIG_HPT34X_AUTODMA is not set
CONFIG_BLK_DEV_HPT366=y
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_NS87415 is not set
CONFIG_BLK_DEV_PDC202XX_OLD=y
# CONFIG_PDC202XX_BURST is not set
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_BLK_DEV_SVWKS=y
CONFIG_BLK_DEV_SIIMAGE=y
CONFIG_BLK_DEV_SIS5513=y
CONFIG_BLK_DEV_SLC90E66=y
# CONFIG_BLK_DEV_TRM290 is not set
CONFIG_BLK_DEV_VIA82CXXX=y
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y

#
# SCSI Transport Attributes
#
CONFIG_SCSI_SPI_ATTRS=m
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set

#
# SCSI low-level drivers
#
CONFIG_BLK_DEV_3W_XXXX_RAID=m
# CONFIG_SCSI_3W_9XXX is not set
CONFIG_SCSI_7000FASST=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AHA152X=m
CONFIG_SCSI_AHA1542=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_ENABLE_RD_STRM is not set
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_DPT_I2O is not set
CONFIG_SCSI_IN2000=m
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_SCSI_SATA=y
# CONFIG_SCSI_SATA_AHCI is not set
CONFIG_SCSI_SATA_SVW=m
CONFIG_SCSI_ATA_PIIX=m
# CONFIG_SCSI_SATA_NV is not set
CONFIG_SCSI_SATA_PROMISE=m
# CONFIG_SCSI_SATA_SX4 is not set
CONFIG_SCSI_SATA_SIL=m
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_ULI is not set
CONFIG_SCSI_SATA_VIA=m
# CONFIG_SCSI_SATA_VITESSE is not set
CONFIG_SCSI_BUSLOGIC=m
# CONFIG_SCSI_OMIT_FLASHPOINT is not set
CONFIG_SCSI_DMX3191D=m
CONFIG_SCSI_DTC3280=m
CONFIG_SCSI_EATA=m
CONFIG_SCSI_EATA_TAGGED_QUEUE=y
# CONFIG_SCSI_EATA_LINKED_COMMANDS is not set
CONFIG_SCSI_EATA_MAX_TAGS=16
CONFIG_SCSI_EATA_PIO=m
CONFIG_SCSI_FUTURE_DOMAIN=m
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_GENERIC_NCR5380=m
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_GENERIC_NCR53C400 is not set
CONFIG_SCSI_IPS=m
# CONFIG_SCSI_INITIO is not set
CONFIG_SCSI_INIA100=m
CONFIG_SCSI_NCR53C406A=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
# CONFIG_SCSI_SYM53C8XX_IOMAPPED is not set
# CONFIG_SCSI_IPR is not set
CONFIG_SCSI_PAS16=m
CONFIG_SCSI_PSI240I=m
CONFIG_SCSI_QLOGIC_FAS=m
CONFIG_SCSI_QLOGIC_ISP=m
CONFIG_SCSI_QLOGIC_FC=m
# CONFIG_SCSI_QLOGIC_FC_FIRMWARE is not set
CONFIG_SCSI_QLOGIC_1280=m
# CONFIG_SCSI_QLOGIC_1280_1040 is not set
CONFIG_SCSI_QLA2XXX=m
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA6322 is not set
CONFIG_SCSI_SYM53C416=m
# CONFIG_SCSI_DC395x is not set
CONFIG_SCSI_DC390T=m
CONFIG_SCSI_T128=m
CONFIG_SCSI_U14_34F=m
# CONFIG_SCSI_U14_34F_TAGGED_QUEUE is not set
# CONFIG_SCSI_U14_34F_LINKED_COMMANDS is not set
CONFIG_SCSI_U14_34F_MAX_TAGS=8
CONFIG_SCSI_ULTRASTOR=m
CONFIG_SCSI_NSP32=m
CONFIG_SCSI_DEBUG=m

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
# CONFIG_MD_RAID10 is not set
CONFIG_MD_RAID5=m
# CONFIG_MD_RAID6 is not set
CONFIG_MD_MULTIPATH=m
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_CRYPT is not set
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_MIRROR is not set
# CONFIG_DM_ZERO is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=y
CONFIG_UNIX=y
# CONFIG_IPMI_SOCKET is not set
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_IP_TCPDIAG=y
# CONFIG_IP_TCPDIAG_IPV6 is not set

#
# IP: Virtual Server Configuration
#
CONFIG_IP_VS=m
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=16

#
# IPVS transport protocol load balancing support
#
# CONFIG_IP_VS_PROTO_TCP is not set
# CONFIG_IP_VS_PROTO_UDP is not set
# CONFIG_IP_VS_PROTO_ESP is not set
# CONFIG_IP_VS_PROTO_AH is not set

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
# CONFIG_IP_VS_SED is not set
# CONFIG_IP_VS_NQ is not set

#
# IPVS application helper
#
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_BRIDGE_NETFILTER=y

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
# CONFIG_IP_NF_CT_ACCT is not set
# CONFIG_IP_NF_CONNTRACK_MARK is not set
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
# CONFIG_IP_NF_MATCH_IPRANGE is not set
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
# CONFIG_IP_NF_MATCH_PHYSDEV is not set
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_REALM is not set
# CONFIG_IP_NF_MATCH_SCTP is not set
# CONFIG_IP_NF_MATCH_COMMENT is not set
# CONFIG_IP_NF_MATCH_HASHLIMIT is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
# CONFIG_IP_NF_TARGET_NETMAP is not set
# CONFIG_IP_NF_TARGET_SAME is not set
# CONFIG_IP_NF_NAT_LOCAL is not set
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
# CONFIG_IP_NF_TARGET_CLASSIFY is not set
# CONFIG_IP_NF_RAW is not set
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IP_NF_COMPAT_IPCHAINS=m
CONFIG_IP_NF_COMPAT_IPFWADM=m

#
# Bridge: Netfilter Configuration
#
# CONFIG_BRIDGE_NF_EBTABLES is not set
CONFIG_XFRM=y
CONFIG_XFRM_USER=m

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
CONFIG_ATM=y
CONFIG_ATM_CLIP=y
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
CONFIG_ATM_BR2684_IPFILTER=y
CONFIG_BRIDGE=m
CONFIG_VLAN_8021Q=m
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
CONFIG_IPX=m
# CONFIG_IPX_INTERN is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=y
CONFIG_LTPC=m
CONFIG_COPS=m
CONFIG_COPS_DAYNA=y
CONFIG_COPS_TANGENT=y
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
CONFIG_NET_DIVERT=y
# CONFIG_ECONET is not set
CONFIG_WAN_ROUTER=m

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
# CONFIG_NET_SCH_CLK_CPU is not set
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
# CONFIG_NET_SCH_HFSC is not set
# CONFIG_NET_SCH_ATM is not set
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
# CONFIG_NET_SCH_NETEM is not set
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
# CONFIG_CLS_U32_PERF is not set
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_CLS_ACT is not set
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_KGDBOE is not set
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_RX is not set
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m
CONFIG_ETHERTAP=m
# CONFIG_NET_SB1000 is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_EL1=m
CONFIG_EL2=m
CONFIG_ELPLUS=m
CONFIG_EL16=m
CONFIG_EL3=m
CONFIG_3C515=m
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_LANCE=m
CONFIG_NET_VENDOR_SMC=y
CONFIG_WD80x3=m
CONFIG_ULTRA=m
CONFIG_SMC9194=m
CONFIG_NET_VENDOR_RACAL=y
CONFIG_NI52=m
CONFIG_NI65=m

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
CONFIG_AT1700=m
CONFIG_DEPCA=m
CONFIG_HP100=m
CONFIG_NET_ISA=y
CONFIG_E2100=m
CONFIG_EWRK3=m
CONFIG_EEXPRESS=m
CONFIG_EEXPRESS_PRO=m
CONFIG_HPLAN_PLUS=m
CONFIG_HPLAN=m
CONFIG_LP486E=m
CONFIG_ETH16I=m
CONFIG_NE2000=m
# CONFIG_ZNET is not set
# CONFIG_SEEQ8005 is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
# CONFIG_AMD8111E_NAPI is not set
CONFIG_ADAPTEC_STARFIRE=m
# CONFIG_ADAPTEC_STARFIRE_NAPI is not set
CONFIG_AC3200=m
CONFIG_APRICOT=m
CONFIG_B44=m
# CONFIG_FORCEDETH is not set
CONFIG_CS89x0=m
CONFIG_DGRS=m
CONFIG_EEPRO100=m
CONFIG_E100=m
# CONFIG_E100_NAPI is not set
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
CONFIG_TLAN=m
CONFIG_VIA_RHINE=m
# CONFIG_VIA_RHINE_MMIO is not set
CONFIG_NET_POCKET=y
CONFIG_ATP=m
CONFIG_DE600=m
CONFIG_DE620=m

#
# Ethernet (1000 Mbit)
#
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=m
CONFIG_E1000_NAPI=y
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
# CONFIG_R8169_NAPI is not set
# CONFIG_R8169_VLAN is not set
CONFIG_SK98LIN=m
# CONFIG_VIA_VELOCITY is not set
CONFIG_TIGON3=m

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
CONFIG_TR=y
CONFIG_IBMTR=m
CONFIG_IBMOL=m
CONFIG_IBMLS=m
CONFIG_3C359=m
CONFIG_TMS380TR=m
CONFIG_TMSPCI=m
# CONFIG_SKISA is not set
# CONFIG_PROTEON is not set
CONFIG_ABYSS=m
CONFIG_SMCTR=m

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set

#
# ATM drivers
#
# CONFIG_ATM_TCP is not set
# CONFIG_ATM_LANAI is not set
# CONFIG_ATM_ENI is not set
# CONFIG_ATM_FIRESTREAM is not set
# CONFIG_ATM_ZATM is not set
# CONFIG_ATM_NICSTAR is not set
# CONFIG_ATM_IDT77252 is not set
# CONFIG_ATM_AMBASSADOR is not set
# CONFIG_ATM_HORIZON is not set
# CONFIG_ATM_IA is not set
# CONFIG_ATM_FORE200E_MAYBE is not set
# CONFIG_ATM_HE is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
# CONFIG_PPP_BSDCOMP is not set
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y
CONFIG_NET_FC=y
CONFIG_SHAPER=m
CONFIG_NETCONSOLE=m

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input I/O drivers
#
# CONFIG_GAMEPORT is not set
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_INPORT is not set
# CONFIG_MOUSE_LOGIBM is not set
# CONFIG_MOUSE_PC110PAD is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_PCSPKR is not set
# CONFIG_INPUT_UINPUT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_ROCKETPORT=m
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
CONFIG_SYNCLINK=m
# CONFIG_SYNCLINKMP is not set
CONFIG_N_HDLC=m
CONFIG_STALDRV=y

#
# Serial drivers
#
CONFIG_SERIAL_8250=m
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=m
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256

#
# IPMI
#
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
# CONFIG_IPMI_SI is not set
CONFIG_IPMI_WATCHDOG=m
# CONFIG_IPMI_POWEROFF is not set

#
# Watchdog Cards
#
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_SC520_WDT=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
CONFIG_WAFER_WDT=m
# CONFIG_I8XX_TCO is not set
CONFIG_SC1200_WDT=m
# CONFIG_SCx200_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_W83627HF_WDT is not set
CONFIG_W83877F_WDT=m
CONFIG_MACHZ_WDT=m

#
# ISA-based Watchdog Cards
#
CONFIG_PCWATCHDOG=m
# CONFIG_MIXCOMWD is not set
CONFIG_WDT=m
# CONFIG_WDT_501 is not set

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
CONFIG_WDTPCI=m
# CONFIG_WDT_501_PCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
# CONFIG_HW_RANDOM is not set
CONFIG_NVRAM=m
CONFIG_RTC=y
CONFIG_RTC_HISTOGRAM=y
CONFIG_DTLK=m
CONFIG_R3964=m
# CONFIG_APPLICOM is not set
CONFIG_SONYPI=m

#
# Ftape, the floppy tape device driver
#
CONFIG_AGP=m
CONFIG_AGP_ALI=m
CONFIG_AGP_ATI=m
CONFIG_AGP_AMD=m
# CONFIG_AGP_AMD64 is not set
CONFIG_AGP_INTEL=m
# CONFIG_AGP_INTEL_MCH is not set
CONFIG_AGP_NVIDIA=m
CONFIG_AGP_SIS=m
CONFIG_AGP_SWORKS=m
CONFIG_AGP_VIA=m
# CONFIG_AGP_EFFICEON is not set
# CONFIG_DRM is not set
CONFIG_MWAVE=m
# CONFIG_RAW_DRIVER is not set
# CONFIG_HANGCHECK_TIMER is not set

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
# CONFIG_I2C_ALI1563 is not set
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
# CONFIG_I2C_AMD756_S4882 is not set
# CONFIG_I2C_AMD8111 is not set
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_ISA=m
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
CONFIG_I2C_PIIX4=m
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_SCx200_ACB is not set
CONFIG_I2C_SIS5595=m
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_STUB is not set
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
# CONFIG_I2C_PCA_ISA is not set

#
# Hardware Sensors Chip support
#
CONFIG_I2C_SENSOR=m
CONFIG_SENSORS_ADM1021=m
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ASB100 is not set
CONFIG_SENSORS_DS1621=m
# CONFIG_SENSORS_FSCHER is not set
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_LM63 is not set
CONFIG_SENSORS_LM75=m
# CONFIG_SENSORS_LM77 is not set
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set

#
# Other I2C Chip support
#
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_SENSORS_PCF8591=m
# CONFIG_SENSORS_RTC8564 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
CONFIG_FB=y
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set
# CONFIG_FB_CIRRUS is not set
CONFIG_FB_PM2=m
# CONFIG_FB_PM2_FIFO_DISCONNECT is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y
CONFIG_FB_HGA=m
# CONFIG_FB_HGA_ACCEL is not set
CONFIG_FB_RIVA=m
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_INTEL is not set
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G450=y
CONFIG_FB_MATROX_G100=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_MATROX_MULTIHEAD=y
# CONFIG_FB_RADEON_OLD is not set
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
# CONFIG_FB_RADEON_DEBUG is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
# CONFIG_FB_ATY_GENERIC_LCD is not set
# CONFIG_FB_ATY_XL_INIT is not set
CONFIG_FB_ATY_GX=y
# CONFIG_FB_SAVAGE is not set
CONFIG_FB_SIS=m
CONFIG_FB_SIS_300=y
CONFIG_FB_SIS_315=y
CONFIG_FB_NEOMAGIC=m
# CONFIG_FB_KYRO is not set
CONFIG_FB_3DFX=m
# CONFIG_FB_3DFX_ACCEL is not set
CONFIG_FB_VOODOO1=m
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_MDA_CONSOLE=m
CONFIG_DUMMY_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE is not set

#
# Logo configuration
#
# CONFIG_LOGO is not set

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
# CONFIG_SND is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_OTG is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y

#
# USB Host Controller Drivers
#
# CONFIG_USB_EHCI_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m

#
# USB Device Class drivers
#
CONFIG_USB_AUDIO=m
# CONFIG_USB_BLUETOOTH_TTY is not set
CONFIG_USB_MIDI=m
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_RW_DETECT is not set
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_HP8200e=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y

#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
# CONFIG_HID_FF is not set
# CONFIG_USB_HIDDEV is not set

#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
CONFIG_USB_AIPTEK=m
CONFIG_USB_WACOM=m
CONFIG_USB_KBTAB=m
CONFIG_USB_POWERMATE=m
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
CONFIG_USB_HPUSBSCSI=m

#
# USB Multimedia devices
#
CONFIG_USB_DABUSB=m

#
# Video4Linux support is needed for USB Multimedia device support
#

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m

#
# USB Host-to-Host Cables
#
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_GENESYS=y
CONFIG_USB_NET1080=y
CONFIG_USB_PL2301=y
CONFIG_USB_KC2190=y

#
# Intelligent USB Devices/Gadgets
#
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_ZAURUS=y
CONFIG_USB_CDCETHER=y

#
# USB Network Adapters
#
CONFIG_USB_AX8817X=y

#
# USB port drivers
#

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
# CONFIG_USB_SERIAL_IPW is not set
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
# CONFIG_USB_SERIAL_KEYSPAN_MPR is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28 is not set
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
# CONFIG_USB_SERIAL_KEYSPAN_USA19 is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA18X is not set
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_SAFE is not set
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_EZUSB=y

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
CONFIG_USB_TIGL=m
CONFIG_USB_AUERSWALD=m
CONFIG_USB_RIO500=m
# CONFIG_USB_LEGOTOWER is not set
CONFIG_USB_LCD=m
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETKIT is not set
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_TEST is not set

#
# USB ATM/DSL drivers
#
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
CONFIG_EXT3_FS=m
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_JBD=m
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISER4_FS is not set
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
CONFIG_REISERFS_PROC_INFO=y
# CONFIG_REISERFS_FS_XATTR is not set
CONFIG_JFS_FS=m
# CONFIG_JFS_POSIX_ACL is not set
CONFIG_JFS_DEBUG=y
# CONFIG_JFS_STATISTICS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
CONFIG_MINIX_FS=m
CONFIG_ROMFS_FS=m
CONFIG_QUOTA=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
# CONFIG_DEVPTS_FS_XATTR is not set
CONFIG_TMPFS=y
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=m
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
# CONFIG_NFS_V4 is not set
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V4 is not set
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_SUNRPC=m
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
CONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not set
# CONFIG_CIFS is not set
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_SMALLDOS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
# CONFIG_CODA_FS_OLD_API is not set
CONFIG_AFS_FS=m
CONFIG_RXRPC=m

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
# CONFIG_EFI_PARTITION is not set

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m

#
# Profiling support
#
CONFIG_PROFILING=y
CONFIG_OPROFILE=m

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_DEBUG_PREEMPT=y
CONFIG_WAKEUP_TIMING=y
CONFIG_PREEMPT_TRACE=y
CONFIG_CRITICAL_PREEMPT_TIMING=y
CONFIG_CRITICAL_IRQSOFF_TIMING=y
CONFIG_CRITICAL_TIMING=y
CONFIG_LATENCY_TIMING=y
CONFIG_LATENCY_TRACE=y
CONFIG_MCOUNT=y
CONFIG_RT_DEADLOCK_DETECT=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
# CONFIG_KGDB is not set

#
# Security options
#
# CONFIG_KEYS is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_CAPABILITIES=y
# CONFIG_SECURITY_ROOTPLUG is not set
# CONFIG_SECURITY_SECLVL is not set
# CONFIG_SECURITY_SELINUX is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
# CONFIG_CRYPTO_WP512 is not set
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_BLOWFISH=m
# CONFIG_CRYPTO_TWOFISH is not set
CONFIG_CRYPTO_SERPENT=m
# CONFIG_CRYPTO_AES_586 is not set
CONFIG_CRYPTO_CAST5=m
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_DEFLATE=m
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_TEST is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=y
# CONFIG_LIBCRC32C is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_PC=y

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> * Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:
> 
> >  - echo 0 > /proc/sys/kernel/preempt_wakeup_timing [entered, but
> > display was frozen at this point and did not see newline nor any
> > further output]
> 
> managed to reproduce this on an SMP box but not on an UP box, so i
> think this is SMP related. It definitely happens almost immediately
> after preempt_wakeup_timing is reset - or after preempt_max_timing is
> reset.  (Perhaps a dump_stack() from the wrong place, or something
> like that.)

The lockup was caused by the mutex wakeup being done under the PI lock,
and if a new critical-section latency is reported within try_to_wake_up
then the trylock done there deadlocked. The NMI watchdog triggered but
the printks done there deadlocked as well.

I fixed both the deadlock scenario, and made the NMI printout path more
robust to get the messages out to the console in even such a case. 

The fixes are in the latest (-30-4) patch which can be found at the
usual place:

  http://redhat.com/~mingo/realtime-preempt/  

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Adam Heath]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 959 bytes --]

On Mon, 22 Nov 2004,  wrote:

> >i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
> >downloaded from the usual place:
> >
> >            http://redhat.com/~mingo/realtime-preempt/

I'm seeing something very odd.  It's against 29-0.  I also seem to recall
seeing something similiar reported earlier.

I'm seeing pauses on my system.  Not certain what is causing it.  Hitting a
key on the keyboard unsticks it.

I don't know if everything stops.  I do know that the runtime displays(remote
xterm displaying continous data, wmnet) stop updating, as did a simple shell
script loop(while date; do sleep 1; done).

Last saturday night, I started a large download.  When I got back to check on
it sunday morning, the time on the machine said 1:15am, while it was 11am.

I am using default RT priorities on the irq threads.

gradall:/home.local/adam# cat /etc/sysctl.conf
kernel/trace_verbose=1
kernel/preempt_max_latency=0

config is attached.

[-- Attachment #2: Type: TEXT/PLAIN, Size: 33419 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.10-rc2-mm2-V0.7.29-0
# Thu Nov 18 12:31:06 2004
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_LOG_BUF_SHIFT=14
# CONFIG_HOTPLUG is not set
CONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not set
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
CONFIG_MK7=y
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
# CONFIG_HPET_TIMER is not set
# CONFIG_SMP is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_UP_APIC=y
# CONFIG_X86_UP_IOAPIC is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_TSC=y
# CONFIG_X86_MCE is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=m

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
# CONFIG_HIGHPTE is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y

#
# Performance-monitoring counters support
#
# CONFIG_PERFCTR is not set
CONFIG_KERN_PHYS_OFFSET=1
# CONFIG_KEXEC is not set

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
# CONFIG_ACPI is not set
CONFIG_ACPI_BLACKLIST_YEAR=0

#
# APM (Advanced Power Management) BIOS Support
#
CONFIG_APM=y
CONFIG_APM_IGNORE_USER_SUSPEND=y
CONFIG_APM_DO_ENABLE=y
CONFIG_APM_CPU_IDLE=y
# CONFIG_APM_DISPLAY_BLANK is not set
CONFIG_APM_RTC_IS_GMT=y
CONFIG_APM_ALLOW_INTS=y
CONFIG_APM_REAL_MODE_POWER_OFF=y

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_AOUT is not set
CONFIG_BINFMT_MISC=m

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play support
#
# CONFIG_PNP is not set

#
# Block devices
#
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_LBD is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=m
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
CONFIG_BLK_DEV_SIIMAGE=m
CONFIG_BLK_DEV_SIS5513=y
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
CONFIG_SCSI_SPI_ATTRS=m
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set

#
# SCSI low-level drivers
#
CONFIG_BLK_DEV_3W_XXXX_RAID=m
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_SCSI_SATA=y
# CONFIG_SCSI_SATA_AHCI is not set
# CONFIG_SCSI_SATA_SVW is not set
# CONFIG_SCSI_ATA_PIIX is not set
# CONFIG_SCSI_SATA_NV is not set
# CONFIG_SCSI_SATA_PROMISE is not set
# CONFIG_SCSI_SATA_SX4 is not set
CONFIG_SCSI_SATA_SIL=m
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_ULI is not set
# CONFIG_SCSI_SATA_VIA is not set
# CONFIG_SCSI_SATA_VITESSE is not set
CONFIG_SCSI_BUSLOGIC=m
# CONFIG_SCSI_OMIT_FLASHPOINT is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
CONFIG_SCSI_GDTH=m
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_NCR53C406A is not set
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
# CONFIG_SCSI_SYM53C8XX_IOMAPPED is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLOGIC_1280_1040 is not set
CONFIG_SCSI_QLA2XXX=m
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA6322 is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID5=m
CONFIG_MD_RAID6=m
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
CONFIG_DM_ZERO=m

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
# CONFIG_NETLINK_DEV is not set
CONFIG_UNIX=y
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
# CONFIG_NET_IPGRE is not set
CONFIG_IP_MROUTE=y
# CONFIG_IP_PIMSM_V1 is not set
# CONFIG_IP_PIMSM_V2 is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_IP_TCPDIAG=y
# CONFIG_IP_TCPDIAG_IPV6 is not set

#
# IP: Virtual Server Configuration
#
CONFIG_IP_VS=m
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_BRIDGE_NETFILTER=y

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_CT_ACCT=y
CONFIG_IP_NF_CONNTRACK_MARK=y
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_PHYSDEV=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
# CONFIG_IP_NF_MATCH_SCTP is not set
CONFIG_IP_NF_MATCH_COMMENT=m
CONFIG_IP_NF_MATCH_CONNMARK=m
CONFIG_IP_NF_MATCH_HASHLIMIT=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_LOCAL=y
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_TARGET_CONNMARK=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# CONFIG_IP_NF_COMPAT_IPCHAINS is not set
# CONFIG_IP_NF_COMPAT_IPFWADM is not set

#
# Bridge: Netfilter Configuration
#
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_XFRM=y
CONFIG_XFRM_USER=m

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
CONFIG_BRIDGE=m
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
# CONFIG_NET_SCH_CLK_CPU is not set
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
# CONFIG_CLS_U32_PERF is not set
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_CLS_ACT is not set
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_KGDBOE is not set
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_RX is not set
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
CONFIG_NET_VENDOR_3COM=y
# CONFIG_EL1 is not set
# CONFIG_EL2 is not set
# CONFIG_ELPLUS is not set
# CONFIG_EL16 is not set
# CONFIG_EL3 is not set
# CONFIG_3C515 is not set
CONFIG_VORTEX=m
# CONFIG_TYPHOON is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_CS89x0 is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
CONFIG_SIS900=m
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_NET_POCKET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPPOE=m
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
CONFIG_NETCONSOLE=y

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=y
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input I/O drivers
#
# CONFIG_GAMEPORT is not set
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
# CONFIG_SERIO_CT82C710 is not set
CONFIG_SERIO_PCIPS2=y
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_INPORT is not set
# CONFIG_MOUSE_LOGIBM is not set
# CONFIG_MOUSE_PC110PAD is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=m
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=m
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
CONFIG_HW_RANDOM=y
# CONFIG_NVRAM is not set
CONFIG_RTC=y
CONFIG_RTC_HISTOGRAM=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
CONFIG_AGP_ATI=m
CONFIG_AGP_AMD=y
CONFIG_AGP_AMD64=m
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_INTEL_MCH is not set
CONFIG_AGP_NVIDIA=m
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
CONFIG_DRM_TDFX=m
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HANGCHECK_TIMER is not set

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_ELEKTOR is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
# CONFIG_I2C_ISA is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
CONFIG_I2C_VOODOO3=m
# CONFIG_I2C_PCA_ISA is not set

#
# Hardware Sensors Chip support
#
# CONFIG_I2C_SENSOR is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set

#
# Other I2C Chip support
#
# CONFIG_SENSORS_EEPROM is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_RTC8564 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
CONFIG_VIDEO_DEV=m

#
# Video For Linux
#

#
# Video Adapters
#
CONFIG_VIDEO_BT848=m
# CONFIG_VIDEO_PMS is not set
# CONFIG_VIDEO_CPIA is not set
# CONFIG_VIDEO_SAA5246A is not set
# CONFIG_VIDEO_SAA5249 is not set
# CONFIG_TUNER_3036 is not set
# CONFIG_VIDEO_STRADIS is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DPC is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_OVCAMCHIP is not set

#
# Radio Adapters
#
# CONFIG_RADIO_CADET is not set
# CONFIG_RADIO_RTRACK is not set
# CONFIG_RADIO_RTRACK2 is not set
# CONFIG_RADIO_AZTECH is not set
# CONFIG_RADIO_GEMTEK is not set
# CONFIG_RADIO_GEMTEK_PCI is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_MAESTRO is not set
# CONFIG_RADIO_SF16FMI is not set
# CONFIG_RADIO_SF16FMR2 is not set
# CONFIG_RADIO_TERRATEC is not set
# CONFIG_RADIO_TRUST is not set
# CONFIG_RADIO_TYPHOON is not set
# CONFIG_RADIO_ZOLTRIX is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEO_BUF=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_IR=m

#
# Graphics support
#
# CONFIG_FB is not set
CONFIG_VIDEO_SELECT=y

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
# CONFIG_SND_SEQ_DUMMY is not set
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set

#
# ISA devices
#
# CONFIG_SND_AD1848 is not set
# CONFIG_SND_CS4231 is not set
# CONFIG_SND_CS4232 is not set
# CONFIG_SND_CS4236 is not set
# CONFIG_SND_ES1688 is not set
# CONFIG_SND_ES18XX is not set
# CONFIG_SND_GUSCLASSIC is not set
# CONFIG_SND_GUSEXTREME is not set
# CONFIG_SND_GUSMAX is not set
# CONFIG_SND_INTERWAVE is not set
# CONFIG_SND_INTERWAVE_STB is not set
# CONFIG_SND_OPTI92X_AD1848 is not set
# CONFIG_SND_OPTI92X_CS4231 is not set
# CONFIG_SND_OPTI93X is not set
# CONFIG_SND_SB8 is not set
# CONFIG_SND_SB16 is not set
# CONFIG_SND_SBAWE is not set
# CONFIG_SND_WAVEFRONT is not set
# CONFIG_SND_CMI8330 is not set
# CONFIG_SND_OPL3SA2 is not set
# CONFIG_SND_SGALAXY is not set
# CONFIG_SND_SSCAPE is not set

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_ALS4000 is not set
CONFIG_SND_CMIPCI=m
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VX222 is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
# CONFIG_USB is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISER4_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_QUOTA is not set
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=m

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
# CONFIG_ZISOFS is not set
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
CONFIG_DEVPTS_FS_XATTR=y
CONFIG_DEVPTS_FS_SECURITY=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
# CONFIG_RPCSEC_GSS_SPKM3 is not set
CONFIG_SMB_FS=m
CONFIG_SMB_NLS_DEFAULT=y
CONFIG_SMB_NLS_REMOTE="cp437"
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_NCP_FS is not set
CONFIG_CODA_FS=m
# CONFIG_CODA_FS_OLD_API is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m

#
# Profiling support
#
# CONFIG_PROFILING is not set

#
# Kernel hacking
#
# CONFIG_DEBUG_KERNEL is not set
CONFIG_DEBUG_PREEMPT=y
CONFIG_WAKEUP_TIMING=y
CONFIG_PREEMPT_TRACE=y
CONFIG_CRITICAL_PREEMPT_TIMING=y
CONFIG_CRITICAL_IRQSOFF_TIMING=y
CONFIG_CRITICAL_TIMING=y
CONFIG_LATENCY_TIMING=y
CONFIG_LATENCY_TRACE=y
CONFIG_MCOUNT=y
CONFIG_RT_DEADLOCK_DETECT=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_CAPABILITIES=y
# CONFIG_SECURITY_SECLVL is not set
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
# CONFIG_SECURITY_SELINUX_DISABLE is not set
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
# CONFIG_SECURITY_SELINUX_MLS is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
# CONFIG_CRYPTO_TEST is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=m
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Adam Heath <doogie@debian.org> wrote:

> > >i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
> > >downloaded from the usual place:
> > >
> > >            http://redhat.com/~mingo/realtime-preempt/
> 
> I'm seeing something very odd.  It's against 29-0.  I also seem to
> recall seeing something similiar reported earlier.
> 
> I'm seeing pauses on my system.  Not certain what is causing it. 
> Hitting a key on the keyboard unsticks it.

at first sight this looks like a scheduling/wakeup anomaly. Please
re-report this if it happens with the current (30-4) kernel too. Also,
could you test the vanilla -mm tree, it has a few scheduler updates too.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Adam Heath]

On Tue, 23 Nov 2004, Ingo Molnar wrote:

>
> * Adam Heath <doogie@debian.org> wrote:
>
> > > >i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
> > > >downloaded from the usual place:
> > > >
> > > >            http://redhat.com/~mingo/realtime-preempt/
> >
> > I'm seeing something very odd.  It's against 29-0.  I also seem to
> > recall seeing something similiar reported earlier.
> >
> > I'm seeing pauses on my system.  Not certain what is causing it.
> > Hitting a key on the keyboard unsticks it.
>
> at first sight this looks like a scheduling/wakeup anomaly. Please
> re-report this if it happens with the current (30-4) kernel too. Also,
> could you test the vanilla -mm tree, it has a few scheduler updates too.

2.6.10-rc1-mm3 doesn't have the same problem.  Didn't have a more recent mm
kernel available last night.  Will compile one, and always keep it available.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0, and 30-9

[Adam Heath]

On Tue, 23 Nov 2004, Adam Heath wrote:

> 2.6.10-rc1-mm3 doesn't have the same problem.  Didn't have a more recent mm
> kernel available last night.  Will compile one, and always keep it available.

Running 30-9.  I'll report any issues that come up.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0, and 30-9

[Steven Rostedt]

Just curious to why you scale the interrupts from 49 down to 25.  What
would be wrong with keeping all of them at 49 (or whatever). Being a
FIFO, no interrupt would preempt another. Why would you want the first
IRQs to be registered have higher priority than (and thus will preempt)
irqs registered later.

-- 
Steven Rostedt
Senior Engineer
Kihon Technologies

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0, and 30-9

[Lee Revell]

On Tue, 2004-11-23 at 16:22 -0500, Steven Rostedt wrote:
> Just curious to why you scale the interrupts from 49 down to 25.  What
> would be wrong with keeping all of them at 49 (or whatever). Being a
> FIFO, no interrupt would preempt another. Why would you want the first
> IRQs to be registered have higher priority than (and thus will preempt)
> irqs registered later.

I raised this issue before.  I agree that all interrupts should get the
same RT prio by default.  Otherwise the default behavior is arbitrary.

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0, and 30-9

[Steven Rostedt]

On Tue, 2004-11-23 at 16:47 -0500, Lee Revell wrote:
> On Tue, 2004-11-23 at 16:22 -0500, Steven Rostedt wrote:
> > Just curious to why you scale the interrupts from 49 down to 25.  What
> > would be wrong with keeping all of them at 49 (or whatever). Being a
> > FIFO, no interrupt would preempt another. Why would you want the first
> > IRQs to be registered have higher priority than (and thus will preempt)
> > irqs registered later.
> 
> I raised this issue before.  I agree that all interrupts should get the
> same RT prio by default.  Otherwise the default behavior is arbitrary.
> 
> Lee

I'll even add that the default behavior slows the system down with extra
scheduling switches.  If IRQ 10 is preempted by IRQ 2 then there's an
extra switch to get back and finish IRQ 10. For every IRQ that comes
during a "lower" priority IRQ there's an extra switch needed. If the
IRQs really don't have a order of priority, then they should be the
same. Some cases you need to set IRQs at different priorities, but that
should be done by the user and not the kernel giving the first irq
preference.

-- 
Steven Rostedt
Senior Engineer
Kihon Technologies

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0, and 30-9

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> On Tue, 2004-11-23 at 16:22 -0500, Steven Rostedt wrote:
> > Just curious to why you scale the interrupts from 49 down to 25.  What
> > would be wrong with keeping all of them at 49 (or whatever). Being a
> > FIFO, no interrupt would preempt another. Why would you want the first
> > IRQs to be registered have higher priority than (and thus will preempt)
> > irqs registered later.
> 
> I raised this issue before.  I agree that all interrupts should get
> the same RT prio by default.  Otherwise the default behavior is
> arbitrary.

i agree that it's arbitrary. There are two reasons for the ordering:

1) _usually_ the IRQs that get registered first are the 'more important'
    ones. E.g. timer and keyboard interrupts will preempt the IDE
    interrupt.  This is in no way a generic thing though.

2) testing: if all IRQs are at the same priority level then alot less 
   inter-IRQ preemption occurs, and testing coverage is lower. With all 
   irqs on different levels the bugs will trigger sooner.

To solve this cleanly some userspace policy code is needed that would
take some settings (e.g. sound_highprio) through which the priority
setup could be configured. It's not a simple task as that could would
have to discover the type of devices that are in the system and their
irqs - possibly a component of udev could do this?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Adam Heath <doogie@debian.org> wrote:

> > > I'm seeing something very odd.  It's against 29-0.  I also seem to
> > > recall seeing something similiar reported earlier.
> > >
> > > I'm seeing pauses on my system.  Not certain what is causing it.
> > > Hitting a key on the keyboard unsticks it.
> >
> > at first sight this looks like a scheduling/wakeup anomaly. Please
> > re-report this if it happens with the current (30-4) kernel too. Also,
> > could you test the vanilla -mm tree, it has a few scheduler updates too.
> 
> 2.6.10-rc1-mm3 doesn't have the same problem.  Didn't have a more
> recent mm kernel available last night.  Will compile one, and always
> keep it available.

-rc2-mm2 would be nice to test - there are a number of new interactivity
fixes from Con being test-driven in -mm right now. In particular, these
patches were added in -rc1-mm4. These are the patches in question:

 sched-adjust_timeslice_granularity.patch
 requeue_granularity.patch
 sched-remove_interactive_credit.patch

you can download them individually from:

 http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/broken-out/

so if these symptoms still occur with vanilla -rc2-mm2, could you try to
unapply them, in reverse order? (there might be rejects when you try
that, due to patch dependencies - let me know if it doesnt work out and
i'll do an undo patch.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Adam Heath]

On Wed, 24 Nov 2004, Ingo Molnar wrote:

>
> * Adam Heath <doogie@debian.org> wrote:
>
> > > > I'm seeing something very odd.  It's against 29-0.  I also seem to
> > > > recall seeing something similiar reported earlier.
> > > >
> > > > I'm seeing pauses on my system.  Not certain what is causing it.
> > > > Hitting a key on the keyboard unsticks it.
> > >
> > > at first sight this looks like a scheduling/wakeup anomaly. Please
> > > re-report this if it happens with the current (30-4) kernel too. Also,
> > > could you test the vanilla -mm tree, it has a few scheduler updates too.
> >
> > 2.6.10-rc1-mm3 doesn't have the same problem.  Didn't have a more
> > recent mm kernel available last night.  Will compile one, and always
> > keep it available.
>
> -rc2-mm2 would be nice to test - there are a number of new interactivity
> fixes from Con being test-driven in -mm right now. In particular, these
> patches were added in -rc1-mm4. These are the patches in question:
>
>  sched-adjust_timeslice_granularity.patch
>  requeue_granularity.patch
>  sched-remove_interactive_credit.patch
>
> you can download them individually from:
>
>  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/broken-out/
>
> so if these symptoms still occur with vanilla -rc2-mm2, could you try to
> unapply them, in reverse order? (there might be rejects when you try
> that, due to patch dependencies - let me know if it doesnt work out and
> i'll do an undo patch.)

The symptoms still occur with 30-9.  I'll be trying rc2-mm2 over the holiday.

Came in this morning, and after hitting a key, my machine said it was 2:38am,
when it was actually 11:10am.  All internet connections had died(obviously).
But the machine started working fine once I hit that key.  No messages in
dmesg.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Adam Heath]

On Wed, 24 Nov 2004, Adam Heath wrote:

> The symptoms still occur with 30-9.  I'll be trying rc2-mm2 over the holiday.

Been running rc2-mm3 all day.  No issues yet.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Adam Heath <doogie@debian.org> wrote:

> > The symptoms still occur with 30-9.  I'll be trying rc2-mm2 over the
> > holiday.
> 
> Been running rc2-mm3 all day.  No issues yet.

thanks, this very much looks like an -RT related scheduling bug. I've
fixed a handful of scheduling problems in recent kernels (latest is
-31-7), you might want to try it. As far as i can tell, none of the bugs
fixed should cause the symptoms you are seeing, but maybe i'm wrong.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Adam Heath]

On Wed, 24 Nov 2004, Adam Heath wrote:

> On Wed, 24 Nov 2004, Adam Heath wrote:
>
> > The symptoms still occur with 30-9.  I'll be trying rc2-mm2 over the holiday.
>
> Been running rc2-mm3 all day.  No issues yet.

Been almost a day now.  Still no odd pause issues.

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Mark_H_Johnson]

>I have a few ideas to simplify the set up to see if I can get some
>useful data out of the system.

It appears the lockup only occurs after I do
  echo 0 > /proc/sys/kernel/preempt_wakeup_latency

I did manage to get a few messages out of the system before the lockup
this time and here is what was on the serial console.

[I didn't do "dmesg -n 1" like I usually do...]

  --Mark
--------

[starts with a couple of the wakeup latency messages, then the
messages from changing the tracing type]
(X/2869/CPU#1): new 76 us maximum-latency wakeup.
(ksoftirqd/1/7/CPU#1): new 110 us maximum-latency wakeup.
(X/2869/CPU#0): new 825 us maximum-latency critical section.
 => started at timestamp 4156290879: <try_to_wake_up+0x379/0x3e0>
 =>   ended at timestamp 4156291705: <__up_mutex+0x469/0x4d0>
 [<c0104e83>] dump_stack+0x23/0x30 (20)
 [<c013d107>] check_critical_timing+0x1d7/0x390 (88)
 [<c013d58d>] trace_irqs_on+0x7d/0x90 (24)
 [<c013ad19>] __up_mutex+0x469/0x4d0 (60)
 [<c013b4c6>] up_mutex+0xb6/0x110 (40)
 [<c016e234>] fget+0x54/0x70 (28)
 [<c0182c37>] do_select+0x207/0x2b0 (120)
 [<c0182fee>] sys_select+0x2be/0x580 (92)
 [<c0103f8d>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c013e32d>] .... print_traces+0x1d/0x60
.....[<c0104e83>] ..   ( <= dump_stack+0x23/0x30)

 =>   dump-end timestamp 4156528745

(IRQ 0/2/CPU#0): new 51 us maximum-latency critical section.
 => started at timestamp 4208681904: <__up_mutex+0x9c/0x4d0>
 =>   ended at timestamp 4208681955: <schedule+0x4c/0x140>
 [<c0104e83>] dump_stack+0x23/0x30 (20)
 [<c013d107>] check_critical_timing+0x1d7/0x390 (88)
 [<c013d58d>] trace_irqs_on+0x7d/0x90 (24)
 [<c032ad7c>] schedule+0x4c/0x140 (36)
 [<c032c386>] __down_mutex+0x2a6/0x310 (84)
 [<c013bccb>] __spin_lock+0x4b/0x60 (24)
 [<c013bcfd>] _spin_lock+0x1d/0x30 (16)
 [<c0109260>] timer_interrupt+0x20/0x110 (32)
 [<c0147b63>] handle_IRQ_event+0x53/0xa0 (40)
 [<c01483d5>] do_hardirq+0xa5/0x100 (40)
 [<c0148571>] do_irqd+0x141/0x210 (48)
 [<c0138b6b>] kthread+0xbb/0xc0 (48)
 [<c0102019>] kernel_thread_helper+0x5/0xc (536952852)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c013e32d>] .... print_traces+0x1d/0x60
.....[<c0104e83>] ..   ( <= dump_stack+0x23/0x30)

 =>   dump-end timestamp 4208969387

(get_ltrace.sh/6305/CPU#1): new 64 us maximum-latency critical section.
 => started at timestamp 4209005150: <do_exit+0x32c/0x5e0>
 =>   ended at timestamp 4209005214: <try_to_wake_up+0x23d/0x3e0>
 [<c0104e83>] dump_stack+0x23/0x30 (20)
 [<c013d107>] check_critical_timing+0x1d7/0x390 (88)
 [<c013d320>] touch_critical_timing+0x60/0x90 (24)
 [<c0118c5d>] try_to_wake_up+0x23d/0x3e0 (72)
 [<c0118e2b>] wake_up_process+0x2b/0x40 (28)
 [<c01201c7>] __mmdrop_delayed+0x67/0xa0 (20)
 [<c01192f7>] finish_task_switch+0x97/0xc0 (24)
 [<c032a857>] __sched_text_start+0x457/0x930 (112)
 [<c032ad70>] schedule+0x40/0x140 (36)
 [<c01257c6>] do_wait+0x1d6/0x540 (140)
 [<c0125c01>] sys_wait4+0x41/0x50 (28)
 [<c0125c3a>] sys_waitpid+0x2a/0x30 (24)
 [<c0103f8d>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00000004 ]
| 4-level deep critical section nesting:
----------------------------------------
.. [<c032a44f>] .... __sched_text_start+0x4f/0x930
.....[<c032ad70>] ..   ( <= schedule+0x40/0x140)
.. [<c012017f>] .... __mmdrop_delayed+0x1f/0xa0
.....[<c01192f7>] ..   ( <= finish_task_switch+0x97/0xc0)
.. [<c032ca3f>] .... _raw_spin_lock+0x1f/0x70
.....[<c0117f62>] ..   ( <= task_rq_lock+0x42/0x80)
.. [<c013e32d>] .... print_traces+0x1d/0x60
.....[<c0104e83>] ..   ( <= dump_stack+0x23/0x30)

 =>   dump-end timestamp 4209394021

(get_ltrace.sh/6305/CPU#1): new 77 us maximum-latency critical section.
 => started at timestamp 4209417650: <__sched_text_start+0x4f/0x930>
 =>   ended at timestamp 4209417728: <preempt_schedule+0x6e/0x80>
 [<c0104e83>] dump_stack+0x23/0x30 (20)
 [<c013d107>] check_critical_timing+0x1d7/0x390 (88)
 [<c013d58d>] trace_irqs_on+0x7d/0x90 (24)
 [<c032aede>] preempt_schedule+0x6e/0x80 (20)
 [<c01190fb>] wake_up_new_task+0x16b/0x240 (44)
 [<c011fce9>] do_fork+0x129/0x1d0 (132)
 [<c01029be>] sys_clone+0x3e/0x50 (32)
 [<c0103f8d>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c013e32d>] .... print_traces+0x1d/0x60
.....[<c0104e83>] ..   ( <= dump_stack+0x23/0x30)

 =>   dump-end timestamp 4209652960

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Mark_H_Johnson]

> [2] variety of long > 1 msec wakeup latencies (see below)
> [3] primary long latencies with ksoftirqd/[01] and IRQ 10 tasks
Never mind on the long latencies.

I forgot to set udma2 on the disk drive. Setting that  changed
the wakeup latencies back to the sub 100 usec range.

Will be trying the (non wakeup) traces next.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Mark_H_Johnson]

>i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
>downloaded from the usual place:
>
>     http://redhat.com/~mingo/realtime-preempt/
>
After simplifying the test, I have some data on wakeup times that don't
look too good. The set up was...
 - booted / telinit 5
 - did NOT change any IRQ nor system task priorities
 - ran data collection script
 - ran a simple script that exercised the disk (write, copy, read)
Nothing was at RT priority except system tasks & data collection script
[script was RT fifo priority 1].

Symptoms seen include:
 [1] still see occasional truncated latency_trace output (see below)
 [2] variety of long > 1 msec wakeup latencies (see below)
 [3] primary long latencies with ksoftirqd/[01] and IRQ 10 tasks
I have over 30 traces in about 5-10 minutes of testing. Let me know if
you want all of them.

  --Mark

Truncated example:

preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.30-2
-------------------------------------------------------
 latency: 2097 us, entries: 1 (1)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:2]
    -----------------
    | task: ksoftirqd/0/4, uid:0 nice:-10 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x379/0x3e0 <c0118d99>
 => ended at:   finish_task_switch+0x4f/0xc0 <c01192af>
=======>
    4 88000001 0.000ms (+0.000ms): trace_stop_sched_switched
(finish_task_switch)

Long trace example [> 2 msec]

preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.30-2
-------------------------------------------------------
 latency: 1537 us, entries: 36 (36)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:2]
    -----------------
    | task: ksoftirqd/0/4, uid:0 nice:-10 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x379/0x3e0 <c0118d99>
 => ended at:   finish_task_switch+0x4f/0xc0 <c01192af>
=======>
    0 88000004 0.000ms (+0.000ms): __trace_start_sched_wakeup
(try_to_wake_up)
    0 88000004 0.000ms (+0.000ms): _raw_spin_unlock (try_to_wake_up)
    0 88000003 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
    0 88000003 0.000ms (+0.000ms): (4) ((0))
    0 88000003 0.001ms (+0.000ms): (105) ((140))
    0 88000003 0.001ms (+0.000ms): try_to_wake_up (wake_up_process)
    0 88000003 0.001ms (+0.000ms): (0) ((1))
    0 88000003 0.001ms (+0.000ms): _raw_spin_unlock (try_to_wake_up)
    0 88000002 0.002ms (+0.000ms): preempt_schedule (try_to_wake_up)
    0 88000002 0.002ms (+0.275ms): wake_up_process (do_softirq)
    0 88010001 0.277ms (+0.207ms): do_nmi (default_idle)
    0 88010001 0.484ms (+0.001ms): do_nmi (__mcount)
    0 88010001 0.486ms (+0.077ms): do_nmi (<00200046>)
    0 88010001 0.563ms (+0.275ms): preempt_schedule (nmi_watchdog_tick)
    0 08000000 0.838ms (+0.000ms): preempt_schedule (cpu_idle)
    0 98000000 0.839ms (+0.000ms): __sched_text_start (preempt_schedule)
    0 98000001 0.839ms (+0.000ms): sched_clock (__sched_text_start)
    0 98000001 0.839ms (+0.000ms): _raw_spin_lock_irq (__sched_text_start)
    0 98000001 0.840ms (+0.000ms): _raw_spin_lock_irqsave
(__sched_text_start)
    0 88000002 0.840ms (+0.000ms): dequeue_task (__sched_text_start)
    0 88000002 0.840ms (+0.000ms): recalc_task_prio (__sched_text_start)
    0 88000002 0.841ms (+0.000ms): effective_prio (recalc_task_prio)
    0 88000002 0.841ms (+0.000ms): enqueue_task (__sched_text_start)
    0 80000002 0.841ms (+0.001ms): trace_array (__sched_text_start)
    0 80000002 0.843ms (+0.000ms): (4) ((105))
    0 80000002 0.843ms (+0.000ms): (0) ((110))
    0 80000002 0.844ms (+0.002ms): trace_array (__sched_text_start)
    4 80000002 0.847ms (+0.000ms): __switch_to (__sched_text_start)
    4 80000002 0.847ms (+0.000ms): (0) ((4))
    4 80000002 0.847ms (+0.000ms): (140) ((105))
    4 80000002 0.847ms (+0.000ms): finish_task_switch (__sched_text_start)
    4 80000002 0.848ms (+0.000ms): _raw_spin_unlock (finish_task_switch)
    4 80000001 0.848ms (+0.000ms): trace_stop_sched_switched
(finish_task_switch)
    4 80000001 0.848ms (+0.000ms): (4) ((105))
    4 80000001 0.848ms (+1.318ms): _raw_spin_lock_irqsave
(trace_stop_sched_switched)
    4 80000001 2.167ms (+0.000ms): trace_stop_sched_switched
(finish_task_switch)

Another long one...

preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.30-2
-------------------------------------------------------
 latency: 1956 us, entries: 36 (36)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:2]
    -----------------
    | task: ksoftirqd/0/4, uid:0 nice:-10 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x379/0x3e0 <c0118d99>
 => ended at:   finish_task_switch+0x4f/0xc0 <c01192af>
=======>
   28 88000003 0.000ms (+0.000ms): __trace_start_sched_wakeup
(try_to_wake_up)
   28 88000003 0.000ms (+0.000ms): _raw_spin_unlock (try_to_wake_up)
   28 88000002 0.000ms (+0.001ms): preempt_schedule (try_to_wake_up)
   28 88000002 0.001ms (+0.000ms): (4) ((28))
   28 88000002 0.002ms (+0.000ms): (105) ((110))
   28 88000002 0.002ms (+0.000ms): try_to_wake_up (wake_up_process)
   28 88000002 0.002ms (+0.000ms): (0) ((1))
   28 88000002 0.002ms (+0.000ms): _raw_spin_unlock (try_to_wake_up)
   28 88000001 0.003ms (+0.000ms): preempt_schedule (try_to_wake_up)
   28 88000001 0.003ms (+0.000ms): wake_up_process (do_softirq)
   28 88000000 0.003ms (+0.000ms): preempt_schedule_irq (need_resched)
   28 98000000 0.004ms (+0.000ms): __sched_text_start
(preempt_schedule_irq)
   28 98000001 0.004ms (+0.000ms): sched_clock (__sched_text_start)
   28 98000001 0.004ms (+0.000ms): _raw_spin_lock_irq (__sched_text_start)
   28 98000001 0.005ms (+0.000ms): _raw_spin_lock_irqsave
(__sched_text_start)
   28 88000002 0.005ms (+0.000ms): dequeue_task (__sched_text_start)
   28 88000002 0.006ms (+0.000ms): recalc_task_prio (__sched_text_start)
   28 88000002 0.006ms (+0.000ms): effective_prio (recalc_task_prio)
   28 88000002 0.006ms (+0.000ms): enqueue_task (__sched_text_start)
   28 80000002 0.006ms (+0.001ms): trace_array (__sched_text_start)
   28 80000002 0.008ms (+0.000ms): (4) ((105))
   28 80000002 0.009ms (+0.000ms): (0) ((110))
   28 80000002 0.009ms (+0.000ms): (28) ((110))
   28 80000002 0.009ms (+0.000ms): (0) ((115))
   28 80000002 0.009ms (+0.000ms): (4344) ((118))
   28 80000002 0.010ms (+0.000ms): (0) ((120))
   28 80000002 0.010ms (+0.052ms): trace_array (__sched_text_start)
    4 80000002 0.063ms (+1.256ms): __switch_to (__sched_text_start)
    4 80000002 1.319ms (+0.000ms): (28) ((4))
    4 80000002 1.319ms (+0.000ms): (110) ((105))
    4 80000002 1.319ms (+0.000ms): finish_task_switch (__sched_text_start)
    4 80000002 1.319ms (+0.139ms): _raw_spin_unlock (finish_task_switch)
    4 80000001 1.459ms (+0.000ms): trace_stop_sched_switched
(finish_task_switch)
    4 80000001 1.460ms (+0.000ms): (4) ((105))
    4 80000001 1.460ms (+0.420ms): _raw_spin_lock_irqsave
(trace_stop_sched_switched)
    4 88000001 1.880ms (+0.000ms): trace_stop_sched_switched
(finish_task_switch)

A third long one, note inconsistent header / total time.

preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.30-2
-------------------------------------------------------
 latency: 1000 us, entries: 32 (32)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:2]
    -----------------
    | task: ksoftirqd/0/4, uid:0 nice:-10 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x379/0x3e0 <c0118d99>
 => ended at:   finish_task_switch+0x4f/0xc0 <c01192af>
=======>
    0 88000004 0.000ms (+0.000ms): __trace_start_sched_wakeup
(try_to_wake_up)
    0 88000004 0.000ms (+0.000ms): _raw_spin_unlock (try_to_wake_up)
    0 88000003 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
    0 88000003 0.000ms (+0.000ms): (4) ((0))
    0 88000003 0.001ms (+0.000ms): (105) ((140))
    0 88000003 0.001ms (+0.000ms): try_to_wake_up (wake_up_process)
    0 88000003 0.001ms (+0.000ms): (0) ((1))
    0 88000003 0.001ms (+0.000ms): _raw_spin_unlock (try_to_wake_up)
    0 88000002 0.002ms (+0.000ms): preempt_schedule (try_to_wake_up)
    0 88000002 0.002ms (+0.000ms): wake_up_process (do_softirq)
    0 08000000 0.003ms (+0.000ms): preempt_schedule (cpu_idle)
    0 98000000 0.003ms (+0.000ms): __sched_text_start (preempt_schedule)
    0 98000001 0.004ms (+0.000ms): sched_clock (__sched_text_start)
    0 98000001 0.004ms (+0.000ms): _raw_spin_lock_irq (__sched_text_start)
    0 98000001 0.004ms (+0.000ms): _raw_spin_lock_irqsave
(__sched_text_start)
    0 88000002 0.005ms (+0.000ms): dequeue_task (__sched_text_start)
    0 88000002 0.005ms (+0.000ms): recalc_task_prio (__sched_text_start)
    0 88000002 0.005ms (+0.000ms): effective_prio (recalc_task_prio)
    0 88000002 0.006ms (+0.000ms): enqueue_task (__sched_text_start)
    0 80000002 0.006ms (+0.001ms): trace_array (__sched_text_start)
    0 80000002 0.008ms (+0.000ms): (4) ((105))
    0 80000002 0.008ms (+0.000ms): (0) ((110))
    0 80000002 0.009ms (+0.002ms): trace_array (__sched_text_start)
    4 80000002 0.011ms (+0.000ms): __switch_to (__sched_text_start)
    4 80000002 0.012ms (+0.000ms): (0) ((4))
    4 80000002 0.012ms (+0.000ms): (140) ((105))
    4 80000002 0.012ms (+0.000ms): finish_task_switch (__sched_text_start)
    4 80000002 0.012ms (+0.000ms): _raw_spin_unlock (finish_task_switch)
    4 80000001 0.013ms (+0.000ms): trace_stop_sched_switched
(finish_task_switch)
    4 80000001 0.013ms (+0.000ms): (4) ((105))
    4 80000001 0.013ms (+1.307ms): _raw_spin_lock_irqsave
(trace_stop_sched_switched)
    4 80000001 1.321ms (+0.000ms): trace_stop_sched_switched
(finish_task_switch)

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Mark_H_Johnson]

>Just did a build with -V0.7.30-2 and was about to start testing when
>the system locked up (no keyboard response, display frozen, etc.). ...

Same symptom with a slightly different set of steps leading to the
problem.
 - boot / telinit 5 OK
 - su'd to get privileges
 - cat /proc/sys/kernel/preempt_wakeup_latency (showed 0)
 - echo 1 > /proc/sys/kernel/preempt_wakeup_latency
 - set RT priorities as before
 - started scripts to record data
 - system-config-soundcard (newline shown)
At this point, the system is locked up again with no response to any
inputs.

One thing I did notice from the previous test, I had two output files
from preempt_trace showing a couple minor (just over 50 usec each)
wakeup traces.

I have a few ideas to simplify the set up to see if I can get some
useful data out of the system.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

[patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.21

[Ingo Molnar]

i have released the -V0.7.21 Real-Time Preemption patch, which can be
downloaded from the usual place:

   http://redhat.com/~mingo/realtime-preempt/

this release includes fixes and debugging-improvements.

Changes since -V0.7.20:

 - reverted the modlist_lock change - it caused more problems than it 
   solved.

 - implemented irqs-off critical section timing/tracing, inspired by the 
   positive results Thomas Gleixner got with a different kind of cli/sti
   tracer. To activate it, enable CONFIG_CRITICAL_TIMING and 
   CONFIG_CRITICAL_IRQSOFF_TIMING and cli/sti latencies will be reported
   'integrated' into the preempt on/off latencies.

 - sped up tracing in a number of ways. Performance of the tracer slowly
   eroded in the past week or two, it needed alignment and size fixes ,
   inlining/branch-prediction updates and i got rid of unnecessary code.
   The max latency is now traced in cycles - this got rid of an
   expensive 64-bit division in the fastpath. (the /proc/sys tunables
   are still in usecs so userspace should not notice anything.) It's
   still not cheap but roughly 5 times faster than -V0.7.20's tracer, on
   a fast desktop box.

 - renamed CONFIG_PREEMPT_REALTIME to CONFIG_PREEMPT_RT - it's shorter.

 - renamed CONFIG_PREEMPT_TIMING to CONFIG_CRITICAL_TIMING and 
   introduced CONFIG_CRITICAL_IRQSOFF_TIMING to enable cli/sti timing.

to create a -V0.7.21 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc1.bz2
   http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc1/2.6.10-rc1-mm3/2.6.10-rc1-mm3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc1-mm3-V0.7.21

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.23

[Ingo Molnar]

i have released the -V0.7.23 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this release includes fixes, development/debugging improvements and
latency improvements and other speedups.

the biggest change is the reworked timing/tracing framework. Wakeup
timing became a compile-time thing and can be selected independently of
the preemption mode - i.e. it can now be used on near-vanilla !PREEMPT
kernels too, providing good wakeup-latency comparison of the various
preemption models.

irqs-off and preempt-off critical section timing/tracing can be selected
if wakeup timing is disabled, the two options can be selected separately
or together as well.

another improvement is that wakeup-timing now works correctly on SMP too
- the tracer 'follows' the highest-priority task in the system as it
gets bounced between CPUs and always traces the CPU where the task is
pending.

other changes since -V0.7.22:

 - semaphore livelock fix: feedback from Mark H. Johnson pinpointed a 
   bug in the down_trylock() semaphore code: if preempted in the wrong
   moment a lower-prio task could cause a higher-prio task to livelock
   indefinitely. This fix could solve the 'keventd looping' problem 
   reported by Mark.

   the fix is to make the down_trylock() code a bit simpler, but this
   also introduces the potential for down_trylock() to get 'spurious' 
   locking-rejects. Hopefully this wont be a big problem - we dont 
   really guarantee that down_trylock() succeeds - but code using higher
   semaphore counts to track resources could be negatively impacted by
   this. We'll see.

 - console assert fix: implemented a different type of fbcon
   RT-preemption handling variant, this could solve the assert reported
   by Amit Shah.

 - debugging improvement: implemented a sequence counter for max latency 
   traces. This has the advantage of solving the 'slow console on SMP
   problem': the latency-timing code used to get confused by another CPU 
   printing a timing message to a slow console and thus delaying that 
   other CPU. Now any latency that gets generated while a maximum is 
   printed is skipped.

 - further shrunk the non-debug size of struct rt_mutex by moving the 
   save_state logic into the debug paths - size is now 4 machine-words.

 - fixed CONFIG_HIGHMEM latencies: all atomic-kmap APIs are now wrapped 
   seemlessly and in a preemptible way.

 - implemented an IO-APIC register cache to speed up the IRQ-redirection
   latency hotpath. Also, made the POST flush a bit faster.

 - disable KGDB if PREEMPT_RT - it's broken for now.

 - move some runtime checks to under DEBUG_PREEMPT - this speeds up 
   CRITICAL_PREEMPT_TIMING.

to create a -V0.7.23 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc1.bz2
   http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc1/2.6.10-rc1-mm3/2.6.10-rc1-mm3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc1-mm3-V0.7.23

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-0

[Ingo Molnar]

i have released the -V0.7.25-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this release includes fixes, new features and latency improvements.

the biggest change is the threading of the lone remaining non-threaded
external interrupt: the timer interrupt (IRQ#0).

It was not threaded until now because unlike device interrupts the timer
interrupt is quite deeply attached to Linux's architecture, driving
process timing, profiling and the scheduler. I separated these into
stuff that can be done from a thread context and stuff that needs to
execute directly.

Fortunately most of the expensive and latency-generating stuff could be
pushed into the irq thread. As a side-effect this enabled the turning of
rtc_lock and xtime_lock into a mutex. Also, it removed some ugly hacks
from rtc.c and should improve worst-case latencies.

the other bigger change is the reworking of the .config space. Instead
of the deep hierarchy of hard-to-understand technical PREEMPT options,
there's now a flattened out choice of 4 preemption models:

                ( ) No Forced Preemption (Server)
                ( ) Voluntary Kernel Preemption (Desktop)
                ( ) Preemptible Kernel (Low-Latency Desktop)
                (X) Complete Preemption (Real-Time)

and updated help texts. Plus for the preemption models where it can be
freely turned off, softirq and hardirq threading is an additional
option.

the third bigger change is the reworking of the tracer to make it easier
to drive it from user-space. There are 3 runtime flags now:

 /proc/sys/kernel/trace_enabled           (default: 1)
 /proc/sys/kernel/trace_user_triggered    (default: 0)
 /proc/sys/kernel/trace_freerunning       (default: 0)

semantics of user-triggered tracing: if enabled then any active tracing
of wakeup and/or critical sections stops and userspace drives start/stop
events via gettimeofday(0,1)/gettimeofday(0,0). The latter saves the
current trace into /proc/latency_trace, the former clears the trace
buffer and starts tracing anew. Doing another gettimeofday(0,1) on an
already running tracer clears the trace and restarts it without saving
the current trace into /proc/latency_trace. Doing a gettimeofday(0,0) on
an already stopped tracer has no effect (i.e. /proc/latency_trace wont
be saved a second time). The tracer does timing for userspace
automatically the same way it does it for the built-in timing
mechanisms, and it can be configured via the preempt_max_latency and
preempt_tresh tunables.

also, wakeup-timing, irq-off and preempt-off critical section timing can
now be done at once again, the /proc/sys/kernel/preempt_wakeup_timing
flag switches between the modes. (default: 1)

other changes since -V0.7.24:

 - debug feature: added the RTC-debug patch sanitized by K.R. Foley,
   plus further cleanups.

 - added upstream fix for kobject related crash, pointed out by Shane 
   Shrybman.

 - cleanup: Kconfig help text fixes from Amit Shah

 - latency improvement: on UP-IOAPIC, when redirecting an interrupt, do
   not ack the APIC. This is the method used for direct interrupts and
   on UP it might as well work out fine. It is certainly faster than
   masking/unmasking, making UP-IOAPIC the fastest PIC mode again.

 - livelock fix: the timer-irq threading unearthed a seqlock related
   livelock scenario, which triggered in do_gettimeoffset() big-time. 
   The solution is to serialize seqlock readers with writers _iff_ the 
   seqlock status is 'invalid'. This is a rare event, but when it
   happens it saves the day.

 - debugging helper: the /proc/sys/kernel/debug_direct_keyboard flag 
   (default: 0) will hack the keyboard IRQ into being direct. NOTE: the 
   keyboard in this mode should only be used to access SysRq 
   functionality that is not possible via the threaded keyboard handler. 
   The direct keyboard IRQ can crash the system.

 - new kernel profiling features: added profile=preempt to profile 
   whether interrupts hit the kernel in preemptible mode or in a 
   critical section. Added /proc/sys/kernel/prof_pid to profile a
   specific PID only. (default: -1, meaning all tasks profiled) Added
   /proc/irq/prof_cpu_mask back.

 - robustness improvement: do not report atomicity-warnings during
   kernel oopses - it's more important to get the oops out to the
   console.

to create a -V0.7.25-0 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc1.bz2
   http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc1/2.6.10-rc1-mm3/2.6.10-rc1-mm3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc1-mm3-V0.7.25-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc1-mm3-V0.7.25-1

[Ingo Molnar]

i have released the -V0.7.25-1 Real-Time Preemption patch, which can be
downloaded from the usual place:

    http://redhat.com/~mingo/realtime-preempt/

this is a fixes-only release that resolves a couple of bugs that slipped
into -V0.7.25-0:

 - lockup/deadlock fix: make debug_direct_keyboard default to 0. It is
   only a debug helper to be used for development, it was never intended
   to be enabled. This fix should resolve the bugs reported by Gunther
   Persoons and Mark H. Johnson.

 - fix symbol export problems in rtc.ko, reported by Remi Colinet, based
   on the patch from K.R. Foley.

 - make preempt_wakeup_timing default to 1 if enabled in the .config, as 
   originally intended.

to create a -V0.7.25-1 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc1.bz2
   http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc1/2.6.10-rc1-mm3/2.6.10-rc1-mm3.bz2
   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc1-mm3-V0.7.25-1

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.27-0

[Ingo Molnar]

i have released the -V0.7.27-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a pure merge of -V0.7.26-5 to 2.6.10-rc2-mm1, there are no other
changes.

to create a -V0.7.27-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.27-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.27-1

[Ingo Molnar]

i have released the -V0.7.27-1 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this quick update fixes a couple of build bugs.

Changes since a -V0.7.27-0:

 - fix iptables compilation error

 - fix selinux compilation error

to create a -V0.7.27-1 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.27-1

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.27-3

[Ingo Molnar]

i have released the -V0.7.27-3 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is another quick update to fix a couple of bugs. Sorry about the
fast pace of updates but these fixes are worth having ASAP:

Changes since a -V0.7.27-1:

 - fix module-put BKL count bug - this could explain/fix the lockups
   reported by Rui Nuno Capela.

 - fixed a netfilter related networking deadlock reported by Mark H. 
   Johnson two weeks ago, it triggered on my testbox today. This (rare)
   bug could potentially explain some of the other lockup reports that
   are still open.

 - fix load average constant +1.0 offset when PREEMPT_RT is enabled. 
   This was an artifact of the IRQ-threading of the timer interrupt.

to create a -V0.7.27-3 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.27-3

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-0

[Ingo Molnar]

i have released the -V0.7.28-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a fixes & latency-reduction release.

Changes since a -V0.7.27-3:

 - made the UP-ioapic code a bit more conservative again - maybe some of
   the lockups are related?

 - removed the BKL from the sound code in a cleaner way and
   removed the quite fragile 'negative ->lock_depth' code. Much less
   intrusive than i originally thought, and much cleaner as well.

 - more fixes to the wakeup-timing logic, 4 false positives fixed in
   total, mostly related to new-task-wakeup not accurately starting the
   tracer.

 - fixed the mmx-memcpy related latency reported by Florian Schmidt and 
   others. Also turned off the MMX/SSE ops in the RAID code, which 
   can introduce similar latencies.

 - kgdb fix from Bill Huey

 - knfsd shutdown with-BKL-held fix

 - highmem compilation fix

 - profiling related crash fix

 - implemented 'direct-path' rescheduling to further reduce scheduling
   latency: the kernel will now in most cases go from try_to_wakeup()
   into the scheduler directly without re-enabling interrupts ever again
   (and thus not giving irq handlers a window to increase latency). This
   is also the final fix for irq nesting and irq-stack recursion.

 - turn off sync wakeups on PREEMPT_RT -> they are latency generators

to create a -V0.7.28-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.28-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm1-V0.7.28-1

[Ingo Molnar]

i have released the -V0.7.28-1 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this should fix the lockup bug reported by Florian Schmidt.

there's a generic PREEMPT bug in the upstream kernel: there exists a
single-instruction race window in __flush_tlb(), if the kernel preempted
exactly there in a lazy-TLB thread and certain other, rare scheduling
and MM properties were true as well (a certain constellation of threads
and lazy-TLB kernel threads occured), and the lazy-TLB task then got
another user TLB to inherit, and switched to a task from which it
inherited that new TLB, thus the wrong cr3 was loaded and inherited by
this next, non-lazy-TLB next task; then (and only then) this scenario
would typically manifest itself in the form of an infinite pagefault
lockup occuring much after the fact, upon the next userspace access (to
the joy of a totally baffled kernel developer). I suspect from the
description you can guess how much fun it was to debug it =B-)

the bug is even more rare in the generic kernel, because there most (but
not all) TLB flush points are in a critical section.

this fix could resolve some of the other 'my box just locked up'
reports.

Changes since a -V0.7.28-0:

 - reverted the UP-ioapic change - it was unrelated to the lockup and it
   is known to cause problems on certain IDE/soundcard combinations.

 - fixed and improved the trace_print_on_crash tracing feature - it was
   highly needed to find the TLB bug ...

to create a -V0.7.28-1 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm1/2.6.10-rc2-mm1.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm1-V0.7.28-1

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.29-0

[Ingo Molnar]

i have released the -V0.7.29-0 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

this is a pure merge of -V0.7.28-2 to 2.6.10-rc2-mm2. -rc2-mm2 itself is
a fixes-only release.

to create a -V0.7.29-0 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.29-0

	Ingo

[patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

the biggest change in this release are fixes for priority-inheritance
bugs uncovered by Esben Nielsen pi_test suite. These bugs could explain
some of the jackd-under-load latencies reported.

Changes since -V0.7.29-0:

 - priority inheritance handling fixes:

    - sort the RT wakees at wakeup time, not at block-time: an RT task
      might have gotten boosted while it slept.

    - fix priority-restoration bug at mutex-release time

    - use task_rt() not p->policy to determine whether a task needs 
      PI handling - a SCHED_OTHER task might be boosted to RT prio.

    - fix mutex_setprio() bug: queue now-RT tasks to the active array, 
      otherwise expired SCHED_OTHER tasks will not be properly boosted.

 - went back to the mask-and-delay method of handling hardirqs on 
   UP-IOAPIC as well. Due to APIC prioritization hardirqs can get
   delayed by another, unacked hardirq, so the quick method needs more 
   work before it can be used.

 - added Thomas Gleixner's semaphore -> completion changes for 
   drv->unload_sem. This fixes the module unload crashes reported by 
   Rui Nuno Capela and Shane Shrybman.

 - dvb mutex updates for RT, this fixes the bug reported by Christian 
   Meder.

 - e100 fix from K.R. Foley - this should fix the boot-time e100
   enable_irq warning.

 - NFS lockd mutex RT fixes from Thomas Gleixner - this could fix some
   of the bugs reported by Bill Huey.

 - PREEMPT_VOLUNTARY fixes - this could fix the boot-time hang reported 
   by Lee Revell.

 - wake up irq thread upon creation - this solves the 'irq thread only 
   changes priority after first interrupt arrives' anomaly reported.

to create a -V0.7.30-2 tree from scratch, the patching order is:

  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.9.tar.bz2
  http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.10-rc2.bz2
  http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.10-rc2/2.6.10-rc2-mm2/2.6.10-rc2-mm2.bz2
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.10-rc2-mm2-V0.7.30-2

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Florian Schmidt]

On Mon, 22 Nov 2004 01:54:11 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
> downloaded from the usual place:

> the biggest change in this release are fixes for priority-inheritance
> bugs uncovered by Esben Nielsen pi_test suite. These bugs could explain
> some of the jackd-under-load latencies reported.

It seems these large load related xruns are gone :) At least i wasn't able
to trigger any during my uptime of 52 min. Will report if i ever see  any of
those again.

flo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> On Mon, 22 Nov 2004 01:54:11 +0100
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> > i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
> > downloaded from the usual place:
> 
> > the biggest change in this release are fixes for priority-inheritance
> > bugs uncovered by Esben Nielsen pi_test suite. These bugs could explain
> > some of the jackd-under-load latencies reported.
> 
> It seems these large load related xruns are gone :) At least i wasn't
> able to trigger any during my uptime of 52 min. Will report if i ever
> see any of those again.

great. I now suspect that some of the xrun problems Rui was observing on
-RT kernel could be (positively) affected by these fixes too.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> * Florian Schmidt wrote:
>
>>
>> > i have released the -V0.7.30-2 Real-Time Preemption patch, which can
>> > be downloaded from the usual place:
>>
>> > the biggest change in this release are fixes for priority-inheritance
>> > bugs uncovered by Esben Nielsen pi_test suite. These bugs could
>> > explain some of the jackd-under-load latencies reported.
>>
>> It seems these large load related xruns are gone :) At least i wasn't
>> able to trigger any during my uptime of 52 min. Will report if i ever
>> see any of those again.
>
> great. I now suspect that some of the xrun problems Rui was observing on
> -RT kernel could be (positively) affected by these fixes too.
>

Just made some test-runs with RT-V0.7.30-2, with my jackd-R + 8*fluidsynth
benchmark on my laptop (P4/UP), and the results don't seem to be eligible
to the hall of fame, at least when compared with RT-0.7.7 as the ones I
last posted here a few weeks ago.

I hate to say this, but the XRUN rate has increased since RT-0.7.7, and
the maximum scheduling delay reported by jackd has also degraded to 1000
usecs (was around 600 usecs).

Please note that this is not unique to latest RT-V0.7.30-2, but also
applies to each one of the previous iterations I've been testing, only not
reported until now. Again, all test conditions were kept the same
(hardware, jackd, alsa), only the kernel has changed (obviously).


OTOH, there's another thing: I don't seem to be able to build an initrd
image under the latest RT kernels. Something related to the loopback
device. When trying to run mkinitrd it stalls, somewhere under this
process:

  mount -t ext2 /root/tmp/initrd.img /root/tmp/initrd.mnt -o loop

This happens on my laptop (P4/UP, Mandrake 10.1c) but not on my desktop
(P4/SMT, SUSE 9.2pro). Speaking of which, the lockups experienced while
unloading the ALSA modules seems to be over, at least as far as I could
try with RT-V0.7.29-4 (probably not enough yet).

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> > great. I now suspect that some of the xrun problems Rui was observing on
> > -RT kernel could be (positively) affected by these fixes too.
> >
> 
> Just made some test-runs with RT-V0.7.30-2, with my jackd-R +
> 8*fluidsynth benchmark on my laptop (P4/UP), and the results don't
> seem to be eligible to the hall of fame, at least when compared with
> RT-0.7.7 as the ones I last posted here a few weeks ago.
> 
> I hate to say this, but the XRUN rate has increased since RT-0.7.7,
> and the maximum scheduling delay reported by jackd has also degraded
> to 1000 usecs (was around 600 usecs).

well, life would be too easy if two bugs were fixed at once ;) These
were nodebug runs, right? Could you give me a description of the precise
commands of how you started jackd and fluidsyth (and their versions) -
so that i could try to reproduce & debug your setup. It is certainly a
complex scheduling scenario.

(perhaps with a link to the .sf2 and .mid files you used, if they are
public - or whether it's fine if i use the VintageDreamsWaves-v2.sf2
sound-fonts that comes with fluidsynth plus a random .mid file from the
net?)

> OTOH, there's another thing: I don't seem to be able to build an
> initrd image under the latest RT kernels. Something related to the
> loopback device. When trying to run mkinitrd it stalls, somewhere
> under this process:
> 
>   mount -t ext2 /root/tmp/initrd.img /root/tmp/initrd.mnt -o loop

Do you know when this started, roughly?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> Rui Nuno Capela wrote:
>
>> > great. I now suspect that some of the xrun problems Rui was observing
>> > on -RT kernel could be (positively) affected by these fixes too.
>> >
>>
>> Just made some test-runs with RT-V0.7.30-2, with my jackd-R +
>> 8*fluidsynth benchmark on my laptop (P4/UP), and the results don't
>> seem to be eligible to the hall of fame, at least when compared with
>> RT-0.7.7 as the ones I last posted here a few weeks ago.
>>
>> I hate to say this, but the XRUN rate has increased since RT-0.7.7,
>> and the maximum scheduling delay reported by jackd has also degraded
>> to 1000 usecs (was around 600 usecs).
>
> well, life would be too easy if two bugs were fixed at once ;) These
> were nodebug runs, right? Could you give me a description of the precise
> commands of how you started jackd and fluidsyth (and their versions) -
> so that i could try to reproduce & debug your setup. It is certainly a
> complex scheduling scenario.
>
> (perhaps with a link to the .sf2 and .mid files you used, if they are
> public - or whether it's fine if i use the VintageDreamsWaves-v2.sf2
> sound-fonts that comes with fluidsynth plus a random .mid file from the
> net?)
>

These are the command-lines of my test suite:

  jackd -R -dalsa -dhw:0 -P20 -r44100 -p64 -n2 -S -P &
  fluidsynth -s -i -a jack -j -o jack.audio.id=fluid1 -o shell.port=9800
ct4mgm.sf2 &
  fluidsynth -s -i -a jack -j -o jack.audio.id=fluid2 -o shell.port=9801
ct4mgm.sf2 &
  .
  .
  .
  fluidsynth -s -i -a jack -j -o jack.audio.id=fluid8 -o shell.port=9807
ct4mgm.sf2 &


Versions are:

  jack 0.99.10cvs
  fluidsynth 1.0.5


>> OTOH, there's another thing: I don't seem to be able to build an
>> initrd image under the latest RT kernels. Something related to the
>> loopback device. When trying to run mkinitrd it stalls, somewhere
>> under this process:
>>
>>   mount -t ext2 /root/tmp/initrd.img /root/tmp/initrd.mnt -o loop
>
> Do you know when this started, roughly?
>

Not sure, but the first time I've noticed it was on RT-0.7.29-2 and that
was purely by chance. Problem is that I've been building the RT kernels
under a non-RT stock kernel, so I can't say how long or when it all
started exactly. I remember however that this is a revisited issue,
thought.

Cheers.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> These are the command-lines of my test suite:
> 
>   jackd -R -dalsa -dhw:0 -P20 -r44100 -p64 -n2 -S -P &
>   fluidsynth -s -i -a jack -j -o jack.audio.id=fluid1 -o shell.port=9800
> ct4mgm.sf2 &
>   fluidsynth -s -i -a jack -j -o jack.audio.id=fluid2 -o shell.port=9801
> ct4mgm.sf2 &

is this enough to generate the xruns in jackd? Shouldnt fluidsynth be
given a MIDI file to play back? (if yes, what is the method i should use
- should i give it on the command line?)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> >   jackd -R -dalsa -dhw:0 -P20 -r44100 -p64 -n2 -S -P &
> >   fluidsynth -s -i -a jack -j -o jack.audio.id=fluid1 -o shell.port=9800
> > ct4mgm.sf2 &
> >   fluidsynth -s -i -a jack -j -o jack.audio.id=fluid2 -o shell.port=9801
> > ct4mgm.sf2 &
> 
> is this enough to generate the xruns in jackd? Shouldnt fluidsynth be
> given a MIDI file to play back? (if yes, what is the method i should
> use - should i give it on the command line?)

ah, i think i understand: fluidsynth has roughly the same CPU overhead
when it is 'silent' (it's generating small static noise in that case),
compared to when it's playing a MIDI file - so i should be able to see
the xruns if i just run jackd and 8 fluidsynth instances, and then load
the box - correct?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> ah, i think i understand: fluidsynth has roughly the same CPU overhead
> when it is 'silent' (it's generating small static noise in that case),
> compared to when it's playing a MIDI file - so i should be able to see
> the xruns if i just run jackd and 8 fluidsynth instances, and then
> load the box - correct?

another question: it seems the fluidsynth threads are running as non-RT
threads - they are soft-synthesizing sound that jackd will mix and play. 
Now, can any delay of these fluidsynth threads (they are non-RT tasks
and can get delayed indefinitely) cause an xrun in Jackd, or should it
'only' make the sound more choppy?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Florian Schmidt]

[-- Attachment #1: Type: text/plain, Size: 2335 bytes --]

On Mon, 22 Nov 2004 14:24:59 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> > Just made some test-runs with RT-V0.7.30-2, with my jackd-R +
> > 8*fluidsynth benchmark on my laptop (P4/UP), and the results don't
> > seem to be eligible to the hall of fame, at least when compared with
> > RT-0.7.7 as the ones I last posted here a few weeks ago.
> > 
> > I hate to say this, but the XRUN rate has increased since RT-0.7.7,
> > and the maximum scheduling delay reported by jackd has also degraded
> > to 1000 usecs (was around 600 usecs).
> 
> well, life would be too easy if two bugs were fixed at once ;) 

Hi,

i just wanted to mention that a good share of jack clients have issues themself, doing all kinds of funky stuff in the RT thread which they shouldn't do. Maybe the RP kernel just exposes this misuse in a greater visible way. I don't know if fluidsynth is one of them. We could only find out by code inspection. 

Another way to test a more complex scenario than just jackd running with an empty graph (assuming that jackd itself isn't to blame) while avoiding the risk of getting bad data due to insane clients would be to code up an example jackd client that does nothing but putting some load onto the jackd graph but in a strictly RT fashion (no blocking stuff whatsoever).

Attached you probably find the most minimal jack client thinkable that does nothing but copy data from its input to its output port. Its only parameter is the time in seconds it will run (default 60). The jack client name is determined by the PID, so it can be started multiple times (jackd requires a unique name for each client).

compile with

g++ -o jack_test jack_test.cc -ljack

This code can easily be adapted to produce more load (just do some math stuff with the data in the process callback).

It seems jackd has a limitation to 14 clients atm (don't ask me why). The 15th kills jackd ;)

Also i wanted to mention that a good share of ALSA drivers have issues, too, and aren't nessecarily suited to low latency audio work. I don't know how to rule these out except for using the ALSA dummy soundcard driver (which might have its own issues, but it's probably simple enough to work reliable. it just doesn't use any hw IRQ's so it's maybe not a good measure for what we want to test) or to use a soundcard with a proven good driver. 

flo

[-- Attachment #2: jack_test.cc --]
[-- Type: text/x-c++src, Size: 1647 bytes --]

#include <jack/jack.h>
#include <iostream>
#include <sstream>
#include <unistd.h>

jack_client_t *client;
jack_port_t *iport;
jack_port_t *oport;

int process(jack_nframes_t frames, void *arg) {
	// std::cout << "process callback" << std::endl;
	jack_default_audio_sample_t *ibuf;
	ibuf = (jack_default_audio_sample_t*)jack_port_get_buffer(iport, frames);

	jack_default_audio_sample_t *obuf;
	obuf = (jack_default_audio_sample_t*)jack_port_get_buffer(oport, frames);

	for (jack_nframes_t frame = 0; frame < frames; frame++) {
		obuf[frame] = ibuf[frame];
	}
        return 1;
}

int main(int argc, char *argv[]) {
	// default = 60 seconds
	unsigned int seconds_to_run = 60;
	if (argc > 1) {
		std::stringstream sec_stream;
		sec_stream << argv[1];
		sec_stream >> seconds_to_run;
	}
	std::cout << "seconds to run: " << seconds_to_run << std::endl;
	
	std::stringstream pid_stream;
	pid_stream << getpid();
	
        std::cout << "client_new" << std::endl;
        client = jack_client_new(pid_stream.str().c_str());

        std::cout << "port_register." << std::endl;
        iport = jack_port_register(client, "in", JACK_DEFAULT_AUDIO_TYPE, JackPortIsInput|JackPortIsTerminal, 0);
	oport = jack_port_register(client, "out", JACK_DEFAULT_AUDIO_TYPE, JackPortIsTerminal|JackPortIsOutput, 0);

        std::cout << "set_process_callback" << std::endl;
        jack_set_process_callback(client, process, 0);

        std::cout << "activate" << std::endl;
        jack_activate(client);

        std::cout << "running" << std::endl;

        // while(1) {sleep(1);};
	sleep(seconds_to_run);

	jack_deactivate(client);
	jack_client_close(client);
}

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Rui Nuno Capela]

Florian Schmidt wrote:

> Ingo Molnar wrote:
>
>> > Just made some test-runs with RT-V0.7.30-2, with my jackd-R +
>> > 8*fluidsynth benchmark on my laptop (P4/UP), and the results don't
>> > seem to be eligible to the hall of fame, at least when compared with
>> > RT-0.7.7 as the ones I last posted here a few weeks ago.
>> >
>> > I hate to say this, but the XRUN rate has increased since RT-0.7.7,
>> > and the maximum scheduling delay reported by jackd has also degraded
>> > to 1000 usecs (was around 600 usecs).
>>
>> well, life would be too easy if two bugs were fixed at once ;)
>
> Hi,
>
> i just wanted to mention that a good share of jack clients have issues
> themself, doing all kinds of funky stuff in the RT thread which they
> shouldn't do. Maybe the RP kernel just exposes this misuse in a greater
> visible way. I don't know if fluidsynth is one of them. We could only find
> out by code inspection.
>

Yes, I know all this, and I warned before that this tests were strictly
and specific to the hardware, jackd and fludisynth code which are
intentionally kept the same along the several RT kernels that have been
issued.

Note that I've kept this consistency to my self, and applies /only/ to my
laptop, where the tests are being evaluated. Again, this test-suite of
mine has the sole intention to compare the jackd workload performance
across kernels, in an almost real softsynth scenario. All kernels tested
are built with no debug options, ressembling production ones as far as
possible.

For example, these are the results-du-jour, which serves as a straight
comparison RT-V0.7.30-2, with the previous posted ones from RT-V0.7.7:

                                  RT-V0.7.7 RT-V0.7.30-2
                                  --------- ------------
  XRUN Rate . . . . . . . . . . :     45.6        292.0  /hour
  Delay Rate (>spare time)  . . :     43.2        265.3  /hour
  Delay Rate (>1000 usecs)  . . :      3.6         29.3  /hour
  Delay Maximum . . . . . . . . :   1249         1045    usecs
  Cycle Maximum . . . . . . . . :    946         1127    usecs
  Average DSP Load. . . . . . . :     55.2         60.1  %
  Average CPU System Load . . . :     13.2         15.5  %
  Average CPU User Load . . . . :     41.9         46.2  %
  Average CPU Nice Load . . . . :      0.0          0.0  %
  Average CPU I/O Wait Load . . :      0.1          0.1  %
  Average CPU IRQ Load  . . . . :      0.0          0.0  %
  Average CPU Soft-IRQ Load . . :      0.0          0.0  %
  Average Interrupt Rate  . . . :   1675.4       1673.8  /sec
  Average Context-Switch Rate . :  13940.9      14894.7  /sec

The only thing that has changed here was the kernel image, as everything
else has remained constant.


> Another way to test a more complex scenario than just jackd running with
> an empty graph (assuming that jackd itself isn't to blame) while avoiding
> the risk of getting bad data due to insane clients would be to code up an
> example jackd client that does nothing but putting some load onto the
> jackd graph but in a strictly RT fashion (no blocking stuff whatsoever).
>
> Attached you probably find the most minimal jack client thinkable that
> does nothing but copy data from its input to its output port. Its only
> parameter is the time in seconds it will run (default 60). The jack client
> name is determined by the PID, so it can be started multiple times (jackd
> requires a unique name for each client).
>
> compile with
>
> g++ -o jack_test jack_test.cc -ljack
>
> This code can easily be adapted to produce more load (just do some math
> stuff with the data in the process callback).
>
> It seems jackd has a limitation to 14 clients atm (don't ask me why). The
> 15th kills jackd ;)
>

So true.

> Also i wanted to mention that a good share of ALSA drivers have issues,
> too, and aren't nessecarily suited to low latency audio work. I don't know
> how to rule these out except for using the ALSA dummy soundcard driver
> (which might have its own issues, but it's probably simple enough to work
> reliable. it just doesn't use any hw IRQ's so it's maybe not a good
> measure for what we want to test) or to use a soundcard with a proven good
> driver.
>

Of course, and the "reference" driver used on my tests is no exception
(snd-ali5451). But again, it's been kept the same on all tests, and the
improvement along the progression of the RT kernel development has been
outstanding nevertheless.

Cheers.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Yes, I know all this, and I warned before that this tests were
> strictly and specific to the hardware, jackd and fludisynth code which
> are intentionally kept the same along the several RT kernels that have
> been issued.

i'm aware of this - and i'm perfectly happy about all the testing that
is done, even if a latency problem in the end it turns out to be some
'side issue', an application or configuration bug. You are doing very
useful QA no matter what the problem turns out to be in the end.

when fixing stuff i tend to go from the simpler testcases to the more
complex ones (it's naturally simpler to validate the simpler ones), but
currently all the simple ones are working fine, so i'm looking at more
complex ones again.

Having said that, (not in small part based on the care you are taking to
keep your test environment consistent) it very much looks like as if the
latency problems you are reporting are related to -RT itself. It could
easily be something else, but as usual, there's only one way to find out
...

anyway, dont worry about presenting me with some problem that in the end
turns out to be something else. As Florian's jackd-xrun report from two
days ago has proven, jackd is still triggering genuine -RT bugs that
none of the simple workloads/apps are triggering. Less than one day
after half a dozen such latency bugs were fixed in the -RT patchset i
have no basis to go and blame Jackd or ALSA for any of the remaining
latency problems =B-)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> > It seems jackd has a limitation to 14 clients atm (don't ask me why). The
> > 15th kills jackd ;)
> >
> 
> So true.

is there any fix for that? Loading 14 jack_test clients only uses up
~33% of CPU time on my testbox. Or should it be possible for me to
trigger xruns using so many clients and 33% CPU utilization already? 

Could you perhaps try 14 instances of jack_test on your box, and see
whether you can generate similar xruns as you could generate with
fluidsynth? jack_test should certainly exclude as much jack-client side
complexity as possible.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Rui Nuno Capela]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

> Ingo Molnar
>
> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
>
>> > It seems jackd has a limitation to 14 clients atm (don't ask me why).
>> The
>> > 15th kills jackd ;)
>> >
>>
>> So true.
>
> is there any fix for that? Loading 14 jack_test clients only uses up
> ~33% of CPU time on my testbox. Or should it be possible for me to
> trigger xruns using so many clients and 33% CPU utilization already?
>
> Could you perhaps try 14 instances of jack_test on your box, and see
> whether you can generate similar xruns as you could generate with
> fluidsynth? jack_test should certainly exclude as much jack-client side
> complexity as possible.
>
> 	Ingo
>

OK. I tried 14 instances of jack_test. I even modded Florian's original
source code, to let each client instance have 4 ins and 4 outs, and to
make things a litle bit heavier, all 4 inputs are mixed into each of the 4
outputs.

Saw at least a couple of XRUNs in a 20 (4*5) minute test-run. CPU load
doesn't get above 30% on my laptop (P4/UP 2.533Ghz).

On attach you may find my "4-multiplex" version of jack_test(.cpp), along
with the jack_test3.sh shell script which has been used for my test runs.

There's also a modified version of nmeter(.c) that served the purpose to
log system performance counters (CPU usage, IRQs  and Context Switch
rate).

Bye.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: jack_test3.tar.gz --]
[-- Type: application/x-gzip-compressed, Size: 9298 bytes --]

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OK. I tried 14 instances of jack_test. I even modded Florian's
> original source code, to let each client instance have 4 ins and 4
> outs, and to make things a litle bit heavier, all 4 inputs are mixed
> into each of the 4 outputs.
> 
> Saw at least a couple of XRUNs in a 20 (4*5) minute test-run. CPU load
> doesn't get above 30% on my laptop (P4/UP 2.533Ghz).

the max CPU load i get here is 46% (your laptop is faster), but no
xruns. The result of a 5-minute run is:

 ************* SUMMARY RESULT ****************
 Timeout Count . . . . . . . . :(    0)
 XRUN Count  . . . . . . . . . :     0
 Delay Count (>spare time) . . :     0
 Delay Count (>1000 usecs) . . :     0
 Delay Maximum . . . . . . . . :     0   usecs
 Cycle Maximum . . . . . . . . :  1016   usecs
 Average DSP Load. . . . . . . :    46.4 %
 Average CPU System Load . . . :    40.5 %
 Average CPU User Load . . . . :     2.3 %
 Average CPU Nice Load . . . . :     0.0 %
 Average CPU I/O Wait Load . . :     0.0 %
 Average CPU IRQ Load  . . . . :     0.0 %
 Average CPU Soft-IRQ Load . . :     0.0 %
 Average Interrupt Rate  . . . :  2374.1 /sec
 Average Context-Switch Rate . : 19172.8 /sec

i suspect i need to activate some option/define in jackd to get some of
the more advanced stats such as delay-maximum?

the kernel i used was -30-6 and i used the snd-via82xx driver. (I had to
do -n3 instead of -n2 when starting up jackd - otherwise i'd get an
endless stream of very small xruns, apparently a via82xx driver bug?)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

> Saw at least a couple of XRUNs in a 20 (4*5) minute test-run. CPU load
> doesn't get above 30% on my laptop (P4/UP 2.533Ghz).

i'm wondering, do you get any xruns (or other bad behavior) if you use
the dummy ALSA driver for the latency test?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> the max CPU load i get here is 46% (your laptop is faster), but no
> xruns. The result of a 5-minute run is:
> 
>  ************* SUMMARY RESULT ****************
>  Timeout Count . . . . . . . . :(    0)
>  XRUN Count  . . . . . . . . . :     0
>  Delay Count (>spare time) . . :     0
>  Delay Count (>1000 usecs) . . :     0
>  Delay Maximum . . . . . . . . :     0   usecs
>  Cycle Maximum . . . . . . . . :  1016   usecs
>  Average DSP Load. . . . . . . :    46.4 %
>  Average CPU System Load . . . :    40.5 %
>  Average CPU User Load . . . . :     2.3 %
>  Average CPU Nice Load . . . . :     0.0 %
>  Average CPU I/O Wait Load . . :     0.0 %
>  Average CPU IRQ Load  . . . . :     0.0 %
>  Average CPU Soft-IRQ Load . . :     0.0 %
>  Average Interrupt Rate  . . . :  2374.1 /sec
>  Average Context-Switch Rate . : 19172.8 /sec

here are the settings i used:

 JACKD_PRIO=20
 JACKD_RATE=44100
 JACKD_PERIOD=64
 JACKD_SECONDS=300
 CLIENT_MAX=14

 jackd 0.99.0

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

another test on the same system, this time completely idle:

 ************* SUMMARY RESULT ****************
 Timeout Count . . . . . . . . :(    0)
 XRUN Count  . . . . . . . . . :     0
 Delay Count (>spare time) . . :     0
 Delay Count (>1000 usecs) . . :     0
 Delay Maximum . . . . . . . . :     0   usecs
 Cycle Maximum . . . . . . . . :   724   usecs
 Average DSP Load. . . . . . . :    32.1 %
 Average CPU System Load . . . :    30.8 %
 Average CPU User Load . . . . :     1.6 %
 Average CPU Nice Load . . . . :     0.0 %
 Average CPU I/O Wait Load . . :     0.0 %
 Average CPU IRQ Load  . . . . :     0.0 %
 Average CPU Soft-IRQ Load . . :     0.0 %
 Average Interrupt Rate  . . . :  1669.0 /sec
 Average Context-Switch Rate . : 16975.4 /sec
 *********************************************

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> another test on the same system, this time completely idle:

ah ... used the jack_test.cc client instead of your new jack_test.cpp.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > another test on the same system, this time completely idle:
> 
> ah ... used the jack_test.cc client instead of your new jack_test.cpp.

no big difference in the results though:

 ************* SUMMARY RESULT ****************
 Timeout Count . . . . . . . . :(    0)
 XRUN Count  . . . . . . . . . :     0
 Delay Count (>spare time) . . :     0
 Delay Count (>1000 usecs) . . :     0
 Delay Maximum . . . . . . . . :     0   usecs
 Cycle Maximum . . . . . . . . :   582   usecs
 Average DSP Load. . . . . . . :    30.7 %
 Average CPU System Load . . . :    22.1 %
 Average CPU User Load . . . . :     8.7 %
 Average CPU Nice Load . . . . :     0.0 %
 Average CPU I/O Wait Load . . :     0.1 %
 Average CPU IRQ Load  . . . . :     0.0 %
 Average CPU Soft-IRQ Load . . :     0.0 %
 Average Interrupt Rate  . . . :  1676.5 /sec
 Average Context-Switch Rate . : 17020.5 /sec
 *********************************************

(i turned off some debugging options in the kernel, hence the small
latency improvement.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Rui Nuno Capela]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

Ingo Molnar wrote:
>
> the max CPU load i get here is 46% (your laptop is faster), but no
> xruns. The result of a 5-minute run is:
>
>  ************* SUMMARY RESULT ****************
>  Timeout Count . . . . . . . . :(    0)
>  XRUN Count  . . . . . . . . . :     0
>  Delay Count (>spare time) . . :     0
>  Delay Count (>1000 usecs) . . :     0
>  Delay Maximum . . . . . . . . :     0   usecs
>  Cycle Maximum . . . . . . . . :  1016   usecs
>  Average DSP Load. . . . . . . :    46.4 %
>  Average CPU System Load . . . :    40.5 %
>  Average CPU User Load . . . . :     2.3 %
>  Average CPU Nice Load . . . . :     0.0 %
>  Average CPU I/O Wait Load . . :     0.0 %
>  Average CPU IRQ Load  . . . . :     0.0 %
>  Average CPU Soft-IRQ Load . . :     0.0 %
>  Average Interrupt Rate  . . . :  2374.1 /sec
>  Average Context-Switch Rate . : 19172.8 /sec
>
> i suspect i need to activate some option/define in jackd to get some of
> the more advanced stats such as delay-maximum?
>

Yes, there's a non-official patch to jackd from Lee Revell's. Without that
you don't get to read the maximum delay from jackd. Sorry. But if you have
the patience to rebuild jack, here comes attached the minimal patch for
just that.

Seeya.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: jack-0.99.10-max_delayed_usecs.patch.gz --]
[-- Type: application/x-gzip-compressed, Size: 1180 bytes --]

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Yes, there's a non-official patch to jackd from Lee Revell's. Without
> that you don't get to read the maximum delay from jackd. Sorry. But if
> you have the patience to rebuild jack, here comes attached the minimal
> patch for just that.

thx, it applied cleanly to jack-cvs. Here's the 5-minute idle-system
test again:

 ************* SUMMARY RESULT ****************
 Timeout Count . . . . . . . . :(    0)
 XRUN Count  . . . . . . . . . :     0
 Delay Count (>spare time) . . :     0
 Delay Count (>1000 usecs) . . :     0
 Delay Maximum . . . . . . . . :    28   usecs
 Cycle Maximum . . . . . . . . :   414   usecs
 Average DSP Load. . . . . . . :    20.6 %
 Average CPU System Load . . . :    11.6 %
 Average CPU User Load . . . . :     8.6 %
 Average CPU Nice Load . . . . :     0.0 %
 Average CPU I/O Wait Load . . :     0.0 %
 Average CPU IRQ Load  . . . . :     0.0 %
 Average CPU Soft-IRQ Load . . :     0.0 %
 Average Interrupt Rate  . . . :  1671.7 /sec
 Average Context-Switch Rate . : 17003.1 /sec
 *********************************************

but i can reproduce xruns on another, much slower box, using just 3-4
jack_test clients. The xruns dont seem to be justified, they happen at
30-40% CPU utilization already.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Rui Nuno Capela]

[-- Attachment #1: Type: text/plain, Size: 2823 bytes --]

Ingo Molnar wrote:
>
> * Rui Nuno Capela wrote:
>
>> Yes, there's a non-official patch to jackd from Lee Revell's. Without
>> that you don't get to read the maximum delay from jackd. Sorry. But if
>> you have the patience to rebuild jack, here comes attached the minimal
>> patch for just that.
>
> thx, it applied cleanly to jack-cvs. Here's the 5-minute idle-system test
> again:
>
>  ************* SUMMARY RESULT ****************
>  Timeout Count . . . . . . . . :(    0)
>  XRUN Count  . . . . . . . . . :     0
>  Delay Count (>spare time) . . :     0
>  Delay Count (>1000 usecs) . . :     0
>  Delay Maximum . . . . . . . . :    28   usecs
>  Cycle Maximum . . . . . . . . :   414   usecs
>  Average DSP Load. . . . . . . :    20.6 %
>  Average CPU System Load . . . :    11.6 %
>  Average CPU User Load . . . . :     8.6 %
>  Average CPU Nice Load . . . . :     0.0 %
>  Average CPU I/O Wait Load . . :     0.0 %
>  Average CPU IRQ Load  . . . . :     0.0 %
>  Average CPU Soft-IRQ Load . . :     0.0 %
>  Average Interrupt Rate  . . . :  1671.7 /sec
>  Average Context-Switch Rate . : 17003.1 /sec
>  *********************************************
>
> but i can reproduce xruns on another, much slower box, using just 3-4
> jack_test clients. The xruns dont seem to be justified, they happen at
> 30-40% CPU utilization already.
>

Now that you're liking, here goes a more contained jackd test-suite (see
attached tarball, jackd_test3.1.tar.gz).

Just launch the provided shell script, from the command line as:

  ./jack_test3_run.sh [secs] [clients] [ports]

where:

  secs    - number of seconds to run jackd workload (default = 300)
  clients - number of test-clients to run (default = 14)
  ports   - number of interface ports per client (default = 4)

As before, each client (jack_test3_client) registers the same number of
input and output ports (default is 4ins x 4outs), where each output is the
audio mix of all inputs.

Note that you can breakup the 14 client barrier, as that limit seems to be
related to the maximum number of client ports jackd can handle by default
(128). The jack_test_run.sh script sets this jackd maximum port limit
number as it sees fit, so any number of clients greater than 14 is
allowed, provided there's enough CPU and/or RAM ;)

Now, with the default workload (14 clients * 4 * 4 ports) I'm reaching 60%
of CPU, and a "fair" number of XRUNs on my P4@2.5G laptop, against the
on-board alsa driver (snd-ali5451), while under RT-V0.7.30-2.

Each test run produces a kernel-timestamped log filename with the complete
captured stdout/err. Consolidated results can be produced by feeding
several of those logfiles into the jack_test3_consolidated.awk script,
just like this:

   cat *.log | awk -f jack_test3_consolidated.awk


Enjoy.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: jack_test3.1.tar.gz --]
[-- Type: application/x-gzip-compressed, Size: 10364 bytes --]

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Now, with the default workload (14 clients * 4 * 4 ports) I'm reaching
> 60% of CPU, and a "fair" number of XRUNs on my P4@2.5G laptop, against
> the on-board alsa driver (snd-ali5451), while under RT-V0.7.30-2.

it would be very interesting to see how the new -30-9 kernel performs
using your workload (both fluidsynth and jackd_test), whether your xruns
are impacted by the fifo fix, and/or whether there are any other large
xrun sources left.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> OK. I tried 14 instances of jack_test. I even modded Florian's
> original source code, to let each client instance have 4 ins and 4
> outs, and to make things a litle bit heavier, all 4 inputs are mixed
> into each of the 4 outputs.

i tried your new test-client, and i've got a generic question: should a
jack client be able to generate an xrun via, other than via overloading
jackd? E.g. i'm wondering about the following behavior: if start up
jackd in the usual way:

  jackd -R -P20 -dalsa -dhw:0 -r44100 -p64 -n2 -S -P

and then i start a single test-client (jack_test.cpp):

 # ./jack_test
 seconds to run: 60
 client_new: jack_test-4215
 port_register
 set_process_callback
 activate
 running

and then if i now Ctrl-Z the Jack client, i get an immediate xrun
message from jackd:

  **** alsa_pcm: xrun of at least 2.119 msecs

and when i 'fg' the client again then jackd sees a big delay:

  **** alsa_pcm: xrun of at least 742.064 msecs

corresponding the amount of time i waited between the Ctrl-Z and the 
'fg'.

since the client runs as SCHED_OTHER, doesnt this mean that simple
delays between SCHED_OTHER tasks could cause xruns in jackd too? A
SCHED_OTHER task can be delayed indefinitely at any stage. So shouldnt
the test-clients have RT priority as well, to guarantee xrun-less jackd?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Florian Schmidt]

On Tue, 23 Nov 2004 15:46:22 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> i tried your new test-client, and i've got a generic question: should a
> jack client be able to generate an xrun via, other than via overloading
> jackd? E.g. i'm wondering about the following behavior: if start up
> jackd in the usual way:

The process() callback in a jackd client is run in a thread created by
libjack. This thread is run with SCHED_FIFO and at the same priority (or
one lower it seems) as the jackd server. Thus a client can only cause an
xrun when it takes a too long time to return from its process callback.

~$ ps  -C jackd -cmL
  PID   LWP CLS PRI TTY          TIME CMD
  975     - -     - ?        00:00:00 jackd
    -   975 TS   19 -        00:00:00 -
    -   976 TS   23 -        00:00:00 -
    -   977 FF  110 -        00:00:00 -
    -   978 FF  100 -        00:00:00 -

~$ ps -C jack_test -cmL
  PID   LWP CLS PRI TTY          TIME CMD
  988     - -     - pts/1    00:00:00 jack_test
    -   988 TS   20 -        00:00:00 -
    -   989 FF   99 -        00:00:00 -

So when you ctrl-z out of jack_test you cause its process() thread to be
suspended, too, thus jackd cannot finish processing its graph.

flo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> So when you ctrl-z out of jack_test you cause its process() thread to
> be suspended, too, thus jackd cannot finish processing its graph.

ok, that makes sense. So under what priority does most of the
CPU-intense processing run in the test client? SCHED_OTHER or
SCHED_FIFO?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> ~$ ps -C jack_test -cmL
>   PID   LWP CLS PRI TTY          TIME CMD
>   988     - -     - pts/1    00:00:00 jack_test
>     -   988 TS   20 -        00:00:00 -
>     -   989 FF   99 -        00:00:00 -
> 
> So when you ctrl-z out of jack_test you cause its process() thread to
> be suspended, too, thus jackd cannot finish processing its graph.

so in theory any scheduling delay of PID 988 in the above setup (the
SCHED_OTHER task) should not be able to negatively influence jackd,
correct? In fact, does in this particular jack_test case PID 988 do
anything substantial?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Florian Schmidt]

On Tue, 23 Nov 2004 16:21:26 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Florian Schmidt <mista.tapas@gmx.net> wrote:
> 
> > ~$ ps -C jack_test -cmL
> >   PID   LWP CLS PRI TTY          TIME CMD
> >   988     - -     - pts/1    00:00:00 jack_test
> >     -   988 TS   20 -        00:00:00 -
> >     -   989 FF   99 -        00:00:00 -
> > 
> > So when you ctrl-z out of jack_test you cause its process() thread to
> > be suspended, too, thus jackd cannot finish processing its graph.
> 
> so in theory any scheduling delay of PID 988 in the above setup (the
> SCHED_OTHER task) should not be able to negatively influence jackd,
> correct? 

correct

> In fact, does in this particular jack_test case PID 988 do
> anything substantial?

Well, it registers the client with jackd, sets up the ports, registers
the process() callback and then simply goes to sleep() for the desired
runtime of the program. All these are non RT ops and should never be
able to cause any xruns.

All the work is done by the process() callback which is called by
libjack in a SCHED_FIFO thread. The process() callback is called once
for each buffer that jackd processes.

I cannot explain the detailed mechanism of how jackd wakes its clients
and communicates with them myself too well, so i'll leave this to Paul
Davis (CC'ed). Care to elaborate, Paul?

flo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Paul Davis]

>On Tue, 23 Nov 2004 16:21:26 +0100
>Ingo Molnar <mingo@elte.hu> wrote:
>
>> 
>> * Florian Schmidt <mista.tapas@gmx.net> wrote:
>> 
>> > ~$ ps -C jack_test -cmL
>> >   PID   LWP CLS PRI TTY          TIME CMD
>> >   988     - -     - pts/1    00:00:00  jack_test
>> >     -   988 TS   20 -        00:00:00 -
>> >     -   989 FF   99 -        00:00:00 -
>> > 
>> > So when you ctrl-z out of jack_test you cause its process() thread to
>> > be suspended, too, thus jackd cannot finish processing its graph.
>> 
>> so in theory any scheduling delay of PID 988 in the above setup (the
>> SCHED_OTHER task) should not be able to negatively influence jackd,
>> correct? 
>
>correct
>
>> In fact, does in this particular jack_test case PID 988 do
>> anything substantial?
>
>Well, it registers the client with jackd, sets up the ports, registers
>the process() callback and then simply goes to sleep() for the desired
>runtime of the program. All these are non RT ops and should never be
>able to cause any xruns.
>
>All the work is done by the process() callback which is called by
>libjack in a SCHED_FIFO thread. The process() callback is called once
>for each buffer that jackd processes.
>
>I cannot explain the detailed mechanism of how jackd wakes its clients
>and communicates with them myself too well, so i'll leave this to Paul
>Davis (CC'ed). Care to elaborate, Paul?

sorry for the delay on this, it ended up in a mail folder that i don't
check very often.

jackd wakes clients by writing a single byte to a FIFO. the first
client is sleeping on the other side of the FIFO. when that client is
done, it writes a single byte to another FIFO. either another client
is sleeping on the other side of that FIFO, or if there are no other
clients, jackd is waiting for the client there.

the "chain" of wakeup FIFOs is rearranged every time the graph
execution order is modified (e.g. by new connections being established
between clients that requires a different execution order).

the chain is executed every time jackd is woken by its "backend"
client, typically the ALSA client that waits on the fd's corresponding
to an audio interface to be readable and/or writable. in other words,
every interrupt from the audio interface.

we know that writes to FIFOs are not really RT-safe, but they are the
closest thing linux has at this time. i have outlined an idea to ingo
that florian and i cooked up one evening on IRC that would provide
true RT-safe IPC mechanisms, but as i recall, he didn't seem to think
that much of it :)

--p

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> we know that writes to FIFOs are not really RT-safe, [...]

in kernels -V0.7.30-9 or later they are RT-safe when PREEMPT_RT is
enabled.

also, the problem is that jackd uses _named_ fifos, which are tied to
the raw FS and might trigger journalling activities. Normal pipes
(unnamed fifos) would not cause such problems. Would it be possible to
change jackd to use a pair of pipes, instead of a fifo?

> [...] i have outlined an idea to ingo that florian and i cooked up one
> evening on IRC that would provide true RT-safe IPC mechanisms, but as
> i recall, he didn't seem to think that much of it :)

actually, my answer (sent on Nov 1) was:

> futexes are nearly lock-free. [and even those locks are short-held so
> combined with priority-inheritance they should be lockfree in
> essence.] Would futexes suit your purposes?

to which suggestion i got no reply yet :-)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Paul Davis]

>in kernels -V0.7.30-9 or later they are RT-safe when PREEMPT_RT is
>enabled.

thats good to hear.

>also, the problem is that jackd uses _named_ fifos, which are tied to
>the raw FS and might trigger journalling activities. Normal pipes
>(unnamed fifos) would not cause such problems. Would it be possible to
>change jackd to use a pair of pipes, instead of a fifo?

i.e. pipe(2) rather than mkfifo(2) ?

it would be a complete pain because the pipes have to be
"discoverable" across processes. we would have to do fd passing, which
is still really quite ugly in linux (and other *nix systems). it
would quite difficult, though not impossible.

>> [...] i have outlined an idea to ingo that florian and i cooked up one
>> evening on IRC that would provide true RT-safe IPC mechanisms, but as
>> i recall, he didn't seem to think that much of it :)
>
>actually, my answer (sent on Nov 1) was:
>
>> futexes are nearly lock-free. [and even those locks are short-held so
>> combined with priority-inheritance they should be lockfree in
>> essence.] Would futexes suit your purposes?
>
>to which suggestion i got no reply yet :-)

i am still trying to find the time to investigate futexes. they seem
close to the desired object, but have a slightly more general semantic
than i can fit into my head right now;)

--p

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> >also, the problem is that jackd uses _named_ fifos, which are tied to
> >the raw FS and might trigger journalling activities. Normal pipes
> >(unnamed fifos) would not cause such problems. Would it be possible to
> >change jackd to use a pair of pipes, instead of a fifo?
> 
> i.e. pipe(2) rather than mkfifo(2) ?
> 
> it would be a complete pain because the pipes have to be
> "discoverable" across processes. we would have to do fd passing, which
> is still really quite ugly in linux (and other *nix systems). it would
> quite difficult, though not impossible.

yeah. And i think mkfifo(2) objects ought to behave atomically as well,
it's an unfortunate side-effect of atime/mtime inode semantics that they
can block.

your point is correct, the best way to have a system-wide namespace for
synchronization objects is ... the filesystem hierarchy. If you create a
unix domain socket then you can distribute your pipe fds, but that's
indeed somewhat painful.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Paul Davis]

>your point is correct, the best way to have a system-wide namespace for
>synchronization objects is ... the filesystem hierarchy. If you create a
>unix domain socket then you can distribute your pipe fds, but that's
>indeed somewhat painful.

this is where Mach ports come in. they were designed to be passed
around from process to process, painlessly, but without any system
wide namespace. you can create ports that can be looked up by anyone,
but not all ports are required meet this condition. this makes it easy
to set up a private communication channel between two processes.

a bit like futexes :)

--p

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Esben Nielsen]

On Wed, 1 Dec 2004, Ingo Molnar wrote:

> 
> * Paul Davis <paul@linuxaudiosystems.com> wrote:
> 
> > >also, the problem is that jackd uses _named_ fifos, which are tied to
> > >the raw FS and might trigger journalling activities. Normal pipes
> > >(unnamed fifos) would not cause such problems. Would it be possible to
> > >change jackd to use a pair of pipes, instead of a fifo?
> > 
> > i.e. pipe(2) rather than mkfifo(2) ?
> > 
> > it would be a complete pain because the pipes have to be
> > "discoverable" across processes. we would have to do fd passing, which
> > is still really quite ugly in linux (and other *nix systems). it would
> > quite difficult, though not impossible.
> 
> yeah. And i think mkfifo(2) objects ought to behave atomically as well,
> it's an unfortunate side-effect of atime/mtime inode semantics that they
> can block.
> 
> your point is correct, the best way to have a system-wide namespace for
> synchronization objects is ... the filesystem hierarchy. If you create a
> unix domain socket then you can distribute your pipe fds, but that's
> indeed somewhat painful.
> 

Don't worry about setting up the stuff. Once you have the pipe it ought to
be RT in the usage of read/write,  but setting it up is something you is
something you do "under boot", just as allocating memory and other
non-real-time stuff. 


> 	Ingo
> 

Esben

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Esben Nielsen <simlo@phys.au.dk> wrote:

> Don't worry about setting up the stuff. Once you have the pipe it
> ought to be RT in the usage of read/write, but setting it up is
> something you is something you do "under boot", just as allocating
> memory and other non-real-time stuff. 

actually, the main problem with fifos was they were _not_ atomic even in
read/write (i myself fully expected them to be that, but they arent). 
That's the bug/misfeature that i fixed in the latest kernels.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Chris Friesen]

Ingo Molnar wrote:

> actually, the main problem with fifos was they were _not_ atomic even in
> read/write (i myself fully expected them to be that, but they arent). 
> That's the bug/misfeature that i fixed in the latest kernels.

Whoa.  pipes (ie from the pipe() call) don't read/write atomicly? Even if you 
write less than a page?

When was this fixed?

Chris

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Esben Nielsen]

On Wed, 1 Dec 2004, Paul Davis wrote:

> [...]
> >
> >> futexes are nearly lock-free. [and even those locks are short-held so
> >> combined with priority-inheritance they should be lockfree in
> >> essence.] Would futexes suit your purposes?
> >
> >to which suggestion i got no reply yet :-)
> 
> i am still trying to find the time to investigate futexes. they seem
> close to the desired object, but have a slightly more general semantic
> than i can fit into my head right now;)
>

I looked into them just to see if they could be used for user-space
priority inheritance. I saw that it takes (read-)lock on mmap_sem.
Isn't that potentially held for a long time? I don't know how the memory
subsystem works at all but my guess is that any changing of the process's
memory is done with mmpa_sem locked. The only way to prevent that is 1)
mlockall() and 2) stop increasing your heap.

I.e. you can't have one thread running real-time using a futex while
another runs non-real-time allocating memory in the same process.

Am I correct?

Another problem is the hashing of the futex. That inherently has a
non-deterministic timing behaviour. The more applications use futex'es
the slower this hashing gets :-( The slowdown might in practise be very
low because looking up in the global hash table can't take very many us!

Can't it just use a file-descriptor in the system call to look up the
futex instead of the hashing? That is RT-O(1), right? But, ofcourse, then
it is hard having a futex in shared mem.

 
> --p

Esben

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

> since the client runs as SCHED_OTHER, doesnt this mean that simple
> delays between SCHED_OTHER tasks could cause xruns in jackd too? A
> SCHED_OTHER task can be delayed indefinitely at any stage. So shouldnt
> the test-clients have RT priority as well, to guarantee xrun-less
> jackd?

if SCHED_OTHER is the goal, then the xruns you are seeing in the
fluidsynth test are likely the result of random fluctuations in
scheduling of SCHED_OTHER tasks.

The way to find out whether that's the main source of xruns would be the
following test: start each fluidsynth instance as "nice -19 fluidsynth
..." to renice it to +19. Nice +19 gives the smoothest timeslices to
CPU-bound tasks (such as fluidsynth), and should give the smallest
global fluctuations. (The system must be idle during the test of course,
a nice +19 task is easily preempted.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Eran Mann]

[-- Attachment #1: Type: text/plain, Size: 1833 bytes --]

Ingo Molnar wrote:
> i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
> downloaded from the usual place:
> 
> 	http://redhat.com/~mingo/realtime-preempt/

Hi,
I´m seeing latencies of up to ~2000 microseconds. see attached traces
file for a small sample. I think I´m missing something obvious
config-wise but I don´t know what...
The ´load´ during the traces consisted of a kernel build in a
gnome-terminal, and 2 browser windows with a heavy site (Flash ads etc.)
in each. This load causes a >1 ms latency every 5 minutes on average.
After the kernel build ended the rate dropped dramatically to ~2 traces
an hour.
The traces were from V-0.7.29-5 but I´ve seen these latencies in all RT
kernels I tested (2.6.9-mm1-RT-V0.2 was the first). I´ll try V0.7.30-2 next.
The machine is a PIII 733 Mhz, 256MB RAM, IDE disks.
I attached the traces and a partial .config.
Relevant items in /proc/sys/kernel:

/proc/sys/kernel/kernel_preemption :
1
...
/proc/sys/kernel/osrelease :
2.6.10-rc2-mm2-V0.7.29-5
...
/proc/sys/kernel/panic :
0
/proc/sys/kernel/panic_on_oops :
0
/proc/sys/kernel/pid_max :
32768
/proc/sys/kernel/preempt_max_latency :
1737
/proc/sys/kernel/preempt_thresh :
1000
/proc/sys/kernel/preempt_wakeup_timing :
1
...
/proc/sys/kernel/real-root-dev :
0
/proc/sys/kernel/sem :
250     32000   32      128
/proc/sys/kernel/shmall :
2097152
/proc/sys/kernel/shmmax :
33554432
/proc/sys/kernel/shmmni :
4096
/proc/sys/kernel/sysrq :
1
/proc/sys/kernel/tainted :
0
/proc/sys/kernel/threads-max :
4095
/proc/sys/kernel/trace_enabled :
1
/proc/sys/kernel/trace_freerunning :
0
/proc/sys/kernel/trace_print_at_crash :
1
/proc/sys/kernel/trace_user_triggered :
0
/proc/sys/kernel/trace_verbose :
0
/proc/sys/kernel/unknown_nmi_panic :
0

Let me know if some more information is required.
Thank,
      Eran Mann


[-- Attachment #2: config --]
[-- Type: text/plain, Size: 2988 bytes --]

# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
CONFIG_MPENTIUMIII=y
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
# CONFIG_SMP is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
# CONFIG_X86_MCE_P4THERMAL is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=y
......
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=m
CONFIG_RTC_HISTOGRAM=m
# CONFIG_GEN_RTC is not set
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
......
#
# Profiling support
#
CONFIG_PROFILING=y
CONFIG_OPROFILE=m

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
CONFIG_WAKEUP_TIMING=y
CONFIG_CRITICAL_PREEMPT_TIMING=y
CONFIG_CRITICAL_IRQSOFF_TIMING=y
CONFIG_CRITICAL_TIMING=y
CONFIG_LATENCY_TIMING=y
CONFIG_LATENCY_TRACE=y
CONFIG_MCOUNT=y
CONFIG_RT_DEADLOCK_DETECT=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
# CONFIG_KGDB is not set
.....
#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y


[-- Attachment #3: traces --]
[-- Type: text/plain, Size: 14074 bytes --]

Nov 21 19:22:42 eran kernel: (multiload-apple/5506/CPU#0): 1694 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1694 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: multiload-apple/5506, uid:500 nice:0 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
  131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): (115) ((116))
  131 88000005 0.000ms (+0.000ms): (5506) ((131))
  131 88000005 0.001ms (+0.000ms): try_to_wake_up (wake_up_process)
  131 88000005 0.001ms (+0.000ms): (0) ((1))
  131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
  131 88000003 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
  131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
  131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
  131 88000002 0.003ms (+1.687ms): dequeue_task (deactivate_task)
 5506 80000002 1.691ms (+0.001ms): __switch_to (__schedule)
 5506 80000002 1.692ms (+0.000ms): (131) ((5506))
 5506 80000002 1.693ms (+0.000ms): (116) ((115))
 5506 80000002 1.693ms (+0.000ms): finish_task_switch (__schedule)
 5506 80000001 1.693ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
 5506 80000001 1.693ms (+0.005ms): (5506) ((115))
 5506 80000001 1.699ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================
Nov 21 19:30:58 eran kernel: (firefox-bin/14054/CPU#0): 1991 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1991 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: firefox-bin/14054, uid:500 nice:0 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
  131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): (115) ((116))
  131 88000005 0.000ms (+0.000ms): <000036e6> ((131))
  131 88000005 0.000ms (+0.000ms): try_to_wake_up (wake_up_process)
  131 88000005 0.001ms (+0.000ms): (0) ((1))
  131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
  131 88000003 0.001ms (+0.000ms): preempt_schedule (__up_mutex)
  131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
  131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
  131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
  131 88000002 0.003ms (+1.986ms): dequeue_task (deactivate_task)
14054 80000002 1.989ms (+0.000ms): __switch_to (__schedule)
14054 80000002 1.989ms (+0.000ms): (131) (<000036e6>)
14054 80000002 1.989ms (+0.000ms): (116) ((115))
14054 80000002 1.990ms (+0.000ms): finish_task_switch (__schedule)
14054 80000001 1.990ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
14054 80000001 1.990ms (+0.006ms): <000036e6> ((115))
14054 80000001 1.996ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================
Nov 21 19:31:44 eran kernel: (IRQ 0/2/CPU#0): 1213 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1213 us, entries: 23 (23)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: IRQ 0/2, uid:0 nice:0 policy:1 rt_prio:49
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
    3 88010003 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
    3 88010002 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
    3 88010002 0.000ms (+0.000ms): (50) ((105))
    3 88010002 0.000ms (+0.000ms): (2) ((3))
    3 88010002 0.000ms (+0.000ms): try_to_wake_up (wake_up_process)
    3 88010002 0.001ms (+0.000ms): (0) ((1))
    3 88010001 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
    3 88010001 0.001ms (+0.000ms): wake_up_process (redirect_hardirq)
    3 88010000 0.001ms (+0.000ms): preempt_schedule (__do_IRQ)
    3 88010000 0.002ms (+0.000ms): irq_exit (do_IRQ)
    3 88000001 0.002ms (+0.000ms): do_softirq (irq_exit)
    3 88000001 0.002ms (+0.000ms): __do_softirq (do_softirq)
    3 88000000 0.002ms (+0.000ms): preempt_schedule_irq (need_resched)
    3 98000000 0.003ms (+0.000ms): __schedule (preempt_schedule_irq)
    3 98000000 0.003ms (+0.000ms): profile_hit (__schedule)
    3 98000001 0.003ms (+0.000ms): sched_clock (__schedule)
    2 80000002 0.004ms (+1.207ms): __switch_to (__schedule)
    2 80000002 1.212ms (+0.000ms): (3) ((2))
    2 80000002 1.212ms (+0.000ms): (105) ((50))
    2 80000002 1.212ms (+0.000ms): finish_task_switch (__schedule)
    2 80000001 1.212ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
    2 80000001 1.212ms (+0.006ms): (2) ((50))
    2 80000001 1.219ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================
Nov 21 19:36:18 eran kernel: (firefox-bin/14054/CPU#0): 1881 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1881 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: firefox-bin/14054, uid:500 nice:0 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
  131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): (115) ((116))
  131 88000005 0.000ms (+0.000ms): <000036e6> ((131))
  131 88000005 0.001ms (+0.000ms): try_to_wake_up (wake_up_process)
  131 88000005 0.001ms (+0.000ms): (0) ((1))
  131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
  131 88000003 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
  131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
  131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
  131 88000002 0.003ms (+1.875ms): dequeue_task (deactivate_task)
14054 80000002 1.878ms (+0.001ms): __switch_to (__schedule)
14054 80000002 1.880ms (+0.000ms): (131) (<000036e6>)
14054 80000002 1.880ms (+0.000ms): (116) ((115))
14054 80000002 1.880ms (+0.000ms): finish_task_switch (__schedule)
14054 80000001 1.880ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
14054 80000001 1.881ms (+0.008ms): <000036e6> ((115))
14054 80000001 1.890ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================
Nov 21 19:37:18 eran kernel: (firefox-bin/14054/CPU#0): 1347 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1347 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: firefox-bin/14054, uid:500 nice:0 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
  131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): (115) ((116))
  131 88000005 0.000ms (+0.000ms): <000036e6> ((131))
  131 88000005 0.000ms (+0.000ms): try_to_wake_up (wake_up_process)
  131 88000005 0.001ms (+0.000ms): (0) ((1))
  131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
  131 88000003 0.001ms (+0.000ms): preempt_schedule (__up_mutex)
  131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
  131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
  131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
  131 88000002 0.003ms (+0.000ms): dequeue_task (deactivate_task)
14054 80000002 0.003ms (+0.000ms): __switch_to (__schedule)
14054 80000002 0.004ms (+0.000ms): (131) (<000036e6>)
14054 80000002 0.004ms (+0.000ms): (116) ((115))
14054 80000002 0.004ms (+0.000ms): finish_task_switch (__schedule)
14054 80000001 0.005ms (+1.342ms): trace_stop_sched_switched (finish_task_switch)
14054 80000001 1.347ms (+0.103ms): <000036e6> ((115))
14054 80000001 1.450ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================
Nov 21 19:49:21 eran kernel: (gnome-terminal/5434/CPU#0): 1683 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 1683 us, entries: 21 (21)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: gnome-terminal/5434, uid:500 nice:0 policy:0 rt_prio:0
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
  131 88000006 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000005 0.000ms (+0.000ms): (115) ((116))
  131 88000005 0.001ms (+0.000ms): (5434) ((131))
  131 88000005 0.001ms (+0.000ms): try_to_wake_up (wake_up_process)
  131 88000005 0.001ms (+0.000ms): (0) ((1))
  131 88000004 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
  131 88000004 0.001ms (+0.000ms): wake_up_process (__up_mutex)
  131 88000003 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 88000002 0.002ms (+0.000ms): preempt_schedule (__up_mutex)
  131 08000001 0.002ms (+0.000ms): preempt_schedule (__schedule)
  131 08000001 0.002ms (+0.000ms): sched_clock (__schedule)
  131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
  131 88000002 0.003ms (+1.676ms): dequeue_task (deactivate_task)
 5434 80000002 1.680ms (+0.001ms): __switch_to (__schedule)
 5434 80000002 1.681ms (+0.000ms): (131) ((5434))
 5434 80000002 1.682ms (+0.000ms): (116) ((115))
 5434 80000002 1.682ms (+0.000ms): finish_task_switch (__schedule)
 5434 80000001 1.682ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
 5434 80000001 1.682ms (+0.006ms): (5434) ((115))
 5434 80000001 1.689ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
==================
Nov 21 19:52:18 eran kernel: (IRQ 0/2/CPU#0): 2009 us wakeup latency violates 1000 us threshold.
preemption latency trace v1.0.7 on 2.6.10-rc2-mm2-V0.7.29-5
-------------------------------------------------------
 latency: 2009 us, entries: 37 (37)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: IRQ 0/2, uid:0 nice:0 policy:1 rt_prio:49
    -----------------
 => started at: try_to_wake_up+0x165/0x1d0 <c0117af5>
 => ended at:   finish_task_switch+0x4f/0xc0 <c0117fbf>
=======>
  783 88010003 0.000ms (+0.000ms): __trace_start_sched_wakeup (try_to_wake_up)
  783 88010002 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  783 88010002 0.000ms (+0.000ms): (50) ((54))
  783 88010002 0.000ms (+0.000ms): (2) ((783))
  783 88010002 0.000ms (+0.000ms): try_to_wake_up (wake_up_process)
  783 88010002 0.001ms (+0.000ms): (0) ((1))
  783 88010001 0.001ms (+0.000ms): preempt_schedule (try_to_wake_up)
  783 88010001 0.001ms (+0.000ms): wake_up_process (redirect_hardirq)
  783 88010000 0.001ms (+0.000ms): preempt_schedule (__do_IRQ)
  783 88010000 0.002ms (+0.000ms): irq_exit (do_IRQ)
  783 88000001 0.002ms (+0.000ms): do_softirq (irq_exit)
  783 88000001 0.002ms (+0.000ms): __do_softirq (do_softirq)
  783 88000001 0.003ms (+0.000ms): wake_up_process (do_softirq)
  783 88000001 0.003ms (+0.000ms): try_to_wake_up (wake_up_process)
  783 88000001 0.003ms (+0.000ms): task_rq_lock (try_to_wake_up)
  783 88000002 0.003ms (+0.000ms): activate_task (try_to_wake_up)
  783 88000002 0.004ms (+0.000ms): sched_clock (activate_task)
  783 88000002 0.004ms (+0.000ms): recalc_task_prio (activate_task)
  783 88000002 0.004ms (+0.000ms): effective_prio (recalc_task_prio)
  783 88000002 0.004ms (+0.000ms): enqueue_task (activate_task)
  783 88000002 0.005ms (+0.000ms): (105) ((54))
  783 88000002 0.005ms (+0.000ms): (3) ((783))
  783 88000002 0.005ms (+0.000ms): try_to_wake_up (wake_up_process)
  783 88000002 0.006ms (+1.999ms): (0) ((1))
  783 88000001 2.005ms (+0.000ms): preempt_schedule (try_to_wake_up)
  783 88000001 2.005ms (+0.000ms): wake_up_process (do_softirq)
  783 88000000 2.006ms (+0.000ms): preempt_schedule_irq (need_resched)
  783 98000000 2.006ms (+0.000ms): __schedule (preempt_schedule_irq)
  783 98000000 2.006ms (+0.000ms): profile_hit (__schedule)
  783 98000001 2.007ms (+0.000ms): sched_clock (__schedule)
    2 80000002 2.007ms (+0.000ms): __switch_to (__schedule)
    2 80000002 2.008ms (+0.000ms): (783) ((2))
    2 80000002 2.008ms (+0.000ms): (54) ((50))
    2 80000002 2.008ms (+0.000ms): finish_task_switch (__schedule)
    2 80000001 2.008ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
    2 80000001 2.009ms (+0.006ms): (2) ((50))
    2 80000001 2.015ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Eran Mann <emann@mrv.com> wrote:

> Ingo Molnar wrote:
> >i have released the -V0.7.30-2 Real-Time Preemption patch, which can be
> >downloaded from the usual place:
> >
> >	http://redhat.com/~mingo/realtime-preempt/
> 
> Hi,
> I´m seeing latencies of up to ~2000 microseconds. see attached traces
> file for a small sample. I think I´m missing something obvious
> config-wise but I don´t know what...

  131 88000002 0.003ms (+0.000ms): deactivate_task (__schedule)
  131 88000002 0.003ms (+1.687ms): dequeue_task (deactivate_task)
 5506 80000002 1.691ms (+0.001ms): __switch_to (__schedule)

this seems to be hardware-generated. As you can see it from the trace,
the codepath between __schedule()'s deactive_task() and __switch_to()
has all interrupts and preemption disabled. The O(1) scheduler there has
constant overhead and that codepath should at most take ~1 usec.

(there is one exception, if both LATENCY_TRACING and RT_DEADLOCK_DETECT
are enabled in -V0.7.30-2 and later kernels then the overhead within
__schedule is O(nr_running), because the tracer adds entries for every
runnable task. But this is not the case for your trace because then
you'd see those entries in /proc/latency_trace.)

> The ´load´ during the traces consisted of a kernel build in a
> gnome-terminal, and 2 browser windows with a heavy site (Flash ads
> etc.) in each. This load causes a >1 ms latency every 5 minutes on
> average. After the kernel build ended the rate dropped dramatically to
> ~2 traces an hour.

this seems to imply IDE DMA related hardware overhead. Apparently what
happens is that with certain motherboards/chipsets, if IDE DMA happens
then that DMA transfer _completely locks up_ the system bus. Nothing 
happens, and the CPU is stalled in essence until the end of the DMA 
request.

there's nothing the kernel can do about a hardware latency like that,
but you can try to work it around. Mark H. Johnson has reported up to
500 usec latencies that had a similar pattern as yours, and he has
experimented with lesser DMA modes (udma2?) via hdparm. YMMV and be
careful with hdparm settings.

> The traces were from V-0.7.29-5 but I´ve seen these latencies in all
> RT kernels I tested (2.6.9-mm1-RT-V0.2 was the first). I´ll try
> V0.7.30-2 next. The machine is a PIII 733 Mhz, 256MB RAM, IDE disks.

this very strongly implies some sort of hardware overhead. Btw., the
likely reason why this often shows up within __schedule() is that 1)
it's a very common operation, especially on the -RT kernel 2) we do a
TLB flush there, which can be quite memory-intense, so if the system bus
(the memory bus) is locked up, there is a high likelyhood that this
function generates a cachemiss.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Eran Mann]

Ingo Molnar wrote:
> * Eran Mann <emann@mrv.com> wrote:
>>
>>Hi,
>>I´m seeing latencies of up to ~2000 microseconds. see attached traces
>>file for a small sample. I think I´m missing something obvious
>>config-wise but I don´t know what...
...

> this seems to imply IDE DMA related hardware overhead. Apparently what
> happens is that with certain motherboards/chipsets, if IDE DMA happens
> then that DMA transfer _completely locks up_ the system bus. Nothing 
> happens, and the CPU is stalled in essence until the end of the DMA 
> request.
> 
....
> 	Ingo

Right on.
After hdparm -d0 I see maximum latency of 35 us after a full kernel 
build with a few GUI apps in the background. I´ll try to find a 
reasonable compromise.
-- 
Eran Mann

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Ingo Molnar]

* Eran Mann <emann@mrv.com> wrote:

> >>I´m seeing latencies of up to ~2000 microseconds. see attached traces
> >>file for a small sample. I think I´m missing something obvious
> >>config-wise but I don´t know what...
> ...
> 
> >this seems to imply IDE DMA related hardware overhead. Apparently what
> >happens is that with certain motherboards/chipsets, if IDE DMA happens
> >then that DMA transfer _completely locks up_ the system bus. Nothing 
> >happens, and the CPU is stalled in essence until the end of the DMA 
> >request.

> Right on.
> After hdparm -d0 I see maximum latency of 35 us after a full kernel 
> build with a few GUI apps in the background. I´ll try to find a 
> reasonable compromise.

it might make sense to report this to the hw vendor as well, as these
latencies dont occur at _every_ IDE DMA, it might be some sort of
chipset (or BIOS) bug they might want to see resolved as well (if this
isnt a ship-and-forget vendor). 2 msec stalls are not nice to a fair
number of applications.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.10-rc2-mm2-V0.7.30-2

[Eran Mann]

Ingo Molnar wrote:
> * Eran Mann <emann@mrv.com> wrote:
> 
>>Right on.
>>After hdparm -d0 I see maximum latency of 35 us after a full kernel 
>>build with a few GUI apps in the background. I´ll try to find a 
>>reasonable compromise.
> 
> it might make sense to report this to the hw vendor as well, as these
> latencies dont occur at _every_ IDE DMA, it might be some sort of
> chipset (or BIOS) bug they might want to see resolved as well (if this
> isnt a ship-and-forget vendor). 2 msec stalls are not nice to a fair
> number of applications.
> 
> 	Ingo

It´s a rather old white-box machine with a noname VIA-based motherboard, 
So I don´t really have whom to report the problem to. On the other hand 
with udma2 I see latencies < 170 us which seems reasonable.
Thanks for the advice.
   Eran.