* Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify
@ 2015-12-08 15:20 Dietmar Putz
2015-12-08 17:01 ` Eric Sandeen
2015-12-08 20:57 ` Dave Chinner
0 siblings, 2 replies; 5+ messages in thread
From: Dietmar Putz @ 2015-12-08 15:20 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 83403 bytes --]
Hello all,
recently we had some updates in a gluster environment as described in :
https://www.gluster.org/pipermail/gluster-users/2015-December/024509.html
since then we are faced to the below show errors.
otherwise as stated in the report on gluster.org not always the same
nodes of the cluster are affected but those who are currently being used
by a gluster-client. all clients running ubuntu 12.04.5 or 14.04.3 with
3.13.0-67 or ~71 and using the same glusterfs-client version as
installed on the glusterfs server (gfs 3.6.7).
we made several tests with different kernel version like 3.13.0-24,
3.13.0-67 and 3.13.0-71. in any case we received error messages as shown
below for gluster-ger-ber-12-int.
After upgrade to kernel 3.19.0-39 the error message turns from "Internal
error xfs_attr3_leaf_write_verify...." to "Metadata corruption detected
at xfs_attr3_leaf_write_verify..." as shown below for
gluster-ger-ber-11-int.
in any case the behavior is the same...the file system becomes
inaccessible and returns an Input/output error. xfs_check does not show
any information and xfs_repair does not show any error. it is sufficient
to remount the file system and it is accessible again.
similar reports are existing :
http://oss.sgi.com/archives/xfs/2015-11/msg00459.html
https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1468039
we decided to downgrade the kernel on the glusterfs-server nodes to
3.8.4-030804-generic #201303201832 SMP Wed Mar 20 22:33:00 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
since then the error disappeared.
for both below shown errors a trace output is available (trace-cmd
record -e xfs\*).
best regards
dietmar
gluster-ger-ber-11-int : (kernel 3.19.0-39)
[56909.996801] XFS (sdc1): xfs_log_force: error -5 returned.
[56910.019110] XFS (sdc1): xfs_log_force: error -5 returned.
[56934.013585] XFS (sdc1): Mounting V4 Filesystem
[56934.149640] XFS (sdc1): Starting recovery (logdev: internal)
[56934.577267] XFS (sdc1): Ending recovery (logdev: internal)
[58674.887753] XFS (sdc1): Mounting V4 Filesystem
[58675.000355] XFS (sdc1): Ending clean mount
[58724.547431] init: statd main process (1490) killed by KILL signal
[58724.547440] init: statd main process ended, respawning
[65828.900368] bash (23057): drop_caches: 3
[67475.972089] nr_pdflush_threads exported in /proc is scheduled for removal
[73715.250762] XFS (sdc1): Metadata corruption detected at
xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x44482d730
[73715.250803] XFS (sdc1): Unmount and run xfs_repair
[73715.250816] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
[73715.250832] ffff8807f7b07000: 00 00 00 00 00 00 00 00 fb ee 00 00 00
00 00 00 ................
[73715.250856] ffff8807f7b07010: 10 00 00 00 00 20 0f e0 00 00 00 00 00
00 00 00 ..... ..........
[73715.250876] ffff8807f7b07020: 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
[73715.250907] ffff8807f7b07030: 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
[73715.250941] XFS (sdc1): xfs_do_force_shutdown(0x8) called from line
1249 of file
/build/linux-lts-vivid-1jarlV/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c.
Return address = 0xffffffffc02bbd22
[73715.251375] XFS (sdc1): Corruption of in-memory data detected.
Shutting down filesystem
[73715.251398] XFS (sdc1): Please umount the filesystem and rectify the
problem(s)
[73715.302287] XFS (sdc1): xfs_log_force: error -5 returned.
[73730.431543] XFS (sdc1): xfs_log_force: error -5 returned.
gluster-ger-ber-12-int : (kernel 3.13.0-67)
[159046.531752] XFS (sdc1): Mounting Filesystem
[159046.675156] XFS (sdc1): Starting recovery (logdev: internal)
[159047.240939] XFS (sdc1): Ending recovery (logdev: internal)
[159064.372532] XFS (sdc1): Mounting Filesystem
[159064.496445] XFS (sdc1): Ending clean mount
[160598.216262] XFS (sdc1): Mounting Filesystem
[160598.403044] XFS (sdc1): Ending clean mount
[160673.912887] init: statd main process (21051) killed by KILL signal
[160673.912897] init: statd main process ended, respawning
[160674.035234] init: statd main process (8122) terminated with status 1
[160674.035240] init: statd main process ended, respawning
[175683.930045] ffff8807a8f59000: 00 00 00 00 00 00 00 00 fb ee 00 00 00
00 00 00 ................
[175683.932979] ffff8807a8f59010: 10 00 00 00 00 20 0f e0 00 00 00 00 00
00 00 00 ..... ..........
[175683.935812] ffff8807a8f59020: 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
[175683.938644] ffff8807a8f59030: 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
[175683.940870] XFS (sdc1): Internal error xfs_attr3_leaf_write_verify
at line 216 of file
/build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
0xffffffffa01936f0
[175683.943857] CPU: 2 PID: 7874 Comm: xfsaild/sdc1 Not tainted
3.13.0-67-generic #110-Ubuntu
[175683.943859] Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 1.1b
11/01/2013
[175683.943861] 0000000000000001 ffff8807ef1ffbd0 ffffffff817240e0
ffff880036846000
[175683.943866] ffff8807ef1ffbe8 ffffffffa01976fb ffffffffa01936f0
ffff8807ef1ffc20
[175683.943870] ffffffffa0197755 000000d80020fc00 ffff88075c2b4900
ffff88025fbea488
[175683.943873] Call Trace:
[175683.943883] [<ffffffff817240e0>] dump_stack+0x45/0x56
[175683.943917] [<ffffffffa01976fb>] xfs_error_report+0x3b/0x40 [xfs]
[175683.943933] [<ffffffffa01936f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[175683.943948] [<ffffffffa0197755>] xfs_corruption_error+0x55/0x80 [xfs]
[175683.943968] [<ffffffffa01b4b70>]
xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[175683.943983] [<ffffffffa01936f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[175683.943998] [<ffffffffa01953d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[175683.944012] [<ffffffffa01936f0>] _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[175683.944017] [<ffffffff81122a1d>] ?
trace_buffer_unlock_commit+0x4d/0x60
[175683.944030] [<ffffffffa01854d2>] ?
ftrace_raw_event_xfs_buf_class+0xf2/0x100 [xfs]
[175683.944045] [<ffffffffa0195322>] ? xfs_buf_iorequest+0x32/0x90 [xfs]
[175683.944059] [<ffffffffa01953d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[175683.944073] [<ffffffffa0195336>] xfs_buf_iorequest+0x46/0x90 [xfs]
[175683.944087] [<ffffffffa01953d5>] xfs_bdstrat_cb+0x55/0xb0 [xfs]
[175683.944102] [<ffffffffa019556b>]
__xfs_buf_delwri_submit+0x13b/0x210 [xfs]
[175683.944117] [<ffffffffa0196000>] ?
xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[175683.944132] [<ffffffffa0196000>]
xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[175683.944157] [<ffffffffa01f4d27>] xfsaild+0x237/0x5c0 [xfs]
[175683.944181] [<ffffffffa01f4af0>] ?
xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[175683.944186] [<ffffffff8108b7d2>] kthread+0xd2/0xf0
[175683.944189] [<ffffffff8108b700>] ? kthread_create_on_node+0x1c0/0x1c0
[175683.944193] [<ffffffff81734c28>] ret_from_fork+0x58/0x90
[175683.944196] [<ffffffff8108b700>] ? kthread_create_on_node+0x1c0/0x1c0
[175683.944199] XFS (sdc1): Corruption detected. Unmount and run xfs_repair
[175683.945754] XFS (sdc1): xfs_do_force_shutdown(0x8) called from line
1320 of file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_buf.c. Return
address = 0xffffffffa019371c
[175683.946133] XFS (sdc1): Corruption of in-memory data detected.
Shutting down filesystem
[175683.947204] XFS (sdc1): Please umount the filesystem and rectify the
problem(s)
[175713.963171] XFS (sdc1): xfs_log_force: error 5 returned.
[175744.014122] XFS (sdc1): xfs_log_force: error 5 returned.
Kernel-Version :
[ 09:07:53 ] - root@gluster-ger-ber-12 /home/admin $uname -a
Linux gluster-ger-ber-12 3.13.0-67-generic #110-Ubuntu SMP Fri Oct 23
13:24:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
xfsprogs Version :
[ 09:20:36 ] - root@gluster-ger-ber-12 /home/admin $xfs_repair -V
xfs_repair version 3.1.9
Number and type of CPU :
1 CPU
model name : Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz
[ 09:21:49 ] - root@gluster-ger-ber-12 /home/admin $more /proc/meminfo
MemTotal: 32907248 kB
MemFree: 258068 kB
Buffers: 99024 kB
Cached: 31446816 kB
SwapCached: 3836 kB
Active: 15800552 kB
Inactive: 15975832 kB
Active(anon): 44396 kB
Inactive(anon): 186212 kB
Active(file): 15756156 kB
Inactive(file): 15789620 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 33510396 kB
SwapFree: 33449196 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 227736 kB
Mapped: 9772 kB
Shmem: 64 kB
Slab: 451404 kB
SReclaimable: 400636 kB
SUnreclaim: 50768 kB
KernelStack: 1560 kB
PageTables: 4128 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49964020 kB
Committed_AS: 598956 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 161180 kB
VmallocChunk: 34359537664 kB
HardwareCorrupted: 0 kB
AnonHugePages: 126976 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 85232 kB
DirectMap2M: 3018752 kB
DirectMap1G: 30408704 kB
[ 09:21:54 ] - root@gluster-ger-ber-12 /home/admin $more /proc/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=16442840k,nr_inodes=4110710,mode=755
0 0
devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=3290728k,mode=755 0 0
/dev/disk/by-uuid/b1db3d58-26c3-4ed1-a53d-34cb11978ba5 / ext4
rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs
rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
/dev/sdb1 /data ext4 rw,relatime,data=ordered 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
systemd /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
gluster-ger-ber-12-int:/ger-ber-01 /sdn fuse.glusterfs
rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
0 0
/dev/sdc1 /gluster-export xfs rw,relatime,attr2,inode64,noquota 0 0
[ 09:22:20 ] - root@gluster-ger-ber-12 /home/admin $
[ 09:22:20 ] - root@gluster-ger-ber-12 /home/admin $more /proc/partitions
major minor #blocks name
8 0 209715200 sda
8 1 176201728 sda1
8 2 1 sda2
8 5 33510400 sda5
8 16 3696754688 sdb
8 17 3696753647 sdb1
8 32 19532349440 sdc
8 33 19532348399 sdc1
[ 09:22:39 ] - root@gluster-ger-ber-12 /home/admin $
[ 09:23:22 ] - root@gluster-ger-ber-12 /home/admin $lspci -v | more
...
05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208
[Thunderbolt] (rev 05)
Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9271-8i
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at b000 [size=256]
Memory at f7160000 (64-bit, non-prefetchable) [size=16K]
Memory at f7100000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at f7140000 [disabled] [size=128K]
Capabilities: [50] Power Management version 3
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [d0] Vital Product Data
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=16 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [1e0] #19
Capabilities: [1c0] Power Budgeting <?>
Capabilities: [190] #16
Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
Kernel driver in use: megaraid_sas
Kernel modules: megaraid_sas
Affected Virtual Drive : 2
[ 09:24:14 ] - root@gluster-ger-ber-12 /home/admin $megacli -LDInfo
-Lall -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
Size : 200.0 GB
Sector Size : 512
Is VD emulated : Yes
Mirror Data : 200.0 GB
State : Optimal
Strip Size : 256 KB
Number Of Drives : 2
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if
Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
PI type: No PI
Is VD Cached: No
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
Size : 3.442 TB
Sector Size : 512
Is VD emulated : Yes
Mirror Data : 3.442 TB
State : Optimal
Strip Size : 256 KB
Number Of Drives : 2
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if
Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
PI type: No PI
Is VD Cached: No
Virtual Drive: 2 (Target Id: 2)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 18.190 TB
Sector Size : 512
Is VD emulated : Yes
Parity Size : 3.637 TB
State : Optimal
Strip Size : 256 KB
Number Of Drives : 6
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if
Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
PI type: No PI
Is VD Cached: No
Exit Code: 0x00
[ 09:24:17 ] - root@gluster-ger-ber-12 /home/admin $
All disks are of same type : HGST Ultrastar 7K4000 512e 4TB
[ 09:25:27 ] - root@gluster-ger-ber-12 /home/admin $megacli -PDList
-aALL | more
...
Enclosure Device ID: 252
Slot Number: 2
Drive's position: DiskGroup: 1, Span: 0, Arm: 0
Enclosure position: N/A
Device Id: 9
WWN: 5000cca23dd7cc1d
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 3.638 TB [0x1d1c0beb0 Sectors]
Non Coerced Size: 3.637 TB [0x1d1b0beb0 Sectors]
Coerced Size: 3.637 TB [0x1d1b00000 Sectors]
Sector Size: 512
Logical Sector Size: 512
Physical Sector Size: 4096
Firmware state: Online, Spun Up
Commissioned Spare : Yes
Emergency Spare : No
Device Firmware Level: A580
Shield Counter: 0
Successful diagnostics completion on : N/A
SAS Address(0): 0x4433221101000000
Connected Port Number: 1(path0)
Inquiry Data: PK1334PBHPAW2PHGST
HUS724040ALE640 MJAOA580
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive: Not Certified
Drive Temperature :43C (109.40 F)
PI Eligibility: No
Drive is formatted for PI information: No
PI: No PI
Drive's NCQ setting : N/A
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No
[ 09:26:17 ] - root@gluster-ger-ber-12 /home/admin $megacli -PDList
-aALL | grep "Inquiry Data"
Inquiry Data: PK1334PBHPAD0PHGST
HUS724040ALE640 MJAOA580
Inquiry Data: PK1334PBHP5X2PHGST
HUS724040ALE640 MJAOA580
Inquiry Data: PK1334PBHPAW2PHGST
HUS724040ALE640 MJAOA580
Inquiry Data: PK1334PBHP5HJPHGST
HUS724040ALE640 MJAOA580
Inquiry Data: PK1334PBHP59ZPHGST
HUS724040ALE640 MJAOA580
Inquiry Data: PK1334PBHP831PHGST
HUS724040ALE640 MJAOA580
Inquiry Data: PK1334PBHP5Z4PHGST
HUS724040ALE640 MJAOA580
Inquiry Data: PK1338P4GNXEEBHGST
HUS724040ALE640 MJAOA580
[ 09:26:29 ] - root@gluster-ger-ber-12 /home/admin $
[ 09:36:05 ] - root@gluster-ger-ber-12 /home/admin $fuser -c
/gluster-export
Cannot stat /gluster-export: Input/output error
[ 09:36:16 ] - root@gluster-ger-ber-12 /home/admin $umount /gluster-export
[ 09:36:25 ] - root@gluster-ger-ber-12 /home/admin $mount /gluster-export/
[ 09:36:29 ] - root@gluster-ger-ber-12 /home/admin $xfs_info
/gluster-export/
meta-data=/dev/sdc1 isize=256 agcount=32, agsize=152596472
blks
= sectsz=512 attr=2
data = bsize=4096 blocks=4883087099, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[ 09:36:36 ] - root@gluster-ger-ber-12 /home/admin $
[ 09:28:20 ] - root@gluster-ger-ber-12 /home/admin $echo w >
/proc/sysrq-trigger
dmesg :
[Fri Dec 4 09:28:34 2015] SysRq : Show Blocked State
[Fri Dec 4 09:28:34 2015] task PC stack pid father
[Fri Dec 4 09:28:34 2015] Sched Debug Version: v0.11, 3.13.0-67-generic
#110-Ubuntu
[Fri Dec 4 09:28:34 2015] ktime : 136344498.523371
[Fri Dec 4 09:28:34 2015] sched_clk : 136212618.303317
[Fri Dec 4 09:28:34 2015] cpu_clk : 136212618.303397
[Fri Dec 4 09:28:34 2015] jiffies : 4328978420
[Fri Dec 4 09:28:34 2015] sched_clock_stable : 1
[Fri Dec 4 09:28:34 2015] sysctl_sched
[Fri Dec 4 09:28:34 2015] .sysctl_sched_latency :
24.000000
[Fri Dec 4 09:28:34 2015] .sysctl_sched_min_granularity :
3.000000
[Fri Dec 4 09:28:34 2015] .sysctl_sched_wakeup_granularity :
4.000000
[Fri Dec 4 09:28:34 2015] .sysctl_sched_child_runs_first : 0
[Fri Dec 4 09:28:34 2015] .sysctl_sched_features : 77435
[Fri Dec 4 09:28:34 2015] .sysctl_sched_tunable_scaling : 1
(logaritmic)
[Fri Dec 4 09:28:34 2015] cpu#0, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .nr_switches : 147348638
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 11437772
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : -28382
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978407
[Fri Dec 4 09:28:34 2015] .curr->pid : 0
[Fri Dec 4 09:28:34 2015] .clock :
136212614.957368
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 3
[Fri Dec 4 09:28:34 2015] .sched_count : 236776897
[Fri Dec 4 09:28:34 2015] .sched_goidle : 66160497
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 72435010
[Fri Dec 4 09:28:34 2015] .ttwu_local : 11357966
[Fri Dec 4 09:28:34 2015] cfs_rq[0]:/autogroup-1402
[Fri Dec 4 09:28:34 2015] .exec_clock : 270.992157
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 269.013532
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-26475674.528669
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 0
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 15
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .se->exec_start :
136212614.939755
[Fri Dec 4 09:28:34 2015] .se->vruntime : 26475931.652185
[Fri Dec 4 09:28:34 2015] .se->sum_exec_runtime : 270.992157
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.exec_max : 1.112867
[Fri Dec 4 09:28:34 2015] .se->statistics.slice_max : 0.021923
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_max : 0.071390
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_sum : 2.037234
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_count : 8501
[Fri Dec 4 09:28:34 2015] .se->load.weight : 2
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_sum : 105
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_period : 47280
[Fri Dec 4 09:28:34 2015] .se->avg.load_avg_contrib : 0
[Fri Dec 4 09:28:34 2015] .se->avg.decay_count : 129902473
[Fri Dec 4 09:28:34 2015] cfs_rq[0]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 3846527.725695
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 26475943.542201
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 : 0.000000
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 98
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 3
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 143
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 47611
[Fri Dec 4 09:28:34 2015] rt_rq[0]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.014935
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] kthreadd 2 26474982.577430
631 120 26474982.577430 29.113277 136195518.581352 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/0 3 26474983.604407
192950 120 26474983.604407 3698.783802 136191450.813456 0 /
[Fri Dec 4 09:28:34 2015] kworker/0:0H 5 3300.494478
6 100 3300.494478 0.071207 6008.439787 0 /
[Fri Dec 4 09:28:34 2015] rcu_sched 7 26475932.336758
14122966 120 26475932.336758 283940.908724 135759709.558946 0 /
[Fri Dec 4 09:28:34 2015] rcu_bh 16 70.281892 2
120 70.281892 0.000707 0.000587 0 /
[Fri Dec 4 09:28:34 2015] rcuob/0 17 72.282352 2
120 72.282352 0.000801 0.000662 0 /
[Fri Dec 4 09:28:34 2015] rcuob/1 18 74.282814 2
120 74.282814 0.000776 0.000634 0 /
[Fri Dec 4 09:28:34 2015] rcuob/2 19 76.283269 2
120 76.283269 0.000848 0.000602 0 /
[Fri Dec 4 09:28:34 2015] rcuob/3 20 78.283650 2
120 78.283650 0.000775 0.000598 0 /
[Fri Dec 4 09:28:34 2015] rcuob/4 21 80.284102 2
120 80.284102 0.000854 0.000663 0 /
[Fri Dec 4 09:28:34 2015] rcuob/5 22 82.284536 2
120 82.284536 0.000809 0.000580 0 /
[Fri Dec 4 09:28:34 2015] rcuob/6 23 84.285021 2
120 84.285021 0.000888 0.000637 0 /
[Fri Dec 4 09:28:34 2015] rcuob/7 24 86.285485 2
120 86.285485 0.000872 0.000639 0 /
[Fri Dec 4 09:28:34 2015] migration/0 25 0.000000 27167
0 0.000000 797.682052 0.000750 0 /
[Fri Dec 4 09:28:34 2015] watchdog/0 26 -5.953704
34090 0 -5.953704 751.540781 13.772317 0 /
[Fri Dec 4 09:28:34 2015] khubd 71 3244.446742
104 120 3244.446742 61.411241 2754.046153 0 /
[Fri Dec 4 09:28:34 2015] kswapd0 77 26205841.066711
30811 120 26205841.066711 76229.909278 134072768.865772 0 /
[Fri Dec 4 09:28:34 2015] khugepaged 79 26475677.591690
13973 139 26475677.591690 2084.204182 136203770.797993 0 /
[Fri Dec 4 09:28:34 2015] kworker/0:1H 379 23223356.272823
57528 100 23223356.272823 1125.377178 107458262.671565 0 /
[Fri Dec 4 09:28:34 2015] systemd-udevd 441 111.205459 1936
120 111.205459 347.271266 136018099.023055 0 /autogroup-90
[Fri Dec 4 09:28:34 2015] kipmi0 511 26475943.542201
464248 139 26475943.542201 822445.570486 135379234.199855 0 /
[Fri Dec 4 09:28:34 2015] in:imuxsock 557 309.267197 5651
120 309.267197 333.499493 136004317.631646 0 /autogroup-112
[Fri Dec 4 09:28:34 2015] acpid 1095 0.855489 5
120 0.855489 0.524418 17.527620 0 /autogroup-224
[Fri Dec 4 09:28:34 2015] kworker/0:0 16096 26475931.584455
957170 120 26475931.584455 20895.761903 53442927.054945 0 /
[Fri Dec 4 09:28:34 2015] kworker/u17:1 25496 26122951.277120
10489 100 26122951.277120 186.192347 46014951.975247 0 /
[Fri Dec 4 09:28:34 2015] glusterd 10248 20892.577731
43829 120 20892.577731 2767.803349 43744498.428498 0
/autogroup-1375
[Fri Dec 4 09:28:34 2015] glusterd 10249 13989.146111
10 120 13989.146111 0.277888 34507872.284027 0
/autogroup-1375
[Fri Dec 4 09:28:34 2015] glusterfs 10350 6719.390861
43778 120 6719.390861 10489.910435 43733358.086887 0
/autogroup-1382
[Fri Dec 4 09:28:34 2015] kworker/0:1 17609
22321475.749296 2 120 22321475.749296 0.050026 0.043033 0 /
[Fri Dec 4 09:28:34 2015] sshd 13731 269.013532 11291
120 269.013532 301.287578 2062023.641370 0 /autogroup-1402
[Fri Dec 4 09:28:34 2015] bash 13732 0.623405 63
120 0.623405 61.403465 5212.815794 0 /autogroup-1403
[Fri Dec 4 09:28:34 2015] cpu#1, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .nr_switches : 140802404
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 11046184
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : -37147
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978404
[Fri Dec 4 09:28:34 2015] .curr->pid : 0
[Fri Dec 4 09:28:34 2015] .clock :
136212543.982417
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 0
[Fri Dec 4 09:28:34 2015] .sched_count : 223822273
[Fri Dec 4 09:28:34 2015] .sched_goidle : 63146083
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 76646395
[Fri Dec 4 09:28:34 2015] .ttwu_local : 10364838
[Fri Dec 4 09:28:34 2015] cfs_rq[1]:/autogroup-161
[Fri Dec 4 09:28:34 2015] .exec_clock : 41.418053
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 40.369477
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-26475903.172724
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 0
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .se->exec_start :
136212390.652070
[Fri Dec 4 09:28:34 2015] .se->vruntime : 25423116.862434
[Fri Dec 4 09:28:34 2015] .se->sum_exec_runtime : 41.418053
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.exec_max : 0.396702
[Fri Dec 4 09:28:34 2015] .se->statistics.slice_max : 0.073704
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_max : 0.182166
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_sum : 1.361087
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_count : 850
[Fri Dec 4 09:28:34 2015] .se->load.weight : 2
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_sum : 2
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_period : 47625
[Fri Dec 4 09:28:34 2015] .se->avg.load_avg_contrib : 0
[Fri Dec 4 09:28:34 2015] .se->avg.decay_count : 129902259
[Fri Dec 4 09:28:34 2015] cfs_rq[1]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 3806336.668940
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 25423128.794116
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 : -1052814.748085
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 92
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 1
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 62
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 46935
[Fri Dec 4 09:28:34 2015] rt_rq[1]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.027093
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] rcuos/0 8 25423118.245296
2404401 120 25423118.245296 101131.182197 136110439.328825 0 /
[Fri Dec 4 09:28:34 2015] rcuos/2 10 25423116.874811
2422938 120 25423116.874811 101151.958178 136110314.686513 0 /
[Fri Dec 4 09:28:34 2015] watchdog/1 27 -2.951694
34090 0 -2.951694 757.129747 0.000729 0 /
[Fri Dec 4 09:28:34 2015] migration/1 28 0.000000 24068
0 0.000000 469.233325 0.000679 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/1 29 25421716.842085
181729 120 25421716.842085 3477.138497 136182012.889144 0 /
[Fri Dec 4 09:28:34 2015] kworker/1:0H 31 4613.719467
7 100 4613.719467 0.037461 10189.369938 0 /
[Fri Dec 4 09:28:34 2015] khungtaskd 76 25421933.541670
1138 120 25421933.541670 84.284528 136187978.841882 0 /
[Fri Dec 4 09:28:34 2015] scsi_eh_5 186 3365.916452
14 120 3365.916452 1.760553 397.345031 0 /
[Fri Dec 4 09:28:34 2015] jbd2/sda1-8 209 25422648.851152
100906 120 25422648.851152 6159.882209 136198241.371272 0 /
[Fri Dec 4 09:28:34 2015] xfslogd 372 3448.379981
2 100 3448.379981 0.004590 0.002710 0 /
[Fri Dec 4 09:28:34 2015] upstart-file-br 436 12.500126 3968
120 12.500126 94.531374 136015292.474727 0 /autogroup-86
[Fri Dec 4 09:28:34 2015] systemd-logind 669 44.037315 300
120 44.037315 248.585829 134260598.704251 0 /autogroup-149
[Fri Dec 4 09:28:34 2015] rpcbind 722 40.369477 4768
120 40.369477 234.292584 136204838.488381 0 /autogroup-161
[Fri Dec 4 09:28:34 2015] getty 1044 0.070244 65
120 0.070244 1.286122 38.280862 0 /autogroup-195
[Fri Dec 4 09:28:34 2015] cron 1104 52.826625 2331
120 52.826625 151.901350 136158801.398533 0 /autogroup-220
[Fri Dec 4 09:28:34 2015] kworker/1:1H 2618 22530547.872807
65641 100 22530547.872807 1320.284728 107453816.441771 0 /
[Fri Dec 4 09:28:34 2015] rpc.statd 27870 0.450345 10
120 0.450345 2.718404 1.565635 0 /autogroup-997
[Fri Dec 4 09:28:34 2015] glusterd 10250 17662.746416
749 120 17662.746416 178.191852 43741936.432719 0
/autogroup-1375
[Fri Dec 4 09:28:34 2015] glusterfs 10345 2653.426150
95 120 2653.426150 2.679007 43685955.773205 0
/autogroup-1380
[Fri Dec 4 09:28:34 2015] kworker/1:0 11396 25423117.034513
515020 120 25423117.034513 12511.346039 42714718.828356 0 /
[Fri Dec 4 09:28:34 2015] kworker/1:1 23002
22479768.454215 2 120 22479768.454215 0.055400 0.015166 0 /
[Fri Dec 4 09:28:34 2015] kworker/u16:1 10511 25422395.165519
12740 120 25422395.165519 833.068573 5769174.355062 0 /
[Fri Dec 4 09:28:34 2015] cpu#2, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .nr_switches : 140954660
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 10911251
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : -32246
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978418
[Fri Dec 4 09:28:34 2015] .curr->pid : 0
[Fri Dec 4 09:28:34 2015] .clock :
136212595.926891
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 0
[Fri Dec 4 09:28:34 2015] .sched_count : 224643151
[Fri Dec 4 09:28:34 2015] .sched_goidle : 63182858
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 75899548
[Fri Dec 4 09:28:34 2015] .ttwu_local : 10349699
[Fri Dec 4 09:28:34 2015] cfs_rq[2]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 3804562.837849
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 25555626.301013
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 : -920317.241188
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 97
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 1
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 54
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 46771
[Fri Dec 4 09:28:34 2015] rt_rq[2]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.021579
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] init 1 425.097653 14333
120 425.097653 3392.591086 136201613.341830 0 /autogroup-2
[Fri Dec 4 09:28:34 2015] rcuos/4 12 25555614.399222
2395326 120 25555614.399222 111227.287682 136100481.939047 0 /
[Fri Dec 4 09:28:34 2015] watchdog/2 32 -5.980436
34089 0 -5.980436 701.341717 0.001035 0 /
[Fri Dec 4 09:28:34 2015] migration/2 33 0.000000 24044
0 0.000000 454.356563 0.000708 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/2 34 25552774.096697
168290 120 25552774.096697 3062.808230 136155517.391477 0 /
[Fri Dec 4 09:28:34 2015] kworker/2:0H 36 2206.597279
7 100 2206.597279 0.043895 10213.196941 0 /
[Fri Dec 4 09:28:34 2015] scsi_eh_1 182 1345.845801
14 120 1345.845801 2.310343 416.263941 0 /
[Fri Dec 4 09:28:34 2015] scsi_eh_2 183 1347.737725
7 120 1347.737725 4.297015 455.843697 0 /
[Fri Dec 4 09:28:34 2015] scsi_eh_6 187 1344.576241
14 120 1344.576241 1.167670 400.858524 0 /
[Fri Dec 4 09:28:34 2015] xfsalloc 370 1491.536262
2 100 1491.536262 0.005613 0.002823 0 /
[Fri Dec 4 09:28:34 2015] xfs_mru_cache 371 1503.541080
2 100 1503.541080 0.005434 0.020169 0 /
[Fri Dec 4 09:28:34 2015] dbus-daemon 562 35.997923 5207
120 35.997923 321.379628 136014456.290568 0 /autogroup-114
[Fri Dec 4 09:28:34 2015] getty 1048 0.798035 66
120 0.798035 1.038112 22.919171 0 /autogroup-199
[Fri Dec 4 09:28:34 2015] rpc.statd 2610 0.113453 7
120 0.113453 2.541659 1.078356 0 /autogroup-287
[Fri Dec 4 09:28:34 2015] kworker/2:1H 2627 22621782.659664
62070 100 22621782.659664 1335.191062 107314569.211629 0 /
[Fri Dec 4 09:28:34 2015] glusterd 10251 15677.807241
772 120 15677.807241 163.402191 43741947.895943 0
/autogroup-1375
[Fri Dec 4 09:28:34 2015] glusterfs 10341 3025.652327
28750 120 3025.652327 4494.806148 43740371.067623 0
/autogroup-1380
[Fri Dec 4 09:28:34 2015] glusterfs 10353 7848.836436
77 120 7848.836436 3.043855 43432185.230605 0
/autogroup-1382
[Fri Dec 4 09:28:34 2015] kworker/2:1 10915 25555614.472845
539514 120 25555614.472845 12494.506682 43240143.702078 0 /
[Fri Dec 4 09:28:34 2015] kworker/2:0 20081
22180323.415652 3 120 22180323.415652 0.062509 7.314314 0 /
[Fri Dec 4 09:28:34 2015] sudo 13768 15.649688 15
120 15.649688 23.750716 2492.693892 0 /autogroup-1403
[Fri Dec 4 09:28:34 2015] cpu#3, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .nr_switches : 141335883
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 10970238
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : -17617
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978392
[Fri Dec 4 09:28:34 2015] .curr->pid : 0
[Fri Dec 4 09:28:34 2015] .clock :
136212595.942930
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 3
[Fri Dec 4 09:28:34 2015] .sched_count : 228167931
[Fri Dec 4 09:28:34 2015] .sched_goidle : 63329458
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 75973716
[Fri Dec 4 09:28:34 2015] .ttwu_local : 10639601
[Fri Dec 4 09:28:34 2015] cfs_rq[3]:/autogroup-1375
[Fri Dec 4 09:28:34 2015] .exec_clock : 18586.219722
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 18069.862107
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-26457873.680094
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 0
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .se->exec_start :
136212425.591062
[Fri Dec 4 09:28:34 2015] .se->vruntime : 26094251.418506
[Fri Dec 4 09:28:34 2015] .se->sum_exec_runtime : 18586.383623
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.exec_max : 3.983940
[Fri Dec 4 09:28:34 2015] .se->statistics.slice_max : 3.673064
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_max : 0.114982
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_sum : 21.381169
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_count : 89939
[Fri Dec 4 09:28:34 2015] .se->load.weight : 2
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_sum : 11
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_period : 47166
[Fri Dec 4 09:28:34 2015] .se->avg.load_avg_contrib : 0
[Fri Dec 4 09:28:34 2015] .se->avg.decay_count : 129902292
[Fri Dec 4 09:28:34 2015] cfs_rq[3]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 3820771.181635
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 26094262.988915
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 : -381680.553286
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 71
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 2
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 95
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 47669
[Fri Dec 4 09:28:34 2015] rt_rq[3]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.022099
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] rcuos/1 9 26094251.119927
2397714 120 26094251.119927 100698.380729 136110789.493453 0 /
[Fri Dec 4 09:28:34 2015] rcuos/3 11 26094251.350073
2404544 120 26094251.350073 100861.304213 136110827.001807 0 /
[Fri Dec 4 09:28:34 2015] rcuos/6 14 26094026.859325
2330692 120 26094026.859325 108121.695870 136100747.700861 0 /
[Fri Dec 4 09:28:34 2015] watchdog/3 37 -5.979836
34090 0 -5.979836 701.406503 0.000996 0 /
[Fri Dec 4 09:28:34 2015] migration/3 38 0.000000 25006
0 0.000000 614.308545 0.000704 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/3 39 26094030.002911
217683 120 26094030.002911 4097.101172 136204388.184594 0 /
[Fri Dec 4 09:28:34 2015] kworker/3:0H 41 4923.752898
7 100 4923.752898 0.038445 10357.144096 0 /
[Fri Dec 4 09:28:34 2015] fsnotify_mark 80 17395580.736262
37 120 17395580.736262 0.282814 92409161.796469 0 /
[Fri Dec 4 09:28:34 2015] upstart-udev-br 397 30.716080 6127
120 30.716080 230.662890 136015275.419172 0 /autogroup-61
[Fri Dec 4 09:28:34 2015] rs:main Q:Reg 559 321.032885 8007
120 321.032885 923.248259 136186466.324339 0 /autogroup-112
[Fri Dec 4 09:28:34 2015] nrpe 1112 11371.926125
12125 120 11371.926125 2129.552313 136190562.066576 0
/autogroup-221
[Fri Dec 4 09:28:34 2015] gdbus 2081 0.744015 8
120 0.744015 0.330701 0.277507 0 /autogroup-266
[Fri Dec 4 09:28:34 2015] kworker/3:1H 2651 23006607.509737
80448 100 23006607.509737 1510.460473 107363258.764220 0 /
[Fri Dec 4 09:28:34 2015] rpc.statd 13685 0.096523 10
120 0.096523 3.059160 47.016680 0 /autogroup-1290
[Fri Dec 4 09:28:34 2015] xfsaild/sdc1 10017 26094253.544465
828662 120 26094253.544465 34547.387217 43827248.923976 0 /
[Fri Dec 4 09:28:34 2015] glusterd 10247 18069.862107
677281 120 18069.862107 142247.429572 43604482.413832 0
/autogroup-1375
[Fri Dec 4 09:28:34 2015] glusterfs 10342 3936.822141
43845 120 3936.822141 10474.508007 43734386.395618 0
/autogroup-1380
[Fri Dec 4 09:28:34 2015] rpc.statd 10411 0.147194 8
120 0.147194 1.668909 41.767193 0 /autogroup-1386
[Fri Dec 4 09:28:34 2015] kworker/3:2 20448
25855231.751517 4 120 25855231.751517 0.089043
30095580.227654 0 /
[Fri Dec 4 09:28:34 2015] kworker/u16:2 13596 26094251.167465
1422 120 26094251.167465 92.576014 2148231.314437 0 /
[Fri Dec 4 09:28:34 2015] kworker/3:0 13638 26094251.086440
24951 120 26094251.086440 579.316363 2063718.158999 0 /
[Fri Dec 4 09:28:34 2015] cpu#4, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .nr_switches : 105568159
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 6630800
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : 51868
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978327
[Fri Dec 4 09:28:34 2015] .curr->pid : 0
[Fri Dec 4 09:28:34 2015] .clock :
136212240.234656
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 0
[Fri Dec 4 09:28:34 2015] .sched_count : 106391214
[Fri Dec 4 09:28:34 2015] .sched_goidle : 45071812
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 54601820
[Fri Dec 4 09:28:34 2015] .ttwu_local : 7541787
[Fri Dec 4 09:28:34 2015] cfs_rq[4]:/autogroup-1382
[Fri Dec 4 09:28:34 2015] .exec_clock : 1228.651419
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 1235.325126
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-26474708.217075
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 0
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 1
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 1
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 4
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 1
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 4
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .se->exec_start :
136212197.612794
[Fri Dec 4 09:28:34 2015] .se->vruntime : 11078932.597413
[Fri Dec 4 09:28:34 2015] .se->sum_exec_runtime : 1228.847115
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.exec_max : 0.335102
[Fri Dec 4 09:28:34 2015] .se->statistics.slice_max : 0.158210
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_max : 0.107315
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_sum : 0.959447
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_count : 25830
[Fri Dec 4 09:28:34 2015] .se->load.weight : 2
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_sum : 191
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_period : 47148
[Fri Dec 4 09:28:34 2015] .se->avg.load_avg_contrib : 1
[Fri Dec 4 09:28:34 2015] .se->avg.decay_count : 129902075
[Fri Dec 4 09:28:34 2015] cfs_rq[4]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 2996672.618810
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 11078943.311719
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-15397000.230482
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 63
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 4
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 226
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 47270
[Fri Dec 4 09:28:34 2015] rt_rq[4]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.022828
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] watchdog/4 42 -8.976676
34089 0 -8.976676 703.393709 0.001054 0 /
[Fri Dec 4 09:28:34 2015] migration/4 43 0.000000 522672
0 0.000000 5586.023859 0.000878 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/4 44 11078931.375157
123872 120 11078931.375157 2699.457434 136206908.221620 0 /
[Fri Dec 4 09:28:34 2015] kworker/4:0H 46 5183.354019
7 100 5183.354019 0.024152 11346.424817 0 /
[Fri Dec 4 09:28:34 2015] md 72 22.959158 2
100 22.959158 0.003847 0.001519 0 /
[Fri Dec 4 09:28:34 2015] devfreq_wq 73 34.962134 2
100 34.962134 0.003250 0.001585 0 /
[Fri Dec 4 09:28:34 2015] ksmd 78 95.024195 2
125 95.024195 0.009110 0.003632 0 /
[Fri Dec 4 09:28:34 2015] ecryptfs-kthrea 81 131.098695 2
120 131.098695 0.009268 0.003717 0 /
[Fri Dec 4 09:28:34 2015] crypto 82 143.106878 2
100 143.106878 0.008849 0.003485 0 /
[Fri Dec 4 09:28:34 2015] kthrotld 94 287.247167 2
100 287.247167 0.008570 0.003546 0 /
[Fri Dec 4 09:28:34 2015] deferwq 121 1852.932827
4 100 1852.932827 0.103774 345.509759 0 /
[Fri Dec 4 09:28:34 2015] scsi_eh_3 184 2330.673890
14 120 2330.673890 1.123806 406.763399 0 /
[Fri Dec 4 09:28:34 2015] upstart-socket- 685 1.976600 2449
120 1.976600 66.649380 136014443.116090 0 /autogroup-158
[Fri Dec 4 09:28:34 2015] whoopsie 1655 2.701519 102
120 2.701519 11.748913 129474990.452082 0 /autogroup-266
[Fri Dec 4 09:28:34 2015] getty 2435 0.212606 2
120 0.212606 1.152749 0.000000 0 /autogroup-272
[Fri Dec 4 09:28:34 2015] kworker/4:1H 2724 11034545.666401
514255 100 11034545.666401 17271.001339 107436722.892382 0 /
[Fri Dec 4 09:28:34 2015] kauditd 2997 37493.612764
2 120 37493.612764 0.033370 0.003881 0 /
[Fri Dec 4 09:28:34 2015] kworker/4:2 1093 11078932.083317
281743 120 11078932.083317 8749.842765 50514657.114231 0 /
[Fri Dec 4 09:28:34 2015] glusterfs 10413 23.728867 4
120 23.728867 0.291486 0.175916 0 /autogroup-1380
[Fri Dec 4 09:28:34 2015] glusterfs 10349 1235.325126
296202 120 1235.325126 11713.948549 43732121.550028 0
/autogroup-1382
[Fri Dec 4 09:28:34 2015] glusterfs 10351 10.962439 1
120 10.962439 0.011024 0.000000 0 /autogroup-1382
[Fri Dec 4 09:28:34 2015] glusterfs 10352 1200.277837
107 120 1200.277837 2.656723 43432185.809267 0
/autogroup-1382
[Fri Dec 4 09:28:34 2015] rpc.statd 10406 0.263949 11
120 0.263949 2.779534 83.848029 0 /autogroup-1384
[Fri Dec 4 09:28:34 2015] kworker/4:0 10440 9518517.060530
30 120 9518517.060530 0.171060 229429.421687 0 /
[Fri Dec 4 09:28:34 2015] kworker/u17:2 11300 11078931.312338
21530 100 11078931.312338 483.672855 42835471.064931 0 /
[Fri Dec 4 09:28:34 2015] cpu#5, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 2
[Fri Dec 4 09:28:34 2015] .load : 2048
[Fri Dec 4 09:28:34 2015] .nr_switches : 100712644
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 6659188
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : 15496
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978415
[Fri Dec 4 09:28:34 2015] .curr->pid : 13770
[Fri Dec 4 09:28:34 2015] .clock :
136212615.014357
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 0
[Fri Dec 4 09:28:34 2015] .sched_count : 101658031
[Fri Dec 4 09:28:34 2015] .sched_goidle : 42813946
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 59500415
[Fri Dec 4 09:28:34 2015] .ttwu_local : 7597598
[Fri Dec 4 09:28:34 2015] cfs_rq[5]:/autogroup-1403
[Fri Dec 4 09:28:34 2015] .exec_clock : 230.543841
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 355.983581
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-26475587.558620
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 41
[Fri Dec 4 09:28:34 2015] .nr_running : 1
[Fri Dec 4 09:28:34 2015] .load : 1024
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .se->exec_start :
136212615.014357
[Fri Dec 4 09:28:34 2015] .se->vruntime : 10971035.472023
[Fri Dec 4 09:28:34 2015] .se->sum_exec_runtime : 230.606599
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.exec_max : 2.722919
[Fri Dec 4 09:28:34 2015] .se->statistics.slice_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_max : 0.109354
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_sum : 3.479993
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_count : 708
[Fri Dec 4 09:28:34 2015] .se->load.weight : 1024
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_sum : 0
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_period : 47759
[Fri Dec 4 09:28:34 2015] .se->avg.load_avg_contrib : 0
[Fri Dec 4 09:28:34 2015] .se->avg.decay_count : 0
[Fri Dec 4 09:28:34 2015] cfs_rq[5]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 2993175.284119
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 10971032.566799
[Fri Dec 4 09:28:34 2015] .min_vruntime : 10971044.027893
[Fri Dec 4 09:28:34 2015] .max_vruntime : 10971032.566799
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-15504899.514308
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 57
[Fri Dec 4 09:28:34 2015] .nr_running : 2
[Fri Dec 4 09:28:34 2015] .load : 2048
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 1
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 92
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 47167
[Fri Dec 4 09:28:34 2015] rt_rq[5]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.022568
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] rcuos/5 13 10971033.584866
2331731 120 10971033.584866 108385.921359 136102683.479070 0 /
[Fri Dec 4 09:28:34 2015] watchdog/5 47 -8.983447
34089 0 -8.983447 695.200256 0.001978 0 /
[Fri Dec 4 09:28:34 2015] migration/5 48 0.000000 494869
0 0.000000 5210.341551 0.000957 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/5 49 10971032.115937
120815 120 10971032.115937 2524.385728 136205075.696253 0 /
[Fri Dec 4 09:28:34 2015] kworker/5:0H 51 1314.433845
7 100 1314.433845 0.019130 5759.479654 0 /
[Fri Dec 4 09:28:34 2015] kworker/5:1H 376 10924705.883613
592663 100 10924705.883613 19824.511923 107439684.007966 0 /
[Fri Dec 4 09:28:34 2015] rpc.idmapd 501 0.876436 2
120 0.876436 0.221789 0.000000 0 /autogroup-103
[Fri Dec 4 09:28:34 2015] in:imklog 558 63.680250 2456
120 63.680250 152.211282 136187204.501111 0 /autogroup-112
[Fri Dec 4 09:28:34 2015] gmain 2004 11.112020 3
120 11.112020 0.033785 0.003892 0 /autogroup-266
[Fri Dec 4 09:28:34 2015] kworker/5:2 17595 10869059.735478
307364 120 10869059.735478 6109.767622 24108162.604994 0 /
[Fri Dec 4 09:28:34 2015] xfs-data/sdc1 10014 7962554.998970
2 100 7962554.998970 0.026140 0.017214 0 /
[Fri Dec 4 09:28:34 2015] xfs-cil/sdc1 10016 7962567.017092
2 100 7962567.017092 0.020740 0.023641 0 /
[Fri Dec 4 09:28:34 2015] kworker/5:1 10568 10971032.566799
128641 120 10971032.566799 3374.264692 43640364.953312 0 /
[Fri Dec 4 09:28:34 2015] R bash 13770 355.983581 605
120 355.983581 135.107161 2054353.015184 0 /autogroup-1403
[Fri Dec 4 09:28:34 2015] kworker/u16:0 15748
10971027.953261 2 120 10971027.953261 0.064451 0.041840 0 /
[Fri Dec 4 09:28:34 2015] cpu#6, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .nr_switches : 101330059
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 6673995
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : 4721
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978308
[Fri Dec 4 09:28:34 2015] .curr->pid : 0
[Fri Dec 4 09:28:34 2015] .clock :
136212587.968700
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 0
[Fri Dec 4 09:28:34 2015] .sched_count : 102357004
[Fri Dec 4 09:28:34 2015] .sched_goidle : 43064796
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 59140573
[Fri Dec 4 09:28:34 2015] .ttwu_local : 7662707
[Fri Dec 4 09:28:34 2015] cfs_rq[6]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 2942612.370030
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 10885288.919952
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-15590654.622249
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 60
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 47343
[Fri Dec 4 09:28:34 2015] rt_rq[6]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.024601
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] watchdog/6 52 -8.983438
34089 0 -8.983438 746.694608 0.001229 0 /
[Fri Dec 4 09:28:34 2015] migration/6 53 0.000000 500995
0 0.000000 5374.899305 0.000708 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/6 54 10885277.093904
227411 120 10885277.093904 3544.811483 136206245.341403 0 /
[Fri Dec 4 09:28:34 2015] kworker/6:0H 56 3696.921460
7 100 3696.921460 0.028692 10119.614540 0 /
[Fri Dec 4 09:28:34 2015] kdevtmpfs 63 3078.228300
209 120 3078.228300 4.819342 7275.772029 0 /
[Fri Dec 4 09:28:34 2015] charger_manager 122 187.040595 2
100 187.040595 0.020085 0.007776 0 /
[Fri Dec 4 09:28:34 2015] scsi_eh_0 181 1007.316465
2 120 1007.316465 0.438704 0.085498 0 /
[Fri Dec 4 09:28:34 2015] scsi_eh_4 185 2377.025540
14 120 2377.025540 2.156366 401.832292 0 /
[Fri Dec 4 09:28:34 2015] ext4-rsv-conver 210 2510.395347
2 100 2510.395347 0.067114 0.041280 0 /
[Fri Dec 4 09:28:34 2015] jbd2/sdb1-8 353 2546.575108
2 120 2546.575108 0.060667 0.035684 0 /
[Fri Dec 4 09:28:34 2015] ext4-rsv-conver 354 2558.630045
2 100 2558.630045 0.055979 0.037283 0 /
[Fri Dec 4 09:28:34 2015] rpciod 459 2638.698180
2 100 2638.698180 0.005582 0.003100 0 /
[Fri Dec 4 09:28:34 2015] nfsiod 464 2650.703138
2 100 2650.703138 0.005447 0.002335 0 /
[Fri Dec 4 09:28:34 2015] kvm-irqfd-clean 553 3025.776387
2 100 3025.776387 0.012426 0.003151 0 /
[Fri Dec 4 09:28:34 2015] ttm_swap 554 3037.781006
2 100 3037.781006 0.005123 0.002415 0 /
[Fri Dec 4 09:28:34 2015] rsyslogd 555 25.224250 13
120 25.224250 1.580098 126966223.143891 0 /autogroup-112
[Fri Dec 4 09:28:34 2015] getty 1053 0.316572 2
120 0.316572 0.942951 0.000000 0 /autogroup-204
[Fri Dec 4 09:28:34 2015] getty 1054 0.367943 2
120 0.367943 0.851462 0.000000 0 /autogroup-205
[Fri Dec 4 09:28:34 2015] getty 1056 0.340820 2
120 0.340820 0.915774 0.000000 0 /autogroup-207
[Fri Dec 4 09:28:34 2015] sshd 1078 327.941777 405
120 327.941777 82.683192 134259926.849503 0 /autogroup-217
[Fri Dec 4 09:28:34 2015] kworker/6:1H 2617 10839732.580783
585771 100 10839732.580783 20183.712003 107397527.261184 0 /
[Fri Dec 4 09:28:34 2015] kworker/6:0 16800 10803459.202410
346761 120 10803459.202410 7185.014710 26461601.428861 0 /
[Fri Dec 4 09:28:34 2015] xfs-conv/sdc1 10015 7888235.754725
2 100 7888235.754725 0.024640 0.007303 0 /
[Fri Dec 4 09:28:34 2015] glusterd 10331 390.262786 4
120 390.262786 0.031971 0.013309 0 /autogroup-1375
[Fri Dec 4 09:28:34 2015] kworker/6:1 11097 10885277.920224
115607 120 10885277.920224 2798.397167 43052617.893022 0 /
[Fri Dec 4 09:28:34 2015] su 13769 101.611241 10
120 101.611241 6.988278 19.152158 0 /autogroup-1403
[Fri Dec 4 09:28:34 2015] cpu#7, 3300.307 MHz
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .nr_switches : 102128665
[Fri Dec 4 09:28:34 2015] .nr_load_updates : 6775064
[Fri Dec 4 09:28:34 2015] .nr_uninterruptible : 43307
[Fri Dec 4 09:28:34 2015] .next_balance : 4328.978170
[Fri Dec 4 09:28:34 2015] .curr->pid : 0
[Fri Dec 4 09:28:34 2015] .clock :
136212276.212312
[Fri Dec 4 09:28:34 2015] .cpu_load[0] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[1] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[2] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[3] : 0
[Fri Dec 4 09:28:34 2015] .cpu_load[4] : 0
[Fri Dec 4 09:28:34 2015] .yld_count : 0
[Fri Dec 4 09:28:34 2015] .sched_count : 103210194
[Fri Dec 4 09:28:34 2015] .sched_goidle : 43510224
[Fri Dec 4 09:28:34 2015] .avg_idle : 1000000
[Fri Dec 4 09:28:34 2015] .ttwu_count : 59131825
[Fri Dec 4 09:28:34 2015] .ttwu_local : 7981886
[Fri Dec 4 09:28:34 2015] cfs_rq[7]:/autogroup-297
[Fri Dec 4 09:28:34 2015] .exec_clock : 53.219583
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 52.171007
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-26475891.371194
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 0
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .se->exec_start :
136212125.099063
[Fri Dec 4 09:28:34 2015] .se->vruntime : 10974006.018491
[Fri Dec 4 09:28:34 2015] .se->sum_exec_runtime : 53.219583
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_start : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.sleep_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.block_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.exec_max : 0.466913
[Fri Dec 4 09:28:34 2015] .se->statistics.slice_max : 0.000000
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_max : 0.002284
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_sum : 0.003771
[Fri Dec 4 09:28:34 2015] .se->statistics.wait_count : 936
[Fri Dec 4 09:28:34 2015] .se->load.weight : 2
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_sum : 2
[Fri Dec 4 09:28:34 2015] .se->avg.runnable_avg_period : 47607
[Fri Dec 4 09:28:34 2015] .se->avg.load_avg_contrib : 0
[Fri Dec 4 09:28:34 2015] .se->avg.decay_count : 129902006
[Fri Dec 4 09:28:34 2015] cfs_rq[7]:/
[Fri Dec 4 09:28:34 2015] .exec_clock : 2988906.266991
[Fri Dec 4 09:28:34 2015] .MIN_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .min_vruntime : 10974017.588045
[Fri Dec 4 09:28:34 2015] .max_vruntime : 0.000001
[Fri Dec 4 09:28:34 2015] .spread : 0.000000
[Fri Dec 4 09:28:34 2015] .spread0 :
-15501925.954156
[Fri Dec 4 09:28:34 2015] .nr_spread_over : 64
[Fri Dec 4 09:28:34 2015] .nr_running : 0
[Fri Dec 4 09:28:34 2015] .load : 0
[Fri Dec 4 09:28:34 2015] .runnable_load_avg : 0
[Fri Dec 4 09:28:34 2015] .blocked_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg_load_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_runnable_contrib : 0
[Fri Dec 4 09:28:34 2015] .tg_load_avg : 0
[Fri Dec 4 09:28:34 2015] .tg->runnable_avg : 12
[Fri Dec 4 09:28:34 2015] .tg->cfs_bandwidth.timer_active: 0
[Fri Dec 4 09:28:34 2015] .throttled : 0
[Fri Dec 4 09:28:34 2015] .throttle_count : 0
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_sum : 3
[Fri Dec 4 09:28:34 2015] .avg->runnable_avg_period : 47050
[Fri Dec 4 09:28:34 2015] rt_rq[7]:
[Fri Dec 4 09:28:34 2015] .rt_nr_running : 0
[Fri Dec 4 09:28:34 2015] .rt_throttled : 0
[Fri Dec 4 09:28:34 2015] .rt_time : 0.023187
[Fri Dec 4 09:28:34 2015] .rt_runtime : 950.000000
[Fri Dec 4 09:28:34 2015] runnable tasks:
[Fri Dec 4 09:28:34 2015] task PID tree-key switches
prio exec-runtime sum-exec sum-sleep
[Fri Dec 4 09:28:34 2015]
----------------------------------------------------------------------------------------------------------
[Fri Dec 4 09:28:34 2015] rcuos/7 15 10974007.991598
2382699 120 10974007.991598 110085.790054 136101357.927665 0 /
[Fri Dec 4 09:28:34 2015] watchdog/7 57 -8.982100
34089 0 -8.982100 703.493670 0.001050 0 /
[Fri Dec 4 09:28:34 2015] migration/7 58 0.000000 526347
0 0.000000 5656.020194 0.000729 0 /
[Fri Dec 4 09:28:34 2015] ksoftirqd/7 59 10974005.644505
124819 120 10974005.644505 2720.084599 136202330.704909 0 /
[Fri Dec 4 09:28:34 2015] kworker/7:0H 61 4090.966692
7 100 4090.966692 0.025188 10735.407280 0 /
[Fri Dec 4 09:28:34 2015] khelper 62 10.956262 2
100 10.956262 0.005331 0.002345 0 /
[Fri Dec 4 09:28:34 2015] netns 64 35.000828 2
100 35.000828 0.003422 0.001643 0 /
[Fri Dec 4 09:28:34 2015] writeback 65 47.003840 2
100 47.003840 0.003331 0.001576 0 /
[Fri Dec 4 09:28:34 2015] kintegrityd 66 59.006725 2
100 59.006725 0.003165 0.001356 0 /
[Fri Dec 4 09:28:34 2015] bioset 67 71.009702 2
100 71.009702 0.003268 0.001227 0 /
[Fri Dec 4 09:28:34 2015] kblockd 69 95.014561 2
100 95.014561 0.003339 0.001176 0 /
[Fri Dec 4 09:28:34 2015] ata_sff 70 103.018850 2
100 103.018850 0.004569 0.001682 0 /
[Fri Dec 4 09:28:34 2015] atd 1105 0.504305 45
120 0.504305 2.077412 133071116.634185 0 /autogroup-222
[Fri Dec 4 09:28:34 2015] irqbalance 1243 524.318196 13688
120 524.318196 20988.724370 136178139.182247 0 /autogroup-261
[Fri Dec 4 09:28:34 2015] kworker/7:1H 2717 10921547.648508
651792 100 10921547.648508 22042.496277 107431179.258583 0 /
[Fri Dec 4 09:28:34 2015] ntpd 2920 52.171007 137327
120 52.171007 8227.627639 136171806.565453 0 /autogroup-297
[Fri Dec 4 09:28:34 2015] kworker/7:0 9210 10974005.838109
218615 120 10974005.838109 4608.186376 44449710.811257 0 /
[Fri Dec 4 09:28:34 2015] glusterfs 10343 10.997945 3
120 10.997945 0.046530 0.092406 0 /autogroup-1380
[Fri Dec 4 09:28:34 2015] glusterfs 10344 1077.778544
77 120 1077.778544 2.718423 43685955.763044 0
/autogroup-1380
[Fri Dec 4 09:28:34 2015] kworker/7:2 12998
10782985.841970 2 120 10782985.841970 0.017175 0.004086 0 /
[Fri Dec 4 09:28:34 2015] sshd 13632 2570.449211
67 120 2570.449211 30.289697 2971.025095 0 /autogroup-1402
[Fri Dec 4 09:28:34 2015]
[ 09:30:50 ] - root@gluster-ger-ber-12 /home/admin $
--
Dietmar Putz
3Q Medien GmbH
Wetzlarer Str. 86
D-14482 Potsdam
Telefax: +49 (0)331 / 2797 866 - 1
Telefon: +49 (0)331 / 2792 866 - 8
Mobile: +49 171 / 90 160 39
Mail: putz@3qmedien.net
[-- Attachment #1.2: Type: text/html, Size: 143450 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify
2015-12-08 15:20 Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify Dietmar Putz
@ 2015-12-08 17:01 ` Eric Sandeen
2015-12-08 20:57 ` Dave Chinner
1 sibling, 0 replies; 5+ messages in thread
From: Eric Sandeen @ 2015-12-08 17:01 UTC (permalink / raw)
To: xfs
On 12/8/15 9:20 AM, Dietmar Putz wrote:
> Hello all,
...
> [73715.250762] XFS (sdc1): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x44482d730
> [73715.250803] XFS (sdc1): Unmount and run xfs_repair
> [73715.250816] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
> [73715.250832] ffff8807f7b07000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................
> [73715.250856] ffff8807f7b07010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........
> [73715.250876] ffff8807f7b07020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [73715.250907] ffff8807f7b07030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
...
> [175683.930045] ffff8807a8f59000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................
^forward ^back ^magic^pad ^count^usedbytes
> [175683.932979] ffff8807a8f59010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........
^firstused
> [175683.935812] ffff8807a8f59020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [175683.938644] ffff8807a8f59030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [175683.940870] XFS (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa01936f0
Line 216 of xfs_attr_leaf.c: in 3.13 is:
if (!xfs_attr3_leaf_verify(bp)) {
XFS_CORRUPTION_ERROR(__func__, XFS_ERRLEVEL_LOW, mp, bp->b_addr);
>From your xfs_info output, this is not a crc-enabled filesystem.
So xfs_attr3_leaf_verify() only checks these 2 things:
if (ichdr.magic != XFS_ATTR_LEAF_MAGIC)
return false;
that magic is 0xfbee, so good there... and:
if (ichdr.count == 0)
return false;
so that's the problem, apparently. If I'm reading it correctly, it's trying
to write an empty attribute block to disk; this shouldn't happen.
If you have traces, perhaps we can see which inode it's working on, and learn something
from examining it in xfs_db...
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify
2015-12-08 15:20 Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify Dietmar Putz
2015-12-08 17:01 ` Eric Sandeen
@ 2015-12-08 20:57 ` Dave Chinner
2015-12-08 22:14 ` Eric Sandeen
1 sibling, 1 reply; 5+ messages in thread
From: Dave Chinner @ 2015-12-08 20:57 UTC (permalink / raw)
To: Dietmar Putz; +Cc: xfs
On Tue, Dec 08, 2015 at 04:20:01PM +0100, Dietmar Putz wrote:
> similar reports are existing :
> http://oss.sgi.com/archives/xfs/2015-11/msg00459.html
> https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1468039
Following a couple of clicky links in that bug leads to:
http://oss.sgi.com/archives/xfs/2015-06/msg00457.html
in which I comment:
| Probably fixed by commit c88547a ("xfs: fix directory hash ordering
| bug"), which also affected attributes in leaf format.
Hence this was probably fixed in 3.15 and pushed back to 3.13.11i
via th eupstream stable kernel process. I have no idea if the ubuntu
maintainers have added that fix to their kernel...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify
2015-12-08 20:57 ` Dave Chinner
@ 2015-12-08 22:14 ` Eric Sandeen
2015-12-08 22:52 ` Dave Chinner
0 siblings, 1 reply; 5+ messages in thread
From: Eric Sandeen @ 2015-12-08 22:14 UTC (permalink / raw)
To: xfs
On 12/8/15 2:57 PM, Dave Chinner wrote:
> On Tue, Dec 08, 2015 at 04:20:01PM +0100, Dietmar Putz wrote:
>> similar reports are existing :
>> http://oss.sgi.com/archives/xfs/2015-11/msg00459.html
>> https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1468039
>
> Following a couple of clicky links in that bug leads to:
>
> http://oss.sgi.com/archives/xfs/2015-06/msg00457.html
>
> in which I comment:
>
> | Probably fixed by commit c88547a ("xfs: fix directory hash ordering
> | bug"), which also affected attributes in leaf format.
>
> Hence this was probably fixed in 3.15 and pushed back to 3.13.11i
> via th eupstream stable kernel process. I have no idea if the ubuntu
> maintainers have added that fix to their kernel...
But:
> After upgrade to kernel 3.19.0-39 the error message turns from "Internal error xfs_attr3_leaf_write_verify...." to
> "Metadata corruption detected at xfs_attr3_leaf_write_verify..." as shown below for gluster-ger-ber-11-int.
and that's 3.19 ...? (and repair found nothing on-disk?)
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify
2015-12-08 22:14 ` Eric Sandeen
@ 2015-12-08 22:52 ` Dave Chinner
0 siblings, 0 replies; 5+ messages in thread
From: Dave Chinner @ 2015-12-08 22:52 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
On Tue, Dec 08, 2015 at 04:14:15PM -0600, Eric Sandeen wrote:
>
>
> On 12/8/15 2:57 PM, Dave Chinner wrote:
> > On Tue, Dec 08, 2015 at 04:20:01PM +0100, Dietmar Putz wrote:
> >> similar reports are existing :
> >> http://oss.sgi.com/archives/xfs/2015-11/msg00459.html
> >> https://bugs.launchpad.net/ubuntu/+source/linux-lts-trusty/+bug/1468039
> >
> > Following a couple of clicky links in that bug leads to:
> >
> > http://oss.sgi.com/archives/xfs/2015-06/msg00457.html
> >
> > in which I comment:
> >
> > | Probably fixed by commit c88547a ("xfs: fix directory hash ordering
> > | bug"), which also affected attributes in leaf format.
> >
> > Hence this was probably fixed in 3.15 and pushed back to 3.13.11i
> > via th eupstream stable kernel process. I have no idea if the ubuntu
> > maintainers have added that fix to their kernel...
>
> But:
>
> > After upgrade to kernel 3.19.0-39 the error message turns from "Internal error xfs_attr3_leaf_write_verify...." to
> > "Metadata corruption detected at xfs_attr3_leaf_write_verify..." as shown below for gluster-ger-ber-11-int.
>
> and that's 3.19 ...? (and repair found nothing on-disk?)
Ah, missed that. too early in the morning/not enough coffee.
I guess we need a reproducer at this point, then...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-12-08 22:52 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-08 15:20 Metadata corruption detected at xfs_attr3_leaf_write_verify / Internal error xfs_attr3_leaf_write_verify Dietmar Putz
2015-12-08 17:01 ` Eric Sandeen
2015-12-08 20:57 ` Dave Chinner
2015-12-08 22:14 ` Eric Sandeen
2015-12-08 22:52 ` Dave Chinner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.