* yum upgrade on btrfs very slow
@ 2009-09-28 12:18 Tomasz Torcz
2009-09-28 13:35 ` Chris Mason
0 siblings, 1 reply; 11+ messages in thread
From: Tomasz Torcz @ 2009-09-28 12:18 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 5838 bytes --]
Hi,
I'm using btrfs as rootfs on my Fedora 12 (rawhide) test system.
Every yum activity is very slow, like 15 minutes for installation of 11
packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64.
Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk
on Intel ICH controller.
/dev/sda2 on / type btrfs (rw)
/dev/sda2 19G 6,7G 13G 36% /
Attached is output from vmstat 1 during yum transaction. I also
have 200MB of "perf record" output, which I can drill down. First screen
of profile is below:
# Samples: 9024736
#
# Overhead Command Shared Object Symbol
# ........ ............... .................................................... ......
#
15.44% perf.2.6.31-33. [kernel] [k] kmem_cache_open
9.74% perf.2.6.31-33. [kernel] [k] print_lock_contention_bug
6.34% perf.2.6.31-33. [kernel] [k] free_kmem_cache_cpus
4.49% perf.2.6.31-33. [kernel] [k] is_module_text_address
4.01% perf.2.6.31-33. [kernel] [k] lock_set_class
3.45% yum-complete-tr [kernel] [k] print_lock_contention_bug
2.50% perf.2.6.31-33. [kernel] [k] _spin_lock
2.48% yum-complete-tr [kernel] [k] kmem_cache_open
2.47% perf.2.6.31-33. [kernel] [k] zlib_tr_flush_block [zlib_deflate]
1.65% yum-complete-tr [kernel] [k] lock_set_class
1.62% perf.2.6.31-33. [kernel] [k] schedule_hrtimeout_range
1.53% perf.2.6.31-33. [kernel] [k] sysfs_slab_add
1.50% yum-complete-tr [kernel] [k] zlib_tr_flush_block [zlib_deflate]
1.46% yum-complete-tr [kernel] [k] free_kmem_cache_cpus
1.40% yum-complete-tr [kernel] [k] _spin_lock
1.34% perf.2.6.31-33. [kernel] [k] __mutex_lock_common
1.27% perf.2.6.31-33. [kernel] [k] btrfs_getxattr [btrfs]
1.06% perf.2.6.31-33. [kernel] [k] lock_contended
0.78% perf.2.6.31-33. [kernel] [k] _raw_read_unlock
0.75% perf.2.6.31-33. [kernel] [k] btrfs_listxattr [btrfs]
0.74% perf.2.6.31-33. [kernel] [k] run_scheduled_bios [btrfs]
0.62% perf.2.6.31-33. [kernel] [k] lockdep_count_backward_deps
0.61% perf.2.6.31-33. [kernel] [k] show_slab_objects
0.58% yum-complete-tr [kernel] [k] sysfs_slab_add
0.48% yum-complete-tr /usr/lib64/libpython2.6.so.1.0 [.] 0x0000000005dab0
0.46% yum-complete-tr [kernel] [k] btrfs_listxattr [btrfs]
0.45% yum-complete-tr /usr/lib64/librpm.so.0.0.0 [.] 0x0000000004877f
0.44% perf.2.6.31-33. [kernel] [k] idr_for_each
0.44% perf.2.6.31-33. [kernel] [k] clear_lock_stats
0.44% yum-complete-tr [kernel] [k] lock_contended
0.43% perf.2.6.31-33. [kernel] [k] btrfs_iget [btrfs]
0.41% yum-complete-tr [kernel] [k] is_module_text_address
0.37% yum-complete-tr /usr/lib64/libpython2.6.so.1.0 [.] PyEval_EvalFrameEx
0.36% yum-complete-tr [kernel] [k] run_scheduled_bios [btrfs]
0.36% yum-complete-tr [kernel] [k] show_slab_objects
0.33% perf.2.6.31-33. [kernel] [k] lock_acquire
0.33% perf.2.6.31-33. [kernel] [k] __module_text_address
0.29% yum-complete-tr [kernel] [k] btrfs_getxattr [btrfs]
0.28% ldconfig [kernel] [k] free_kmem_cache_cpus
0.27% perf.2.6.31-33. [kernel] [k] btrfs_set_device_sector_size [btrfs]
0.26% yum-complete-tr [kernel] [k] _raw_read_unlock
0.25% perf.2.6.31-33. [kernel] [k] _spin_lock_bh
0.25% yum-complete-tr [kernel] [k] btrfs_drop_extents [btrfs]
0.23% perf.2.6.31-33. [kernel] [k] btrfs_comp_cpu_keys [btrfs]
0.23% yum-complete-tr /lib64/libc-2.10.90.so [.] __GI_memcpy
--
Tomasz Torcz 72->| 80->|
xmpp: zdzichubg@chrome.pl 72->| 80->|
[-- Attachment #2: yum-vmstat_1.txt --]
[-- Type: text/plain, Size: 42267 bytes --]
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 31876 14724 28 470292 0 2 75 44 134 129 2 2 88 8 0
0 1 31876 14528 28 470676 0 0 384 0 266 412 0 2 50 49 0
0 1 31876 14196 28 471004 0 0 336 0 265 390 1 1 50 48 0
0 1 31876 14204 28 471264 0 0 296 7668 1142 614 5 27 29 40 0
0 1 31876 11600 28 472140 0 0 872 0 515 586 0 6 50 44 0
0 1 31876 11104 28 472696 0 0 516 0 469 467 0 4 52 43 0
0 1 31876 10732 28 473100 0 0 408 0 481 450 1 5 51 43 0
2 1 31876 10236 28 473492 0 0 392 0 458 491 1 4 49 46 0
1 0 31876 9864 28 473848 0 0 356 0 412 450 1 5 49 46 0
0 1 31876 9368 28 474416 0 0 568 0 456 547 1 4 48 46 0
0 1 31876 8872 28 474840 0 0 424 0 428 504 1 4 48 46 0
0 1 32184 10828 28 472224 0 308 400 308 752 527 1 8 47 45 0
0 1 32184 10456 28 472732 0 0 504 0 475 495 0 4 51 44 0
0 1 32184 9712 28 473452 0 0 720 0 493 579 0 5 50 44 0
0 3 32412 10164 28 472668 0 228 392 1860 1008 1372 0 10 28 62 0
0 2 32412 10208 28 472892 0 0 200 240 448 629 0 3 21 76 0
0 2 32412 10320 28 473028 0 0 136 8 216 373 0 1 23 75 0
0 1 32412 10072 28 473316 0 0 304 0 387 452 1 3 37 58 0
0 1 32412 9576 28 473772 0 0 448 0 447 546 0 4 49 46 0
1 0 32496 9584 28 473872 0 84 1088 84 787 814 0 8 48 43 0
0 1 33076 9748 28 473048 0 580 492 580 1162 532 1 15 46 38 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 33108 10732 28 472416 0 32 492 32 574 530 1 6 49 44 0
1 0 33108 9676 28 472600 0 0 140 0 1650 309 29 10 50 12 0
2 0 33772 9212 28 468528 0 664 8 3420 2553 414 41 21 38 0 0
1 0 34260 8960 28 468188 0 488 16 488 2533 194 18 36 44 2 0
0 1 34284 10020 28 467532 0 24 32 92 1688 243 14 26 50 10 0
1 0 35940 10020 28 463200 0 1656 8 1740 2469 326 14 29 41 16 0
2 0 37652 13964 28 443076 0 1712 124 1712 3769 565 25 41 31 3 0
1 0 37652 12512 28 443364 0 0 0 640 2065 231 33 22 46 0 0
1 0 38068 9692 28 440108 0 416 0 416 2647 356 38 22 41 0 0
1 0 38652 8948 28 434144 0 584 0 584 2699 427 45 16 40 0 0
1 0 38664 9228 28 433660 0 12 0 12 2131 304 45 9 46 0 0
0 1 38664 27800 28 431240 0 0 0 612 1521 579 20 15 47 18 0
0 1 38664 27808 28 431020 0 0 4 1624 516 495 0 14 46 41 0
0 1 38664 28056 28 430860 0 0 4 1624 1391 1417 1 33 38 28 0
0 1 38664 27776 28 431424 0 0 360 108 208 273 0 2 49 49 0
0 1 38664 27320 28 431832 0 0 164 740 526 653 1 9 48 43 0
0 1 38664 25828 28 433276 0 0 1480 0 283 388 0 2 50 49 0
1 1 38664 25668 28 433604 0 0 180 308 490 615 0 6 49 45 0
1 1 38664 25088 28 433836 0 0 148 684 677 642 4 7 49 41 0
0 1 38664 24964 28 434056 0 0 68 460 379 460 0 3 50 47 0
0 1 38664 24216 28 434604 0 0 548 0 292 379 0 1 50 49 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 38664 23184 28 435476 0 0 932 0 343 394 0 2 50 47 0
0 1 38664 23020 28 435632 0 0 100 0 269 451 0 1 50 50 0
0 1 38664 22856 28 435972 0 0 352 0 254 403 0 1 50 48 0
0 1 38664 22032 28 436772 0 0 488 0 270 405 0 2 49 48 0
0 1 38664 21856 28 437176 0 0 416 0 271 421 0 2 50 48 0
0 3 38664 20180 28 437268 0 0 180 1732 734 1234 0 12 10 78 0
0 3 38664 20016 28 437272 0 0 8 1224 605 1161 1 8 21 71 0
0 3 38664 20304 28 437272 0 0 0 1472 372 511 0 0 0 100 0
0 3 38664 20440 28 437272 0 0 0 1560 402 547 0 1 2 97 0
0 3 38664 20872 28 437272 0 0 0 1344 392 654 0 1 0 99 0
0 3 38664 21436 28 436660 0 0 4 648 665 540 0 8 38 54 0
2 1 38664 21212 28 436892 0 0 4 912 1128 670 0 30 37 32 0
1 1 38664 21492 28 436984 0 0 0 800 440 732 0 3 52 46 0
0 1 38664 16476 28 441760 0 0 4716 0 390 282 0 6 49 45 0
0 1 38664 15544 28 442456 0 0 404 0 510 419 0 6 51 43 0
0 1 38664 13884 28 443708 0 0 524 0 569 482 0 7 51 42 0
5 1 39244 9356 28 423352 0 580 344 3968 1920 1118 1 48 25 26 0
1 1 39348 9356 28 415888 0 104 152 1456 1085 433 10 18 0 71 0
0 1 39408 42600 28 414708 0 60 496 60 1210 576 13 14 37 36 0
0 1 39408 40216 28 416104 0 0 412 0 600 521 1 9 49 41 0
0 1 39408 40320 28 416536 0 0 200 1144 723 706 1 8 50 41 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 39408 40472 28 416196 0 0 16 3476 892 1369 0 6 17 77 0
0 1 39408 35040 28 418160 0 0 1340 0 1338 404 11 21 38 30 0
0 2 39396 31968 28 420084 96 0 1424 0 1364 540 12 18 34 36 0
1 1 39396 28176 28 421752 0 0 152 11716 2100 1236 19 31 36 14 0
0 1 39396 28312 28 421740 0 0 0 4608 579 627 0 10 50 41 0
2 1 39396 28460 28 421560 0 0 0 888 618 835 0 10 47 42 0
1 1 39396 28576 28 421684 0 0 88 888 604 671 0 12 48 39 0
2 0 39396 28584 28 421908 0 0 8 568 487 520 0 6 47 46 0
1 0 39396 28336 28 422272 0 0 180 816 699 653 2 12 43 42 0
1 0 39396 27964 28 422764 0 0 4 1108 528 680 0 9 47 44 0
0 1 39396 27312 28 423380 0 0 368 544 307 469 0 3 50 46 0
1 1 39396 26900 28 423828 0 0 432 552 406 1000 0 5 49 45 0
0 1 39396 26468 28 424340 0 0 0 1216 1085 866 0 21 45 35 0
0 1 39396 22708 28 427380 0 0 3016 0 518 405 0 7 49 44 0
0 1 39396 19840 28 429812 0 0 2352 2300 1405 436 9 22 48 21 0
0 1 39396 19340 28 430336 0 0 476 0 489 365 1 2 50 47 0
1 0 39396 18468 28 430928 0 0 628 0 584 441 0 4 50 45 0
0 1 39396 16640 28 432604 0 0 1276 0 764 562 1 6 49 44 0
0 1 39396 14532 28 433688 0 0 632 0 870 486 2 9 51 38 0
0 3 39568 14184 28 429480 0 172 392 1972 1471 1804 0 30 35 34 0
1 3 39568 14180 28 429512 0 0 24 1048 461 586 0 2 0 98 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 3 39568 14212 28 429536 0 0 20 1112 394 422 0 1 0 98 0
0 3 39568 14692 28 429572 0 0 44 1076 360 440 0 1 0 98 0
0 3 39568 14712 28 429624 0 0 20 1148 399 562 0 2 0 99 0
0 3 39568 14856 28 429648 0 0 16 1220 321 367 0 1 0 99 0
0 3 39568 15108 28 429664 0 0 20 1372 346 460 0 1 0 99 0
0 3 39568 14972 28 429692 0 0 28 1216 478 570 0 3 6 91 0
1 2 39568 15088 28 429744 0 0 36 1160 379 492 0 1 50 48 0
0 3 39568 15028 28 429916 0 0 60 1628 383 385 0 2 50 47 0
0 2 39568 15372 28 430092 0 0 24 1072 431 712 0 3 30 67 0
0 2 39568 15596 28 430236 0 0 48 8 279 424 1 2 23 75 0
0 1 39568 14804 28 432704 0 0 296 184 1167 455 2 25 35 38 0
0 1 39568 14120 28 433016 0 0 12 188 811 458 7 10 49 34 0
2 0 39568 13752 28 433380 0 0 20 4460 1144 1220 0 31 36 33 0
0 2 39568 13596 28 433484 0 0 12 2328 1020 984 0 14 36 50 0
0 2 39568 13552 28 433512 0 0 0 0 297 206 0 3 0 97 0
0 1 39568 13524 28 433660 0 0 4 640 608 714 0 10 44 45 0
0 1 39568 13400 28 433944 0 0 4 984 789 770 1 14 45 40 0
1 1 39568 13400 28 434136 0 0 36 5136 686 768 0 11 47 42 0
1 0 39568 12532 28 434532 0 0 28 440 1209 504 1 28 14 57 0
0 1 39568 12300 28 434796 0 0 0 584 497 578 1 7 49 43 0
0 1 39568 11804 28 435256 0 0 4 936 459 673 0 6 50 44 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 40060 11612 28 433412 0 492 4 492 2476 293 35 20 45 0 0
1 0 40220 10132 28 431588 0 160 36 160 2216 340 39 16 45 1 0
1 0 40444 10132 28 428448 0 224 0 224 2248 321 40 16 45 0 0
1 0 40700 9644 28 426108 0 256 0 256 2380 313 39 16 46 0 0
1 0 41096 9396 28 423284 0 396 0 396 2471 343 39 17 44 0 0
1 1 41096 14628 28 417780 0 0 96 300 1618 432 25 14 45 15 0
0 1 41096 13480 28 418936 0 0 952 108 514 511 2 6 49 42 0
1 0 41096 13244 28 419068 0 0 0 768 442 709 0 6 50 45 0
0 1 41096 13304 28 418532 0 0 20 1676 1148 1478 1 26 34 39 0
1 1 41096 13368 28 418540 0 0 0 16 175 342 0 1 49 50 0
1 1 41096 13492 28 418396 0 0 32 936 468 666 0 8 47 45 0
1 0 41096 12800 28 419000 0 0 588 108 284 411 0 2 51 46 0
0 1 41096 12872 28 418964 0 0 24 1016 571 638 2 8 46 44 0
0 1 41096 12996 28 418964 0 0 4 888 363 632 0 4 50 46 0
1 1 41096 11688 28 420240 0 0 1324 0 355 463 0 2 51 46 0
0 1 41096 11224 28 420712 0 0 452 0 281 448 0 2 50 49 0
0 1 41212 12436 28 417108 0 116 1476 116 550 473 1 8 45 46 0
0 1 41212 12380 28 418124 0 0 1028 0 356 484 0 2 50 47 0
0 1 41212 12372 28 418960 0 0 880 0 295 453 0 2 49 49 0
1 1 41212 12372 28 419388 0 0 68 2184 446 554 0 6 24 70 0
0 4 42028 31824 28 398344 0 816 284 2308 2245 2177 0 41 3 55 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 5 42028 31732 28 398588 0 0 296 1248 441 571 0 3 44 52 0
1 5 42028 31168 28 398904 0 0 252 500 300 444 0 2 0 98 0
0 5 42028 31172 28 399356 0 0 488 684 297 411 0 2 0 98 0
0 4 42020 30560 28 400212 32 0 1000 928 456 452 0 3 0 97 0
0 4 42020 30232 28 400580 0 0 284 992 384 557 0 1 0 99 0
0 4 42020 30416 28 400792 0 0 252 1456 381 466 0 4 0 96 0
0 5 42020 30724 28 400700 0 0 28 1204 386 398 0 3 0 97 0
0 4 42020 30956 28 400644 0 0 16 1052 301 567 0 2 10 88 0
0 5 42020 31100 28 400712 0 0 40 1064 445 424 0 3 10 87 0
0 3 42020 31412 28 400740 0 0 24 976 410 484 0 1 0 98 0
0 5 42020 31532 28 400968 0 0 68 6536 812 690 0 11 0 89 0
0 6 42020 30916 28 401228 32 0 228 652 415 443 0 3 39 57 0
0 6 42012 30864 28 401696 12 0 528 264 602 971 1 8 0 92 0
0 7 42012 30824 28 401888 0 0 152 948 663 365 0 5 0 94 0
1 7 42012 30804 28 402220 0 0 100 8 554 666 0 7 0 93 0
1 2 42012 27292 28 405828 0 0 3580 8764 1378 841 0 42 1 56 0
0 3 42012 26816 28 406100 0 0 116 4604 1275 744 0 35 9 57 0
0 6 42012 26552 28 406428 0 0 268 576 619 876 0 3 22 75 0
0 3 42012 28012 28 406516 0 0 240 0 666 531 1 12 0 86 0
1 0 42012 23228 28 408432 0 0 288 0 1369 415 7 23 26 43 0
0 1 42012 27588 28 406880 0 0 16 1144 487 620 1 7 48 45 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 42012 27580 28 406884 0 0 0 2296 529 993 0 5 49 47 0
0 1 42012 27584 28 406984 0 0 0 1704 715 896 0 11 50 38 0
0 1 42012 27704 28 406996 0 0 0 8 148 245 0 0 51 49 0
0 1 42012 27464 28 407676 0 0 384 828 586 600 2 7 49 42 0
0 1 42012 27340 28 407796 0 0 0 1152 454 797 0 5 49 45 0
0 1 42012 27208 28 407348 0 0 36 580 466 650 0 9 42 48 0
0 1 42012 27084 28 407428 0 0 44 340 272 466 0 3 51 46 0
0 1 42012 27084 28 407472 0 0 8 744 496 699 1 7 46 45 0
1 0 42012 27084 28 407600 0 0 44 512 371 627 0 4 48 48 0
0 1 42012 26836 28 408128 0 0 0 1432 852 939 0 13 48 39 0
0 1 42012 26316 28 408348 0 0 64 480 506 496 0 6 51 43 0
0 1 42012 26092 28 408636 0 0 56 784 398 585 0 3 47 50 0
1 0 42012 25968 28 408852 0 0 0 848 391 537 0 4 50 47 0
0 1 42012 25452 28 409140 0 0 0 1900 781 668 0 13 49 38 0
1 0 42012 25340 28 409668 0 0 308 16 706 462 6 6 49 38 0
1 0 42012 24224 28 411016 0 0 4 0 2156 443 32 22 46 0 0
1 0 42012 23604 28 411468 0 0 0 0 2051 236 35 17 49 0 0
1 0 42012 23480 28 411660 0 0 4 0 2025 204 35 15 49 0 0
0 1 42012 23356 28 411760 0 0 24 1824 2162 530 26 22 38 14 0
0 1 42012 22620 28 412476 0 0 696 0 228 324 0 2 50 48 0
0 1 42012 22000 28 413088 0 0 132 1056 689 877 0 11 47 42 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 42012 21636 28 413420 0 0 180 184 257 334 1 2 49 48 0
1 2 42012 20864 28 413932 0 0 372 712 643 1161 0 11 39 50 0
0 2 42012 20892 28 414240 0 0 72 2780 883 466 0 24 4 73 0
2 1 42012 20576 28 414248 0 0 44 1380 570 811 0 9 20 72 0
0 2 42012 18036 28 414280 0 0 0 1240 859 1847 0 15 7 79 0
0 2 42012 18324 28 414280 0 0 0 1080 476 754 0 2 0 98 0
0 2 42012 18580 28 414280 0 0 0 1088 347 387 0 1 0 99 0
0 2 42012 18804 28 414280 0 0 0 1784 488 748 0 3 0 97 0
1 2 42880 10176 28 390804 0 868 40 2016 1944 521 3 33 0 64 0
1 2 42880 43384 28 390812 0 0 0 1416 1076 728 17 4 27 52 0
0 3 42880 43720 28 391044 0 0 296 1956 324 366 0 2 45 53 0
0 1 42880 44024 28 391108 0 0 0 1024 266 756 0 1 38 61 0
4 1 42880 44236 28 391008 0 0 8 244 216 241 0 4 33 63 0
1 0 42880 43856 28 390928 0 0 92 732 604 557 0 11 44 44 0
0 1 42880 43692 28 391624 0 0 528 188 445 536 0 6 49 45 0
0 1 42880 43740 28 391772 0 0 48 536 413 481 0 6 47 47 0
0 1 42880 41268 28 394284 0 0 252 200 384 486 0 5 49 46 0
0 1 42880 41080 28 394416 0 0 172 1264 1398 1308 1 35 27 39 0
0 1 42880 41136 28 394424 0 0 0 16 163 350 0 1 50 50 0
0 1 42880 40516 28 394636 0 0 84 360 463 462 0 6 50 43 0
0 1 42880 40268 28 394940 0 0 4 1176 603 634 0 10 49 42 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 42880 40152 28 395036 0 0 28 876 505 680 1 7 48 44 0
0 1 42880 40028 28 395064 0 0 0 764 422 660 0 6 49 45 0
0 1 42880 40020 28 395096 0 0 0 1052 497 762 0 8 47 44 0
1 1 42880 39904 28 395476 0 0 8 576 381 590 1 5 49 46 0
0 1 42880 39764 28 395608 0 0 0 1176 668 976 0 10 50 41 0
1 1 42880 39772 28 395640 0 0 0 512 336 582 0 3 49 47 0
0 3 42880 39656 28 395720 0 0 12 1804 575 976 0 11 27 61 0
0 1 42880 39752 28 395476 0 0 12 1292 512 755 0 9 19 73 0
0 1 42880 39896 28 395592 0 0 0 328 283 448 0 2 50 48 0
0 1 42880 39904 28 395624 0 0 0 728 286 491 0 4 50 46 0
0 1 42880 39896 28 395800 0 0 0 1232 630 688 0 10 47 42 0
1 0 42880 37384 28 398512 0 0 2000 1500 1994 461 22 26 34 20 0
1 0 42880 34812 28 401264 0 0 884 0 2055 189 30 21 49 0 0
1 0 42880 33076 28 402984 0 0 0 0 2044 176 29 21 50 0 0
1 0 42880 31340 28 404704 0 0 0 0 2049 177 30 20 50 0 0
1 0 42880 29480 28 406720 0 0 0 0 2045 198 29 22 49 0 0
1 0 42880 27000 28 409172 0 0 0 0 2073 243 31 21 49 0 0
1 0 42880 25140 28 410996 0 0 0 0 2051 193 29 22 49 0 0
2 0 42880 23024 28 412808 0 0 0 0 2048 197 30 21 49 0 0
1 0 42880 20376 28 414752 0 0 4 0 2055 194 29 22 49 0 0
1 0 42880 17436 28 416820 0 0 4 0 2053 192 30 21 49 0 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 42880 15412 28 418088 0 0 0 0 2033 202 27 24 49 0 0
1 2 42892 13020 28 417220 0 12 0 2188 2420 2695 27 41 5 26 0
1 2 42892 13284 28 416496 0 0 0 1256 2254 634 29 25 6 40 0
1 2 42948 13976 28 415256 0 56 0 1120 2308 387 29 26 1 45 0
1 2 42948 13080 28 416112 0 0 0 1272 2258 723 30 25 0 46 0
1 2 42948 12884 28 415936 0 0 0 1464 2285 754 29 25 22 26 0
1 2 42968 13764 28 414620 0 20 0 1116 2327 382 29 26 3 43 0
1 2 42968 13056 28 415548 0 0 0 2016 2283 649 28 26 0 46 0
1 1 42968 11692 28 416412 0 0 0 1392 2299 652 29 26 0 44 0
1 1 42996 13052 28 414748 0 28 0 36 2237 402 26 28 6 41 0
1 0 43028 12860 28 414372 0 32 0 56 2157 190 25 33 29 13 0
1 0 43040 12836 28 413816 0 12 16 12 2041 219 29 23 48 1 0
1 0 43040 12044 28 413832 0 0 0 0 2061 222 30 23 48 0 0
1 0 43120 12708 28 412764 0 80 0 80 2134 226 30 23 47 0 0
1 0 43120 13148 28 412048 0 0 28 0 2003 220 31 19 48 2 0
0 1 43120 12980 28 411788 0 0 0 1160 1709 721 23 16 49 13 0
0 1 43120 12700 28 412152 0 0 292 0 200 312 0 1 50 49 0
0 1 43120 12088 28 412780 0 0 672 0 240 386 0 2 50 48 0
0 1 43220 14476 28 410064 0 100 1072 100 563 575 0 6 48 46 0
0 1 43220 13724 28 411168 0 0 1224 0 303 457 0 2 51 47 0
0 2 43220 12948 28 411976 0 0 872 1080 592 540 0 9 30 61 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 1 43220 12884 28 411500 0 0 120 1144 905 687 0 17 41 42 0
2 0 43220 12980 28 411564 0 0 0 112 316 288 0 5 50 45 0
0 1 43220 12988 28 411600 0 0 0 1080 399 734 0 4 48 48 0
1 1 43220 12544 28 411972 0 0 452 104 353 414 1 3 50 45 0
2 1 43272 13820 28 410440 0 52 284 1624 1205 851 2 27 28 43 0
2 3 43272 14356 28 409320 0 0 104 6372 2473 2257 7 70 3 22 0
0 6 43272 14436 28 408904 0 0 64 3020 674 662 1 13 0 87 0
0 5 43272 14884 28 408504 0 0 196 0 586 408 0 10 0 90 0
0 6 43272 14760 28 408720 0 0 72 0 256 256 0 3 0 97 0
0 7 43272 14884 28 408644 0 0 76 376 264 266 0 4 0 97 0
1 4 43272 15004 28 408656 0 0 408 336 953 715 0 19 0 81 0
0 3 43272 14924 28 408592 0 0 124 2272 901 695 1 20 22 58 0
0 5 43272 14956 28 408800 0 0 16 284 275 273 0 3 1 96 0
0 4 43272 14920 28 408936 0 0 68 12 223 275 0 2 30 68 0
0 3 43272 14924 28 409348 0 0 160 1304 558 690 0 10 0 90 0
1 4 43272 14420 28 410048 0 0 248 2184 1028 1309 0 23 3 74 0
1 5 43272 14484 28 410120 0 0 12 0 154 187 1 1 0 99 0
0 3 43272 14032 28 410500 0 0 172 436 455 532 0 7 0 93 0
0 3 43272 13856 28 410744 0 0 92 1772 562 772 0 12 4 84 0
3 5 43540 21448 28 398752 0 268 388 3668 1601 2976 1 41 5 52 0
0 4 44024 31092 28 387276 0 484 32 1296 1698 1902 0 25 5 69 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 5 44024 30860 28 387488 0 0 24 1044 413 585 0 5 0 95 0
0 6 44024 30840 28 387600 0 0 12 820 329 382 0 4 0 96 0
0 7 44024 30812 28 387884 0 0 28 576 546 528 0 6 0 94 0
0 7 44024 30676 28 388116 0 0 48 928 472 538 0 5 0 95 0
0 5 44024 30940 28 388144 0 0 20 1000 306 395 0 3 0 98 0
2 3 44024 31064 28 388188 0 0 16 1288 449 731 0 3 0 96 0
0 6 44024 30936 28 388540 0 0 48 1484 498 426 0 6 0 94 0
0 7 44024 31288 28 388684 0 0 52 1024 379 468 0 3 0 97 0
0 6 44024 31196 28 389112 224 0 484 1256 609 561 0 6 0 94 0
0 4 44024 31556 28 389140 0 0 44 728 385 502 1 2 0 98 0
0 4 44024 31644 28 389420 0 0 200 1080 341 440 0 2 0 98 0
0 4 44024 31896 28 389596 0 0 68 1784 368 509 0 2 44 54 0
0 3 44024 32088 28 389700 0 0 20 1136 493 765 0 5 50 44 0
1 3 44024 32336 28 389672 0 0 4 1224 323 548 0 1 50 49 0
0 3 44024 32712 28 389892 0 0 8 1176 558 505 0 6 0 94 0
0 3 44024 32912 28 389908 0 0 0 760 257 387 0 2 0 99 0
0 3 44024 33164 28 390008 0 0 4 1128 601 478 0 8 0 92 0
0 4 44020 33212 28 390424 32 0 316 3120 840 490 5 8 5 81 0
0 3 44020 33440 28 390380 0 0 12 2756 381 466 0 4 0 96 0
0 2 44020 33736 28 390420 0 0 8 1568 256 318 0 1 0 99 0
1 3 44020 34236 28 390528 0 0 12 76 316 555 0 4 2 94 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 1 44020 34584 28 390860 0 0 108 3236 1272 1562 1 34 23 41 0
1 1 44020 34076 28 391212 0 0 0 2320 825 1247 0 17 0 83 0
0 1 44020 33712 28 391528 0 0 0 504 347 394 0 6 44 50 0
0 3 44020 33356 28 392124 0 0 16 3076 945 856 0 18 24 58 0
0 2 44020 31960 28 393000 0 0 952 1156 1140 1211 0 5 1 94 0
0 3 44020 31596 28 392932 0 0 48 2296 588 702 0 8 0 91 0
0 2 44020 31236 28 393476 0 0 480 572 384 459 0 5 26 69 0
0 1 44020 31712 28 393532 0 0 84 576 297 466 0 2 50 49 0
0 1 44020 31456 28 394120 0 0 564 0 288 383 0 2 50 47 0
0 1 44020 31200 28 394372 0 0 344 0 278 416 0 3 50 46 0
1 0 44020 27512 28 398360 0 0 168 0 1284 289 5 24 49 22 0
0 4 44020 27008 28 398364 0 0 12 4716 1095 1714 2 24 14 60 0
0 5 44020 26704 28 398448 0 0 72 2112 571 260 0 10 0 90 0
0 5 44020 26464 28 398428 0 0 164 0 759 362 1 16 0 84 0
1 3 44020 26524 28 398512 0 0 76 0 372 258 0 8 0 92 0
1 0 44020 26996 28 398864 0 0 268 0 1122 482 0 26 34 40 0
0 1 44020 26888 28 398872 0 0 0 2288 480 909 0 6 48 46 0
0 1 44020 26888 28 398904 0 0 4 1712 408 731 0 4 49 47 0
0 1 44020 26896 28 398888 0 0 4 1528 408 630 0 4 50 46 0
0 1 44020 26896 28 399084 0 0 0 488 267 470 1 2 49 49 0
1 0 44020 26524 28 399556 0 0 156 752 591 579 1 8 51 39 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 45832 9520 28 385864 0 1812 32 3484 3791 693 33 58 8 2 0
1 0 45832 41428 28 386696 0 0 4 0 2080 312 47 22 31 1 0
0 1 45832 41304 28 387104 0 0 4 188 1629 293 26 13 49 11 0
0 1 45832 40676 28 387856 0 0 376 640 455 530 0 7 49 44 0
0 1 45832 40436 28 388056 0 0 32 1304 1168 1220 1 25 35 39 0
0 1 45832 40188 28 388308 0 0 4 408 233 383 0 3 49 49 0
1 1 45832 39364 28 389136 0 0 692 424 296 467 0 4 50 46 0
0 1 45832 39188 28 389344 0 0 12 408 408 501 0 6 47 47 0
0 3 45832 32792 28 389452 0 0 128 1724 1364 2759 0 27 25 48 0
0 3 45832 33104 28 389452 0 0 0 1040 442 561 0 1 3 96 0
0 3 45832 33360 28 389452 0 0 0 1320 403 536 0 1 0 99 0
0 3 45832 33608 28 389452 0 0 0 1352 448 724 1 1 0 98 0
1 3 45832 33928 28 389452 0 0 0 1128 398 650 0 1 0 99 0
0 3 45832 34176 28 389452 0 0 0 1496 317 403 0 1 0 99 0
0 3 45832 34556 28 389452 0 0 0 1424 358 591 0 1 0 99 0
0 3 45832 34900 28 389452 0 0 0 992 332 495 0 1 0 99 0
0 3 45832 35216 28 389752 0 0 284 1376 353 465 0 1 0 99 0
0 3 45832 35676 28 389736 0 0 4 1576 420 769 0 2 0 98 0
0 3 45832 36088 28 389736 0 0 4 1608 459 653 0 1 0 99 0
1 3 45832 36524 28 389736 0 0 0 1016 408 391 0 1 0 99 0
0 3 45832 36984 28 389736 0 0 0 2492 484 858 0 2 0 98 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 2 45832 37420 28 389736 0 0 0 968 312 368 0 1 0 98 0
0 1 45832 38032 28 389444 0 0 0 576 419 538 0 6 33 62 0
0 1 45832 38748 28 389444 0 0 92 828 597 501 0 12 41 47 0
1 0 45832 38436 28 389632 0 0 180 188 250 323 0 4 50 46 0
0 1 45832 37812 28 390124 0 0 500 0 273 419 0 2 51 48 0
2 1 45832 37972 28 390076 0 0 4 1024 1320 1370 1 36 35 29 0
0 1 45832 37684 28 390124 0 0 20 88 353 259 0 4 52 43 0
0 1 45832 37568 28 390324 0 0 0 528 355 523 0 3 51 46 0
0 1 45832 37336 28 390708 0 0 0 896 529 662 0 8 49 43 0
1 0 45832 36716 28 391524 0 0 312 0 1479 303 21 13 50 16 0
1 0 45832 35972 28 392392 0 0 4 0 2108 328 31 20 49 1 0
1 0 45832 35724 28 392628 0 0 0 0 2052 191 36 14 49 0 0
0 1 45832 35352 28 393016 0 0 308 180 1003 373 13 10 49 29 0
0 1 45832 35224 28 393268 0 0 12 792 482 677 0 7 49 44 0
0 1 45832 33444 28 395044 0 0 4 3992 1784 1860 1 42 32 25 0
0 1 45832 33604 28 394892 0 0 0 16 273 411 0 2 50 48 0
0 1 45832 33496 28 395124 0 0 0 640 381 574 0 5 49 46 0
0 1 45832 33084 28 395516 0 0 172 556 467 551 0 7 50 43 0
0 1 45832 32424 28 396168 0 0 412 808 552 729 1 8 49 42 0
0 1 45832 32008 28 396636 0 0 208 608 470 672 0 4 50 45 0
0 1 45832 31508 28 397012 0 0 72 1556 1213 1289 1 25 40 35 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 45832 31588 28 397004 0 0 0 16 306 438 0 1 48 51 0
0 1 45832 30760 28 397172 0 0 0 520 518 509 1 7 49 44 0
1 1 45832 29956 28 397700 0 0 0 1552 752 730 0 10 50 40 0
1 0 45832 30008 28 398364 0 0 212 16 514 499 4 5 50 42 0
0 1 45832 29416 28 398728 0 0 140 468 1315 565 12 16 48 24 0
0 1 45832 29152 28 399076 0 0 152 536 607 737 3 6 30 61 0
2 0 45832 29144 28 399120 0 0 0 672 358 553 1 5 48 47 0
1 1 45832 29176 28 399004 0 0 0 1080 784 919 0 13 46 41 0
1 0 45832 28780 28 399656 0 0 156 1116 755 764 0 15 45 40 0
0 1 45832 28680 28 399836 0 0 0 996 526 759 3 7 46 44 0
0 1 45832 28432 28 400092 0 0 28 712 398 676 0 4 50 46 0
0 1 45832 28432 28 400116 0 0 8 1504 901 1088 0 16 46 39 0
0 3 45832 23440 28 400200 0 0 184 2144 1197 1977 0 18 36 46 0
0 3 45832 23632 28 400400 0 0 204 1048 431 581 0 3 0 97 0
0 3 45832 23640 28 400420 0 0 20 884 360 373 0 1 0 98 0
0 3 45832 23112 28 400524 0 0 104 1120 425 560 1 2 0 97 0
1 2 45832 20404 28 400852 0 0 328 1200 763 491 5 6 0 89 0
0 3 45832 18432 28 403084 0 0 4 1376 860 518 4 11 0 85 0
0 3 45832 18688 28 403084 0 0 4 1244 343 444 0 1 0 99 0
0 3 45832 19120 28 403084 0 0 0 1312 467 691 0 1 0 99 0
0 3 45832 19412 28 403084 0 0 0 1300 346 373 0 1 0 99 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 3 45832 19960 28 403084 0 0 0 1616 320 407 0 1 0 99 0
1 2 45832 20284 28 403084 0 0 0 4296 457 1021 0 4 39 57 0
0 3 45832 20900 28 402504 0 0 4 192 584 532 0 9 44 46 0
3 1 45832 21084 28 402584 0 0 12 56 363 265 0 9 43 48 0
0 1 45832 27456 28 400260 0 0 4 1152 670 772 1 11 43 45 0
2 0 45832 27472 28 400256 0 0 0 1168 319 525 0 3 50 47 0
3 0 45832 27340 28 400336 0 0 44 2524 943 627 1 19 47 34 0
0 1 45832 50264 28 377312 4 0 1620 136 1212 405 3 26 44 27 0
2 1 45832 46792 28 379684 0 0 2336 0 740 882 3 10 48 40 0
0 1 45832 46560 28 380504 0 0 540 0 797 596 3 13 47 36 0
0 1 45832 45940 28 381168 0 0 656 0 259 424 0 2 50 49 0
0 1 45832 45196 28 381772 0 0 652 4 273 397 0 2 49 49 0
0 1 45832 44032 28 382904 0 0 1112 0 282 392 0 2 49 49 0
0 1 45832 43460 28 383576 0 0 716 0 247 360 0 2 48 50 0
0 1 45832 42584 28 384404 0 0 836 0 311 445 0 2 50 47 0
0 1 45832 41724 28 385416 0 0 984 0 291 443 1 3 49 48 0
0 1 45832 39736 28 386684 0 0 1236 760 322 514 0 4 49 47 0
1 1 45832 39228 28 387636 0 0 1216 0 405 358 0 5 50 44 0
0 1 45832 38616 28 388316 0 0 700 0 244 363 1 2 49 49 0
0 1 45832 37624 28 389260 0 0 984 0 273 380 0 2 51 46 0
1 1 45832 35104 28 390648 0 0 1284 0 287 448 0 2 50 48 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 45832 34972 28 392032 0 0 1448 0 224 341 0 2 50 48 0
0 1 45832 33492 28 393404 0 0 1360 0 261 362 0 2 50 47 0
0 1 45832 32912 28 394136 0 0 748 0 246 375 0 2 49 49 0
1 0 45832 31920 28 394932 0 0 776 0 299 435 0 1 52 46 0
0 1 45832 31300 28 395812 0 0 924 0 291 432 1 3 49 47 0
0 1 45832 30300 28 396776 0 0 928 0 253 399 0 1 50 48 0
0 1 45832 28904 28 398100 0 0 1336 0 235 354 0 2 49 48 0
0 1 45832 28076 28 399044 0 0 1000 0 238 374 0 2 51 47 0
2 1 45832 26828 28 400256 0 0 1136 0 242 388 0 1 50 49 0
0 1 45832 25844 28 401172 0 0 900 0 276 448 0 2 50 48 0
0 1 45832 24688 28 402488 0 0 1352 0 297 444 0 2 50 47 0
0 1 45832 23408 28 403572 0 0 1084 0 223 380 0 2 50 49 0
0 1 45832 22416 28 404608 0 0 1056 0 227 379 0 2 50 48 0
0 3 45832 21236 28 404560 0 0 28 1784 657 1243 0 11 43 47 0
0 3 45832 21332 28 404600 0 0 24 880 357 462 0 1 30 68 0
2 3 45832 21688 28 404632 0 0 16 1356 424 736 1 2 26 72 0
0 2 45832 21816 28 404676 0 0 16 1224 445 603 0 1 0 99 0
0 2 45832 21992 28 404920 0 0 248 8 400 534 0 2 14 84 0
0 1 45832 21380 28 405556 0 0 644 0 339 320 0 4 49 47 0
0 1 45832 20388 28 406624 0 0 1040 0 345 497 0 1 51 48 0
3 1 45832 19100 28 407796 0 0 1196 1388 539 573 0 10 47 43 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 45832 18148 28 408636 0 0 952 0 383 452 1 5 47 47 0
0 1 45832 16868 28 409988 0 0 1336 0 283 404 0 3 48 49 0
0 1 45832 15628 28 411224 0 0 1220 0 278 396 0 2 49 49 0
0 1 45832 14668 28 412300 0 0 1116 0 295 428 0 2 49 49 0
0 1 45832 13436 28 413496 0 0 1208 0 265 410 0 2 49 48 0
0 1 45832 12444 28 414564 0 0 1028 0 267 458 0 2 48 49 0
0 1 45832 9132 28 416432 0 0 1856 0 305 442 0 3 50 47 0
0 1 45832 8912 28 417824 0 0 1408 0 258 369 0 2 49 49 0
0 1 46028 10124 28 416172 0 196 892 196 498 419 0 4 47 48 0
0 1 46028 9880 28 417088 0 0 916 4 242 403 0 2 48 51 0
0 1 46060 10376 28 415904 0 32 1084 32 340 396 0 4 49 47 0
0 1 46060 9780 28 417012 0 0 1124 0 301 407 1 2 51 46 0
0 1 46060 9288 28 417908 0 0 856 0 260 359 1 2 49 49 0
1 1 46128 10004 28 416876 0 68 936 68 413 461 0 3 47 50 0
1 1 46652 10680 28 384712 0 524 1256 524 1645 515 4 37 31 29 0
0 1 46652 43352 28 385368 0 0 680 0 931 497 15 4 48 33 0
0 1 46652 41948 28 386740 0 0 1404 0 303 441 1 3 49 48 0
0 1 46652 41032 28 387612 0 0 832 0 296 432 0 2 48 50 0
0 1 46652 40220 28 388436 0 0 828 0 275 377 0 2 49 48 0
0 1 46652 39344 28 389408 0 0 972 0 311 422 0 2 50 47 0
0 1 46652 38276 28 390512 0 0 1144 0 280 388 0 2 50 47 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 46652 37276 28 391456 0 0 968 0 295 469 0 2 50 47 0
1 0 46652 36044 28 392576 0 0 1040 0 301 414 0 2 50 47 0
0 1 46652 35292 28 393552 0 0 988 0 326 444 1 2 50 47 0
0 1 46652 34220 28 394608 0 0 1060 0 276 397 0 2 50 48 0
0 1 46652 33060 28 395964 0 0 1356 0 268 371 1 2 50 47 0
1 0 46652 32068 28 396972 0 0 1016 0 310 472 0 2 50 48 0
1 3 46652 30024 28 398452 0 0 1480 1164 637 1023 0 7 31 62 0
0 3 46652 30024 28 398620 0 0 156 584 281 395 0 1 0 99 0
0 2 46652 29816 28 399116 0 0 552 200 319 462 0 1 0 98 0
0 1 46652 29828 28 399292 0 0 160 8 336 625 0 2 4 94 0
0 1 46652 27692 28 401204 0 0 1860 0 335 468 0 3 49 48 0
2 2 46648 25872 28 402680 32 0 1504 72 365 497 1 3 38 58 0
0 2 46648 25124 28 403512 0 0 764 56 283 404 0 2 29 68 0
0 2 46648 23564 28 404664 0 0 1064 1348 568 632 1 10 25 64 0
0 2 46648 22432 28 406328 0 0 1584 0 314 424 0 2 49 48 0
0 2 46648 19176 28 409604 0 0 3280 0 383 451 0 4 48 48 0
0 2 46648 18124 28 410640 0 0 1028 0 291 422 1 2 48 49 0
0 2 46648 17428 28 411488 0 0 884 0 342 451 1 2 49 48 0
1 1 46648 15696 28 412944 0 0 1424 0 405 436 2 3 49 46 0
0 2 46648 15288 28 413552 0 0 600 0 310 437 0 2 49 49 0
1 2 46648 13792 28 415000 0 0 1488 0 608 542 6 4 21 69 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 2 46648 11476 28 416312 0 0 1144 0 1092 545 15 7 17 60 0
1 1 46648 10112 28 417260 0 0 384 0 855 556 2 11 0 87 0
0 2 46772 10372 28 416320 0 124 312 124 1093 570 4 12 6 78 0
1 2 46772 9592 28 416740 0 0 1232 0 916 598 2 12 12 74 0
4 2 46776 9808 28 416548 0 4 4428 4 1085 692 4 16 10 70 0
0 2 46860 12468 28 411260 0 84 744 84 1203 697 7 18 1 73 0
0 2 46860 10472 28 413164 0 0 1540 0 770 643 1 9 0 90 0
0 2 46860 9280 28 413844 0 0 444 0 685 608 3 6 0 91 0
0 2 46952 9580 28 413052 0 92 732 92 1154 734 4 15 7 74 0
0 2 47004 9536 28 411804 0 52 924 52 1012 627 3 14 1 81 0
0 2 47004 9800 28 410460 0 0 1532 0 979 727 3 15 13 69 0
0 2 47008 10072 28 409348 0 4 676 4 977 609 4 10 2 84 0
0 2 47044 10320 28 407568 0 36 828 36 1058 690 4 14 8 74 0
0 2 47044 9784 28 407568 0 0 1224 0 927 613 3 11 13 72 0
1 1 47044 10772 28 405224 0 0 764 0 974 627 3 13 14 70 0
0 2 47112 9852 28 402716 0 68 756 68 1239 650 6 17 1 75 0
0 2 47280 11748 28 400732 0 168 1360 168 1257 713 4 16 1 79 0
2 0 47280 14768 28 396496 0 0 1084 4 1126 689 4 19 2 75 0
0 4 47372 16612 28 393920 0 92 616 1132 1238 930 4 18 6 72 0
2 3 47372 16796 28 394004 0 0 68 224 528 769 1 3 9 88 0
0 2 47372 15976 28 394364 0 0 240 56 503 393 1 6 2 90 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 47372 36112 28 394680 0 0 164 188 708 477 5 8 0 86 0
0 1 47372 154288 28 395300 0 0 416 0 1615 442 23 18 29 29 0
0 1 47372 153680 28 395768 0 0 388 0 521 522 1 6 49 44 0
1 0 47372 152536 28 395920 0 0 352 0 641 520 2 8 48 42 0
0 1 47372 151552 28 396788 0 0 908 0 340 520 1 2 50 46 0
0 1 47372 150100 28 397144 0 0 324 0 254 490 0 2 50 48 0
1 1 47372 148948 28 397808 0 0 688 1376 564 647 1 10 51 39 0
0 1 47372 147924 28 398504 0 0 620 0 323 502 0 3 48 49 0
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-09-28 12:18 yum upgrade on btrfs very slow Tomasz Torcz
@ 2009-09-28 13:35 ` Chris Mason
2009-09-29 6:18 ` Tomasz Torcz
0 siblings, 1 reply; 11+ messages in thread
From: Chris Mason @ 2009-09-28 13:35 UTC (permalink / raw)
To: Tomasz Torcz; +Cc: linux-btrfs
On Mon, Sep 28, 2009 at 02:18:27PM +0200, Tomasz Torcz wrote:
>
> Hi,
>
> I'm using btrfs as rootfs on my Fedora 12 (rawhide) test system.
> Every yum activity is very slow, like 15 minutes for installation of 11
> packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64.
> Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk
> on Intel ICH controller.
You're doing quite a lot of reads, and some writes. Could you please
capture the output of sysrq-w at 5s intervals during the upgrade?
-chris
>
> /dev/sda2 on / type btrfs (rw)
>
> /dev/sda2 19G 6,7G 13G 36% /
>
> Attached is output from vmstat 1 during yum transaction. I also
> have 200MB of "perf record" output, which I can drill down. First screen
> of profile is below:
>
> # Samples: 9024736
> #
> # Overhead Command Shared Object Symbol
> # ........ ............... .................................................... ......
> #
> 15.44% perf.2.6.31-33. [kernel] [k] kmem_cache_open
> 9.74% perf.2.6.31-33. [kernel] [k] print_lock_contention_bug
> 6.34% perf.2.6.31-33. [kernel] [k] free_kmem_cache_cpus
> 4.49% perf.2.6.31-33. [kernel] [k] is_module_text_address
> 4.01% perf.2.6.31-33. [kernel] [k] lock_set_class
> 3.45% yum-complete-tr [kernel] [k] print_lock_contention_bug
> 2.50% perf.2.6.31-33. [kernel] [k] _spin_lock
> 2.48% yum-complete-tr [kernel] [k] kmem_cache_open
> 2.47% perf.2.6.31-33. [kernel] [k] zlib_tr_flush_block [zlib_deflate]
> 1.65% yum-complete-tr [kernel] [k] lock_set_class
> 1.62% perf.2.6.31-33. [kernel] [k] schedule_hrtimeout_range
> 1.53% perf.2.6.31-33. [kernel] [k] sysfs_slab_add
> 1.50% yum-complete-tr [kernel] [k] zlib_tr_flush_block [zlib_deflate]
> 1.46% yum-complete-tr [kernel] [k] free_kmem_cache_cpus
> 1.40% yum-complete-tr [kernel] [k] _spin_lock
> 1.34% perf.2.6.31-33. [kernel] [k] __mutex_lock_common
> 1.27% perf.2.6.31-33. [kernel] [k] btrfs_getxattr [btrfs]
> 1.06% perf.2.6.31-33. [kernel] [k] lock_contended
> 0.78% perf.2.6.31-33. [kernel] [k] _raw_read_unlock
> 0.75% perf.2.6.31-33. [kernel] [k] btrfs_listxattr [btrfs]
> 0.74% perf.2.6.31-33. [kernel] [k] run_scheduled_bios [btrfs]
> 0.62% perf.2.6.31-33. [kernel] [k] lockdep_count_backward_deps
> 0.61% perf.2.6.31-33. [kernel] [k] show_slab_objects
> 0.58% yum-complete-tr [kernel] [k] sysfs_slab_add
> 0.48% yum-complete-tr /usr/lib64/libpython2.6.so.1.0 [.] 0x0000000005dab0
> 0.46% yum-complete-tr [kernel] [k] btrfs_listxattr [btrfs]
> 0.45% yum-complete-tr /usr/lib64/librpm.so.0.0.0 [.] 0x0000000004877f
> 0.44% perf.2.6.31-33. [kernel] [k] idr_for_each
> 0.44% perf.2.6.31-33. [kernel] [k] clear_lock_stats
> 0.44% yum-complete-tr [kernel] [k] lock_contended
> 0.43% perf.2.6.31-33. [kernel] [k] btrfs_iget [btrfs]
> 0.41% yum-complete-tr [kernel] [k] is_module_text_address
> 0.37% yum-complete-tr /usr/lib64/libpython2.6.so.1.0 [.] PyEval_EvalFrameEx
> 0.36% yum-complete-tr [kernel] [k] run_scheduled_bios [btrfs]
> 0.36% yum-complete-tr [kernel] [k] show_slab_objects
> 0.33% perf.2.6.31-33. [kernel] [k] lock_acquire
> 0.33% perf.2.6.31-33. [kernel] [k] __module_text_address
> 0.29% yum-complete-tr [kernel] [k] btrfs_getxattr [btrfs]
> 0.28% ldconfig [kernel] [k] free_kmem_cache_cpus
> 0.27% perf.2.6.31-33. [kernel] [k] btrfs_set_device_sector_size [btrfs]
> 0.26% yum-complete-tr [kernel] [k] _raw_read_unlock
> 0.25% perf.2.6.31-33. [kernel] [k] _spin_lock_bh
> 0.25% yum-complete-tr [kernel] [k] btrfs_drop_extents [btrfs]
> 0.23% perf.2.6.31-33. [kernel] [k] btrfs_comp_cpu_keys [btrfs]
> 0.23% yum-complete-tr /lib64/libc-2.10.90.so [.] __GI_memcpy
>
>
>
> --
> Tomasz Torcz 72->| 80->|
> xmpp: zdzichubg@chrome.pl 72->| 80->|
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-09-28 13:35 ` Chris Mason
@ 2009-09-29 6:18 ` Tomasz Torcz
2009-09-29 13:25 ` Chris Mason
0 siblings, 1 reply; 11+ messages in thread
From: Tomasz Torcz @ 2009-09-29 6:18 UTC (permalink / raw)
To: linux-btrfs
On Mon, Sep 28, 2009 at 09:35:43AM -0400, Chris Mason wrote:
> On Mon, Sep 28, 2009 at 02:18:27PM +0200, Tomasz Torcz wrote:
> >
> > Hi,
> >
> > I'm using btrfs as rootfs on my Fedora 12 (rawhide) test system.
> > Every yum activity is very slow, like 15 minutes for installation of 11
> > packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64.
> > Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk
> > on Intel ICH controller.
>
> You're doing quite a lot of reads, and some writes. Could you please
> capture the output of sysrq-w at 5s intervals during the upgrade?
It got quite big (over 100 KiB, over 1 MiB after unpacking), so I
put it at http://pipebreaker.pl/dump/sysrq-w_every_5_sec.txt.bz2
--
Tomasz Torcz Only gods can safely risk perfection,
xmpp: zdzichubg@chrome.pl it's a dangerous thing for a man. -- Alia
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-09-29 6:18 ` Tomasz Torcz
@ 2009-09-29 13:25 ` Chris Mason
2009-09-30 19:43 ` Tomasz Torcz
0 siblings, 1 reply; 11+ messages in thread
From: Chris Mason @ 2009-09-29 13:25 UTC (permalink / raw)
To: Tomasz Torcz; +Cc: linux-btrfs
On Tue, Sep 29, 2009 at 08:18:22AM +0200, Tomasz Torcz wrote:
> On Mon, Sep 28, 2009 at 09:35:43AM -0400, Chris Mason wrote:
> > On Mon, Sep 28, 2009 at 02:18:27PM +0200, Tomasz Torcz wrote:
> > >
> > > Hi,
> > >
> > > I'm using btrfs as rootfs on my Fedora 12 (rawhide) test system.
> > > Every yum activity is very slow, like 15 minutes for installation of 11
> > > packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64.
> > > Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk
> > > on Intel ICH controller.
> >
> > You're doing quite a lot of reads, and some writes. Could you please
> > capture the output of sysrq-w at 5s intervals during the upgrade?
>
> It got quite big (over 100 KiB, over 1 MiB after unpacking), so I
> put it at http://pipebreaker.pl/dump/sysrq-w_every_5_sec.txt.bz2
Ok, you've got 158 sysrq-w runs in here, and 119 of them involve fsync.
This is good because it is what Josef expected the problem to be.
The first thing I would try is mount -o ssd. I think what is happening
is that we are seeking around while fsyncing files. mount -o ssd isn't
really the right long term answer but it will tell us if I've got it
right.
If that works, I'd try the code in the master branch of btrfs-unstable.
This has a change that makes the allocator a little smarter, even
without mount -o ssd.
If you're comparing w/ext3 and wondering why btrfs is sooooooo much
slower it might be because btrfs has barriers on by default and ext3
doesn't. You could mount -o nobarrier for btrfs or mount -o barrier=1
for ext3 for a proper comparison.
(Assuming your dmesg doesn't have messages from btrfs about disabling
barriers).
-chris
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-09-29 13:25 ` Chris Mason
@ 2009-09-30 19:43 ` Tomasz Torcz
2009-10-01 9:01 ` Tomasz Torcz
2009-10-01 9:04 ` Jens Axboe
0 siblings, 2 replies; 11+ messages in thread
From: Tomasz Torcz @ 2009-09-30 19:43 UTC (permalink / raw)
To: linux-btrfs
On Tue, Sep 29, 2009 at 09:25:38AM -0400, Chris Mason wrote:
> On Tue, Sep 29, 2009 at 08:18:22AM +0200, Tomasz Torcz wrote:
> > On Mon, Sep 28, 2009 at 09:35:43AM -0400, Chris Mason wrote:
> > > > Every yum activity is very slow, like 15 minutes for installation of 11
> > > > packages 25MB in size. Kernel is 2.6.31.1-48.fc12.x86_64, btrfs-progs-0.19-7.fc12.x86_64.
> > > > Hardware is pentium 4 3.0 GHz (Hyperthreading, 64 bit), with single IDE disk
> > > > on Intel ICH controller.
> > >
> > > You're doing quite a lot of reads, and some writes. Could you please
> > > capture the output of sysrq-w at 5s intervals during the upgrade?
> >
> > It got quite big (over 100 KiB, over 1 MiB after unpacking), so I
> > put it at http://pipebreaker.pl/dump/sysrq-w_every_5_sec.txt.bz2
>
> Ok, you've got 158 sysrq-w runs in here, and 119 of them involve fsync.
> This is good because it is what Josef expected the problem to be.
>
> The first thing I would try is mount -o ssd. I think what is happening
> is that we are seeking around while fsyncing files. mount -o ssd isn't
> really the right long term answer but it will tell us if I've got it
> right.
Tried that. Not noticable faster. Dumps are here:
http://pipebreaker.pl/dump/sysrq_w-with_ssd.txt.bz2
http://pipebreaker.pl/dump/vmstat-with.ssd.txt.bz2
>
> If you're comparing w/ext3 and wondering why btrfs is sooooooo much
> slower it might be because btrfs has barriers on by default and ext3
> doesn't. You could mount -o nobarrier for btrfs or mount -o barrier=1
> for ext3 for a proper comparison.
>
> (Assuming your dmesg doesn't have messages from btrfs about disabling
> barriers).
I wouldn't expect barriers to work here (reminder, this is PATA drive
on ICH7 sata controller), but I will test tomorrow with nobarrier.
Then I probably check his "yum upgrade" under seekwatcher on friday.
--
Tomasz Torcz Morality must always be based on practicality.
xmpp: zdzichubg@chrome.pl -- Baron Vladimir Harkonnen
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-09-30 19:43 ` Tomasz Torcz
@ 2009-10-01 9:01 ` Tomasz Torcz
2009-10-21 9:27 ` Tomasz Torcz
2009-10-01 9:04 ` Jens Axboe
1 sibling, 1 reply; 11+ messages in thread
From: Tomasz Torcz @ 2009-10-01 9:01 UTC (permalink / raw)
To: linux-btrfs
On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote:
> >
> > If you're comparing w/ext3 and wondering why btrfs is sooooooo much
> > slower it might be because btrfs has barriers on by default and ext3
> > doesn't. You could mount -o nobarrier for btrfs or mount -o barrier=1
> > for ext3 for a proper comparison.
> >
> > (Assuming your dmesg doesn't have messages from btrfs about disabling
> > barriers).
>
> I wouldn't expect barriers to work here (reminder, this is PATA drive
> on ICH7 sata controller), but I will test tomorrow with nobarrier.
> Then I probably check his "yum upgrade" under seekwatcher on friday.
Nobarrier result are here:
http://pipebreaker.pl/dump/sysrq_w-nobarrier.txt.bz2
http://pipebreaker.pl/dump/vmstat-nobarrier.txt.bz2
My observation: still slow. Just a note on my unscientific methodology:
I've got two very similar machines with Rawhide, one on ext4, second
on btrfs. Every day I do "yum upgrade" on both. Ext4 one finishes much
faster thatn btrfs. This is especially visible while yum is in "cleanup phase".
On ext4 most packages are cleaned'up in under one second time. On btrfs
this takes few seconds.
Whole transaction ends in couple of minutes on ext4. On btrfs it takes 5x
longer.
--
Tomasz Torcz RIP is irrevelant. Spoofing is futile.
xmpp: zdzichubg@chrome.pl Your routes will be aggreggated. -- Alex Yuriev
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-09-30 19:43 ` Tomasz Torcz
2009-10-01 9:01 ` Tomasz Torcz
@ 2009-10-01 9:04 ` Jens Axboe
2009-10-05 12:09 ` Ric Wheeler
1 sibling, 1 reply; 11+ messages in thread
From: Jens Axboe @ 2009-10-01 9:04 UTC (permalink / raw)
To: Tomasz Torcz; +Cc: linux-btrfs
On Wed, Sep 30 2009, Tomasz Torcz wrote:
> I wouldn't expect barriers to work here (reminder, this is PATA drive
> on ICH7 sata controller), but I will test tomorrow with nobarrier.
> Then I probably check his "yum upgrade" under seekwatcher on friday.
Barriers should work fine on that setup. In fact, barriers were first
implemented for PATA back in the day :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-10-01 9:04 ` Jens Axboe
@ 2009-10-05 12:09 ` Ric Wheeler
0 siblings, 0 replies; 11+ messages in thread
From: Ric Wheeler @ 2009-10-05 12:09 UTC (permalink / raw)
To: Jens Axboe; +Cc: Tomasz Torcz, linux-btrfs
On 10/01/2009 05:04 AM, Jens Axboe wrote:
> On Wed, Sep 30 2009, Tomasz Torcz wrote:
>
>> I wouldn't expect barriers to work here (reminder, this is PATA drive
>> on ICH7 sata controller), but I will test tomorrow with nobarrier.
>> Then I probably check his "yum upgrade" under seekwatcher on friday.
>>
> Barriers should work fine on that setup. In fact, barriers were first
> implemented for PATA back in the day :-)
>
>
I think that we first tested them on ich5 & old promise controllers with
old (250GB) P-ATA disks :-)
ric
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-10-01 9:01 ` Tomasz Torcz
@ 2009-10-21 9:27 ` Tomasz Torcz
2009-10-26 8:55 ` Chris Mason
0 siblings, 1 reply; 11+ messages in thread
From: Tomasz Torcz @ 2009-10-21 9:27 UTC (permalink / raw)
To: linux-btrfs
On Thu, Oct 01, 2009 at 11:01:46AM +0200, Tomasz Torcz wrote:
> On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote:
> > >
> > > If you're comparing w/ext3 and wondering why btrfs is sooooooo much
> > > slower it might be because btrfs has barriers on by default and ext3
> > > doesn't. You could mount -o nobarrier for btrfs or mount -o barrier=1
> > > for ext3 for a proper comparison.
Last update: things got noticably faster with 2.6.31.4-88.fc12.x86_64,
which contains backport of latest mainline btrfs (thank you, Josef!).
No more comlains from me (for now ;).
--
Tomasz Torcz Morality must always be based on practicality.
xmpp: zdzichubg@chrome.pl -- Baron Vladimir Harkonnen
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-10-21 9:27 ` Tomasz Torcz
@ 2009-10-26 8:55 ` Chris Mason
2009-10-26 13:35 ` Tomasz Torcz
0 siblings, 1 reply; 11+ messages in thread
From: Chris Mason @ 2009-10-26 8:55 UTC (permalink / raw)
To: Tomasz Torcz, linux-btrfs
On Wed, Oct 21, 2009 at 11:27:33AM +0200, Tomasz Torcz wrote:
> On Thu, Oct 01, 2009 at 11:01:46AM +0200, Tomasz Torcz wrote:
> > On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote:
> > > >
> > > > If you're comparing w/ext3 and wondering why btrfs is sooooooo much
> > > > slower it might be because btrfs has barriers on by default and ext3
> > > > doesn't. You could mount -o nobarrier for btrfs or mount -o barrier=1
> > > > for ext3 for a proper comparison.
>
> Last update: things got noticably faster with 2.6.31.4-88.fc12.x86_64,
> which contains backport of latest mainline btrfs (thank you, Josef!).
> No more comlains from me (for now ;).
Fantastic. Are you running with or without barriers now?
-chris
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: yum upgrade on btrfs very slow
2009-10-26 8:55 ` Chris Mason
@ 2009-10-26 13:35 ` Tomasz Torcz
0 siblings, 0 replies; 11+ messages in thread
From: Tomasz Torcz @ 2009-10-26 13:35 UTC (permalink / raw)
To: linux-btrfs; +Cc: Chris Mason
On Mon, Oct 26, 2009 at 05:55:45PM +0900, Chris Mason wrote:
> On Wed, Oct 21, 2009 at 11:27:33AM +0200, Tomasz Torcz wrote:
> > On Thu, Oct 01, 2009 at 11:01:46AM +0200, Tomasz Torcz wrote:
> > > On Wed, Sep 30, 2009 at 09:43:51PM +0200, Tomasz Torcz wrote:
> > > > >
> > > > > If you're comparing w/ext3 and wondering why btrfs is sooooooo much
> > > > > slower it might be because btrfs has barriers on by default and ext3
> > > > > doesn't. You could mount -o nobarrier for btrfs or mount -o barrier=1
> > > > > for ext3 for a proper comparison.
> >
> > Last update: things got noticably faster with 2.6.31.4-88.fc12.x86_64,
> > which contains backport of latest mainline btrfs (thank you, Josef!).
> > No more comlains from me (for now ;).
>
> Fantastic. Are you running with or without barriers now?
I stick to default settings, that means barriers I think. Overall, my
test hardware is so slow by itself, that barriers are not really noticable.
(It's 3 GHz Pentium 4 in 64 bit with single IDE 20GB drive).
--
Tomasz Torcz RIP is irrevelant. Spoofing is futile.
xmpp: zdzichubg@chrome.pl Your routes will be aggreggated. -- Alex Yuriev
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2009-10-26 13:35 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-09-28 12:18 yum upgrade on btrfs very slow Tomasz Torcz
2009-09-28 13:35 ` Chris Mason
2009-09-29 6:18 ` Tomasz Torcz
2009-09-29 13:25 ` Chris Mason
2009-09-30 19:43 ` Tomasz Torcz
2009-10-01 9:01 ` Tomasz Torcz
2009-10-21 9:27 ` Tomasz Torcz
2009-10-26 8:55 ` Chris Mason
2009-10-26 13:35 ` Tomasz Torcz
2009-10-01 9:04 ` Jens Axboe
2009-10-05 12:09 ` Ric Wheeler
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.