btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Marc MERLIN]

My server died last night during a btrfs send/receive to a btrfs radi5 array

Here are the logs. Is this anything known or with a possible workaround?

Thanks,
Marc

btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000
 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffff88007ff7ce00
 0000000000000000 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff81014650>] ? dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff812f9fa2>] ? blkg_path.isra.80.constprop.90+0x17/0x38
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 171
CPU    1: hi:  186, btch:  31 usd: 190
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:94189 isolated_file:32
 unevictable:1260 dirty:38914 writeback:49596 unstable:0
 free:15999 slab_reclaimable:8198 slab_unreclaimable:9741
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8084kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:2040kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56080kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:374588kB unevictable:4940kB isolated(anon):0kB isolated(file):128kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197112kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38628kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 85*4kB (UEM) 22*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8100kB
Node 0 DMA32: 13004*4kB (M) 16*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56240kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
164139 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d690 ffffffff816090b3 0000000000000000
 ffff88000549d718 ffffffff811037b0 00000001fffffffe ffff88000549d6c8
 ffffffff8160e9c4 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff8160e9c4>] ? _raw_spin_trylock+0x20/0x50
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff81093ce9>] ? console_unlock+0x2f6/0x302
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff811063d7>] ? get_page_from_freelist+0x549/0x71d
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812fc16d>] cfq_insert_request+0x391/0x3b5
 [<ffffffff812e002f>] ? perf_trace_block_rq_with_error+0x45/0x14f
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812dc08c>] __elv_add_request+0x1fc/0x276
 [<ffffffff812e1c6c>] blk_queue_bio+0x237/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 179
CPU    1: hi:  186, btch:  31 usd: 213
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:93709 isolated_file:32
 unevictable:1260 dirty:38914 writeback:49627 unstable:0
 free:16338 slab_reclaimable:8198 slab_unreclaimable:9741
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8084kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:2040kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:57352kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:372680kB unevictable:4940kB isolated(anon):0kB isolated(file):116kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38628kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 85*4kB (UEM) 22*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8100kB
Node 0 DMA32: 13283*4kB (M) 16*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 57356kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163658 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000
 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffff880000000000
 0000000000000046 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff811063d7>] ? get_page_from_freelist+0x549/0x71d
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 169
CPU    1: hi:  186, btch:  31 usd: 185
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:93277 isolated_file:32
 unevictable:1260 dirty:38914 writeback:49627 unstable:0
 free:16250 slab_reclaimable:8198 slab_unreclaimable:9762
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8080kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:1980kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56920kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:371128kB unevictable:4940kB isolated(anon):0kB isolated(file):128kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38712kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 86*4kB (UEM) 19*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8080kB
Node 0 DMA32: 13210*4kB (M) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56984kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163317 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000
 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffffffff00000029
 ffffffff81e8fe40 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 169
CPU    1: hi:  186, btch:  31 usd: 185
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:93298 isolated_file:0
 unevictable:1260 dirty:38914 writeback:49627 unstable:0
 free:16250 slab_reclaimable:8198 slab_unreclaimable:9762
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8080kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:1980kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56920kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:371212kB unevictable:4940kB isolated(anon):0kB isolated(file):0kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38712kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 86*4kB (UEM) 19*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8080kB
Node 0 DMA32: 13210*4kB (M) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56984kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163317 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned


-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Chris Mason]

On 03/19/2014 11:45 AM, Marc MERLIN wrote:
> My server died last night during a btrfs send/receive to a btrfs radi5 array
>
> Here are the logs. Is this anything known or with a possible workaround?
>
> Thanks,
> Marc
>
> btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

This is an order 1 atomic allocation from the mvs driver, we really 
should not be depending on that to get IO done.  A quick search and it 
looks like we're allocating MVS_SLOT_BUF_SZ (8192) bytes.

You could try bumping the lowmem reserves.

-chris

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Marc MERLIN]

On Wed, Mar 19, 2014 at 12:20:08PM -0400, Chris Mason wrote:
> On 03/19/2014 11:45 AM, Marc MERLIN wrote:
> >My server died last night during a btrfs send/receive to a btrfs radi5 
> >array
> >
> >Here are the logs. Is this anything known or with a possible workaround?
> >
> >Thanks,
> >Marc
> >
> >btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
> 
> This is an order 1 atomic allocation from the mvs driver, we really 
> should not be depending on that to get IO done.  A quick search and it 
> looks like we're allocating MVS_SLOT_BUF_SZ (8192) bytes.
> 
> You could try bumping the lowmem reserves.

Thanks for the info.

So for now, I have
CONFIG_X86_RESERVE_LOW=64

This is the option we're talking about, right?

Should I double it?

For now, I have the copy running again, and it's been going for 8 hours
without failure on the old kernel but of course that doesn't mean my 2TB
copy will complete without hitting the bug again.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Chris Mason]

On 3/19/14, 6:37 PM, "Marc MERLIN" <marc@merlins.org> wrote:

>On Wed, Mar 19, 2014 at 12:20:08PM -0400, Chris Mason wrote:
>> On 03/19/2014 11:45 AM, Marc MERLIN wrote:
>> >My server died last night during a btrfs send/receive to a btrfs radi5
>> >array
>> >
>> >Here are the logs. Is this anything known or with a possible
>>workaround?
>> >
>> >Thanks,
>> >Marc
>> >
>> >btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
>> 
>> This is an order 1 atomic allocation from the mvs driver, we really
>> should not be depending on that to get IO done.  A quick search and it
>> looks like we're allocating MVS_SLOT_BUF_SZ (8192) bytes.
>> 
>> You could try bumping the lowmem reserves.
>
>Thanks for the info.
>
>So for now, I have
>CONFIG_X86_RESERVE_LOW=64
>
>This is the option we're talking about, right?
>
>Should I double it?
>
>For now, I have the copy running again, and it's been going for 8 hours
>without failure on the old kernel but of course that doesn't mean my 2TB
>copy will complete without hitting the bug again.

Sorry, I misspoke, you should bump /proc/sys/vm/min_free_kbytes.  Honestly
though, it¹s just a bug in the mvs driver.  Atomic 8K allocations are
doomed to fail eventually.

The driver should either busy loop until the allocation completes (really
not a great choice), gracefully deal with the failure (looks tricky), or
preallocate the space (like the rest of the block layer).

-chris

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Marc MERLIN]

On Thu, Mar 20, 2014 at 12:13:36AM +0000, Chris Mason wrote:
> >Should I double it?
> >
> >For now, I have the copy running again, and it's been going for 8 hours
> >without failure on the old kernel but of course that doesn't mean my 2TB
> >copy will complete without hitting the bug again.
> 
> Sorry, I misspoke, you should bump /proc/sys/vm/min_free_kbytes.  Honestly
> though, it¹s just a bug in the mvs driver.  Atomic 8K allocations are
> doomed to fail eventually.

Gotcha
polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes
45056
polgara:/mnt/btrfs_backupcopy# echo 100000 > /proc/sys/vm/min_free_kbytes
polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes
100000
polgara:/mnt/btrfs_backupcopy# 
 
> The driver should either busy loop until the allocation completes (really
> not a great choice), gracefully deal with the failure (looks tricky), or
> preallocate the space (like the rest of the block layer).

Gotcha. I'll report this to the folks maintaining the marvel driver.

So just to make sure I got you right, although the page allocation failure
was shown in btrfs, it's really the underlying marvel driver at fault here,
and there isn't really anything to change on the btrfs side, correct?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Chris Mason]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="euc-kr", Size: 1785 bytes --]



On 3/19/14, 8:20 PM, "Marc MERLIN" <marc@merlins.org> wrote:

>On Thu, Mar 20, 2014 at 12:13:36AM +0000, Chris Mason wrote:
>> >Should I double it?
>> >
>> >For now, I have the copy running again, and it's been going for 8 hours
>> >without failure on the old kernel but of course that doesn't mean my
>>2TB
>> >copy will complete without hitting the bug again.
>> 
>> Sorry, I misspoke, you should bump /proc/sys/vm/min_free_kbytes.
>>Honestly
>> though, it©ös just a bug in the mvs driver.  Atomic 8K allocations are
>> doomed to fail eventually.
>
>Gotcha
>polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes
>45056
>polgara:/mnt/btrfs_backupcopy# echo 100000 > /proc/sys/vm/min_free_kbytes
>polgara:/mnt/btrfs_backupcopy# cat /proc/sys/vm/min_free_kbytes
>100000
>polgara:/mnt/btrfs_backupcopy#
> 
>> The driver should either busy loop until the allocation completes
>>(really
>> not a great choice), gracefully deal with the failure (looks tricky), or
>> preallocate the space (like the rest of the block layer).
>
>Gotcha. I'll report this to the folks maintaining the marvel driver.
>
>So just to make sure I got you right, although the page allocation failure
>was shown in btrfs, it's really the underlying marvel driver at fault
>here,
>and there isn't really anything to change on the btrfs side, correct?

The process is a btrfs worker, and the IO was started by btrfs, but the
allocation failure is all inside the mvs driver.  There¡¯s even the printk
in there from mvs about the allocation failing.

The only reason it¡¯s btrfs instead of a regular process is because for
raid5/6 the rmw is farmed out to helper threads.

-chris

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Duncan]

Chris Mason posted on Thu, 20 Mar 2014 01:01:35 +0000 as excerpted:

> Message-ID: <CF4FB5F3.2116%clm@fb.com>
> Accept-Language: en-US
> Content-Type: text/plain; charset=euc-kr
> Content-Transfer-Encoding: base64
> Content-Language: en-US

>>> Sorry, I misspoke, you should bump /proc/sys/vm/min_free_kbytes.
>>> Honestly though, it©ös just a bug in the mvs driver.  Atomic 8K
>>> allocations are doomed to fail eventually.

> The process is a btrfs worker, and the IO was started by btrfs, but the
> allocation failure is all inside the mvs driver.  There¡¯s even the
> printk in there from mvs about the allocation failing.

> -chris
> 
> N‹§²æìrž›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±nÚß²)í…æèw*\x1fjg¬±š\x1e¶‰šŽŠÝ¢
j/êäz¹Þ–Šà2ŠÞ™šè­Ú&¢)ß¡«a¶Ú\x7fþø\x1e®G«éh®\x0fæj:+v‰šŠwè†Ù¥


Chris, you might wish to take a look at this and/or have one of the FB 
techs familiar with your mail transport layers (and/or your mail client, 
and/or perhaps it's vger's list-serv bot) look at it.  That list-sig came 
thru as garbage at least here, and your "it's" and "there's" appear to 
have strange apostrophes as well.  As you can see I also included the 
headers I think might be relevant, plus the Message-ID.

The problem wasn't a big one here, but if it starts corrupting patches or 
something too...

I'd guess it has something to do with that content-type charset=euc-kr in 
the headers, which looks really strange combined with the accept-language 
and content-language both being en-US.

The other messages of yours (with an uncorrupted list sig) I checked had 
a content-type charset=iso-8859-1 header, which seems to be most common 
in English messages, anyway.  Why this one had euc-kr I don't know.

However, it's also worth noting that the third-level quoting above, also 
from you, has an "it's", and when I checked /that/ message, the 
apostrophe was a super-script-1, as it was when quoted in Marc's reply, 
but that got changed to the copyright symbol (plus something else) in 
your quote, which at least as I'm posting the quote, is showing the same 
way.

Meanwhile, the apostrophe in the "there's" in your message's new content 
(which is thus first-level quote in this message) is different, an i 
followed by an over-line.  I'd /guess/ that's what actually triggered the 
Korean (?) charset in your message, in ordered to handle that or perhaps 
some other character I missed, while your other messages are iso-8859-1, 
which thus changed the auto-inserted list-sig into garbage, since the 
list-bot presumably inserted it as the usual iso-8859-1 while your 
message claimed to be in eur-kr.

(It'll be interesting to see if my message, with both quotes, looks the 
same when I read it on the list via gmane, as it does when I send it, or 
if it's further garbled, and if my charset gets set to something exotic 
too.  FWIW, my client, which should be visible in my headers, is pan, via 
gmane.org's list2news service.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Chris Mason]

On 03/20/2014 02:19 AM, Duncan wrote:
> Chris Mason posted on Thu, 20 Mar 2014 01:01:35 +0000 as excerpted:
>
>> Message-ID: <CF4FB5F3.2116%clm@fb.com>
>> Accept-Language: en-US
>> Content-Type: text/plain; charset=euc-kr
>> Content-Transfer-Encoding: base64
>> Content-Language: en-US
>
>>>> Sorry, I misspoke, you should bump /proc/sys/vm/min_free_kbytes.
>>>> Honestly though, it©ös just a bug in the mvs driver.  Atomic 8K
>>>> allocations are doomed to fail eventually.
>
>> The process is a btrfs worker, and the IO was started by btrfs, but the
>> allocation failure is all inside the mvs driver.  There¡¯s even the
>> printk in there from mvs about the allocation failing.
>
>> -chris
>>
>> N‹§²æìrž›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±nÚß²)í…æèw*\x1fjg¬±š\x1e¶‰šŽŠÝ¢
> j/êäz¹Þ–Šà2ŠÞ™šè­Ú&¢)ß¡«a¶Ú\x7fþø\x1e®G«éh®\x0fæj:+v‰šŠwè†Ù¥
>
>
> Chris, you might wish to take a look at this and/or have one of the FB
> techs familiar with your mail transport layers (and/or your mail client,
> and/or perhaps it's vger's list-serv bot) look at it.  That list-sig came
> thru as garbage at least here, and your "it's" and "there's" appear to
> have strange apostrophes as well.  As you can see I also included the
> headers I think might be relevant, plus the Message-ID.
>

So the list-sig is a vger feature.  It just tacks on the list sig 
regardless of the encoding of the message, which generates garbage some 
times.

The rest sounds like a facebook feature.  I've got thunderbird 
configured to force saner encodings, hopefully it'll be more consistent now.

Thanks

-chris

Re: btrfs-rmw-2: page allocation failure: order:1, mode:0x8020

[Marc MERLIN]

+linux-kernel since I got no answer.

Hi,

I see you are maintainers of/contributors to drivers/scsi/mvsas

The btrfs folks pointed out that the problem below is due to the MVS driver,
namely:
----------------------------------------------------------------------------
From: Chris Mason <clm@fb.com>

This is an order 1 atomic allocation from the mvs driver, we really
should not be depending on that to get IO done.  A quick search and it
looks like we're allocating MVS_SLOT_BUF_SZ (8192) bytes.

You could try bumping the lowmem reserves.

-chris
----------------------------------------------------------------------------

Would you be able to modify the driver to avoid these low memory problems?

Thanks,
Marc


----- Forwarded message from Marc MERLIN <marc@merlins.org> -----

From: Marc MERLIN <marc@merlins.org>
To: linux-btrfs@vger.kernel.org

My server died last night during a btrfs send/receive to a btrfs radi5 array

Here are the logs. Is this anything known or with a possible workaround?

Thanks,
Marc

btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000
 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffff88007ff7ce00
 0000000000000000 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff81014650>] ? dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff812f9fa2>] ? blkg_path.isra.80.constprop.90+0x17/0x38
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 171
CPU    1: hi:  186, btch:  31 usd: 190
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:94189 isolated_file:32
 unevictable:1260 dirty:38914 writeback:49596 unstable:0
 free:15999 slab_reclaimable:8198 slab_unreclaimable:9741
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8084kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:2040kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56080kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:374588kB unevictable:4940kB isolated(anon):0kB isolated(file):128kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197112kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38628kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 85*4kB (UEM) 22*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8100kB
Node 0 DMA32: 13004*4kB (M) 16*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56240kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
164139 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d690 ffffffff816090b3 0000000000000000
 ffff88000549d718 ffffffff811037b0 00000001fffffffe ffff88000549d6c8
 ffffffff8160e9c4 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff8160e9c4>] ? _raw_spin_trylock+0x20/0x50
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff81093ce9>] ? console_unlock+0x2f6/0x302
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff811063d7>] ? get_page_from_freelist+0x549/0x71d
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812fc16d>] cfq_insert_request+0x391/0x3b5
 [<ffffffff812e002f>] ? perf_trace_block_rq_with_error+0x45/0x14f
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812dc08c>] __elv_add_request+0x1fc/0x276
 [<ffffffff812e1c6c>] blk_queue_bio+0x237/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 179
CPU    1: hi:  186, btch:  31 usd: 213
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:93709 isolated_file:32
 unevictable:1260 dirty:38914 writeback:49627 unstable:0
 free:16338 slab_reclaimable:8198 slab_unreclaimable:9741
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8084kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:2040kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:57352kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:372680kB unevictable:4940kB isolated(anon):0kB isolated(file):116kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38628kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:29 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 85*4kB (UEM) 22*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8100kB
Node 0 DMA32: 13283*4kB (M) 16*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 57356kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163658 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000
 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffff880000000000
 0000000000000046 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff811063d7>] ? get_page_from_freelist+0x549/0x71d
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 169
CPU    1: hi:  186, btch:  31 usd: 185
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:93277 isolated_file:32
 unevictable:1260 dirty:38914 writeback:49627 unstable:0
 free:16250 slab_reclaimable:8198 slab_unreclaimable:9762
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8080kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:1980kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56920kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:371128kB unevictable:4940kB isolated(anon):0kB isolated(file):128kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38712kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 86*4kB (UEM) 19*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8080kB
Node 0 DMA32: 13210*4kB (M) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56984kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163317 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned
mvsas 0000:01:00.0: mvsas prep failed[0]!
btrfs-rmw-2: page allocation failure: order:1, mode:0x8020
CPU: 1 PID: 12499 Comm: btrfs-rmw-2 Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
Hardware name: System manufacturer P5KC/P5KC, BIOS 0502    05/24/2007
 0000000000000000 ffff88000549d780 ffffffff816090b3 0000000000000000
 ffff88000549d808 ffffffff811037b0 00000001fffffffe ffffffff00000029
 ffffffff81e8fe40 0000000200000000 0000003000000000 ffff88007ff7ce00
Call Trace:
 [<ffffffff816090b3>] dump_stack+0x4e/0x7a
 [<ffffffff811037b0>] warn_alloc_failed+0x111/0x125
 [<ffffffff81106cb2>] __alloc_pages_nodemask+0x707/0x854
 [<ffffffff81014650>] dma_generic_alloc_coherent+0xa7/0x11c
 [<ffffffff811354e8>] dma_pool_alloc+0x10a/0x1cb
 [<ffffffffa0084aa0>] mvs_task_prep+0x192/0xa42 [mvsas]
 [<ffffffff8114069b>] ? ____cache_alloc+0x1c/0x29b
 [<ffffffffa00853ad>] mvs_task_exec.isra.9+0x5d/0xc9 [mvsas]
 [<ffffffffa0085a76>] mvs_queue_command+0x3d/0x29b [mvsas]
 [<ffffffff8114118d>] ? kmem_cache_alloc+0xe3/0x161
 [<ffffffffa0064d1c>] sas_ata_qc_issue+0x1cd/0x235 [libsas]
 [<ffffffff814a9598>] ata_qc_issue+0x291/0x2f1
 [<ffffffff814af413>] ? ata_scsiop_mode_sense+0x29c/0x29c
 [<ffffffff814b049e>] __ata_scsi_queuecmd+0x184/0x1e0
 [<ffffffff814b05a5>] ata_sas_queuecmd+0x31/0x4d
 [<ffffffffa00637ba>] sas_queuecommand+0x98/0x1fe [libsas]
 [<ffffffff8148fdee>] scsi_dispatch_cmd+0x14f/0x22e
 [<ffffffff814964da>] scsi_request_fn+0x4da/0x507
 [<ffffffff812e512c>] ? blk_recount_segments+0x1e/0x2e
 [<ffffffff812e01a3>] __blk_run_queue_uncond+0x22/0x2b
 [<ffffffff812e01c5>] __blk_run_queue+0x19/0x1b
 [<ffffffff812e1c74>] blk_queue_bio+0x23f/0x256
 [<ffffffff812df92c>] generic_make_request+0x9c/0xdb
 [<ffffffff812dfa7d>] submit_bio+0x112/0x131
 [<ffffffff8128274c>] rmw_work+0x112/0x162
 [<ffffffff8125073f>] worker_loop+0x168/0x4d8
 [<ffffffff812505d7>] ? btrfs_queue_worker+0x283/0x283
 [<ffffffff8106bc56>] kthread+0xae/0xb6
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
 [<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 169
CPU    1: hi:  186, btch:  31 usd: 185
active_anon:17298 inactive_anon:21061 isolated_anon:0
 active_file:67491 inactive_file:93298 isolated_file:0
 unevictable:1260 dirty:38914 writeback:49627 unstable:0
 free:16250 slab_reclaimable:8198 slab_unreclaimable:9762
 mapped:12981 shmem:1661 pagetables:2711 bounce:0
 free_cma:0
Node 0 DMA free:8080kB min:348kB low:432kB high:520kB active_anon:360kB inactive_anon:764kB active_file:288kB inactive_file:1980kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:100kB dirty:0kB writeback:1272kB mapped:252kB shmem:8kB slab_reclaimable:168kB slab_unreclaimable:336kB kernel_stack:88kB pagetables:128kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 1987 1987 1987
Node 0 DMA32 free:56920kB min:44704kB low:55880kB high:67056kB active_anon:68832kB inactive_anon:83480kB active_file:269676kB inactive_file:371212kB unevictable:4940kB isolated(anon):0kB isolated(file):0kB present:2080256kB managed:2039064kB mlocked:4940kB dirty:155668kB writeback:197236kB mapped:51672kB shmem:6636kB slab_reclaimable:32624kB slab_unreclaimable:38712kB kernel_stack:2912kB pagetables:10716kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 86*4kB (UEM) 19*8kB (UEM) 62*16kB (UEM) 6*32kB (UM) 2*64kB (UE) 5*128kB (UEM) 6*256kB (UEM) 4*512kB (EM) 0*1024kB 1*2048kB (R) 0*4096kB = 8080kB
Node 0 DMA32: 13210*4kB (M) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 56984kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
163317 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 9255932kB
Total swap = 9255932kB
524058 pages RAM
0 pages HighMem/MovableOnly
10298 pages reserved
0 pages hwpoisoned

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/