How to favor memory allocations for WQ_MEM_RECLAIM threads?

How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tetsuo Handa]

Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :

While I was testing a patch which avoids infinite too_many_isolated() loop in
shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
waiting for memory allocation. I guess that we overlooked a basic thing about
WQ_MEM_RECLAIM.

  WQ_MEM_RECLAIM helps only when the cause of failing to complete
  a work item is lack of "struct task_struct" to run that work item, for
  WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
  will not be blocked waiting for memory allocation for "struct task_struct".

  WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
  item is blocked waiting for memory allocation (or is indirectly blocked
  on a lock where the owner of that lock is blocked waiting for memory
  allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
  if memory allocation (including indirect memory allocation via
  locks/completions) is needed.

In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
"xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.

What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
workqueues (which are RESCUER) got stuck waiting for memory allocation.

----------------------------------------
[ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
[ 1095.633626] xfs-data/sda1   R  running task    12696   451      2 0x00000000
[ 1095.633663] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.633665] Call Trace:
[ 1095.633668]  __schedule+0x336/0xe00
[ 1095.633671]  schedule+0x3d/0x90
[ 1095.633672]  schedule_timeout+0x20d/0x510
[ 1095.633675]  ? lock_timer_base+0xa0/0xa0
[ 1095.633678]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.633680]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.633687]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.633699]  alloc_pages_current+0x97/0x1b0
[ 1095.633702]  new_slab+0x4cb/0x6b0
[ 1095.633706]  ___slab_alloc+0x3a3/0x620
[ 1095.633728]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633730]  ? ___slab_alloc+0x5c6/0x620
[ 1095.633732]  ? cpuacct_charge+0x38/0x1e0
[ 1095.633767]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633770]  __slab_alloc+0x46/0x7d
[ 1095.633773]  __kmalloc+0x301/0x3b0
[ 1095.633802]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.633804]  ? kfree+0x1fa/0x330
[ 1095.633842]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.633864]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.633883]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.633901]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.633936]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.633954]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.633971]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.633973]  process_one_work+0x22b/0x760
[ 1095.633975]  ? process_one_work+0x194/0x760
[ 1095.633997]  rescuer_thread+0x1f2/0x3d0
[ 1095.634002]  kthread+0x10f/0x150
[ 1095.634003]  ? worker_thread+0x4b0/0x4b0
[ 1095.634004]  ? kthread_create_on_node+0x70/0x70
[ 1095.634007]  ret_from_fork+0x31/0x40
[ 1095.634013] MemAlloc: xfs-eofblocks/s(456) flags=0x4228860 switches=15435 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=293074
[ 1095.634014] xfs-eofblocks/s R  running task    12032   456      2 0x00000000
[ 1095.634037] Workqueue: xfs-eofblocks/sda1 xfs_eofblocks_worker [xfs]
[ 1095.634038] Call Trace:
[ 1095.634040]  ? _raw_spin_lock+0x3d/0x80
[ 1095.634042]  ? vmpressure+0xd0/0x120
[ 1095.634044]  ? vmpressure+0xd0/0x120
[ 1095.634047]  ? vmpressure_prio+0x21/0x30
[ 1095.634049]  ? do_try_to_free_pages+0x70/0x300
[ 1095.634052]  ? try_to_free_pages+0x131/0x3f0
[ 1095.634058]  ? __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.634065]  ? __alloc_pages_nodemask+0x3e4/0x460
[ 1095.634069]  ? alloc_pages_current+0x97/0x1b0
[ 1095.634111]  ? xfs_buf_allocate_memory+0x160/0x2a3 [xfs]
[ 1095.634133]  ? xfs_buf_get_map+0x2be/0x480 [xfs]
[ 1095.634169]  ? xfs_buf_read_map+0x2c/0x400 [xfs]
[ 1095.634204]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
[ 1095.634222]  ? xfs_btree_read_buf_block.constprop.34+0x78/0xc0 [xfs]
[ 1095.634239]  ? xfs_btree_lookup_get_block+0x8a/0x180 [xfs]
[ 1095.634257]  ? xfs_btree_lookup+0xd0/0x3f0 [xfs]
[ 1095.634296]  ? kmem_zone_alloc+0x96/0x120 [xfs]
[ 1095.634299]  ? _raw_spin_unlock+0x27/0x40
[ 1095.634315]  ? xfs_bmbt_lookup_eq+0x1f/0x30 [xfs]
[ 1095.634348]  ? xfs_bmap_del_extent+0x1b2/0x1610 [xfs]
[ 1095.634380]  ? kmem_zone_alloc+0x96/0x120 [xfs]
[ 1095.634400]  ? __xfs_bunmapi+0x4db/0xda0 [xfs]
[ 1095.634421]  ? xfs_bunmapi+0x2b/0x40 [xfs]
[ 1095.634459]  ? xfs_itruncate_extents+0x1df/0x780 [xfs]
[ 1095.634502]  ? xfs_rename+0xc70/0x1080 [xfs]
[ 1095.634525]  ? xfs_free_eofblocks+0x1c4/0x230 [xfs]
[ 1095.634546]  ? xfs_inode_free_eofblocks+0x18d/0x280 [xfs]
[ 1095.634565]  ? xfs_inode_ag_walk.isra.13+0x2b5/0x620 [xfs]
[ 1095.634582]  ? xfs_inode_ag_walk.isra.13+0x91/0x620 [xfs]
[ 1095.634618]  ? xfs_inode_clear_eofblocks_tag+0x1a0/0x1a0 [xfs]
[ 1095.634630]  ? radix_tree_next_chunk+0x10b/0x2d0
[ 1095.634635]  ? radix_tree_gang_lookup_tag+0xd7/0x150
[ 1095.634672]  ? xfs_perag_get_tag+0x11d/0x370 [xfs]
[ 1095.634690]  ? xfs_perag_get_tag+0x5/0x370 [xfs]
[ 1095.634709]  ? xfs_inode_ag_iterator_tag+0x71/0xa0 [xfs]
[ 1095.634726]  ? xfs_inode_clear_eofblocks_tag+0x1a0/0x1a0 [xfs]
[ 1095.634744]  ? __xfs_icache_free_eofblocks+0x3b/0x40 [xfs]
[ 1095.634759]  ? xfs_eofblocks_worker+0x27/0x40 [xfs]
[ 1095.634762]  ? process_one_work+0x22b/0x760
[ 1095.634763]  ? process_one_work+0x194/0x760
[ 1095.634784]  ? rescuer_thread+0x1f2/0x3d0
[ 1095.634788]  ? kthread+0x10f/0x150
[ 1095.634789]  ? worker_thread+0x4b0/0x4b0
[ 1095.634790]  ? kthread_create_on_node+0x70/0x70
[ 1095.634793]  ? ret_from_fork+0x31/0x40

[ 1116.758199] Showing busy workqueues and worker pools:
[ 1116.759630] workqueue events: flags=0x0
[ 1116.760882]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=5/256
[ 1116.762484]     pending: vmpressure_work_fn, vmstat_shepherd, vmw_fb_dirty_flush [vmwgfx], check_corruption, console_callback
[ 1116.765197]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256
[ 1116.766841]     pending: drain_local_pages_wq BAR(9595), e1000_watchdog [e1000]
[ 1116.768755]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
[ 1116.770778]     in-flight: 7418:rht_deferred_worker
[ 1116.772255]     pending: rht_deferred_worker
[ 1116.773648] workqueue events_long: flags=0x0
[ 1116.776610]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.778302]     pending: gc_worker [nf_conntrack]
[ 1116.779857] workqueue events_power_efficient: flags=0x80
[ 1116.781485]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.783175]     pending: fb_flashcursor
[ 1116.784452]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
[ 1116.786265]     pending: neigh_periodic_work, neigh_periodic_work
[ 1116.788056] workqueue events_freezable_power_: flags=0x84
[ 1116.789748]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.791490]     in-flight: 27:disk_events_workfn
[ 1116.793092] workqueue writeback: flags=0x4e
[ 1116.794471]   pwq 256: cpus=0-127 flags=0x4 nice=0 active=2/256
[ 1116.796209]     in-flight: 8444:wb_workfn wb_workfn
[ 1116.798747] workqueue mpt_poll_0: flags=0x8
[ 1116.800223]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.801997]     pending: mpt_fault_reset_work [mptbase]
[ 1116.803780] workqueue xfs-data/sda1: flags=0xc
[ 1116.805324]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=27/256 MAYDAY
[ 1116.807272]     in-flight: 5356:xfs_end_io [xfs], 451(RESCUER):xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs], 10498:xfs_end_io [xfs], 6386:xfs_end_io [xfs]
[ 1116.812145]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.820988]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=21/256 MAYDAY
[ 1116.823105]     in-flight: 535:xfs_end_io [xfs], 7416:xfs_end_io [xfs], 7415:xfs_end_io [xfs], 65:xfs_end_io [xfs]
[ 1116.826062]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.834549]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=4/256 MAYDAY
[ 1116.837139]     in-flight: 5357:xfs_end_io [xfs], 193:xfs_end_io [xfs], 52:xfs_end_io [xfs], 5358:xfs_end_io [xfs]
[ 1116.840182]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
[ 1116.842297]     in-flight: 2486:xfs_end_io [xfs]
[ 1116.844230] workqueue xfs-reclaim/sda1: flags=0xc
[ 1116.846168]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.848323]     pending: xfs_reclaim_worker [xfs]
[ 1116.850280] workqueue xfs-eofblocks/sda1: flags=0xc
[ 1116.852358]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.854601]     in-flight: 456(RESCUER):xfs_eofblocks_worker [xfs]
[ 1116.856826] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=0s workers=3 idle: 3 6387
[ 1116.859293] pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=670s workers=6 manager: 19
[ 1116.861762] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=673s workers=6 manager: 157
[ 1116.864240] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=673s workers=4 manager: 10499
[ 1116.866876] pool 256: cpus=0-127 flags=0x4 nice=0 hung=670s workers=3 idle: 425 426
----------------------------------------

kthreadd (PID = 2) is trying to allocate "struct task_struct" requested by
workqueue managers (PID = 19, 157, 10499) but is blocked on memory allocation.
Since "struct task_struct" cannot be allocated for workqueues, RESCUER threads
(PID = 451, 456) are responsible for making forward progress.

----------------------------------------
[ 1039.876146] kworker/1:0     D12872    19      2 0x00000000
[ 1039.877743] Call Trace:
[ 1039.878717]  __schedule+0x336/0xe00
[ 1039.879908]  schedule+0x3d/0x90
[ 1039.881016]  schedule_timeout+0x26a/0x510
[ 1039.882306]  ? wait_for_completion_killable+0x56/0x1e0
[ 1039.883831]  wait_for_completion_killable+0x166/0x1e0
[ 1039.885331]  ? wake_up_q+0x80/0x80
[ 1039.886512]  ? process_one_work+0x760/0x760
[ 1039.887827]  __kthread_create_on_node+0x194/0x240
[ 1039.889255]  kthread_create_on_node+0x49/0x70
[ 1039.890604]  create_worker+0xca/0x1a0
[ 1039.891823]  worker_thread+0x34d/0x4b0
[ 1039.893050]  kthread+0x10f/0x150
[ 1039.894186]  ? process_one_work+0x760/0x760
[ 1039.895518]  ? kthread_create_on_node+0x70/0x70
[ 1039.896921]  ret_from_fork+0x31/0x40

[ 1040.575900] kworker/2:2     D12504   157      2 0x00000000
[ 1040.577485] Call Trace:
[ 1040.578469]  __schedule+0x336/0xe00
[ 1040.579646]  schedule+0x3d/0x90
[ 1040.580748]  schedule_timeout+0x26a/0x510
[ 1040.582031]  ? wait_for_completion_killable+0x56/0x1e0
[ 1040.583548]  wait_for_completion_killable+0x166/0x1e0
[ 1040.585041]  ? wake_up_q+0x80/0x80
[ 1040.586199]  ? process_one_work+0x760/0x760
[ 1040.587859]  __kthread_create_on_node+0x194/0x240
[ 1040.589294]  kthread_create_on_node+0x49/0x70
[ 1040.590662]  create_worker+0xca/0x1a0
[ 1040.591884]  worker_thread+0x34d/0x4b0
[ 1040.593123]  kthread+0x10f/0x150
[ 1040.594243]  ? process_one_work+0x760/0x760
[ 1040.595572]  ? kthread_create_on_node+0x70/0x70
[ 1040.596974]  ret_from_fork+0x31/0x40

[ 1090.480342] kworker/3:1     D11280 10499      2 0x00000080
[ 1090.481966] Call Trace:
[ 1090.482984]  __schedule+0x336/0xe00
[ 1090.484228]  ? account_entity_enqueue+0xdb/0x110
[ 1090.487424]  schedule+0x3d/0x90
[ 1090.488575]  schedule_timeout+0x26a/0x510
[ 1090.489881]  ? wait_for_completion_killable+0x56/0x1e0
[ 1090.491615]  wait_for_completion_killable+0x166/0x1e0
[ 1090.493143]  ? wake_up_q+0x80/0x80
[ 1090.494334]  ? process_one_work+0x760/0x760
[ 1090.495679]  __kthread_create_on_node+0x194/0x240
[ 1090.498150]  kthread_create_on_node+0x49/0x70
[ 1090.499518]  create_worker+0xca/0x1a0
[ 1090.500737]  worker_thread+0x34d/0x4b0
[ 1090.501986]  kthread+0x10f/0x150
[ 1090.503126]  ? process_one_work+0x760/0x760
[ 1090.504459]  ? kthread_create_on_node+0x70/0x70
[ 1090.505857]  ? do_syscall_64+0x195/0x200
[ 1090.507132]  ret_from_fork+0x31/0x40

[ 1095.631379] MemAlloc: kthreadd(2) flags=0x208840 switches=313 seq=5 gfp=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK) order=2 delay=652085 uninterruptible
[ 1095.631380] kthreadd        D11000     2      0 0x00000000
[ 1095.631384] Call Trace:
[ 1095.631387]  __schedule+0x336/0xe00
[ 1095.631388]  ? __mutex_lock+0x1a2/0x9c0
[ 1095.631391]  schedule+0x3d/0x90
[ 1095.631393]  schedule_preempt_disabled+0x15/0x20
[ 1095.631394]  __mutex_lock+0x2ed/0x9c0
[ 1095.631395]  ? __mutex_lock+0xce/0x9c0
[ 1095.631397]  ? radix_tree_gang_lookup_tag+0xd7/0x150
[ 1095.631417]  ? xfs_reclaim_inodes_ag+0x3c6/0x4f0 [xfs]
[ 1095.631439]  ? xfs_perag_get_tag+0x5/0x370 [xfs]
[ 1095.631441]  mutex_lock_nested+0x1b/0x20
[ 1095.631473]  xfs_reclaim_inodes_ag+0x3c6/0x4f0 [xfs]
[ 1095.631500]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
[ 1095.631507]  ? _raw_spin_unlock_irqrestore+0x3b/0x60
[ 1095.631510]  ? try_to_wake_up+0x59/0x7a0
[ 1095.631514]  ? wake_up_process+0x15/0x20
[ 1095.631557]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 1095.631580]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
[ 1095.631582]  super_cache_scan+0x181/0x190
[ 1095.631585]  shrink_slab+0x29f/0x6d0
[ 1095.631591]  shrink_node+0x2fa/0x310
[ 1095.631594]  do_try_to_free_pages+0xe1/0x300
[ 1095.631597]  try_to_free_pages+0x131/0x3f0
[ 1095.631602]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.631606]  ? _raw_spin_unlock+0x27/0x40
[ 1095.631610]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.631611]  ? cpumask_next_and+0x47/0xa0
[ 1095.631615]  new_slab+0x450/0x6b0
[ 1095.631618]  ___slab_alloc+0x3a3/0x620
[ 1095.631622]  ? find_busiest_group+0x47/0x4d0
[ 1095.631624]  ? copy_process.part.31+0x122/0x21e0
[ 1095.631627]  ? copy_process.part.31+0x122/0x21e0
[ 1095.631629]  __slab_alloc+0x46/0x7d
[ 1095.631631]  kmem_cache_alloc_node+0xab/0x3a0
[ 1095.631633]  ? load_balance+0x1e7/0xb50
[ 1095.631635]  copy_process.part.31+0x122/0x21e0
[ 1095.631638]  ? pick_next_task_fair+0x6c6/0x890
[ 1095.631641]  ? kthread_create_on_node+0x70/0x70
[ 1095.631642]  ? finish_task_switch+0x70/0x240
[ 1095.631644]  _do_fork+0xf3/0x750
[ 1095.631647]  ? kthreadd+0x2f2/0x3c0
[ 1095.631650]  kernel_thread+0x29/0x30
[ 1095.631651]  kthreadd+0x35a/0x3c0
[ 1095.631652]  ? ret_from_fork+0x31/0x40
[ 1095.631656]  ? kthread_create_on_cpu+0xb0/0xb0
[ 1095.631658]  ret_from_fork+0x31/0x40
----------------------------------------

As usual, GFP_NOIO allocation by disk_events_workfn (as a crying
canary indicating a lockup) is stalling there.

----------------------------------------
[ 1095.631687] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=38727 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=652160
[ 1095.631688] kworker/2:0     R  running task    12680    27      2 0x00000000
[ 1095.631739] Workqueue: events_freezable_power_ disk_events_workfn
[ 1095.631740] Call Trace:
[ 1095.631743]  __schedule+0x336/0xe00
[ 1095.631746]  preempt_schedule_common+0x1f/0x31
[ 1095.631747]  _cond_resched+0x1c/0x30
[ 1095.631749]  shrink_slab+0x339/0x6d0
[ 1095.631754]  shrink_node+0x2fa/0x310
[ 1095.631758]  do_try_to_free_pages+0xe1/0x300
[ 1095.631761]  try_to_free_pages+0x131/0x3f0
[ 1095.631765]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.631771]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.631775]  alloc_pages_current+0x97/0x1b0
[ 1095.631779]  bio_copy_kern+0xc9/0x180
[ 1095.631830]  blk_rq_map_kern+0x70/0x140
[ 1095.631835]  __scsi_execute.isra.22+0x13a/0x1e0
[ 1095.631838]  scsi_execute_req_flags+0x94/0x100
[ 1095.631844]  sr_check_events+0xbf/0x2b0 [sr_mod]
[ 1095.631852]  cdrom_check_events+0x18/0x30 [cdrom]
[ 1095.631854]  sr_block_check_events+0x2a/0x30 [sr_mod]
[ 1095.631856]  disk_check_events+0x60/0x170
[ 1095.631859]  disk_events_workfn+0x1c/0x20
[ 1095.631862]  process_one_work+0x22b/0x760
[ 1095.631863]  ? process_one_work+0x194/0x760
[ 1095.631867]  worker_thread+0x137/0x4b0
[ 1095.631887]  kthread+0x10f/0x150
[ 1095.631889]  ? process_one_work+0x760/0x760
[ 1095.631890]  ? kthread_create_on_node+0x70/0x70
[ 1095.631893]  ret_from_fork+0x31/0x40
----------------------------------------

Due to __GFP_NOWARN, warn_alloc() cannot warn allocation stalls.
Due to order <= PAGE_ALLOC_COSTLY_ORDER without __GFP_NORETRY, the
"%s(%u) possible memory allocation deadlock size %u in %s (mode:0x%x)"
message cannot be printed because __alloc_pages_nodemask() does not return.
And due to GFP_NOFS with neither __GFP_HIGH nor __GFP_NOFAIL, memory cannot
be allocated to RESCUER threads which are trying to allocate memory for
reclaiming memory.

Setting PF_MEMALLOC will disable direct reclaim and allow access to memory
reserves, but allocation failure is not acceptable due to !KM_MAYFAIL.
Passing __GFP_HIGH and/or __GFP_NOFAIL will allow access to memory reserves,
but allocation might still loop forever inside the page allocator (especially
if order > 0) because commit 06ad276ac18742c6 ("mm, oom: do not enforce OOM
killer for __GFP_NOFAIL automatically") deprived WQ_MEM_RECLAIM users of
the last resort for invoking the OOM killer. If order > 0 allocation request
with __GFP_NOFAIL fails due to fragmentation, there is nothing we can do.
Passing __GFP_NORETRY will allow the
"%s(%u) possible memory allocation deadlock size %u in %s (mode:0x%x)" message
to be printed but that helps only emitting warning.

So, how can we avoid this situation?

How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tetsuo Handa]

Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :

While I was testing a patch which avoids infinite too_many_isolated() loop in
shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
waiting for memory allocation. I guess that we overlooked a basic thing about
WQ_MEM_RECLAIM.

  WQ_MEM_RECLAIM helps only when the cause of failing to complete
  a work item is lack of "struct task_struct" to run that work item, for
  WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
  will not be blocked waiting for memory allocation for "struct task_struct".

  WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
  item is blocked waiting for memory allocation (or is indirectly blocked
  on a lock where the owner of that lock is blocked waiting for memory
  allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
  if memory allocation (including indirect memory allocation via
  locks/completions) is needed.

In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
"xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.

What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
workqueues (which are RESCUER) got stuck waiting for memory allocation.

----------------------------------------
[ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
[ 1095.633626] xfs-data/sda1   R  running task    12696   451      2 0x00000000
[ 1095.633663] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.633665] Call Trace:
[ 1095.633668]  __schedule+0x336/0xe00
[ 1095.633671]  schedule+0x3d/0x90
[ 1095.633672]  schedule_timeout+0x20d/0x510
[ 1095.633675]  ? lock_timer_base+0xa0/0xa0
[ 1095.633678]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.633680]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.633687]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.633699]  alloc_pages_current+0x97/0x1b0
[ 1095.633702]  new_slab+0x4cb/0x6b0
[ 1095.633706]  ___slab_alloc+0x3a3/0x620
[ 1095.633728]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633730]  ? ___slab_alloc+0x5c6/0x620
[ 1095.633732]  ? cpuacct_charge+0x38/0x1e0
[ 1095.633767]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633770]  __slab_alloc+0x46/0x7d
[ 1095.633773]  __kmalloc+0x301/0x3b0
[ 1095.633802]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.633804]  ? kfree+0x1fa/0x330
[ 1095.633842]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.633864]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.633883]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.633901]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.633936]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.633954]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.633971]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.633973]  process_one_work+0x22b/0x760
[ 1095.633975]  ? process_one_work+0x194/0x760
[ 1095.633997]  rescuer_thread+0x1f2/0x3d0
[ 1095.634002]  kthread+0x10f/0x150
[ 1095.634003]  ? worker_thread+0x4b0/0x4b0
[ 1095.634004]  ? kthread_create_on_node+0x70/0x70
[ 1095.634007]  ret_from_fork+0x31/0x40
[ 1095.634013] MemAlloc: xfs-eofblocks/s(456) flags=0x4228860 switches=15435 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=293074
[ 1095.634014] xfs-eofblocks/s R  running task    12032   456      2 0x00000000
[ 1095.634037] Workqueue: xfs-eofblocks/sda1 xfs_eofblocks_worker [xfs]
[ 1095.634038] Call Trace:
[ 1095.634040]  ? _raw_spin_lock+0x3d/0x80
[ 1095.634042]  ? vmpressure+0xd0/0x120
[ 1095.634044]  ? vmpressure+0xd0/0x120
[ 1095.634047]  ? vmpressure_prio+0x21/0x30
[ 1095.634049]  ? do_try_to_free_pages+0x70/0x300
[ 1095.634052]  ? try_to_free_pages+0x131/0x3f0
[ 1095.634058]  ? __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.634065]  ? __alloc_pages_nodemask+0x3e4/0x460
[ 1095.634069]  ? alloc_pages_current+0x97/0x1b0
[ 1095.634111]  ? xfs_buf_allocate_memory+0x160/0x2a3 [xfs]
[ 1095.634133]  ? xfs_buf_get_map+0x2be/0x480 [xfs]
[ 1095.634169]  ? xfs_buf_read_map+0x2c/0x400 [xfs]
[ 1095.634204]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
[ 1095.634222]  ? xfs_btree_read_buf_block.constprop.34+0x78/0xc0 [xfs]
[ 1095.634239]  ? xfs_btree_lookup_get_block+0x8a/0x180 [xfs]
[ 1095.634257]  ? xfs_btree_lookup+0xd0/0x3f0 [xfs]
[ 1095.634296]  ? kmem_zone_alloc+0x96/0x120 [xfs]
[ 1095.634299]  ? _raw_spin_unlock+0x27/0x40
[ 1095.634315]  ? xfs_bmbt_lookup_eq+0x1f/0x30 [xfs]
[ 1095.634348]  ? xfs_bmap_del_extent+0x1b2/0x1610 [xfs]
[ 1095.634380]  ? kmem_zone_alloc+0x96/0x120 [xfs]
[ 1095.634400]  ? __xfs_bunmapi+0x4db/0xda0 [xfs]
[ 1095.634421]  ? xfs_bunmapi+0x2b/0x40 [xfs]
[ 1095.634459]  ? xfs_itruncate_extents+0x1df/0x780 [xfs]
[ 1095.634502]  ? xfs_rename+0xc70/0x1080 [xfs]
[ 1095.634525]  ? xfs_free_eofblocks+0x1c4/0x230 [xfs]
[ 1095.634546]  ? xfs_inode_free_eofblocks+0x18d/0x280 [xfs]
[ 1095.634565]  ? xfs_inode_ag_walk.isra.13+0x2b5/0x620 [xfs]
[ 1095.634582]  ? xfs_inode_ag_walk.isra.13+0x91/0x620 [xfs]
[ 1095.634618]  ? xfs_inode_clear_eofblocks_tag+0x1a0/0x1a0 [xfs]
[ 1095.634630]  ? radix_tree_next_chunk+0x10b/0x2d0
[ 1095.634635]  ? radix_tree_gang_lookup_tag+0xd7/0x150
[ 1095.634672]  ? xfs_perag_get_tag+0x11d/0x370 [xfs]
[ 1095.634690]  ? xfs_perag_get_tag+0x5/0x370 [xfs]
[ 1095.634709]  ? xfs_inode_ag_iterator_tag+0x71/0xa0 [xfs]
[ 1095.634726]  ? xfs_inode_clear_eofblocks_tag+0x1a0/0x1a0 [xfs]
[ 1095.634744]  ? __xfs_icache_free_eofblocks+0x3b/0x40 [xfs]
[ 1095.634759]  ? xfs_eofblocks_worker+0x27/0x40 [xfs]
[ 1095.634762]  ? process_one_work+0x22b/0x760
[ 1095.634763]  ? process_one_work+0x194/0x760
[ 1095.634784]  ? rescuer_thread+0x1f2/0x3d0
[ 1095.634788]  ? kthread+0x10f/0x150
[ 1095.634789]  ? worker_thread+0x4b0/0x4b0
[ 1095.634790]  ? kthread_create_on_node+0x70/0x70
[ 1095.634793]  ? ret_from_fork+0x31/0x40

[ 1116.758199] Showing busy workqueues and worker pools:
[ 1116.759630] workqueue events: flags=0x0
[ 1116.760882]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=5/256
[ 1116.762484]     pending: vmpressure_work_fn, vmstat_shepherd, vmw_fb_dirty_flush [vmwgfx], check_corruption, console_callback
[ 1116.765197]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256
[ 1116.766841]     pending: drain_local_pages_wq BAR(9595), e1000_watchdog [e1000]
[ 1116.768755]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
[ 1116.770778]     in-flight: 7418:rht_deferred_worker
[ 1116.772255]     pending: rht_deferred_worker
[ 1116.773648] workqueue events_long: flags=0x0
[ 1116.776610]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.778302]     pending: gc_worker [nf_conntrack]
[ 1116.779857] workqueue events_power_efficient: flags=0x80
[ 1116.781485]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.783175]     pending: fb_flashcursor
[ 1116.784452]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
[ 1116.786265]     pending: neigh_periodic_work, neigh_periodic_work
[ 1116.788056] workqueue events_freezable_power_: flags=0x84
[ 1116.789748]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.791490]     in-flight: 27:disk_events_workfn
[ 1116.793092] workqueue writeback: flags=0x4e
[ 1116.794471]   pwq 256: cpus=0-127 flags=0x4 nice=0 active=2/256
[ 1116.796209]     in-flight: 8444:wb_workfn wb_workfn
[ 1116.798747] workqueue mpt_poll_0: flags=0x8
[ 1116.800223]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.801997]     pending: mpt_fault_reset_work [mptbase]
[ 1116.803780] workqueue xfs-data/sda1: flags=0xc
[ 1116.805324]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=27/256 MAYDAY
[ 1116.807272]     in-flight: 5356:xfs_end_io [xfs], 451(RESCUER):xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs], 10498:xfs_end_io [xfs], 6386:xfs_end_io [xfs]
[ 1116.812145]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.820988]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=21/256 MAYDAY
[ 1116.823105]     in-flight: 535:xfs_end_io [xfs], 7416:xfs_end_io [xfs], 7415:xfs_end_io [xfs], 65:xfs_end_io [xfs]
[ 1116.826062]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.834549]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=4/256 MAYDAY
[ 1116.837139]     in-flight: 5357:xfs_end_io [xfs], 193:xfs_end_io [xfs], 52:xfs_end_io [xfs], 5358:xfs_end_io [xfs]
[ 1116.840182]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
[ 1116.842297]     in-flight: 2486:xfs_end_io [xfs]
[ 1116.844230] workqueue xfs-reclaim/sda1: flags=0xc
[ 1116.846168]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.848323]     pending: xfs_reclaim_worker [xfs]
[ 1116.850280] workqueue xfs-eofblocks/sda1: flags=0xc
[ 1116.852358]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.854601]     in-flight: 456(RESCUER):xfs_eofblocks_worker [xfs]
[ 1116.856826] pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=0s workers=3 idle: 3 6387
[ 1116.859293] pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=670s workers=6 manager: 19
[ 1116.861762] pool 4: cpus=2 node=0 flags=0x0 nice=0 hung=673s workers=6 manager: 157
[ 1116.864240] pool 6: cpus=3 node=0 flags=0x0 nice=0 hung=673s workers=4 manager: 10499
[ 1116.866876] pool 256: cpus=0-127 flags=0x4 nice=0 hung=670s workers=3 idle: 425 426
----------------------------------------

kthreadd (PID = 2) is trying to allocate "struct task_struct" requested by
workqueue managers (PID = 19, 157, 10499) but is blocked on memory allocation.
Since "struct task_struct" cannot be allocated for workqueues, RESCUER threads
(PID = 451, 456) are responsible for making forward progress.

----------------------------------------
[ 1039.876146] kworker/1:0     D12872    19      2 0x00000000
[ 1039.877743] Call Trace:
[ 1039.878717]  __schedule+0x336/0xe00
[ 1039.879908]  schedule+0x3d/0x90
[ 1039.881016]  schedule_timeout+0x26a/0x510
[ 1039.882306]  ? wait_for_completion_killable+0x56/0x1e0
[ 1039.883831]  wait_for_completion_killable+0x166/0x1e0
[ 1039.885331]  ? wake_up_q+0x80/0x80
[ 1039.886512]  ? process_one_work+0x760/0x760
[ 1039.887827]  __kthread_create_on_node+0x194/0x240
[ 1039.889255]  kthread_create_on_node+0x49/0x70
[ 1039.890604]  create_worker+0xca/0x1a0
[ 1039.891823]  worker_thread+0x34d/0x4b0
[ 1039.893050]  kthread+0x10f/0x150
[ 1039.894186]  ? process_one_work+0x760/0x760
[ 1039.895518]  ? kthread_create_on_node+0x70/0x70
[ 1039.896921]  ret_from_fork+0x31/0x40

[ 1040.575900] kworker/2:2     D12504   157      2 0x00000000
[ 1040.577485] Call Trace:
[ 1040.578469]  __schedule+0x336/0xe00
[ 1040.579646]  schedule+0x3d/0x90
[ 1040.580748]  schedule_timeout+0x26a/0x510
[ 1040.582031]  ? wait_for_completion_killable+0x56/0x1e0
[ 1040.583548]  wait_for_completion_killable+0x166/0x1e0
[ 1040.585041]  ? wake_up_q+0x80/0x80
[ 1040.586199]  ? process_one_work+0x760/0x760
[ 1040.587859]  __kthread_create_on_node+0x194/0x240
[ 1040.589294]  kthread_create_on_node+0x49/0x70
[ 1040.590662]  create_worker+0xca/0x1a0
[ 1040.591884]  worker_thread+0x34d/0x4b0
[ 1040.593123]  kthread+0x10f/0x150
[ 1040.594243]  ? process_one_work+0x760/0x760
[ 1040.595572]  ? kthread_create_on_node+0x70/0x70
[ 1040.596974]  ret_from_fork+0x31/0x40

[ 1090.480342] kworker/3:1     D11280 10499      2 0x00000080
[ 1090.481966] Call Trace:
[ 1090.482984]  __schedule+0x336/0xe00
[ 1090.484228]  ? account_entity_enqueue+0xdb/0x110
[ 1090.487424]  schedule+0x3d/0x90
[ 1090.488575]  schedule_timeout+0x26a/0x510
[ 1090.489881]  ? wait_for_completion_killable+0x56/0x1e0
[ 1090.491615]  wait_for_completion_killable+0x166/0x1e0
[ 1090.493143]  ? wake_up_q+0x80/0x80
[ 1090.494334]  ? process_one_work+0x760/0x760
[ 1090.495679]  __kthread_create_on_node+0x194/0x240
[ 1090.498150]  kthread_create_on_node+0x49/0x70
[ 1090.499518]  create_worker+0xca/0x1a0
[ 1090.500737]  worker_thread+0x34d/0x4b0
[ 1090.501986]  kthread+0x10f/0x150
[ 1090.503126]  ? process_one_work+0x760/0x760
[ 1090.504459]  ? kthread_create_on_node+0x70/0x70
[ 1090.505857]  ? do_syscall_64+0x195/0x200
[ 1090.507132]  ret_from_fork+0x31/0x40

[ 1095.631379] MemAlloc: kthreadd(2) flags=0x208840 switches=313 seq=5 gfp=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK) order=2 delay=652085 uninterruptible
[ 1095.631380] kthreadd        D11000     2      0 0x00000000
[ 1095.631384] Call Trace:
[ 1095.631387]  __schedule+0x336/0xe00
[ 1095.631388]  ? __mutex_lock+0x1a2/0x9c0
[ 1095.631391]  schedule+0x3d/0x90
[ 1095.631393]  schedule_preempt_disabled+0x15/0x20
[ 1095.631394]  __mutex_lock+0x2ed/0x9c0
[ 1095.631395]  ? __mutex_lock+0xce/0x9c0
[ 1095.631397]  ? radix_tree_gang_lookup_tag+0xd7/0x150
[ 1095.631417]  ? xfs_reclaim_inodes_ag+0x3c6/0x4f0 [xfs]
[ 1095.631439]  ? xfs_perag_get_tag+0x5/0x370 [xfs]
[ 1095.631441]  mutex_lock_nested+0x1b/0x20
[ 1095.631473]  xfs_reclaim_inodes_ag+0x3c6/0x4f0 [xfs]
[ 1095.631500]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
[ 1095.631507]  ? _raw_spin_unlock_irqrestore+0x3b/0x60
[ 1095.631510]  ? try_to_wake_up+0x59/0x7a0
[ 1095.631514]  ? wake_up_process+0x15/0x20
[ 1095.631557]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 1095.631580]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
[ 1095.631582]  super_cache_scan+0x181/0x190
[ 1095.631585]  shrink_slab+0x29f/0x6d0
[ 1095.631591]  shrink_node+0x2fa/0x310
[ 1095.631594]  do_try_to_free_pages+0xe1/0x300
[ 1095.631597]  try_to_free_pages+0x131/0x3f0
[ 1095.631602]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.631606]  ? _raw_spin_unlock+0x27/0x40
[ 1095.631610]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.631611]  ? cpumask_next_and+0x47/0xa0
[ 1095.631615]  new_slab+0x450/0x6b0
[ 1095.631618]  ___slab_alloc+0x3a3/0x620
[ 1095.631622]  ? find_busiest_group+0x47/0x4d0
[ 1095.631624]  ? copy_process.part.31+0x122/0x21e0
[ 1095.631627]  ? copy_process.part.31+0x122/0x21e0
[ 1095.631629]  __slab_alloc+0x46/0x7d
[ 1095.631631]  kmem_cache_alloc_node+0xab/0x3a0
[ 1095.631633]  ? load_balance+0x1e7/0xb50
[ 1095.631635]  copy_process.part.31+0x122/0x21e0
[ 1095.631638]  ? pick_next_task_fair+0x6c6/0x890
[ 1095.631641]  ? kthread_create_on_node+0x70/0x70
[ 1095.631642]  ? finish_task_switch+0x70/0x240
[ 1095.631644]  _do_fork+0xf3/0x750
[ 1095.631647]  ? kthreadd+0x2f2/0x3c0
[ 1095.631650]  kernel_thread+0x29/0x30
[ 1095.631651]  kthreadd+0x35a/0x3c0
[ 1095.631652]  ? ret_from_fork+0x31/0x40
[ 1095.631656]  ? kthread_create_on_cpu+0xb0/0xb0
[ 1095.631658]  ret_from_fork+0x31/0x40
----------------------------------------

As usual, GFP_NOIO allocation by disk_events_workfn (as a crying
canary indicating a lockup) is stalling there.

----------------------------------------
[ 1095.631687] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=38727 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=652160
[ 1095.631688] kworker/2:0     R  running task    12680    27      2 0x00000000
[ 1095.631739] Workqueue: events_freezable_power_ disk_events_workfn
[ 1095.631740] Call Trace:
[ 1095.631743]  __schedule+0x336/0xe00
[ 1095.631746]  preempt_schedule_common+0x1f/0x31
[ 1095.631747]  _cond_resched+0x1c/0x30
[ 1095.631749]  shrink_slab+0x339/0x6d0
[ 1095.631754]  shrink_node+0x2fa/0x310
[ 1095.631758]  do_try_to_free_pages+0xe1/0x300
[ 1095.631761]  try_to_free_pages+0x131/0x3f0
[ 1095.631765]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.631771]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.631775]  alloc_pages_current+0x97/0x1b0
[ 1095.631779]  bio_copy_kern+0xc9/0x180
[ 1095.631830]  blk_rq_map_kern+0x70/0x140
[ 1095.631835]  __scsi_execute.isra.22+0x13a/0x1e0
[ 1095.631838]  scsi_execute_req_flags+0x94/0x100
[ 1095.631844]  sr_check_events+0xbf/0x2b0 [sr_mod]
[ 1095.631852]  cdrom_check_events+0x18/0x30 [cdrom]
[ 1095.631854]  sr_block_check_events+0x2a/0x30 [sr_mod]
[ 1095.631856]  disk_check_events+0x60/0x170
[ 1095.631859]  disk_events_workfn+0x1c/0x20
[ 1095.631862]  process_one_work+0x22b/0x760
[ 1095.631863]  ? process_one_work+0x194/0x760
[ 1095.631867]  worker_thread+0x137/0x4b0
[ 1095.631887]  kthread+0x10f/0x150
[ 1095.631889]  ? process_one_work+0x760/0x760
[ 1095.631890]  ? kthread_create_on_node+0x70/0x70
[ 1095.631893]  ret_from_fork+0x31/0x40
----------------------------------------

Due to __GFP_NOWARN, warn_alloc() cannot warn allocation stalls.
Due to order <= PAGE_ALLOC_COSTLY_ORDER without __GFP_NORETRY, the
"%s(%u) possible memory allocation deadlock size %u in %s (mode:0x%x)"
message cannot be printed because __alloc_pages_nodemask() does not return.
And due to GFP_NOFS with neither __GFP_HIGH nor __GFP_NOFAIL, memory cannot
be allocated to RESCUER threads which are trying to allocate memory for
reclaiming memory.

Setting PF_MEMALLOC will disable direct reclaim and allow access to memory
reserves, but allocation failure is not acceptable due to !KM_MAYFAIL.
Passing __GFP_HIGH and/or __GFP_NOFAIL will allow access to memory reserves,
but allocation might still loop forever inside the page allocator (especially
if order > 0) because commit 06ad276ac18742c6 ("mm, oom: do not enforce OOM
killer for __GFP_NOFAIL automatically") deprived WQ_MEM_RECLAIM users of
the last resort for invoking the OOM killer. If order > 0 allocation request
with __GFP_NOFAIL fails due to fragmentation, there is nothing we can do.
Passing __GFP_NORETRY will allow the
"%s(%u) possible memory allocation deadlock size %u in %s (mode:0x%x)" message
to be printed but that helps only emitting warning.

So, how can we avoid this situation?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Michal Hocko]

On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> 
> While I was testing a patch which avoids infinite too_many_isolated() loop in
> shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> waiting for memory allocation. I guess that we overlooked a basic thing about
> WQ_MEM_RECLAIM.
> 
>   WQ_MEM_RECLAIM helps only when the cause of failing to complete
>   a work item is lack of "struct task_struct" to run that work item, for
>   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
>   will not be blocked waiting for memory allocation for "struct task_struct".
> 
>   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
>   item is blocked waiting for memory allocation (or is indirectly blocked
>   on a lock where the owner of that lock is blocked waiting for memory
>   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
>   if memory allocation (including indirect memory allocation via
>   locks/completions) is needed.
> 
> In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> 
> What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> workqueues (which are RESCUER) got stuck waiting for memory allocation.

If those workers are really required for a further progress of the
memory reclaim then they shouldn't block on allocation at all and either
use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
that only very limited amount of memory is allocated from that context
and there will be at least the same amount of memory freed as a result
in a reasonable time.

This is something for xfs people to answer though. Please note that I
didn't really have time to look through the below traces so the above
note is rather generic. It would be really helpful if you could provide
a high level dependency chains to see why those rescuers are necessary
for the forward progress because it is really easy to get lost in so
many traces.
-- 
Michal Hocko
SUSE Labs

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Michal Hocko]

On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> 
> While I was testing a patch which avoids infinite too_many_isolated() loop in
> shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> waiting for memory allocation. I guess that we overlooked a basic thing about
> WQ_MEM_RECLAIM.
> 
>   WQ_MEM_RECLAIM helps only when the cause of failing to complete
>   a work item is lack of "struct task_struct" to run that work item, for
>   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
>   will not be blocked waiting for memory allocation for "struct task_struct".
> 
>   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
>   item is blocked waiting for memory allocation (or is indirectly blocked
>   on a lock where the owner of that lock is blocked waiting for memory
>   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
>   if memory allocation (including indirect memory allocation via
>   locks/completions) is needed.
> 
> In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> 
> What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> workqueues (which are RESCUER) got stuck waiting for memory allocation.

If those workers are really required for a further progress of the
memory reclaim then they shouldn't block on allocation at all and either
use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
that only very limited amount of memory is allocated from that context
and there will be at least the same amount of memory freed as a result
in a reasonable time.

This is something for xfs people to answer though. Please note that I
didn't really have time to look through the below traces so the above
note is rather generic. It would be really helpful if you could provide
a high level dependency chains to see why those rescuers are necessary
for the forward progress because it is really easy to get lost in so
many traces.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Fri, Mar 03, 2017 at 02:39:51PM +0100, Michal Hocko wrote:
> On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> > Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> > 
> > While I was testing a patch which avoids infinite too_many_isolated() loop in
> > shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> > waiting for memory allocation. I guess that we overlooked a basic thing about
> > WQ_MEM_RECLAIM.
> > 
> >   WQ_MEM_RECLAIM helps only when the cause of failing to complete
> >   a work item is lack of "struct task_struct" to run that work item, for
> >   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
> >   will not be blocked waiting for memory allocation for "struct task_struct".
> > 
> >   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
> >   item is blocked waiting for memory allocation (or is indirectly blocked
> >   on a lock where the owner of that lock is blocked waiting for memory
> >   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
> >   if memory allocation (including indirect memory allocation via
> >   locks/completions) is needed.
> > 
> > In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> > "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> > workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> > 
> > What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> > I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> > workqueues (which are RESCUER) got stuck waiting for memory allocation.
> 
> If those workers are really required for a further progress of the
> memory reclaim then they shouldn't block on allocation at all and either
> use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
> that only very limited amount of memory is allocated from that context
> and there will be at least the same amount of memory freed as a result
> in a reasonable time.
> 
> This is something for xfs people to answer though. Please note that I
> didn't really have time to look through the below traces so the above
> note is rather generic. It would be really helpful if you could provide
> a high level dependency chains to see why those rescuers are necessary
> for the forward progress because it is really easy to get lost in so
> many traces.

Hmm, I can't claim to fully grok the wq internals, but my understanding
of the WQ_MEM_RECLAIM setting used on the XFS side was to create rescuer
threads for workqueues to deal with the case where the kthread
allocation required memory reclaim, and memory reclaim progress is made
via a workqueue item. IIRC, this was originally targeted for the
xfs-reclaim wq based on the report in commit 7a29ac474a ("xfs: give all
workqueues rescuer threads"), but it looks like that patch was modified
before it was committed to give all workqueues such a thread as well. I
don't know that we explicitly need this flag for all XFS wq's, but I
also don't recall particular meaning behind WQ_MEM_RECLAIM that
suggested the associated wq should guarantee progress, so perhaps that's
why it was added as such.

That aside, looking through some of the traces in this case...

- kswapd0 is waiting on an inode flush lock. This means somebody else
  flushed the inode and it won't be unlocked until the underlying buffer
  I/O is completed. This context is also holding pag_ici_reclaim_lock
  which is what probably blocks other contexts from getting into inode
  reclaim.
- xfsaild is in xfs_iflush(), which means it has the inode flush lock.
  It's waiting on reading the underlying inode buffer. The buffer read
  sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
  queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
  work item is what eventually triggers the I/O completion in
  xfs_buf_ioend().

So at this point reclaim is waiting on a read I/O completion. It's not
clear to me whether the read had completed and the work item was queued
or not. I do see the following in the workqueue lockup BUG output:

[  273.412600] workqueue xfs-buf/sda1: flags=0xc
[  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
[  273.416415]     pending: xfs_buf_ioend_work [xfs]

... which suggests that it was queued..? I suppose this could be one of
the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
appears to be idle:

[ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
[ 1041.556813] Call Trace:
[ 1041.557796]  __schedule+0x336/0xe00
[ 1041.558983]  schedule+0x3d/0x90
[ 1041.560085]  rescuer_thread+0x322/0x3d0
[ 1041.561333]  kthread+0x10f/0x150
[ 1041.562464]  ? worker_thread+0x4b0/0x4b0
[ 1041.563732]  ? kthread_create_on_node+0x70/0x70
[ 1041.565123]  ret_from_fork+0x31/0x40

So shouldn't that thread pick up the work item if that is the case?

Brian

> -- 
> Michal Hocko
> SUSE Labs
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Fri, Mar 03, 2017 at 02:39:51PM +0100, Michal Hocko wrote:
> On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> > Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> > 
> > While I was testing a patch which avoids infinite too_many_isolated() loop in
> > shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> > waiting for memory allocation. I guess that we overlooked a basic thing about
> > WQ_MEM_RECLAIM.
> > 
> >   WQ_MEM_RECLAIM helps only when the cause of failing to complete
> >   a work item is lack of "struct task_struct" to run that work item, for
> >   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
> >   will not be blocked waiting for memory allocation for "struct task_struct".
> > 
> >   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
> >   item is blocked waiting for memory allocation (or is indirectly blocked
> >   on a lock where the owner of that lock is blocked waiting for memory
> >   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
> >   if memory allocation (including indirect memory allocation via
> >   locks/completions) is needed.
> > 
> > In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> > "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> > workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> > 
> > What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> > I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> > workqueues (which are RESCUER) got stuck waiting for memory allocation.
> 
> If those workers are really required for a further progress of the
> memory reclaim then they shouldn't block on allocation at all and either
> use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
> that only very limited amount of memory is allocated from that context
> and there will be at least the same amount of memory freed as a result
> in a reasonable time.
> 
> This is something for xfs people to answer though. Please note that I
> didn't really have time to look through the below traces so the above
> note is rather generic. It would be really helpful if you could provide
> a high level dependency chains to see why those rescuers are necessary
> for the forward progress because it is really easy to get lost in so
> many traces.

Hmm, I can't claim to fully grok the wq internals, but my understanding
of the WQ_MEM_RECLAIM setting used on the XFS side was to create rescuer
threads for workqueues to deal with the case where the kthread
allocation required memory reclaim, and memory reclaim progress is made
via a workqueue item. IIRC, this was originally targeted for the
xfs-reclaim wq based on the report in commit 7a29ac474a ("xfs: give all
workqueues rescuer threads"), but it looks like that patch was modified
before it was committed to give all workqueues such a thread as well. I
don't know that we explicitly need this flag for all XFS wq's, but I
also don't recall particular meaning behind WQ_MEM_RECLAIM that
suggested the associated wq should guarantee progress, so perhaps that's
why it was added as such.

That aside, looking through some of the traces in this case...

- kswapd0 is waiting on an inode flush lock. This means somebody else
  flushed the inode and it won't be unlocked until the underlying buffer
  I/O is completed. This context is also holding pag_ici_reclaim_lock
  which is what probably blocks other contexts from getting into inode
  reclaim.
- xfsaild is in xfs_iflush(), which means it has the inode flush lock.
  It's waiting on reading the underlying inode buffer. The buffer read
  sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
  queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
  work item is what eventually triggers the I/O completion in
  xfs_buf_ioend().

So at this point reclaim is waiting on a read I/O completion. It's not
clear to me whether the read had completed and the work item was queued
or not. I do see the following in the workqueue lockup BUG output:

[  273.412600] workqueue xfs-buf/sda1: flags=0xc
[  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
[  273.416415]     pending: xfs_buf_ioend_work [xfs]

... which suggests that it was queued..? I suppose this could be one of
the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
appears to be idle:

[ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
[ 1041.556813] Call Trace:
[ 1041.557796]  __schedule+0x336/0xe00
[ 1041.558983]  schedule+0x3d/0x90
[ 1041.560085]  rescuer_thread+0x322/0x3d0
[ 1041.561333]  kthread+0x10f/0x150
[ 1041.562464]  ? worker_thread+0x4b0/0x4b0
[ 1041.563732]  ? kthread_create_on_node+0x70/0x70
[ 1041.565123]  ret_from_fork+0x31/0x40

So shouldn't that thread pick up the work item if that is the case?

Brian

> -- 
> Michal Hocko
> SUSE Labs
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Michal Hocko]

On Fri 03-03-17 10:37:21, Brian Foster wrote:
[...]
> That aside, looking through some of the traces in this case...
> 
> - kswapd0 is waiting on an inode flush lock. This means somebody else
>   flushed the inode and it won't be unlocked until the underlying buffer
>   I/O is completed. This context is also holding pag_ici_reclaim_lock
>   which is what probably blocks other contexts from getting into inode
>   reclaim.
> - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
>   It's waiting on reading the underlying inode buffer. The buffer read
>   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
>   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
>   work item is what eventually triggers the I/O completion in
>   xfs_buf_ioend().
> 
> So at this point reclaim is waiting on a read I/O completion. It's not
> clear to me whether the read had completed and the work item was queued
> or not. I do see the following in the workqueue lockup BUG output:
> 
> [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> 
> ... which suggests that it was queued..? I suppose this could be one of
> the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> appears to be idle:
> 
> [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> [ 1041.556813] Call Trace:
> [ 1041.557796]  __schedule+0x336/0xe00
> [ 1041.558983]  schedule+0x3d/0x90
> [ 1041.560085]  rescuer_thread+0x322/0x3d0
> [ 1041.561333]  kthread+0x10f/0x150
> [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> [ 1041.565123]  ret_from_fork+0x31/0x40
> 
> So shouldn't that thread pick up the work item if that is the case?

Is it possible that the progress is done but tediously slow? Keep in
mind that the test case is doing write from 1k processes while one
process basically consumes all the memory. So I wouldn't be surprised
if this just made system to crawl on any attempt to do an IO.
-- 
Michal Hocko
SUSE Labs

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Michal Hocko]

On Fri 03-03-17 10:37:21, Brian Foster wrote:
[...]
> That aside, looking through some of the traces in this case...
> 
> - kswapd0 is waiting on an inode flush lock. This means somebody else
>   flushed the inode and it won't be unlocked until the underlying buffer
>   I/O is completed. This context is also holding pag_ici_reclaim_lock
>   which is what probably blocks other contexts from getting into inode
>   reclaim.
> - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
>   It's waiting on reading the underlying inode buffer. The buffer read
>   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
>   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
>   work item is what eventually triggers the I/O completion in
>   xfs_buf_ioend().
> 
> So at this point reclaim is waiting on a read I/O completion. It's not
> clear to me whether the read had completed and the work item was queued
> or not. I do see the following in the workqueue lockup BUG output:
> 
> [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> 
> ... which suggests that it was queued..? I suppose this could be one of
> the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> appears to be idle:
> 
> [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> [ 1041.556813] Call Trace:
> [ 1041.557796]  __schedule+0x336/0xe00
> [ 1041.558983]  schedule+0x3d/0x90
> [ 1041.560085]  rescuer_thread+0x322/0x3d0
> [ 1041.561333]  kthread+0x10f/0x150
> [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> [ 1041.565123]  ret_from_fork+0x31/0x40
> 
> So shouldn't that thread pick up the work item if that is the case?

Is it possible that the progress is done but tediously slow? Keep in
mind that the test case is doing write from 1k processes while one
process basically consumes all the memory. So I wouldn't be surprised
if this just made system to crawl on any attempt to do an IO.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Fri, Mar 03, 2017 at 04:52:58PM +0100, Michal Hocko wrote:
> On Fri 03-03-17 10:37:21, Brian Foster wrote:
> [...]
> > That aside, looking through some of the traces in this case...
> > 
> > - kswapd0 is waiting on an inode flush lock. This means somebody else
> >   flushed the inode and it won't be unlocked until the underlying buffer
> >   I/O is completed. This context is also holding pag_ici_reclaim_lock
> >   which is what probably blocks other contexts from getting into inode
> >   reclaim.
> > - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
> >   It's waiting on reading the underlying inode buffer. The buffer read
> >   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
> >   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
> >   work item is what eventually triggers the I/O completion in
> >   xfs_buf_ioend().
> > 
> > So at this point reclaim is waiting on a read I/O completion. It's not
> > clear to me whether the read had completed and the work item was queued
> > or not. I do see the following in the workqueue lockup BUG output:
> > 
> > [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> > [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> > [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> > 
> > ... which suggests that it was queued..? I suppose this could be one of
> > the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> > appears to be idle:
> > 
> > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > [ 1041.556813] Call Trace:
> > [ 1041.557796]  __schedule+0x336/0xe00
> > [ 1041.558983]  schedule+0x3d/0x90
> > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > [ 1041.561333]  kthread+0x10f/0x150
> > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > [ 1041.565123]  ret_from_fork+0x31/0x40
> > 
> > So shouldn't that thread pick up the work item if that is the case?
> 
> Is it possible that the progress is done but tediously slow? Keep in
> mind that the test case is doing write from 1k processes while one
> process basically consumes all the memory. So I wouldn't be surprised
> if this just made system to crawl on any attempt to do an IO.

That would seem like a possibility to me.. either waiting on an actual
I/O (no guarantee that the pending xfs-buf item is the one we care about
I suppose) completion or waiting for whatever needs to happen for the wq
infrastructure to kick off the rescuer. Though I think that's probably
something Tetsuo would ultimately have to confirm on his setup..

Brian

> -- 
> Michal Hocko
> SUSE Labs
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Fri, Mar 03, 2017 at 04:52:58PM +0100, Michal Hocko wrote:
> On Fri 03-03-17 10:37:21, Brian Foster wrote:
> [...]
> > That aside, looking through some of the traces in this case...
> > 
> > - kswapd0 is waiting on an inode flush lock. This means somebody else
> >   flushed the inode and it won't be unlocked until the underlying buffer
> >   I/O is completed. This context is also holding pag_ici_reclaim_lock
> >   which is what probably blocks other contexts from getting into inode
> >   reclaim.
> > - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
> >   It's waiting on reading the underlying inode buffer. The buffer read
> >   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
> >   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
> >   work item is what eventually triggers the I/O completion in
> >   xfs_buf_ioend().
> > 
> > So at this point reclaim is waiting on a read I/O completion. It's not
> > clear to me whether the read had completed and the work item was queued
> > or not. I do see the following in the workqueue lockup BUG output:
> > 
> > [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> > [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> > [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> > 
> > ... which suggests that it was queued..? I suppose this could be one of
> > the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> > appears to be idle:
> > 
> > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > [ 1041.556813] Call Trace:
> > [ 1041.557796]  __schedule+0x336/0xe00
> > [ 1041.558983]  schedule+0x3d/0x90
> > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > [ 1041.561333]  kthread+0x10f/0x150
> > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > [ 1041.565123]  ret_from_fork+0x31/0x40
> > 
> > So shouldn't that thread pick up the work item if that is the case?
> 
> Is it possible that the progress is done but tediously slow? Keep in
> mind that the test case is doing write from 1k processes while one
> process basically consumes all the memory. So I wouldn't be surprised
> if this just made system to crawl on any attempt to do an IO.

That would seem like a possibility to me.. either waiting on an actual
I/O (no guarantee that the pending xfs-buf item is the one we care about
I suppose) completion or waiting for whatever needs to happen for the wq
infrastructure to kick off the rescuer. Though I think that's probably
something Tetsuo would ultimately have to confirm on his setup..

Brian

> -- 
> Michal Hocko
> SUSE Labs
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Dave Chinner]

On Fri, Mar 03, 2017 at 02:39:51PM +0100, Michal Hocko wrote:
> On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> > Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> > 
> > While I was testing a patch which avoids infinite too_many_isolated() loop in
> > shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> > waiting for memory allocation. I guess that we overlooked a basic thing about
> > WQ_MEM_RECLAIM.
> > 
> >   WQ_MEM_RECLAIM helps only when the cause of failing to complete
> >   a work item is lack of "struct task_struct" to run that work item, for
> >   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
> >   will not be blocked waiting for memory allocation for "struct task_struct".
> > 
> >   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
> >   item is blocked waiting for memory allocation (or is indirectly blocked
> >   on a lock where the owner of that lock is blocked waiting for memory
> >   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
> >   if memory allocation (including indirect memory allocation via
> >   locks/completions) is needed.
> > 
> > In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> > "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> > workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> > 
> > What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> > I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> > workqueues (which are RESCUER) got stuck waiting for memory allocation.
> 
> If those workers are really required for a further progress of the
> memory reclaim then they shouldn't block on allocation at all and either
> use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
> that only very limited amount of memory is allocated from that context
> and there will be at least the same amount of memory freed as a result
> in a reasonable time.
> 
> This is something for xfs people to answer though. Please note that I
> didn't really have time to look through the below traces so the above
> note is rather generic. It would be really helpful if you could provide
> a high level dependency chains to see why those rescuers are necessary
> for the forward progress because it is really easy to get lost in so
> many traces.

Data IO completion is required to make progress to free memory. IO
completion is done via work queues, so they need rescuer threads to
ensure work can be run.

IO completion can require transactions to run. Transactions require
memory allocation. Freeing memory therefore requires IO completion
to have access to memory reserves if it's occurring from rescuer
threads to allow progress to be made.

That means metadata IO completion require rescuer threads, because
data IO completion can be dependent on metadata buffers being
available. e.g. reserving space in the log for the transaction can
require waiting on metadata IO dispatch and completion. Hence the
buffer IO completion workqueues need rescuer threads.

Transactions can also require log forces and flushes to occur, which
means they require the log workqueues (both the CIL flush and IO
completion workqueues) to make progress.  Log flushes also require
both IO and memory allocation to make progress to complete. Again,
this means the log workqueues need rescuer threads. It also needs
the log workqueues to be high priority so that they can make
progress before IO completion work that is dependent on
transactions making progress are processed.

IOWs, pretty much all the XFS workqueues are involved in memory
reclaim in one way or another.

The real problem here is that the XFS code has /no idea/ of what
workqueue context it is operating in - the fact it is in a rescuer
thread is completely hidden from the executing context. It seems to
me that the workqueue infrastructure's responsibility to tell memory
reclaim that the rescuer thread needs special access to the memory
reserves to allow the work it is running to allow forwards progress
to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
something similar...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Dave Chinner]

On Fri, Mar 03, 2017 at 02:39:51PM +0100, Michal Hocko wrote:
> On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> > Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> > 
> > While I was testing a patch which avoids infinite too_many_isolated() loop in
> > shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> > waiting for memory allocation. I guess that we overlooked a basic thing about
> > WQ_MEM_RECLAIM.
> > 
> >   WQ_MEM_RECLAIM helps only when the cause of failing to complete
> >   a work item is lack of "struct task_struct" to run that work item, for
> >   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
> >   will not be blocked waiting for memory allocation for "struct task_struct".
> > 
> >   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
> >   item is blocked waiting for memory allocation (or is indirectly blocked
> >   on a lock where the owner of that lock is blocked waiting for memory
> >   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
> >   if memory allocation (including indirect memory allocation via
> >   locks/completions) is needed.
> > 
> > In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> > "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> > workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> > 
> > What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> > I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> > workqueues (which are RESCUER) got stuck waiting for memory allocation.
> 
> If those workers are really required for a further progress of the
> memory reclaim then they shouldn't block on allocation at all and either
> use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
> that only very limited amount of memory is allocated from that context
> and there will be at least the same amount of memory freed as a result
> in a reasonable time.
> 
> This is something for xfs people to answer though. Please note that I
> didn't really have time to look through the below traces so the above
> note is rather generic. It would be really helpful if you could provide
> a high level dependency chains to see why those rescuers are necessary
> for the forward progress because it is really easy to get lost in so
> many traces.

Data IO completion is required to make progress to free memory. IO
completion is done via work queues, so they need rescuer threads to
ensure work can be run.

IO completion can require transactions to run. Transactions require
memory allocation. Freeing memory therefore requires IO completion
to have access to memory reserves if it's occurring from rescuer
threads to allow progress to be made.

That means metadata IO completion require rescuer threads, because
data IO completion can be dependent on metadata buffers being
available. e.g. reserving space in the log for the transaction can
require waiting on metadata IO dispatch and completion. Hence the
buffer IO completion workqueues need rescuer threads.

Transactions can also require log forces and flushes to occur, which
means they require the log workqueues (both the CIL flush and IO
completion workqueues) to make progress.  Log flushes also require
both IO and memory allocation to make progress to complete. Again,
this means the log workqueues need rescuer threads. It also needs
the log workqueues to be high priority so that they can make
progress before IO completion work that is dependent on
transactions making progress are processed.

IOWs, pretty much all the XFS workqueues are involved in memory
reclaim in one way or another.

The real problem here is that the XFS code has /no idea/ of what
workqueue context it is operating in - the fact it is in a rescuer
thread is completely hidden from the executing context. It seems to
me that the workqueue infrastructure's responsibility to tell memory
reclaim that the rescuer thread needs special access to the memory
reserves to allow the work it is running to allow forwards progress
to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
something similar...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tetsuo Handa]

Brian Foster wrote:
> On Fri, Mar 03, 2017 at 04:52:58PM +0100, Michal Hocko wrote:
> > On Fri 03-03-17 10:37:21, Brian Foster wrote:
> > [...]
> > > That aside, looking through some of the traces in this case...
> > > 
> > > - kswapd0 is waiting on an inode flush lock. This means somebody else
> > >   flushed the inode and it won't be unlocked until the underlying buffer
> > >   I/O is completed. This context is also holding pag_ici_reclaim_lock
> > >   which is what probably blocks other contexts from getting into inode
> > >   reclaim.
> > > - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
> > >   It's waiting on reading the underlying inode buffer. The buffer read
> > >   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
> > >   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
> > >   work item is what eventually triggers the I/O completion in
> > >   xfs_buf_ioend().
> > > 
> > > So at this point reclaim is waiting on a read I/O completion. It's not
> > > clear to me whether the read had completed and the work item was queued
> > > or not. I do see the following in the workqueue lockup BUG output:
> > > 
> > > [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> > > [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> > > [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> > > 
> > > ... which suggests that it was queued..? I suppose this could be one of
> > > the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> > > appears to be idle:
> > > 
> > > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > > [ 1041.556813] Call Trace:
> > > [ 1041.557796]  __schedule+0x336/0xe00
> > > [ 1041.558983]  schedule+0x3d/0x90
> > > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > > [ 1041.561333]  kthread+0x10f/0x150
> > > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > > [ 1041.565123]  ret_from_fork+0x31/0x40
> > > 
> > > So shouldn't that thread pick up the work item if that is the case?
> > 
> > Is it possible that the progress is done but tediously slow? Keep in
> > mind that the test case is doing write from 1k processes while one
> > process basically consumes all the memory. So I wouldn't be surprised
> > if this just made system to crawl on any attempt to do an IO.
> 
> That would seem like a possibility to me.. either waiting on an actual
> I/O (no guarantee that the pending xfs-buf item is the one we care about
> I suppose) completion or waiting for whatever needs to happen for the wq
> infrastructure to kick off the rescuer. Though I think that's probably
> something Tetsuo would ultimately have to confirm on his setup..

This lockup began from uptime = 444. Thus, please ignore logs up to

[  444.281177] Killed process 9477 (a.out) total-vm:4168kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB
[  444.287046] oom_reaper: reaped process 9477 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

line because memory reclaim was making progress and the OOM killer was being invoked.
Also, while it is true that this stressing program tries to create 1024 child processes,
majority of child processes are simply waiting at open() syscall

[ 1056.479865] a.out           D13992  9655   9473 0x00000080
[ 1056.481471] Call Trace:
[ 1056.482474]  __schedule+0x336/0xe00
[ 1056.483690]  schedule+0x3d/0x90
[ 1056.484825]  rwsem_down_write_failed+0x240/0x470
[ 1056.486252]  ? rwsem_down_write_failed+0x65/0x470
[ 1056.487703]  call_rwsem_down_write_failed+0x17/0x30
[ 1056.489185]  ? path_openat+0x60b/0xd50
[ 1056.490441]  down_write+0x95/0xc0
[ 1056.491614]  ? path_openat+0x60b/0xd50
[ 1056.492865]  path_openat+0x60b/0xd50
[ 1056.494089]  ? ___slab_alloc+0x5c6/0x620
[ 1056.495370]  do_filp_open+0x91/0x100
[ 1056.496574]  ? _raw_spin_unlock+0x27/0x40
[ 1056.497873]  ? __alloc_fd+0xf7/0x210
[ 1056.499086]  do_sys_open+0x124/0x210
[ 1056.500290]  SyS_open+0x1e/0x20
[ 1056.501402]  do_syscall_64+0x6c/0x200
[ 1056.502611]  entry_SYSCALL64_slow_path+0x25/0x25

which means that these child processes are irrelevant to this problem.
They are simply trying to make this stressing program last for a few minutes
in order to eliminate time waiting for memory consumer process to consume
all memory again and again.

Only 184 processes were doing memory allocation when lockup began.

[  518.090012] MemAlloc-Info: stalling=184 dying=1 exiting=0 victim=1 oom_count=8441307
[  553.070829] MemAlloc-Info: stalling=184 dying=1 exiting=0 victim=1 oom_count=10318507
[  616.394649] MemAlloc-Info: stalling=186 dying=1 exiting=0 victim=1 oom_count=13908219
[  642.266252] MemAlloc-Info: stalling=186 dying=1 exiting=0 victim=1 oom_count=15180673
[  702.412189] MemAlloc-Info: stalling=187 dying=1 exiting=0 victim=1 oom_count=18732529
[  736.787879] MemAlloc-Info: stalling=187 dying=1 exiting=0 victim=1 oom_count=20565244
[  800.715759] MemAlloc-Info: stalling=188 dying=1 exiting=0 victim=1 oom_count=24411576
[  837.571405] MemAlloc-Info: stalling=188 dying=1 exiting=0 victim=1 oom_count=26463562
[  899.021495] MemAlloc-Info: stalling=189 dying=1 exiting=0 victim=1 oom_count=30144879
[  936.282709] MemAlloc-Info: stalling=189 dying=1 exiting=0 victim=1 oom_count=32129234
[  997.328119] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=35657983
[ 1033.977265] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=37659912
[ 1095.630961] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=40639677
[ 1095.821248] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=40646791

But most of processes doing memory allocation were blocked on locks when lockup began.

[  518.092627] MemAlloc: kthreadd(2) flags=0x208840 switches=313 seq=5 gfp=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK) order=2 delay=74546 uninterruptible
[  518.319917] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=14350 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=74621
[  518.437336] MemAlloc: khugepaged(47) flags=0x200840 switches=26 seq=8 gfp=0x4742ca(GFP_TRANSHUGE|__GFP_THISNODE) order=9 delay=36854 uninterruptible
[  518.605804] MemAlloc: kworker/1:1(52) flags=0x4228060 switches=4112 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74544
[  518.756891] MemAlloc: kworker/2:1(65) flags=0x4228860 switches=2749 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74504
[  518.900038] MemAlloc: kswapd0(69) flags=0xa40840 switches=23883 uninterruptible
[  518.963034] MemAlloc: kworker/1:2(193) flags=0x4228060 switches=4426 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545 uninterruptible
[  519.049597] MemAlloc: xfs-data/sda1(451) flags=0x4228860 switches=6525 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74534
[  519.135487] MemAlloc: systemd-journal(526) flags=0x400900 switches=22443 seq=43228 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74224 uninterruptible
[  519.371973] MemAlloc: kworker/2:3(535) flags=0x4228060 switches=29263 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545 uninterruptible
[  519.502080] MemAlloc: auditd(564) flags=0x400900 switches=897 seq=250 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=59753 uninterruptible
[  519.681032] MemAlloc: tuned(2526) flags=0x400840 switches=22780 seq=42374 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74205 uninterruptible
[  519.850986] MemAlloc: irqbalance(726) flags=0x400900 switches=17821 seq=18779 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73843 uninterruptible
[  520.003760] MemAlloc: vmtoolsd(737) flags=0x400900 switches=23616 seq=38031 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74364 uninterruptible
[  520.171223] MemAlloc: systemd-logind(817) flags=0x400900 switches=2878 seq=7734 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=56256 uninterruptible
[  520.374034] MemAlloc: crond(859) flags=0x400900 switches=5562 seq=6922 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=43400 uninterruptible
[  520.526133] MemAlloc: gmain(2873) flags=0x400840 switches=19609 seq=19603 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74080 uninterruptible
[  520.749935] MemAlloc: kworker/0:3(2486) flags=0x4228060 switches=37566 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74525 uninterruptible
[  520.814386] MemAlloc: master(2491) flags=0x400940 switches=5275 seq=4376 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=46072 uninterruptible
[  520.909697] MemAlloc: vmtoolsd(2497) flags=0x400900 switches=22477 seq=33640 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74173 uninterruptible
[  521.011345] MemAlloc: nmbd(2521) flags=0x400940 switches=22908 seq=36949 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=72001 uninterruptible
[  521.243774] MemAlloc: lpqd(3032) flags=0x400840 switches=7889 seq=11697 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=58023 uninterruptible
[  521.472130] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=16212 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74547
[  521.536605] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=6647 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.692946] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=3718 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.817708] MemAlloc: kworker/3:5(6386) flags=0x4228060 switches=30373 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548
[  521.955723] MemAlloc: kworker/2:4(7415) flags=0x4228860 switches=3304 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74539
[  522.115239] MemAlloc: kworker/2:5(7416) flags=0x4228060 switches=2851 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74530 uninterruptible
[  522.246011] MemAlloc: kworker/1:5(7418) flags=0x4208860 switches=17514 seq=5 gfp=0x14002c2(GFP_KERNEL|__GFP_HIGHMEM|__GFP_NOWARN) order=0 delay=73160 uninterruptible
[  522.418809] MemAlloc: a.out(9473) flags=0x400800 switches=153 seq=10699 gfp=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO) order=0 delay=75047 uninterruptible
[  522.576919] MemAlloc: a.out(9477) flags=0x420040 switches=27 uninterruptible dying victim
[  522.797074] MemAlloc: a.out(9480) flags=0x400840 switches=39 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  523.008893] MemAlloc: a.out(9481) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  523.230713] MemAlloc: a.out(9482) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  523.440192] MemAlloc: a.out(9483) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  523.622972] MemAlloc: a.out(9485) flags=0x400840 switches=32 seq=5 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  523.802630] MemAlloc: a.out(9486) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  523.977233] MemAlloc: a.out(9492) flags=0x400840 switches=28 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  524.143441] MemAlloc: a.out(9494) flags=0x400840 switches=30 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  524.303139] MemAlloc: a.out(9495) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  524.477860] MemAlloc: a.out(9496) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  524.605941] MemAlloc: a.out(9497) flags=0x400840 switches=53 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  524.818506] MemAlloc: a.out(9498) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  525.053989] MemAlloc: a.out(9499) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  525.209989] MemAlloc: a.out(9500) flags=0x400840 switches=28 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  525.398782] MemAlloc: a.out(9501) flags=0x400840 switches=34 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  525.575204] MemAlloc: a.out(9502) flags=0x400840 switches=44 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  525.865094] MemAlloc: a.out(9503) flags=0x400840 switches=30 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.048023] MemAlloc: a.out(9504) flags=0x400840 switches=30 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.228154] MemAlloc: a.out(9505) flags=0x400840 switches=32 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.359385] MemAlloc: a.out(9507) flags=0x400840 switches=30 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73842 uninterruptible
[  526.568938] MemAlloc: a.out(9508) flags=0x400840 switches=28 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  526.744746] MemAlloc: a.out(9509) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.945986] MemAlloc: a.out(9510) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73963 uninterruptible
[  527.166154] MemAlloc: a.out(9511) flags=0x400040 switches=1996 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74941
[  527.324939] MemAlloc: a.out(9512) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  527.526047] MemAlloc: a.out(9513) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  527.726877] MemAlloc: a.out(9514) flags=0x400840 switches=40 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  527.952047] MemAlloc: a.out(9515) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  528.132160] MemAlloc: a.out(9516) flags=0x400840 switches=63 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73962 uninterruptible
[  528.329563] MemAlloc: a.out(9517) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  528.501754] MemAlloc: a.out(9519) flags=0x400840 switches=24 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  528.649021] MemAlloc: a.out(9520) flags=0x400840 switches=29 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  528.842687] MemAlloc: a.out(9522) flags=0x400840 switches=37 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73961 uninterruptible
[  529.014041] MemAlloc: a.out(9523) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  529.193899] MemAlloc: a.out(9524) flags=0x400840 switches=24 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  529.360486] MemAlloc: a.out(9525) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  529.572691] MemAlloc: a.out(9526) flags=0x400840 switches=39 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  529.924087] MemAlloc: a.out(9527) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.137611] MemAlloc: a.out(9528) flags=0x400840 switches=27 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.359847] MemAlloc: a.out(9529) flags=0x400840 switches=26 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.569215] MemAlloc: a.out(9530) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.784715] MemAlloc: a.out(9531) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  530.993554] MemAlloc: a.out(9532) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  531.163695] MemAlloc: a.out(9533) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  531.340946] MemAlloc: a.out(9534) flags=0x400840 switches=27 seq=5 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  531.517626] MemAlloc: a.out(9536) flags=0x400840 switches=21 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  531.683190] MemAlloc: a.out(9537) flags=0x400840 switches=74 seq=191 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74134 uninterruptible
[  531.843838] MemAlloc: a.out(9538) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  532.005303] MemAlloc: a.out(9541) flags=0x400840 switches=40 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74231 uninterruptible
[  532.195906] MemAlloc: a.out(9542) flags=0x400840 switches=31 seq=35 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74256 uninterruptible
[  532.430946] MemAlloc: a.out(9545) flags=0x420840 switches=2014 seq=7 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=1 delay=74581
[  532.642039] MemAlloc: a.out(9546) flags=0x400840 switches=23 seq=79 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  532.875209] MemAlloc: a.out(9550) flags=0x400840 switches=21 seq=5 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  532.995120] MemAlloc: a.out(9551) flags=0x420040 switches=2151 seq=3 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74577
[  533.160574] MemAlloc: a.out(9554) flags=0x400840 switches=20 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73962 uninterruptible
[  533.383124] MemAlloc: a.out(9555) flags=0x400040 switches=2059 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74949
[  533.548238] MemAlloc: a.out(9556) flags=0x400840 switches=22 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  533.793954] MemAlloc: a.out(9558) flags=0x400840 switches=22 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  534.027669] MemAlloc: a.out(9560) flags=0x400840 switches=22 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  534.274691] MemAlloc: a.out(9561) flags=0x400840 switches=21 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  534.479135] MemAlloc: a.out(9562) flags=0x400840 switches=17 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  534.644896] MemAlloc: a.out(9563) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  534.924052] MemAlloc: a.out(9564) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  535.202235] MemAlloc: a.out(9565) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  535.436619] MemAlloc: a.out(9566) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  535.638534] MemAlloc: a.out(9567) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  535.886729] MemAlloc: a.out(9568) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  536.123130] MemAlloc: a.out(9569) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  536.358902] MemAlloc: a.out(9570) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  536.603860] MemAlloc: a.out(9571) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  536.709749] MemAlloc: a.out(9572) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  536.942101] MemAlloc: a.out(9573) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  537.127875] MemAlloc: a.out(9574) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  537.310662] MemAlloc: a.out(9575) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  537.455864] MemAlloc: a.out(9576) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  537.675090] MemAlloc: a.out(9579) flags=0x400040 switches=2251 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74933
[  537.779100] MemAlloc: a.out(9581) flags=0x400840 switches=26 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  537.925518] MemAlloc: a.out(9583) flags=0x400840 switches=46 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  538.169936] MemAlloc: a.out(9586) flags=0x400840 switches=32 seq=129 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74232 uninterruptible
[  538.406918] MemAlloc: a.out(9587) flags=0x400840 switches=28 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  538.642036] MemAlloc: a.out(9591) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  538.866188] MemAlloc: a.out(9592) flags=0x400840 switches=49 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  539.119566] MemAlloc: a.out(9593) flags=0x400840 switches=65 seq=35 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73960 uninterruptible
[  539.378634] MemAlloc: a.out(9595) flags=0x400040 switches=32 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74939 uninterruptible
[  539.556915] MemAlloc: a.out(9597) flags=0x400840 switches=48 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  539.787355] MemAlloc: a.out(9599) flags=0x400840 switches=24 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.015041] MemAlloc: a.out(9600) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  540.148695] MemAlloc: a.out(9605) flags=0x400840 switches=27 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  540.237056] MemAlloc: a.out(9606) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  540.333568] MemAlloc: a.out(9607) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  540.434293] MemAlloc: a.out(9608) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.590569] MemAlloc: a.out(9609) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.732138] MemAlloc: a.out(9610) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.961649] MemAlloc: a.out(9611) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  541.218220] MemAlloc: a.out(9612) flags=0x400840 switches=16 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  541.375479] MemAlloc: a.out(9613) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  541.571384] MemAlloc: a.out(9614) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  541.754231] MemAlloc: a.out(9615) flags=0x400840 switches=17 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  541.896925] MemAlloc: a.out(9616) flags=0x400840 switches=18 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  542.056536] MemAlloc: a.out(9617) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  542.229652] MemAlloc: a.out(9618) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  542.364170] MemAlloc: a.out(9619) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  542.493472] MemAlloc: a.out(9620) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  542.679770] MemAlloc: a.out(9621) flags=0x400840 switches=16 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  542.865175] MemAlloc: a.out(9622) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  542.972020] MemAlloc: a.out(9623) flags=0x400840 switches=15 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  543.135246] MemAlloc: a.out(9624) flags=0x400840 switches=18 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  543.358968] MemAlloc: a.out(9625) flags=0x400840 switches=16 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  543.583466] MemAlloc: a.out(9626) flags=0x400840 switches=17 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  543.809065] MemAlloc: a.out(9627) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.017351] MemAlloc: a.out(9628) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.217362] MemAlloc: a.out(9629) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.462068] MemAlloc: a.out(9630) flags=0x400840 switches=21 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  544.699780] MemAlloc: a.out(9631) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.895071] MemAlloc: a.out(9632) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  545.080182] MemAlloc: a.out(9633) flags=0x400840 switches=31 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  545.352169] MemAlloc: a.out(9634) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  545.493870] MemAlloc: a.out(9635) flags=0x400840 switches=14 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  545.609266] MemAlloc: a.out(9636) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  545.725966] MemAlloc: a.out(9637) flags=0x400840 switches=18 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73963 uninterruptible
[  545.812656] MemAlloc: a.out(9638) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  545.984346] MemAlloc: a.out(9639) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  546.170187] MemAlloc: a.out(9640) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  546.351502] MemAlloc: a.out(9641) flags=0x400840 switches=14 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  546.529103] MemAlloc: a.out(9646) flags=0x400840 switches=11 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  546.710185] MemAlloc: a.out(9647) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  546.852608] MemAlloc: a.out(9648) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.014298] MemAlloc: a.out(9650) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.188938] MemAlloc: a.out(9651) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.362058] MemAlloc: a.out(9652) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.471970] MemAlloc: a.out(9653) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.706983] MemAlloc: a.out(9654) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.870033] MemAlloc: a.out(9656) flags=0x400840 switches=12 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.079204] MemAlloc: a.out(9662) flags=0x400840 switches=12 seq=209 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.305820] MemAlloc: a.out(9664) flags=0x400840 switches=20 seq=302 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.525819] MemAlloc: a.out(9666) flags=0x400840 switches=11 seq=25 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.793002] MemAlloc: a.out(9668) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  549.006725] MemAlloc: a.out(9670) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.232993] MemAlloc: a.out(9671) flags=0x400840 switches=10 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.469884] MemAlloc: a.out(9672) flags=0x400840 switches=11 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.727175] MemAlloc: a.out(9673) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.987420] MemAlloc: a.out(9674) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  550.214963] MemAlloc: a.out(9675) flags=0x400840 switches=10 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  550.421057] MemAlloc: a.out(9676) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  550.654050] MemAlloc: a.out(9677) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  550.841175] MemAlloc: a.out(9678) flags=0x400840 switches=13 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  551.030916] MemAlloc: a.out(9679) flags=0x400840 switches=13 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  551.213400] MemAlloc: a.out(9681) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  551.401337] MemAlloc: a.out(9682) flags=0x400840 switches=13 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  551.604094] MemAlloc: a.out(9683) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  551.783882] MemAlloc: a.out(9684) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  551.971142] MemAlloc: a.out(9685) flags=0x420840 switches=2357 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74818
[  552.137039] MemAlloc: a.out(9686) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  552.342766] MemAlloc: a.out(9689) flags=0x400040 switches=2370 seq=3 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74818
[  552.453767] MemAlloc: kworker/3:0(10498) flags=0x4228860 switches=4184 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548

Only 17 out of 186 processes were able to loop inside page allocator.

[  518.319917] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=14350 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=74621
[  518.605804] MemAlloc: kworker/1:1(52) flags=0x4228060 switches=4112 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74544
[  518.756891] MemAlloc: kworker/2:1(65) flags=0x4228860 switches=2749 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74504
[  519.049597] MemAlloc: xfs-data/sda1(451) flags=0x4228860 switches=6525 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74534
[  521.472130] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=16212 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74547
[  521.536605] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=6647 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.692946] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=3718 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.817708] MemAlloc: kworker/3:5(6386) flags=0x4228060 switches=30373 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548
[  521.955723] MemAlloc: kworker/2:4(7415) flags=0x4228860 switches=3304 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74539
[  527.166154] MemAlloc: a.out(9511) flags=0x400040 switches=1996 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74941
[  532.430946] MemAlloc: a.out(9545) flags=0x420840 switches=2014 seq=7 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=1 delay=74581
[  532.995120] MemAlloc: a.out(9551) flags=0x420040 switches=2151 seq=3 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74577
[  533.383124] MemAlloc: a.out(9555) flags=0x400040 switches=2059 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74949
[  537.675090] MemAlloc: a.out(9579) flags=0x400040 switches=2251 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74933
[  551.971142] MemAlloc: a.out(9685) flags=0x420840 switches=2357 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74818
[  552.342766] MemAlloc: a.out(9689) flags=0x400040 switches=2370 seq=3 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74818
[  552.453767] MemAlloc: kworker/3:0(10498) flags=0x4228860 switches=4184 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548

Situation was similar when I gave up waiting.
Only 20 out of 192 processes were able to loop inside page allocator.

[ 1095.631687] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=38727 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=652160
[ 1095.632186] MemAlloc: kworker/1:1(52) flags=0x4228860 switches=28036 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652083
[ 1095.632604] MemAlloc: kworker/2:1(65) flags=0x4228060 switches=22879 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652043
[ 1095.633213] MemAlloc: kworker/1:2(193) flags=0x4228860 switches=28494 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
[ 1095.634013] MemAlloc: xfs-eofblocks/s(456) flags=0x4228860 switches=15435 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=293074
[ 1095.635223] MemAlloc: kworker/2:3(535) flags=0x4228860 switches=49285 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.638807] MemAlloc: kworker/0:3(2486) flags=0x4228860 switches=76240 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652064
[ 1095.644936] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=29192 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652086
[ 1095.645241] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=30893 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.645506] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=27329 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.647293] MemAlloc: kworker/3:5(6386) flags=0x4228860 switches=43427 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
[ 1095.691898] MemAlloc: a.out(9511) flags=0x400040 switches=14710 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652480
[ 1095.726301] MemAlloc: a.out(9545) flags=0x420040 switches=14120 seq=7 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=1 delay=652120
[ 1095.727373] MemAlloc: a.out(9551) flags=0x420840 switches=14617 seq=3 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=652116
[ 1095.728228] MemAlloc: a.out(9555) flags=0x400040 switches=14575 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652488
[ 1095.739636] MemAlloc: a.out(9579) flags=0x400840 switches=14791 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652472
[ 1095.805447] MemAlloc: a.out(9685) flags=0x420840 switches=14309 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=652357
[ 1095.806316] MemAlloc: a.out(9689) flags=0x400840 switches=14131 seq=3 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652357
[ 1095.806606] MemAlloc: kworker/3:0(10498) flags=0x4228060 switches=16222 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087

You can't say that 1024 processes were doing write() requests. Many were still
blocked at open(). Only 20 or so processes were able to consume CPU time for
memory allocation. I don't think this is a too insane stress to wait. This is not
a stress which can justify stalls over 10 minutes. The kworker/2:0(27) line
indicates that order-0 GFP_NOIO allocation request was unable to find a page
for more than 10 minutes. Under such situation, how can other GFP_NOFS allocation
requests find a page without boosting priority (unless someone releases memory
voluntarily) ?

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tetsuo Handa]

Brian Foster wrote:
> On Fri, Mar 03, 2017 at 04:52:58PM +0100, Michal Hocko wrote:
> > On Fri 03-03-17 10:37:21, Brian Foster wrote:
> > [...]
> > > That aside, looking through some of the traces in this case...
> > > 
> > > - kswapd0 is waiting on an inode flush lock. This means somebody else
> > >   flushed the inode and it won't be unlocked until the underlying buffer
> > >   I/O is completed. This context is also holding pag_ici_reclaim_lock
> > >   which is what probably blocks other contexts from getting into inode
> > >   reclaim.
> > > - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
> > >   It's waiting on reading the underlying inode buffer. The buffer read
> > >   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
> > >   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
> > >   work item is what eventually triggers the I/O completion in
> > >   xfs_buf_ioend().
> > > 
> > > So at this point reclaim is waiting on a read I/O completion. It's not
> > > clear to me whether the read had completed and the work item was queued
> > > or not. I do see the following in the workqueue lockup BUG output:
> > > 
> > > [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> > > [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> > > [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> > > 
> > > ... which suggests that it was queued..? I suppose this could be one of
> > > the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> > > appears to be idle:
> > > 
> > > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > > [ 1041.556813] Call Trace:
> > > [ 1041.557796]  __schedule+0x336/0xe00
> > > [ 1041.558983]  schedule+0x3d/0x90
> > > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > > [ 1041.561333]  kthread+0x10f/0x150
> > > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > > [ 1041.565123]  ret_from_fork+0x31/0x40
> > > 
> > > So shouldn't that thread pick up the work item if that is the case?
> > 
> > Is it possible that the progress is done but tediously slow? Keep in
> > mind that the test case is doing write from 1k processes while one
> > process basically consumes all the memory. So I wouldn't be surprised
> > if this just made system to crawl on any attempt to do an IO.
> 
> That would seem like a possibility to me.. either waiting on an actual
> I/O (no guarantee that the pending xfs-buf item is the one we care about
> I suppose) completion or waiting for whatever needs to happen for the wq
> infrastructure to kick off the rescuer. Though I think that's probably
> something Tetsuo would ultimately have to confirm on his setup..

This lockup began from uptime = 444. Thus, please ignore logs up to

[  444.281177] Killed process 9477 (a.out) total-vm:4168kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB
[  444.287046] oom_reaper: reaped process 9477 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

line because memory reclaim was making progress and the OOM killer was being invoked.
Also, while it is true that this stressing program tries to create 1024 child processes,
majority of child processes are simply waiting at open() syscall

[ 1056.479865] a.out           D13992  9655   9473 0x00000080
[ 1056.481471] Call Trace:
[ 1056.482474]  __schedule+0x336/0xe00
[ 1056.483690]  schedule+0x3d/0x90
[ 1056.484825]  rwsem_down_write_failed+0x240/0x470
[ 1056.486252]  ? rwsem_down_write_failed+0x65/0x470
[ 1056.487703]  call_rwsem_down_write_failed+0x17/0x30
[ 1056.489185]  ? path_openat+0x60b/0xd50
[ 1056.490441]  down_write+0x95/0xc0
[ 1056.491614]  ? path_openat+0x60b/0xd50
[ 1056.492865]  path_openat+0x60b/0xd50
[ 1056.494089]  ? ___slab_alloc+0x5c6/0x620
[ 1056.495370]  do_filp_open+0x91/0x100
[ 1056.496574]  ? _raw_spin_unlock+0x27/0x40
[ 1056.497873]  ? __alloc_fd+0xf7/0x210
[ 1056.499086]  do_sys_open+0x124/0x210
[ 1056.500290]  SyS_open+0x1e/0x20
[ 1056.501402]  do_syscall_64+0x6c/0x200
[ 1056.502611]  entry_SYSCALL64_slow_path+0x25/0x25

which means that these child processes are irrelevant to this problem.
They are simply trying to make this stressing program last for a few minutes
in order to eliminate time waiting for memory consumer process to consume
all memory again and again.

Only 184 processes were doing memory allocation when lockup began.

[  518.090012] MemAlloc-Info: stalling=184 dying=1 exiting=0 victim=1 oom_count=8441307
[  553.070829] MemAlloc-Info: stalling=184 dying=1 exiting=0 victim=1 oom_count=10318507
[  616.394649] MemAlloc-Info: stalling=186 dying=1 exiting=0 victim=1 oom_count=13908219
[  642.266252] MemAlloc-Info: stalling=186 dying=1 exiting=0 victim=1 oom_count=15180673
[  702.412189] MemAlloc-Info: stalling=187 dying=1 exiting=0 victim=1 oom_count=18732529
[  736.787879] MemAlloc-Info: stalling=187 dying=1 exiting=0 victim=1 oom_count=20565244
[  800.715759] MemAlloc-Info: stalling=188 dying=1 exiting=0 victim=1 oom_count=24411576
[  837.571405] MemAlloc-Info: stalling=188 dying=1 exiting=0 victim=1 oom_count=26463562
[  899.021495] MemAlloc-Info: stalling=189 dying=1 exiting=0 victim=1 oom_count=30144879
[  936.282709] MemAlloc-Info: stalling=189 dying=1 exiting=0 victim=1 oom_count=32129234
[  997.328119] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=35657983
[ 1033.977265] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=37659912
[ 1095.630961] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=40639677
[ 1095.821248] MemAlloc-Info: stalling=190 dying=1 exiting=0 victim=1 oom_count=40646791

But most of processes doing memory allocation were blocked on locks when lockup began.

[  518.092627] MemAlloc: kthreadd(2) flags=0x208840 switches=313 seq=5 gfp=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK) order=2 delay=74546 uninterruptible
[  518.319917] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=14350 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=74621
[  518.437336] MemAlloc: khugepaged(47) flags=0x200840 switches=26 seq=8 gfp=0x4742ca(GFP_TRANSHUGE|__GFP_THISNODE) order=9 delay=36854 uninterruptible
[  518.605804] MemAlloc: kworker/1:1(52) flags=0x4228060 switches=4112 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74544
[  518.756891] MemAlloc: kworker/2:1(65) flags=0x4228860 switches=2749 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74504
[  518.900038] MemAlloc: kswapd0(69) flags=0xa40840 switches=23883 uninterruptible
[  518.963034] MemAlloc: kworker/1:2(193) flags=0x4228060 switches=4426 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545 uninterruptible
[  519.049597] MemAlloc: xfs-data/sda1(451) flags=0x4228860 switches=6525 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74534
[  519.135487] MemAlloc: systemd-journal(526) flags=0x400900 switches=22443 seq=43228 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74224 uninterruptible
[  519.371973] MemAlloc: kworker/2:3(535) flags=0x4228060 switches=29263 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545 uninterruptible
[  519.502080] MemAlloc: auditd(564) flags=0x400900 switches=897 seq=250 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=59753 uninterruptible
[  519.681032] MemAlloc: tuned(2526) flags=0x400840 switches=22780 seq=42374 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74205 uninterruptible
[  519.850986] MemAlloc: irqbalance(726) flags=0x400900 switches=17821 seq=18779 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73843 uninterruptible
[  520.003760] MemAlloc: vmtoolsd(737) flags=0x400900 switches=23616 seq=38031 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74364 uninterruptible
[  520.171223] MemAlloc: systemd-logind(817) flags=0x400900 switches=2878 seq=7734 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=56256 uninterruptible
[  520.374034] MemAlloc: crond(859) flags=0x400900 switches=5562 seq=6922 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=43400 uninterruptible
[  520.526133] MemAlloc: gmain(2873) flags=0x400840 switches=19609 seq=19603 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74080 uninterruptible
[  520.749935] MemAlloc: kworker/0:3(2486) flags=0x4228060 switches=37566 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74525 uninterruptible
[  520.814386] MemAlloc: master(2491) flags=0x400940 switches=5275 seq=4376 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=46072 uninterruptible
[  520.909697] MemAlloc: vmtoolsd(2497) flags=0x400900 switches=22477 seq=33640 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74173 uninterruptible
[  521.011345] MemAlloc: nmbd(2521) flags=0x400940 switches=22908 seq=36949 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=72001 uninterruptible
[  521.243774] MemAlloc: lpqd(3032) flags=0x400840 switches=7889 seq=11697 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=58023 uninterruptible
[  521.472130] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=16212 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74547
[  521.536605] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=6647 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.692946] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=3718 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.817708] MemAlloc: kworker/3:5(6386) flags=0x4228060 switches=30373 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548
[  521.955723] MemAlloc: kworker/2:4(7415) flags=0x4228860 switches=3304 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74539
[  522.115239] MemAlloc: kworker/2:5(7416) flags=0x4228060 switches=2851 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74530 uninterruptible
[  522.246011] MemAlloc: kworker/1:5(7418) flags=0x4208860 switches=17514 seq=5 gfp=0x14002c2(GFP_KERNEL|__GFP_HIGHMEM|__GFP_NOWARN) order=0 delay=73160 uninterruptible
[  522.418809] MemAlloc: a.out(9473) flags=0x400800 switches=153 seq=10699 gfp=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO) order=0 delay=75047 uninterruptible
[  522.576919] MemAlloc: a.out(9477) flags=0x420040 switches=27 uninterruptible dying victim
[  522.797074] MemAlloc: a.out(9480) flags=0x400840 switches=39 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  523.008893] MemAlloc: a.out(9481) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  523.230713] MemAlloc: a.out(9482) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  523.440192] MemAlloc: a.out(9483) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  523.622972] MemAlloc: a.out(9485) flags=0x400840 switches=32 seq=5 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  523.802630] MemAlloc: a.out(9486) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  523.977233] MemAlloc: a.out(9492) flags=0x400840 switches=28 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  524.143441] MemAlloc: a.out(9494) flags=0x400840 switches=30 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  524.303139] MemAlloc: a.out(9495) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  524.477860] MemAlloc: a.out(9496) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  524.605941] MemAlloc: a.out(9497) flags=0x400840 switches=53 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  524.818506] MemAlloc: a.out(9498) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  525.053989] MemAlloc: a.out(9499) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  525.209989] MemAlloc: a.out(9500) flags=0x400840 switches=28 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  525.398782] MemAlloc: a.out(9501) flags=0x400840 switches=34 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  525.575204] MemAlloc: a.out(9502) flags=0x400840 switches=44 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  525.865094] MemAlloc: a.out(9503) flags=0x400840 switches=30 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.048023] MemAlloc: a.out(9504) flags=0x400840 switches=30 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.228154] MemAlloc: a.out(9505) flags=0x400840 switches=32 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.359385] MemAlloc: a.out(9507) flags=0x400840 switches=30 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73842 uninterruptible
[  526.568938] MemAlloc: a.out(9508) flags=0x400840 switches=28 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  526.744746] MemAlloc: a.out(9509) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  526.945986] MemAlloc: a.out(9510) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73963 uninterruptible
[  527.166154] MemAlloc: a.out(9511) flags=0x400040 switches=1996 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74941
[  527.324939] MemAlloc: a.out(9512) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  527.526047] MemAlloc: a.out(9513) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  527.726877] MemAlloc: a.out(9514) flags=0x400840 switches=40 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  527.952047] MemAlloc: a.out(9515) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  528.132160] MemAlloc: a.out(9516) flags=0x400840 switches=63 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73962 uninterruptible
[  528.329563] MemAlloc: a.out(9517) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  528.501754] MemAlloc: a.out(9519) flags=0x400840 switches=24 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  528.649021] MemAlloc: a.out(9520) flags=0x400840 switches=29 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  528.842687] MemAlloc: a.out(9522) flags=0x400840 switches=37 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73961 uninterruptible
[  529.014041] MemAlloc: a.out(9523) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  529.193899] MemAlloc: a.out(9524) flags=0x400840 switches=24 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  529.360486] MemAlloc: a.out(9525) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  529.572691] MemAlloc: a.out(9526) flags=0x400840 switches=39 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  529.924087] MemAlloc: a.out(9527) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.137611] MemAlloc: a.out(9528) flags=0x400840 switches=27 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.359847] MemAlloc: a.out(9529) flags=0x400840 switches=26 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.569215] MemAlloc: a.out(9530) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  530.784715] MemAlloc: a.out(9531) flags=0x400840 switches=25 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  530.993554] MemAlloc: a.out(9532) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  531.163695] MemAlloc: a.out(9533) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  531.340946] MemAlloc: a.out(9534) flags=0x400840 switches=27 seq=5 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  531.517626] MemAlloc: a.out(9536) flags=0x400840 switches=21 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  531.683190] MemAlloc: a.out(9537) flags=0x400840 switches=74 seq=191 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74134 uninterruptible
[  531.843838] MemAlloc: a.out(9538) flags=0x400840 switches=26 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  532.005303] MemAlloc: a.out(9541) flags=0x400840 switches=40 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74231 uninterruptible
[  532.195906] MemAlloc: a.out(9542) flags=0x400840 switches=31 seq=35 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74256 uninterruptible
[  532.430946] MemAlloc: a.out(9545) flags=0x420840 switches=2014 seq=7 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=1 delay=74581
[  532.642039] MemAlloc: a.out(9546) flags=0x400840 switches=23 seq=79 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  532.875209] MemAlloc: a.out(9550) flags=0x400840 switches=21 seq=5 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  532.995120] MemAlloc: a.out(9551) flags=0x420040 switches=2151 seq=3 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74577
[  533.160574] MemAlloc: a.out(9554) flags=0x400840 switches=20 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73962 uninterruptible
[  533.383124] MemAlloc: a.out(9555) flags=0x400040 switches=2059 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74949
[  533.548238] MemAlloc: a.out(9556) flags=0x400840 switches=22 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  533.793954] MemAlloc: a.out(9558) flags=0x400840 switches=22 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  534.027669] MemAlloc: a.out(9560) flags=0x400840 switches=22 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  534.274691] MemAlloc: a.out(9561) flags=0x400840 switches=21 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  534.479135] MemAlloc: a.out(9562) flags=0x400840 switches=17 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  534.644896] MemAlloc: a.out(9563) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  534.924052] MemAlloc: a.out(9564) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  535.202235] MemAlloc: a.out(9565) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  535.436619] MemAlloc: a.out(9566) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  535.638534] MemAlloc: a.out(9567) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  535.886729] MemAlloc: a.out(9568) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  536.123130] MemAlloc: a.out(9569) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  536.358902] MemAlloc: a.out(9570) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  536.603860] MemAlloc: a.out(9571) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  536.709749] MemAlloc: a.out(9572) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  536.942101] MemAlloc: a.out(9573) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  537.127875] MemAlloc: a.out(9574) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  537.310662] MemAlloc: a.out(9575) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  537.455864] MemAlloc: a.out(9576) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  537.675090] MemAlloc: a.out(9579) flags=0x400040 switches=2251 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74933
[  537.779100] MemAlloc: a.out(9581) flags=0x400840 switches=26 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  537.925518] MemAlloc: a.out(9583) flags=0x400840 switches=46 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  538.169936] MemAlloc: a.out(9586) flags=0x400840 switches=32 seq=129 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=74232 uninterruptible
[  538.406918] MemAlloc: a.out(9587) flags=0x400840 switches=28 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  538.642036] MemAlloc: a.out(9591) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  538.866188] MemAlloc: a.out(9592) flags=0x400840 switches=49 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  539.119566] MemAlloc: a.out(9593) flags=0x400840 switches=65 seq=35 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73960 uninterruptible
[  539.378634] MemAlloc: a.out(9595) flags=0x400040 switches=32 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74939 uninterruptible
[  539.556915] MemAlloc: a.out(9597) flags=0x400840 switches=48 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  539.787355] MemAlloc: a.out(9599) flags=0x400840 switches=24 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.015041] MemAlloc: a.out(9600) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  540.148695] MemAlloc: a.out(9605) flags=0x400840 switches=27 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  540.237056] MemAlloc: a.out(9606) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  540.333568] MemAlloc: a.out(9607) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73841 uninterruptible
[  540.434293] MemAlloc: a.out(9608) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.590569] MemAlloc: a.out(9609) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.732138] MemAlloc: a.out(9610) flags=0x400840 switches=29 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  540.961649] MemAlloc: a.out(9611) flags=0x400840 switches=27 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  541.218220] MemAlloc: a.out(9612) flags=0x400840 switches=16 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  541.375479] MemAlloc: a.out(9613) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  541.571384] MemAlloc: a.out(9614) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  541.754231] MemAlloc: a.out(9615) flags=0x400840 switches=17 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  541.896925] MemAlloc: a.out(9616) flags=0x400840 switches=18 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  542.056536] MemAlloc: a.out(9617) flags=0x400840 switches=17 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  542.229652] MemAlloc: a.out(9618) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  542.364170] MemAlloc: a.out(9619) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  542.493472] MemAlloc: a.out(9620) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  542.679770] MemAlloc: a.out(9621) flags=0x400840 switches=16 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  542.865175] MemAlloc: a.out(9622) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  542.972020] MemAlloc: a.out(9623) flags=0x400840 switches=15 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  543.135246] MemAlloc: a.out(9624) flags=0x400840 switches=18 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  543.358968] MemAlloc: a.out(9625) flags=0x400840 switches=16 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  543.583466] MemAlloc: a.out(9626) flags=0x400840 switches=17 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  543.809065] MemAlloc: a.out(9627) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.017351] MemAlloc: a.out(9628) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.217362] MemAlloc: a.out(9629) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.462068] MemAlloc: a.out(9630) flags=0x400840 switches=21 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  544.699780] MemAlloc: a.out(9631) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  544.895071] MemAlloc: a.out(9632) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  545.080182] MemAlloc: a.out(9633) flags=0x400840 switches=31 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  545.352169] MemAlloc: a.out(9634) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  545.493870] MemAlloc: a.out(9635) flags=0x400840 switches=14 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  545.609266] MemAlloc: a.out(9636) flags=0x400840 switches=16 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  545.725966] MemAlloc: a.out(9637) flags=0x400840 switches=18 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73963 uninterruptible
[  545.812656] MemAlloc: a.out(9638) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73735 uninterruptible
[  545.984346] MemAlloc: a.out(9639) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  546.170187] MemAlloc: a.out(9640) flags=0x400840 switches=15 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  546.351502] MemAlloc: a.out(9641) flags=0x400840 switches=14 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  546.529103] MemAlloc: a.out(9646) flags=0x400840 switches=11 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  546.710185] MemAlloc: a.out(9647) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  546.852608] MemAlloc: a.out(9648) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.014298] MemAlloc: a.out(9650) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.188938] MemAlloc: a.out(9651) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.362058] MemAlloc: a.out(9652) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.471970] MemAlloc: a.out(9653) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.706983] MemAlloc: a.out(9654) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  547.870033] MemAlloc: a.out(9656) flags=0x400840 switches=12 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.079204] MemAlloc: a.out(9662) flags=0x400840 switches=12 seq=209 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.305820] MemAlloc: a.out(9664) flags=0x400840 switches=20 seq=302 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.525819] MemAlloc: a.out(9666) flags=0x400840 switches=11 seq=25 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  548.793002] MemAlloc: a.out(9668) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  549.006725] MemAlloc: a.out(9670) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.232993] MemAlloc: a.out(9671) flags=0x400840 switches=10 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.469884] MemAlloc: a.out(9672) flags=0x400840 switches=11 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.727175] MemAlloc: a.out(9673) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  549.987420] MemAlloc: a.out(9674) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73839 uninterruptible
[  550.214963] MemAlloc: a.out(9675) flags=0x400840 switches=10 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  550.421057] MemAlloc: a.out(9676) flags=0x400840 switches=11 seq=2 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73835 uninterruptible
[  550.654050] MemAlloc: a.out(9677) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  550.841175] MemAlloc: a.out(9678) flags=0x400840 switches=13 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  551.030916] MemAlloc: a.out(9679) flags=0x400840 switches=13 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  551.213400] MemAlloc: a.out(9681) flags=0x400840 switches=13 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  551.401337] MemAlloc: a.out(9682) flags=0x400840 switches=13 seq=4 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  551.604094] MemAlloc: a.out(9683) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  551.783882] MemAlloc: a.out(9684) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73838 uninterruptible
[  551.971142] MemAlloc: a.out(9685) flags=0x420840 switches=2357 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74818
[  552.137039] MemAlloc: a.out(9686) flags=0x400840 switches=12 seq=3 gfp=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD) order=0 delay=73836 uninterruptible
[  552.342766] MemAlloc: a.out(9689) flags=0x400040 switches=2370 seq=3 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74818
[  552.453767] MemAlloc: kworker/3:0(10498) flags=0x4228860 switches=4184 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548

Only 17 out of 186 processes were able to loop inside page allocator.

[  518.319917] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=14350 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=74621
[  518.605804] MemAlloc: kworker/1:1(52) flags=0x4228060 switches=4112 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74544
[  518.756891] MemAlloc: kworker/2:1(65) flags=0x4228860 switches=2749 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74504
[  519.049597] MemAlloc: xfs-data/sda1(451) flags=0x4228860 switches=6525 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74534
[  521.472130] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=16212 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74547
[  521.536605] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=6647 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.692946] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=3718 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74545
[  521.817708] MemAlloc: kworker/3:5(6386) flags=0x4228060 switches=30373 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548
[  521.955723] MemAlloc: kworker/2:4(7415) flags=0x4228860 switches=3304 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74539
[  527.166154] MemAlloc: a.out(9511) flags=0x400040 switches=1996 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74941
[  532.430946] MemAlloc: a.out(9545) flags=0x420840 switches=2014 seq=7 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=1 delay=74581
[  532.995120] MemAlloc: a.out(9551) flags=0x420040 switches=2151 seq=3 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74577
[  533.383124] MemAlloc: a.out(9555) flags=0x400040 switches=2059 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74949
[  537.675090] MemAlloc: a.out(9579) flags=0x400040 switches=2251 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74933
[  551.971142] MemAlloc: a.out(9685) flags=0x420840 switches=2357 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=74818
[  552.342766] MemAlloc: a.out(9689) flags=0x400040 switches=2370 seq=3 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=74818
[  552.453767] MemAlloc: kworker/3:0(10498) flags=0x4228860 switches=4184 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=74548

Situation was similar when I gave up waiting.
Only 20 out of 192 processes were able to loop inside page allocator.

[ 1095.631687] MemAlloc: kworker/2:0(27) flags=0x4208860 switches=38727 seq=21 gfp=0x1400000(GFP_NOIO) order=0 delay=652160
[ 1095.632186] MemAlloc: kworker/1:1(52) flags=0x4228860 switches=28036 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652083
[ 1095.632604] MemAlloc: kworker/2:1(65) flags=0x4228060 switches=22879 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652043
[ 1095.633213] MemAlloc: kworker/1:2(193) flags=0x4228860 switches=28494 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
[ 1095.634013] MemAlloc: xfs-eofblocks/s(456) flags=0x4228860 switches=15435 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=293074
[ 1095.635223] MemAlloc: kworker/2:3(535) flags=0x4228860 switches=49285 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.638807] MemAlloc: kworker/0:3(2486) flags=0x4228860 switches=76240 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652064
[ 1095.644936] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=29192 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652086
[ 1095.645241] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=30893 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.645506] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=27329 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.647293] MemAlloc: kworker/3:5(6386) flags=0x4228860 switches=43427 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
[ 1095.691898] MemAlloc: a.out(9511) flags=0x400040 switches=14710 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652480
[ 1095.726301] MemAlloc: a.out(9545) flags=0x420040 switches=14120 seq=7 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=1 delay=652120
[ 1095.727373] MemAlloc: a.out(9551) flags=0x420840 switches=14617 seq=3 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=652116
[ 1095.728228] MemAlloc: a.out(9555) flags=0x400040 switches=14575 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652488
[ 1095.739636] MemAlloc: a.out(9579) flags=0x400840 switches=14791 seq=2 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652472
[ 1095.805447] MemAlloc: a.out(9685) flags=0x420840 switches=14309 seq=1 gfp=0x1400240(GFP_NOFS|__GFP_NOWARN) order=0 delay=652357
[ 1095.806316] MemAlloc: a.out(9689) flags=0x400840 switches=14131 seq=3 gfp=0x1c2004a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE) order=0 delay=652357
[ 1095.806606] MemAlloc: kworker/3:0(10498) flags=0x4228060 switches=16222 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087

You can't say that 1024 processes were doing write() requests. Many were still
blocked at open(). Only 20 or so processes were able to consume CPU time for
memory allocation. I don't think this is a too insane stress to wait. This is not
a stress which can justify stalls over 10 minutes. The kworker/2:0(27) line
indicates that order-0 GFP_NOIO allocation request was unable to find a page
for more than 10 minutes. Under such situation, how can other GFP_NOFS allocation
requests find a page without boosting priority (unless someone releases memory
voluntarily) ?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Sat, Mar 04, 2017 at 11:54:56PM +0900, Tetsuo Handa wrote:
> Brian Foster wrote:
> > On Fri, Mar 03, 2017 at 04:52:58PM +0100, Michal Hocko wrote:
> > > On Fri 03-03-17 10:37:21, Brian Foster wrote:
> > > [...]
> > > > That aside, looking through some of the traces in this case...
> > > > 
> > > > - kswapd0 is waiting on an inode flush lock. This means somebody else
> > > >   flushed the inode and it won't be unlocked until the underlying buffer
> > > >   I/O is completed. This context is also holding pag_ici_reclaim_lock
> > > >   which is what probably blocks other contexts from getting into inode
> > > >   reclaim.
> > > > - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
> > > >   It's waiting on reading the underlying inode buffer. The buffer read
> > > >   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
> > > >   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
> > > >   work item is what eventually triggers the I/O completion in
> > > >   xfs_buf_ioend().
> > > > 
> > > > So at this point reclaim is waiting on a read I/O completion. It's not
> > > > clear to me whether the read had completed and the work item was queued
> > > > or not. I do see the following in the workqueue lockup BUG output:
> > > > 
> > > > [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> > > > [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> > > > [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> > > > 
> > > > ... which suggests that it was queued..? I suppose this could be one of
> > > > the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> > > > appears to be idle:
> > > > 
> > > > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > > > [ 1041.556813] Call Trace:
> > > > [ 1041.557796]  __schedule+0x336/0xe00
> > > > [ 1041.558983]  schedule+0x3d/0x90
> > > > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > > > [ 1041.561333]  kthread+0x10f/0x150
> > > > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > > > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > > > [ 1041.565123]  ret_from_fork+0x31/0x40
> > > > 
> > > > So shouldn't that thread pick up the work item if that is the case?
> > > 
> > > Is it possible that the progress is done but tediously slow? Keep in
> > > mind that the test case is doing write from 1k processes while one
> > > process basically consumes all the memory. So I wouldn't be surprised
> > > if this just made system to crawl on any attempt to do an IO.
> > 
> > That would seem like a possibility to me.. either waiting on an actual
> > I/O (no guarantee that the pending xfs-buf item is the one we care about
> > I suppose) completion or waiting for whatever needs to happen for the wq
> > infrastructure to kick off the rescuer. Though I think that's probably
> > something Tetsuo would ultimately have to confirm on his setup..
> 
> This lockup began from uptime = 444. Thus, please ignore logs up to
> 
> [  444.281177] Killed process 9477 (a.out) total-vm:4168kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB
> [  444.287046] oom_reaper: reaped process 9477 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> 
...
> 
> You can't say that 1024 processes were doing write() requests. Many were still
> blocked at open(). Only 20 or so processes were able to consume CPU time for
> memory allocation. I don't think this is a too insane stress to wait. This is not
> a stress which can justify stalls over 10 minutes. The kworker/2:0(27) line
> indicates that order-0 GFP_NOIO allocation request was unable to find a page
> for more than 10 minutes. Under such situation, how can other GFP_NOFS allocation
> requests find a page without boosting priority (unless someone releases memory
> voluntarily) ?

As noted in my previous reply, I'm not sure there's enough here to point
at allocation failure as the root cause. E.g., kswapd is stuck here:

[ 1095.632985] kswapd0         D10776    69      2 0x00000000
[ 1095.632988] Call Trace:
[ 1095.632991]  __schedule+0x336/0xe00
[ 1095.632994]  schedule+0x3d/0x90
[ 1095.632996]  io_schedule+0x16/0x40
[ 1095.633017]  __xfs_iflock+0x129/0x140 [xfs]
[ 1095.633021]  ? autoremove_wake_function+0x60/0x60
[ 1095.633051]  xfs_reclaim_inode+0x162/0x440 [xfs]
[ 1095.633072]  xfs_reclaim_inodes_ag+0x2cf/0x4f0 [xfs]
[ 1095.633106]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
[ 1095.633114]  ? trace_hardirqs_on+0xd/0x10
[ 1095.633116]  ? try_to_wake_up+0x59/0x7a0
[ 1095.633120]  ? wake_up_process+0x15/0x20
[ 1095.633156]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 1095.633178]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
[ 1095.633180]  super_cache_scan+0x181/0x190
[ 1095.633183]  shrink_slab+0x29f/0x6d0
[ 1095.633189]  shrink_node+0x2fa/0x310
[ 1095.633193]  kswapd+0x362/0x9b0
[ 1095.633200]  kthread+0x10f/0x150
[ 1095.633201]  ? mem_cgroup_shrink_node+0x3b0/0x3b0
[ 1095.633202]  ? kthread_create_on_node+0x70/0x70
[ 1095.633205]  ret_from_fork+0x31/0x40

... which is waiting on an inode flush lock. It can't get the lock
(presumably) because xfsaild has it:

[ 1041.772095] xfsaild/sda1    D13216   457      2 0x00000000
[ 1041.773726] Call Trace:
[ 1041.774734]  __schedule+0x336/0xe00
[ 1041.775956]  schedule+0x3d/0x90
[ 1041.777105]  schedule_timeout+0x26a/0x510
[ 1041.778426]  ? wait_for_completion+0x4c/0x190
[ 1041.779824]  wait_for_completion+0x12c/0x190
[ 1041.781273]  ? wake_up_q+0x80/0x80
[ 1041.782597]  ? _xfs_buf_read+0x44/0x90 [xfs]
[ 1041.784086]  xfs_buf_submit_wait+0xe9/0x5c0 [xfs]
[ 1041.785659]  _xfs_buf_read+0x44/0x90 [xfs]
[ 1041.787067]  xfs_buf_read_map+0xfa/0x400 [xfs]
[ 1041.788501]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
[ 1041.790103]  xfs_trans_read_buf_map+0x186/0x830 [xfs]
[ 1041.791672]  xfs_imap_to_bp+0x71/0x110 [xfs]
[ 1041.793090]  xfs_iflush+0x122/0x3b0 [xfs]
[ 1041.794444]  xfs_inode_item_push+0x108/0x1c0 [xfs]
[ 1041.795956]  xfsaild_push+0x1d8/0xb70 [xfs]
[ 1041.797344]  xfsaild+0x150/0x270 [xfs]
[ 1041.798623]  kthread+0x10f/0x150
[ 1041.799819]  ? xfsaild_push+0xb70/0xb70 [xfs]
[ 1041.801217]  ? kthread_create_on_node+0x70/0x70
[ 1041.802652]  ret_from_fork+0x31/0x40

xfsaild is flushing an inode, but is waiting on a read of the underlying
inode cluster buffer such that it can flush out the in-core inode data
structure. I cannot tell if the read had actually completed and is
blocked somewhere else before running the completion. As Dave notes
earlier, buffer I/O completion relies on the xfs-buf wq. What is evident
from the logs is that xfs-buf has a rescuer thread that is sitting idle:

[ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
[ 1041.556813] Call Trace:
[ 1041.557796]  __schedule+0x336/0xe00
[ 1041.558983]  schedule+0x3d/0x90
[ 1041.560085]  rescuer_thread+0x322/0x3d0
[ 1041.561333]  kthread+0x10f/0x150
[ 1041.562464]  ? worker_thread+0x4b0/0x4b0
[ 1041.563732]  ? kthread_create_on_node+0x70/0x70
[ 1041.565123]  ret_from_fork+0x31/0x40

So AFAICT if the buffer I/O completion would run, it would allow xfsaild
to progress, which would eventually flush the underlying buffer, write
it, release the flush lock and allow kswapd to continue. The question is
has the actually I/O completed? If so, is the xfs-buf workqueue stuck
(waiting on an allocation perhaps)? And if that is the case, why is the
xfs-buf rescuer thread not doing anything?

Brian

> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Sat, Mar 04, 2017 at 11:54:56PM +0900, Tetsuo Handa wrote:
> Brian Foster wrote:
> > On Fri, Mar 03, 2017 at 04:52:58PM +0100, Michal Hocko wrote:
> > > On Fri 03-03-17 10:37:21, Brian Foster wrote:
> > > [...]
> > > > That aside, looking through some of the traces in this case...
> > > > 
> > > > - kswapd0 is waiting on an inode flush lock. This means somebody else
> > > >   flushed the inode and it won't be unlocked until the underlying buffer
> > > >   I/O is completed. This context is also holding pag_ici_reclaim_lock
> > > >   which is what probably blocks other contexts from getting into inode
> > > >   reclaim.
> > > > - xfsaild is in xfs_iflush(), which means it has the inode flush lock.
> > > >   It's waiting on reading the underlying inode buffer. The buffer read
> > > >   sets b_ioend_wq to the xfs-buf wq, which is ultimately going to be
> > > >   queued in xfs_buf_bio_end_io()->xfs_buf_ioend_async(). The associated
> > > >   work item is what eventually triggers the I/O completion in
> > > >   xfs_buf_ioend().
> > > > 
> > > > So at this point reclaim is waiting on a read I/O completion. It's not
> > > > clear to me whether the read had completed and the work item was queued
> > > > or not. I do see the following in the workqueue lockup BUG output:
> > > > 
> > > > [  273.412600] workqueue xfs-buf/sda1: flags=0xc
> > > > [  273.414486]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/1
> > > > [  273.416415]     pending: xfs_buf_ioend_work [xfs]
> > > > 
> > > > ... which suggests that it was queued..? I suppose this could be one of
> > > > the workqueues waiting on a kthread, but xfs-buf also has a rescuer that
> > > > appears to be idle:
> > > > 
> > > > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > > > [ 1041.556813] Call Trace:
> > > > [ 1041.557796]  __schedule+0x336/0xe00
> > > > [ 1041.558983]  schedule+0x3d/0x90
> > > > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > > > [ 1041.561333]  kthread+0x10f/0x150
> > > > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > > > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > > > [ 1041.565123]  ret_from_fork+0x31/0x40
> > > > 
> > > > So shouldn't that thread pick up the work item if that is the case?
> > > 
> > > Is it possible that the progress is done but tediously slow? Keep in
> > > mind that the test case is doing write from 1k processes while one
> > > process basically consumes all the memory. So I wouldn't be surprised
> > > if this just made system to crawl on any attempt to do an IO.
> > 
> > That would seem like a possibility to me.. either waiting on an actual
> > I/O (no guarantee that the pending xfs-buf item is the one we care about
> > I suppose) completion or waiting for whatever needs to happen for the wq
> > infrastructure to kick off the rescuer. Though I think that's probably
> > something Tetsuo would ultimately have to confirm on his setup..
> 
> This lockup began from uptime = 444. Thus, please ignore logs up to
> 
> [  444.281177] Killed process 9477 (a.out) total-vm:4168kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB
> [  444.287046] oom_reaper: reaped process 9477 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
> 
...
> 
> You can't say that 1024 processes were doing write() requests. Many were still
> blocked at open(). Only 20 or so processes were able to consume CPU time for
> memory allocation. I don't think this is a too insane stress to wait. This is not
> a stress which can justify stalls over 10 minutes. The kworker/2:0(27) line
> indicates that order-0 GFP_NOIO allocation request was unable to find a page
> for more than 10 minutes. Under such situation, how can other GFP_NOFS allocation
> requests find a page without boosting priority (unless someone releases memory
> voluntarily) ?

As noted in my previous reply, I'm not sure there's enough here to point
at allocation failure as the root cause. E.g., kswapd is stuck here:

[ 1095.632985] kswapd0         D10776    69      2 0x00000000
[ 1095.632988] Call Trace:
[ 1095.632991]  __schedule+0x336/0xe00
[ 1095.632994]  schedule+0x3d/0x90
[ 1095.632996]  io_schedule+0x16/0x40
[ 1095.633017]  __xfs_iflock+0x129/0x140 [xfs]
[ 1095.633021]  ? autoremove_wake_function+0x60/0x60
[ 1095.633051]  xfs_reclaim_inode+0x162/0x440 [xfs]
[ 1095.633072]  xfs_reclaim_inodes_ag+0x2cf/0x4f0 [xfs]
[ 1095.633106]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
[ 1095.633114]  ? trace_hardirqs_on+0xd/0x10
[ 1095.633116]  ? try_to_wake_up+0x59/0x7a0
[ 1095.633120]  ? wake_up_process+0x15/0x20
[ 1095.633156]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 1095.633178]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
[ 1095.633180]  super_cache_scan+0x181/0x190
[ 1095.633183]  shrink_slab+0x29f/0x6d0
[ 1095.633189]  shrink_node+0x2fa/0x310
[ 1095.633193]  kswapd+0x362/0x9b0
[ 1095.633200]  kthread+0x10f/0x150
[ 1095.633201]  ? mem_cgroup_shrink_node+0x3b0/0x3b0
[ 1095.633202]  ? kthread_create_on_node+0x70/0x70
[ 1095.633205]  ret_from_fork+0x31/0x40

... which is waiting on an inode flush lock. It can't get the lock
(presumably) because xfsaild has it:

[ 1041.772095] xfsaild/sda1    D13216   457      2 0x00000000
[ 1041.773726] Call Trace:
[ 1041.774734]  __schedule+0x336/0xe00
[ 1041.775956]  schedule+0x3d/0x90
[ 1041.777105]  schedule_timeout+0x26a/0x510
[ 1041.778426]  ? wait_for_completion+0x4c/0x190
[ 1041.779824]  wait_for_completion+0x12c/0x190
[ 1041.781273]  ? wake_up_q+0x80/0x80
[ 1041.782597]  ? _xfs_buf_read+0x44/0x90 [xfs]
[ 1041.784086]  xfs_buf_submit_wait+0xe9/0x5c0 [xfs]
[ 1041.785659]  _xfs_buf_read+0x44/0x90 [xfs]
[ 1041.787067]  xfs_buf_read_map+0xfa/0x400 [xfs]
[ 1041.788501]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
[ 1041.790103]  xfs_trans_read_buf_map+0x186/0x830 [xfs]
[ 1041.791672]  xfs_imap_to_bp+0x71/0x110 [xfs]
[ 1041.793090]  xfs_iflush+0x122/0x3b0 [xfs]
[ 1041.794444]  xfs_inode_item_push+0x108/0x1c0 [xfs]
[ 1041.795956]  xfsaild_push+0x1d8/0xb70 [xfs]
[ 1041.797344]  xfsaild+0x150/0x270 [xfs]
[ 1041.798623]  kthread+0x10f/0x150
[ 1041.799819]  ? xfsaild_push+0xb70/0xb70 [xfs]
[ 1041.801217]  ? kthread_create_on_node+0x70/0x70
[ 1041.802652]  ret_from_fork+0x31/0x40

xfsaild is flushing an inode, but is waiting on a read of the underlying
inode cluster buffer such that it can flush out the in-core inode data
structure. I cannot tell if the read had actually completed and is
blocked somewhere else before running the completion. As Dave notes
earlier, buffer I/O completion relies on the xfs-buf wq. What is evident
from the logs is that xfs-buf has a rescuer thread that is sitting idle:

[ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
[ 1041.556813] Call Trace:
[ 1041.557796]  __schedule+0x336/0xe00
[ 1041.558983]  schedule+0x3d/0x90
[ 1041.560085]  rescuer_thread+0x322/0x3d0
[ 1041.561333]  kthread+0x10f/0x150
[ 1041.562464]  ? worker_thread+0x4b0/0x4b0
[ 1041.563732]  ? kthread_create_on_node+0x70/0x70
[ 1041.565123]  ret_from_fork+0x31/0x40

So AFAICT if the buffer I/O completion would run, it would allow xfsaild
to progress, which would eventually flush the underlying buffer, write
it, release the flush lock and allow kswapd to continue. The question is
has the actually I/O completed? If so, is the xfs-buf workqueue stuck
(waiting on an allocation perhaps)? And if that is the case, why is the
xfs-buf rescuer thread not doing anything?

Brian

> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tetsuo Handa]

Brian Foster wrote:
> As noted in my previous reply, I'm not sure there's enough here to point
> at allocation failure as the root cause. E.g., kswapd is stuck here:
> 
> [ 1095.632985] kswapd0         D10776    69      2 0x00000000
> [ 1095.632988] Call Trace:
> [ 1095.632991]  __schedule+0x336/0xe00
> [ 1095.632994]  schedule+0x3d/0x90
> [ 1095.632996]  io_schedule+0x16/0x40
> [ 1095.633017]  __xfs_iflock+0x129/0x140 [xfs]
> [ 1095.633021]  ? autoremove_wake_function+0x60/0x60
> [ 1095.633051]  xfs_reclaim_inode+0x162/0x440 [xfs]
> [ 1095.633072]  xfs_reclaim_inodes_ag+0x2cf/0x4f0 [xfs]
> [ 1095.633106]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
> [ 1095.633114]  ? trace_hardirqs_on+0xd/0x10
> [ 1095.633116]  ? try_to_wake_up+0x59/0x7a0
> [ 1095.633120]  ? wake_up_process+0x15/0x20
> [ 1095.633156]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> [ 1095.633178]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
> [ 1095.633180]  super_cache_scan+0x181/0x190
> [ 1095.633183]  shrink_slab+0x29f/0x6d0
> [ 1095.633189]  shrink_node+0x2fa/0x310
> [ 1095.633193]  kswapd+0x362/0x9b0
> [ 1095.633200]  kthread+0x10f/0x150
> [ 1095.633201]  ? mem_cgroup_shrink_node+0x3b0/0x3b0
> [ 1095.633202]  ? kthread_create_on_node+0x70/0x70
> [ 1095.633205]  ret_from_fork+0x31/0x40
> 
> ... which is waiting on an inode flush lock. It can't get the lock
> (presumably) because xfsaild has it:
> 
> [ 1041.772095] xfsaild/sda1    D13216   457      2 0x00000000
> [ 1041.773726] Call Trace:
> [ 1041.774734]  __schedule+0x336/0xe00
> [ 1041.775956]  schedule+0x3d/0x90
> [ 1041.777105]  schedule_timeout+0x26a/0x510
> [ 1041.778426]  ? wait_for_completion+0x4c/0x190
> [ 1041.779824]  wait_for_completion+0x12c/0x190
> [ 1041.781273]  ? wake_up_q+0x80/0x80
> [ 1041.782597]  ? _xfs_buf_read+0x44/0x90 [xfs]
> [ 1041.784086]  xfs_buf_submit_wait+0xe9/0x5c0 [xfs]
> [ 1041.785659]  _xfs_buf_read+0x44/0x90 [xfs]
> [ 1041.787067]  xfs_buf_read_map+0xfa/0x400 [xfs]
> [ 1041.788501]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
> [ 1041.790103]  xfs_trans_read_buf_map+0x186/0x830 [xfs]
> [ 1041.791672]  xfs_imap_to_bp+0x71/0x110 [xfs]
> [ 1041.793090]  xfs_iflush+0x122/0x3b0 [xfs]
> [ 1041.794444]  xfs_inode_item_push+0x108/0x1c0 [xfs]
> [ 1041.795956]  xfsaild_push+0x1d8/0xb70 [xfs]
> [ 1041.797344]  xfsaild+0x150/0x270 [xfs]
> [ 1041.798623]  kthread+0x10f/0x150
> [ 1041.799819]  ? xfsaild_push+0xb70/0xb70 [xfs]
> [ 1041.801217]  ? kthread_create_on_node+0x70/0x70
> [ 1041.802652]  ret_from_fork+0x31/0x40
> 
> xfsaild is flushing an inode, but is waiting on a read of the underlying
> inode cluster buffer such that it can flush out the in-core inode data
> structure. I cannot tell if the read had actually completed and is
> blocked somewhere else before running the completion. As Dave notes
> earlier, buffer I/O completion relies on the xfs-buf wq. What is evident
> from the logs is that xfs-buf has a rescuer thread that is sitting idle:
> 
> [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> [ 1041.556813] Call Trace:
> [ 1041.557796]  __schedule+0x336/0xe00
> [ 1041.558983]  schedule+0x3d/0x90
> [ 1041.560085]  rescuer_thread+0x322/0x3d0
> [ 1041.561333]  kthread+0x10f/0x150
> [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> [ 1041.565123]  ret_from_fork+0x31/0x40
> 
> So AFAICT if the buffer I/O completion would run, it would allow xfsaild
> to progress, which would eventually flush the underlying buffer, write
> it, release the flush lock and allow kswapd to continue. The question is
> has the actually I/O completed? If so, is the xfs-buf workqueue stuck
> (waiting on an allocation perhaps)? And if that is the case, why is the
> xfs-buf rescuer thread not doing anything?

[ 1116.803780] workqueue xfs-data/sda1: flags=0xc
[ 1116.805324]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=27/256 MAYDAY
[ 1116.807272]     in-flight: 5356:xfs_end_io [xfs], 451(RESCUER):xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs], 10498:xfs_end_io [xfs], 6386:xfs_end_io [xfs]
[ 1116.812145]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
 [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.820988]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=21/256 MAYDAY
[ 1116.823105]     in-flight: 535:xfs_end_io [xfs], 7416:xfs_end_io [xfs], 7415:xfs_end_io [xfs], 65:xfs_end_io [xfs]
[ 1116.826062]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
 [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.834549]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=4/256 MAYDAY
[ 1116.837139]     in-flight: 5357:xfs_end_io [xfs], 193:xfs_end_io [xfs], 52:xfs_end_io [xfs], 5358:xfs_end_io [xfs]
[ 1116.840182]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
[ 1116.842297]     in-flight: 2486:xfs_end_io [xfs]
[ 1116.844230] workqueue xfs-reclaim/sda1: flags=0xc
[ 1116.846168]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.848323]     pending: xfs_reclaim_worker [xfs]
[ 1116.850280] workqueue xfs-eofblocks/sda1: flags=0xc
[ 1116.852358]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.854601]     in-flight: 456(RESCUER):xfs_eofblocks_worker [xfs]

As listed below, all in-flight workqueues for processing xfs_end_io (PID = 5356,
451(RESCUER), 10498, 6386, 535, 7416, 7415, 65, 5357, 193, 52, 5358, 2486) are
stuck at memory allocation, and thus cannot call complete() when xfsaild/sda1
(PID = 457) is waiting at wait_for_completion(&bp->b_iowait) in xfs_buf_submit_wait().

[ 1095.644936] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=29192 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652086
[ 1095.644937] kworker/3:3     R  running task    12760  5356      2 0x00000080
[ 1095.644959] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.644960] Call Trace:
[ 1095.644962]  __schedule+0x336/0xe00
[ 1095.644965]  preempt_schedule_common+0x1f/0x31
[ 1095.644966]  _cond_resched+0x1c/0x30
[ 1095.644968]  shrink_slab+0x339/0x6d0
[ 1095.644973]  shrink_node+0x2fa/0x310
[ 1095.644977]  do_try_to_free_pages+0xe1/0x300
[ 1095.644979]  try_to_free_pages+0x131/0x3f0
[ 1095.644984]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.644985]  ? cpuacct_charge+0xf3/0x1e0
[ 1095.644986]  ? cpuacct_charge+0x38/0x1e0
[ 1095.644992]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.644995]  alloc_pages_current+0x97/0x1b0
[ 1095.644998]  new_slab+0x4cb/0x6b0
[ 1095.645001]  ___slab_alloc+0x3a3/0x620
[ 1095.645023]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645026]  ? ___slab_alloc+0x5c6/0x620
[ 1095.645027]  ? trace_hardirqs_on+0xd/0x10
[ 1095.645058]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645066]  __slab_alloc+0x46/0x7d
[ 1095.645069]  __kmalloc+0x301/0x3b0
[ 1095.645088]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.645089]  ? kfree+0x1fa/0x330
[ 1095.645110]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.645132]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.645151]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.645169]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.645188]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.645204]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.645221]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.645223]  process_one_work+0x22b/0x760
[ 1095.645224]  ? process_one_work+0x194/0x760
[ 1095.645228]  worker_thread+0x137/0x4b0
[ 1095.645231]  kthread+0x10f/0x150
[ 1095.645232]  ? process_one_work+0x760/0x760
[ 1095.645233]  ? kthread_create_on_node+0x70/0x70
[ 1095.645236]  ret_from_fork+0x31/0x40

[ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
[ 1095.633626] xfs-data/sda1   R  running task    12696   451      2 0x00000000
[ 1095.633663] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.633665] Call Trace:
[ 1095.633668]  __schedule+0x336/0xe00
[ 1095.633671]  schedule+0x3d/0x90
[ 1095.633672]  schedule_timeout+0x20d/0x510
[ 1095.633675]  ? lock_timer_base+0xa0/0xa0
[ 1095.633678]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.633680]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.633687]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.633699]  alloc_pages_current+0x97/0x1b0
[ 1095.633702]  new_slab+0x4cb/0x6b0
[ 1095.633706]  ___slab_alloc+0x3a3/0x620
[ 1095.633728]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633730]  ? ___slab_alloc+0x5c6/0x620
[ 1095.633732]  ? cpuacct_charge+0x38/0x1e0
[ 1095.633767]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633770]  __slab_alloc+0x46/0x7d
[ 1095.633773]  __kmalloc+0x301/0x3b0
[ 1095.633802]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.633804]  ? kfree+0x1fa/0x330
[ 1095.633842]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.633864]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.633883]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.633901]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.633936]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.633954]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.633971]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.633973]  process_one_work+0x22b/0x760
[ 1095.633975]  ? process_one_work+0x194/0x760
[ 1095.633997]  rescuer_thread+0x1f2/0x3d0
[ 1095.634002]  kthread+0x10f/0x150
[ 1095.634003]  ? worker_thread+0x4b0/0x4b0
[ 1095.634004]  ? kthread_create_on_node+0x70/0x70
[ 1095.634007]  ret_from_fork+0x31/0x40

[ 1095.806606] MemAlloc: kworker/3:0(10498) flags=0x4228060 switches=16222 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
[ 1095.806607] kworker/3:0     R  running task    11352 10498      2 0x00000080
[ 1095.806631] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.806633] Call Trace:
[ 1095.806636]  __schedule+0x336/0xe00
[ 1095.806640]  schedule+0x3d/0x90
[ 1095.806641]  schedule_timeout+0x20d/0x510
[ 1095.806644]  ? lock_timer_base+0xa0/0xa0
[ 1095.806649]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.806651]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.806652]  ? cpuacct_charge+0xf3/0x1e0
[ 1095.806654]  ? cpuacct_charge+0x38/0x1e0
[ 1095.806663]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.806667]  alloc_pages_current+0x97/0x1b0
[ 1095.806671]  new_slab+0x4cb/0x6b0
[ 1095.806675]  ___slab_alloc+0x3a3/0x620
[ 1095.806698]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.806701]  ? ___slab_alloc+0x5c6/0x620
[ 1095.806721]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.806724]  __slab_alloc+0x46/0x7d
[ 1095.806727]  __kmalloc+0x301/0x3b0
[ 1095.806747]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.806749]  ? kfree+0x1fa/0x330
[ 1095.806771]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.806794]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.806813]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.806832]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.806852]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.806868]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.806886]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.806889]  process_one_work+0x22b/0x760
[ 1095.806890]  ? process_one_work+0x194/0x760
[ 1095.806896]  worker_thread+0x137/0x4b0
[ 1095.806899]  kthread+0x10f/0x150
[ 1095.806901]  ? process_one_work+0x760/0x760
[ 1095.806902]  ? kthread_create_on_node+0x70/0x70
[ 1095.806904]  ? do_syscall_64+0x195/0x200
[ 1095.806906]  ret_from_fork+0x31/0x40

[ 1095.647293] MemAlloc: kworker/3:5(6386) flags=0x4228860 switches=43427 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
[ 1095.647294] kworker/3:5     R  running task    11048  6386      2 0x00000080
[ 1095.647754] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.648203] Call Trace:
[ 1095.648207]  __schedule+0x336/0xe00
[ 1095.648504]  preempt_schedule_common+0x1f/0x31
[ 1095.648806]  _cond_resched+0x1c/0x30
[ 1095.648952]  shrink_slab+0x339/0x6d0
[ 1095.649252]  shrink_node+0x2fa/0x310
[ 1095.649694]  do_try_to_free_pages+0xe1/0x300
[ 1095.649699]  try_to_free_pages+0x131/0x3f0
[ 1095.649996]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.650296]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.650743]  alloc_pages_current+0x97/0x1b0
[ 1095.650892]  new_slab+0x4cb/0x6b0
[ 1095.651193]  ___slab_alloc+0x3a3/0x620
[ 1095.652087]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.652400]  ? ___slab_alloc+0x5c6/0x620
[ 1095.653457]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.653901]  __slab_alloc+0x46/0x7d
[ 1095.654050]  __kmalloc+0x301/0x3b0
[ 1095.654948]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.655396]  ? kfree+0x1fa/0x330
[ 1095.656589]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.656919]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.656941]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.656961]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.658006]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.659053]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.660098]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.660846]  process_one_work+0x22b/0x760
[ 1095.660994]  ? process_one_work+0x194/0x760
[ 1095.661292]  worker_thread+0x137/0x4b0
[ 1095.661439]  kthread+0x10f/0x150
[ 1095.661741]  ? process_one_work+0x760/0x760
[ 1095.661743]  ? kthread_create_on_node+0x70/0x70
[ 1095.661891]  ret_from_fork+0x31/0x40

[ 1095.635223] MemAlloc: kworker/2:3(535) flags=0x4228860 switches=49285 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.635224] kworker/2:3     R  running task    12272   535      2 0x00000000
[ 1095.635246] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.635247] Call Trace:
[ 1095.635250]  __schedule+0x336/0xe00
[ 1095.635253]  preempt_schedule_common+0x1f/0x31
[ 1095.635254]  _cond_resched+0x1c/0x30
[ 1095.635256]  shrink_slab+0x339/0x6d0
[ 1095.635263]  shrink_node+0x2fa/0x310
[ 1095.635267]  do_try_to_free_pages+0xe1/0x300
[ 1095.635271]  try_to_free_pages+0x131/0x3f0
[ 1095.635293]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.635301]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.635305]  alloc_pages_current+0x97/0x1b0
[ 1095.635308]  new_slab+0x4cb/0x6b0
[ 1095.635312]  ___slab_alloc+0x3a3/0x620
[ 1095.635351]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.635355]  ? ___slab_alloc+0x5c6/0x620
[ 1095.635375]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.635377]  __slab_alloc+0x46/0x7d
[ 1095.635380]  __kmalloc+0x301/0x3b0
[ 1095.635399]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.635400]  ? kfree+0x1fa/0x330
[ 1095.635421]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.635460]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.635494]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.635529]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.635549]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.635566]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.635583]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.635586]  process_one_work+0x22b/0x760
[ 1095.635587]  ? process_one_work+0x194/0x760
[ 1095.635591]  worker_thread+0x137/0x4b0
[ 1095.635594]  kthread+0x10f/0x150
[ 1095.635595]  ? process_one_work+0x760/0x760
[ 1095.635596]  ? kthread_create_on_node+0x70/0x70
[ 1095.635598]  ret_from_fork+0x31/0x40

[ 1095.663708] MemAlloc: kworker/2:5(7416) flags=0x4228060 switches=22830 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652069 uninterruptible
[ 1095.663708] kworker/2:5     D12272  7416      2 0x00000080
[ 1095.663738] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.663740] Call Trace:
[ 1095.663743]  __schedule+0x336/0xe00
[ 1095.663746]  schedule+0x3d/0x90
[ 1095.663747]  schedule_timeout+0x20d/0x510
[ 1095.663749]  ? lock_timer_base+0xa0/0xa0
[ 1095.663753]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.663755]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.663761]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.663765]  alloc_pages_current+0x97/0x1b0
[ 1095.663768]  new_slab+0x4cb/0x6b0
[ 1095.663771]  ___slab_alloc+0x3a3/0x620
[ 1095.663793]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663795]  ? ___slab_alloc+0x5c6/0x620
[ 1095.663815]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663817]  __slab_alloc+0x46/0x7d
[ 1095.663820]  __kmalloc+0x301/0x3b0
[ 1095.663859]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.663861]  ? kfree+0x1fa/0x330
[ 1095.663883]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.663905]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.663924]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.663942]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.663961]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.663996]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.664015]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.664017]  process_one_work+0x22b/0x760
[ 1095.664018]  ? process_one_work+0x194/0x760
[ 1095.664022]  worker_thread+0x137/0x4b0
[ 1095.664025]  kthread+0x10f/0x150
[ 1095.664026]  ? process_one_work+0x760/0x760
[ 1095.664027]  ? kthread_create_on_node+0x70/0x70
[ 1095.664030]  ret_from_fork+0x31/0x40

[ 1095.663334] MemAlloc: kworker/2:4(7415) flags=0x4228060 switches=23221 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652078 uninterruptible
[ 1095.663335] kworker/2:4     D11496  7415      2 0x00000080
[ 1095.663361] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.663362] Call Trace:
[ 1095.663366]  __schedule+0x336/0xe00
[ 1095.663369]  schedule+0x3d/0x90
[ 1095.663371]  schedule_timeout+0x20d/0x510
[ 1095.663407]  ? lock_timer_base+0xa0/0xa0
[ 1095.663412]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.663414]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.663421]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.663425]  alloc_pages_current+0x97/0x1b0
[ 1095.663428]  new_slab+0x4cb/0x6b0
[ 1095.663431]  ___slab_alloc+0x3a3/0x620
[ 1095.663457]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663459]  ? ___slab_alloc+0x5c6/0x620
[ 1095.663481]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663483]  __slab_alloc+0x46/0x7d
[ 1095.663485]  __kmalloc+0x301/0x3b0
[ 1095.663505]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.663507]  ? kfree+0x1fa/0x330
[ 1095.663548]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.663572]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.663591]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.663611]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.663630]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.663647]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.663664]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.663667]  process_one_work+0x22b/0x760
[ 1095.663668]  ? process_one_work+0x194/0x760
[ 1095.663672]  worker_thread+0x137/0x4b0
[ 1095.663674]  kthread+0x10f/0x150
[ 1095.663676]  ? process_one_work+0x760/0x760
[ 1095.663696]  ? kthread_create_on_node+0x70/0x70
[ 1095.663699]  ? do_syscall_64+0x195/0x200
[ 1095.663701]  ret_from_fork+0x31/0x40

[ 1095.632604] MemAlloc: kworker/2:1(65) flags=0x4228060 switches=22879 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652043
[ 1095.632604] kworker/2:1     R  running task    12184    65      2 0x00000000
[ 1095.632641] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.632642] Call Trace:
[ 1095.632646]  __schedule+0x336/0xe00
[ 1095.632649]  schedule+0x3d/0x90
[ 1095.632651]  schedule_timeout+0x20d/0x510
[ 1095.632654]  ? lock_timer_base+0xa0/0xa0
[ 1095.632658]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.632660]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.632668]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.632672]  alloc_pages_current+0x97/0x1b0
[ 1095.632675]  new_slab+0x4cb/0x6b0
[ 1095.632678]  ___slab_alloc+0x3a3/0x620
[ 1095.632700]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632702]  ? ___slab_alloc+0x5c6/0x620
[ 1095.632704]  ? free_debug_processing+0x27d/0x2af
[ 1095.632723]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632725]  __slab_alloc+0x46/0x7d
[ 1095.632728]  __kmalloc+0x301/0x3b0
[ 1095.632765]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.632767]  ? kfree+0x1fa/0x330
[ 1095.632804]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.632828]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.632847]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.632865]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.632884]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.632916]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.632935]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.632938]  process_one_work+0x22b/0x760
[ 1095.632939]  ? process_one_work+0x194/0x760
[ 1095.632944]  worker_thread+0x137/0x4b0
[ 1095.632947]  kthread+0x10f/0x150
[ 1095.632949]  ? process_one_work+0x760/0x760
[ 1095.632960]  ? kthread_create_on_node+0x70/0x70
[ 1095.632963]  ret_from_fork+0x31/0x40

[ 1095.645241] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=30893 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.645242] kworker/1:3     R  running task    12184  5357      2 0x00000080
[ 1095.645261] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.645262] Call Trace:
[ 1095.645263]  ? shrink_node+0x2fa/0x310
[ 1095.645267]  ? do_try_to_free_pages+0xe1/0x300
[ 1095.645270]  ? try_to_free_pages+0x131/0x3f0
[ 1095.645274]  ? __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.645280]  ? __alloc_pages_nodemask+0x3e4/0x460
[ 1095.645283]  ? alloc_pages_current+0x97/0x1b0
[ 1095.645286]  ? new_slab+0x4cb/0x6b0
[ 1095.645289]  ? ___slab_alloc+0x3a3/0x620
[ 1095.645310]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645312]  ? ___slab_alloc+0x5c6/0x620
[ 1095.645331]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645333]  ? __slab_alloc+0x46/0x7d
[ 1095.645335]  ? __kmalloc+0x301/0x3b0
[ 1095.645355]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645356]  ? kfree+0x1fa/0x330
[ 1095.645377]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.645398]  ? __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.645417]  ? xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.645435]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.645453]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.645470]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.645486]  ? xfs_end_io+0x81/0x110 [xfs]
[ 1095.645489]  ? process_one_work+0x22b/0x760
[ 1095.645490]  ? process_one_work+0x194/0x760
[ 1095.645493]  ? worker_thread+0x137/0x4b0
[ 1095.645496]  ? kthread+0x10f/0x150
[ 1095.645497]  ? process_one_work+0x760/0x760
[ 1095.645498]  ? kthread_create_on_node+0x70/0x70
[ 1095.645501]  ? ret_from_fork+0x31/0x40

[ 1095.633213] MemAlloc: kworker/1:2(193) flags=0x4228860 switches=28494 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.633213] kworker/1:2     R  running task    12760   193      2 0x00000000
[ 1095.633236] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.633237] Call Trace:
[ 1095.633240]  __schedule+0x336/0xe00
[ 1095.633260]  preempt_schedule_common+0x1f/0x31
[ 1095.633261]  _cond_resched+0x1c/0x30
[ 1095.633263]  shrink_slab+0x339/0x6d0
[ 1095.633270]  shrink_node+0x2fa/0x310
[ 1095.633275]  do_try_to_free_pages+0xe1/0x300
[ 1095.633278]  try_to_free_pages+0x131/0x3f0
[ 1095.633283]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.633292]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.633295]  alloc_pages_current+0x97/0x1b0
[ 1095.633298]  new_slab+0x4cb/0x6b0
[ 1095.633318]  ___slab_alloc+0x3a3/0x620
[ 1095.633349]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633353]  ? ___slab_alloc+0x5c6/0x620
[ 1095.633373]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633375]  __slab_alloc+0x46/0x7d
[ 1095.633378]  __kmalloc+0x301/0x3b0
[ 1095.633397]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.633398]  ? kfree+0x1fa/0x330
[ 1095.633436]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.633464]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.633500]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.633520]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.633538]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.633555]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.633572]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.633575]  process_one_work+0x22b/0x760
[ 1095.633576]  ? process_one_work+0x194/0x760
[ 1095.633597]  worker_thread+0x137/0x4b0
[ 1095.633602]  kthread+0x10f/0x150
[ 1095.633603]  ? process_one_work+0x760/0x760
[ 1095.633604]  ? kthread_create_on_node+0x70/0x70
[ 1095.633607]  ret_from_fork+0x31/0x40

[ 1095.632186] MemAlloc: kworker/1:1(52) flags=0x4228860 switches=28036 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652083
[ 1095.632186] kworker/1:1     R  running task    12760    52      2 0x00000000
[ 1095.632209] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.632210] Call Trace:
[ 1095.632213]  ? _raw_spin_lock+0x69/0x80
[ 1095.632214]  ? __list_lru_count_one.isra.2+0x22/0x80
[ 1095.632216]  ? __list_lru_count_one.isra.2+0x22/0x80
[ 1095.632217]  ? list_lru_count_one+0x23/0x30
[ 1095.632219]  ? super_cache_count+0x6c/0xe0
[ 1095.632221]  ? shrink_slab+0x15c/0x6d0
[ 1095.632241]  ? mem_cgroup_iter+0x14d/0x720
[ 1095.632244]  ? css_next_child+0x17/0xd0
[ 1095.632247]  ? shrink_node+0x2fa/0x310
[ 1095.632251]  ? do_try_to_free_pages+0xe1/0x300
[ 1095.632254]  ? try_to_free_pages+0x131/0x3f0
[ 1095.632258]  ? __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.632265]  ? __alloc_pages_nodemask+0x3e4/0x460
[ 1095.632268]  ? alloc_pages_current+0x97/0x1b0
[ 1095.632271]  ? new_slab+0x4cb/0x6b0
[ 1095.632274]  ? ___slab_alloc+0x3a3/0x620
[ 1095.632313]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632317]  ? ___slab_alloc+0x5c6/0x620
[ 1095.632337]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632339]  ? __slab_alloc+0x46/0x7d
[ 1095.632341]  ? __kmalloc+0x301/0x3b0
[ 1095.632371]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632373]  ? kfree+0x1fa/0x330
[ 1095.632412]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.632443]  ? __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.632480]  ? xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.632500]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.632519]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.632536]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.632553]  ? xfs_end_io+0x81/0x110 [xfs]
[ 1095.632556]  ? process_one_work+0x22b/0x760
[ 1095.632558]  ? process_one_work+0x194/0x760
[ 1095.632588]  ? worker_thread+0x137/0x4b0
[ 1095.632591]  ? kthread+0x10f/0x150
[ 1095.632592]  ? process_one_work+0x760/0x760
[ 1095.632593]  ? kthread_create_on_node+0x70/0x70
[ 1095.632596]  ? ret_from_fork+0x31/0x40

[ 1095.645506] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=27329 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.645506] kworker/1:4     R  running task    12272  5358      2 0x00000080
[ 1095.645525] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.645526] Call Trace:
[ 1095.645528]  __schedule+0x336/0xe00
[ 1095.645531]  schedule+0x3d/0x90
[ 1095.645532]  schedule_timeout+0x20d/0x510
[ 1095.645534]  ? lock_timer_base+0xa0/0xa0
[ 1095.645538]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.645540]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.645546]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.645550]  alloc_pages_current+0x97/0x1b0
[ 1095.645552]  new_slab+0x4cb/0x6b0
[ 1095.645555]  ___slab_alloc+0x3a3/0x620
[ 1095.645576]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645578]  ? ___slab_alloc+0x5c6/0x620
[ 1095.645597]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645599]  __slab_alloc+0x46/0x7d
[ 1095.645601]  __kmalloc+0x301/0x3b0
[ 1095.645620]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.645621]  ? kfree+0x1fa/0x330
[ 1095.645642]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.645663]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.645681]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.645699]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.646636]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.647249]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.647269]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.647273]  process_one_work+0x22b/0x760
[ 1095.647274]  ? process_one_work+0x194/0x760
[ 1095.647277]  worker_thread+0x137/0x4b0
[ 1095.647280]  kthread+0x10f/0x150
[ 1095.647281]  ? process_one_work+0x760/0x760
[ 1095.647283]  ? kthread_create_on_node+0x70/0x70
[ 1095.647285]  ret_from_fork+0x31/0x40

[ 1095.638807] MemAlloc: kworker/0:3(2486) flags=0x4228860 switches=76240 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652064
[ 1095.638807] kworker/0:3     R  running task    11608  2486      2 0x00000080
[ 1095.638829] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.638830] Call Trace:
[ 1095.638833]  __schedule+0x336/0xe00
[ 1095.638837]  preempt_schedule_common+0x1f/0x31
[ 1095.638838]  _cond_resched+0x1c/0x30
[ 1095.638839]  shrink_slab+0x339/0x6d0
[ 1095.638846]  shrink_node+0x2fa/0x310
[ 1095.638850]  do_try_to_free_pages+0xe1/0x300
[ 1095.638854]  try_to_free_pages+0x131/0x3f0
[ 1095.638859]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.638866]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.638871]  alloc_pages_current+0x97/0x1b0
[ 1095.638874]  new_slab+0x4cb/0x6b0
[ 1095.638878]  ___slab_alloc+0x3a3/0x620
[ 1095.638900]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.638903]  ? ___slab_alloc+0x5c6/0x620
[ 1095.638923]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.638925]  __slab_alloc+0x46/0x7d
[ 1095.638928]  __kmalloc+0x301/0x3b0
[ 1095.638947]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.638948]  ? kfree+0x1fa/0x330
[ 1095.638970]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.638992]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.639011]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.639030]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.639066]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.639083]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.639100]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.639103]  process_one_work+0x22b/0x760
[ 1095.639104]  ? process_one_work+0x194/0x760
[ 1095.639108]  worker_thread+0x137/0x4b0
[ 1095.639111]  kthread+0x10f/0x150
[ 1095.639112]  ? process_one_work+0x760/0x760
[ 1095.639113]  ? kthread_create_on_node+0x70/0x70
[ 1095.639116]  ? do_syscall_64+0x6c/0x200
[ 1095.639118]  ret_from_fork+0x31/0x40

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tetsuo Handa]

Brian Foster wrote:
> As noted in my previous reply, I'm not sure there's enough here to point
> at allocation failure as the root cause. E.g., kswapd is stuck here:
> 
> [ 1095.632985] kswapd0         D10776    69      2 0x00000000
> [ 1095.632988] Call Trace:
> [ 1095.632991]  __schedule+0x336/0xe00
> [ 1095.632994]  schedule+0x3d/0x90
> [ 1095.632996]  io_schedule+0x16/0x40
> [ 1095.633017]  __xfs_iflock+0x129/0x140 [xfs]
> [ 1095.633021]  ? autoremove_wake_function+0x60/0x60
> [ 1095.633051]  xfs_reclaim_inode+0x162/0x440 [xfs]
> [ 1095.633072]  xfs_reclaim_inodes_ag+0x2cf/0x4f0 [xfs]
> [ 1095.633106]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
> [ 1095.633114]  ? trace_hardirqs_on+0xd/0x10
> [ 1095.633116]  ? try_to_wake_up+0x59/0x7a0
> [ 1095.633120]  ? wake_up_process+0x15/0x20
> [ 1095.633156]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> [ 1095.633178]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
> [ 1095.633180]  super_cache_scan+0x181/0x190
> [ 1095.633183]  shrink_slab+0x29f/0x6d0
> [ 1095.633189]  shrink_node+0x2fa/0x310
> [ 1095.633193]  kswapd+0x362/0x9b0
> [ 1095.633200]  kthread+0x10f/0x150
> [ 1095.633201]  ? mem_cgroup_shrink_node+0x3b0/0x3b0
> [ 1095.633202]  ? kthread_create_on_node+0x70/0x70
> [ 1095.633205]  ret_from_fork+0x31/0x40
> 
> ... which is waiting on an inode flush lock. It can't get the lock
> (presumably) because xfsaild has it:
> 
> [ 1041.772095] xfsaild/sda1    D13216   457      2 0x00000000
> [ 1041.773726] Call Trace:
> [ 1041.774734]  __schedule+0x336/0xe00
> [ 1041.775956]  schedule+0x3d/0x90
> [ 1041.777105]  schedule_timeout+0x26a/0x510
> [ 1041.778426]  ? wait_for_completion+0x4c/0x190
> [ 1041.779824]  wait_for_completion+0x12c/0x190
> [ 1041.781273]  ? wake_up_q+0x80/0x80
> [ 1041.782597]  ? _xfs_buf_read+0x44/0x90 [xfs]
> [ 1041.784086]  xfs_buf_submit_wait+0xe9/0x5c0 [xfs]
> [ 1041.785659]  _xfs_buf_read+0x44/0x90 [xfs]
> [ 1041.787067]  xfs_buf_read_map+0xfa/0x400 [xfs]
> [ 1041.788501]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
> [ 1041.790103]  xfs_trans_read_buf_map+0x186/0x830 [xfs]
> [ 1041.791672]  xfs_imap_to_bp+0x71/0x110 [xfs]
> [ 1041.793090]  xfs_iflush+0x122/0x3b0 [xfs]
> [ 1041.794444]  xfs_inode_item_push+0x108/0x1c0 [xfs]
> [ 1041.795956]  xfsaild_push+0x1d8/0xb70 [xfs]
> [ 1041.797344]  xfsaild+0x150/0x270 [xfs]
> [ 1041.798623]  kthread+0x10f/0x150
> [ 1041.799819]  ? xfsaild_push+0xb70/0xb70 [xfs]
> [ 1041.801217]  ? kthread_create_on_node+0x70/0x70
> [ 1041.802652]  ret_from_fork+0x31/0x40
> 
> xfsaild is flushing an inode, but is waiting on a read of the underlying
> inode cluster buffer such that it can flush out the in-core inode data
> structure. I cannot tell if the read had actually completed and is
> blocked somewhere else before running the completion. As Dave notes
> earlier, buffer I/O completion relies on the xfs-buf wq. What is evident
> from the logs is that xfs-buf has a rescuer thread that is sitting idle:
> 
> [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> [ 1041.556813] Call Trace:
> [ 1041.557796]  __schedule+0x336/0xe00
> [ 1041.558983]  schedule+0x3d/0x90
> [ 1041.560085]  rescuer_thread+0x322/0x3d0
> [ 1041.561333]  kthread+0x10f/0x150
> [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> [ 1041.565123]  ret_from_fork+0x31/0x40
> 
> So AFAICT if the buffer I/O completion would run, it would allow xfsaild
> to progress, which would eventually flush the underlying buffer, write
> it, release the flush lock and allow kswapd to continue. The question is
> has the actually I/O completed? If so, is the xfs-buf workqueue stuck
> (waiting on an allocation perhaps)? And if that is the case, why is the
> xfs-buf rescuer thread not doing anything?

[ 1116.803780] workqueue xfs-data/sda1: flags=0xc
[ 1116.805324]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=27/256 MAYDAY
[ 1116.807272]     in-flight: 5356:xfs_end_io [xfs], 451(RESCUER):xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs], 10498:xfs_end_io [xfs], 6386:xfs_end_io [xfs]
[ 1116.812145]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
 [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.820988]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=21/256 MAYDAY
[ 1116.823105]     in-flight: 535:xfs_end_io [xfs], 7416:xfs_end_io [xfs], 7415:xfs_end_io [xfs], 65:xfs_end_io [xfs]
[ 1116.826062]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
 [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
[ 1116.834549]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=4/256 MAYDAY
[ 1116.837139]     in-flight: 5357:xfs_end_io [xfs], 193:xfs_end_io [xfs], 52:xfs_end_io [xfs], 5358:xfs_end_io [xfs]
[ 1116.840182]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
[ 1116.842297]     in-flight: 2486:xfs_end_io [xfs]
[ 1116.844230] workqueue xfs-reclaim/sda1: flags=0xc
[ 1116.846168]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[ 1116.848323]     pending: xfs_reclaim_worker [xfs]
[ 1116.850280] workqueue xfs-eofblocks/sda1: flags=0xc
[ 1116.852358]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[ 1116.854601]     in-flight: 456(RESCUER):xfs_eofblocks_worker [xfs]

As listed below, all in-flight workqueues for processing xfs_end_io (PID = 5356,
451(RESCUER), 10498, 6386, 535, 7416, 7415, 65, 5357, 193, 52, 5358, 2486) are
stuck at memory allocation, and thus cannot call complete() when xfsaild/sda1
(PID = 457) is waiting at wait_for_completion(&bp->b_iowait) in xfs_buf_submit_wait().

[ 1095.644936] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=29192 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652086
[ 1095.644937] kworker/3:3     R  running task    12760  5356      2 0x00000080
[ 1095.644959] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.644960] Call Trace:
[ 1095.644962]  __schedule+0x336/0xe00
[ 1095.644965]  preempt_schedule_common+0x1f/0x31
[ 1095.644966]  _cond_resched+0x1c/0x30
[ 1095.644968]  shrink_slab+0x339/0x6d0
[ 1095.644973]  shrink_node+0x2fa/0x310
[ 1095.644977]  do_try_to_free_pages+0xe1/0x300
[ 1095.644979]  try_to_free_pages+0x131/0x3f0
[ 1095.644984]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.644985]  ? cpuacct_charge+0xf3/0x1e0
[ 1095.644986]  ? cpuacct_charge+0x38/0x1e0
[ 1095.644992]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.644995]  alloc_pages_current+0x97/0x1b0
[ 1095.644998]  new_slab+0x4cb/0x6b0
[ 1095.645001]  ___slab_alloc+0x3a3/0x620
[ 1095.645023]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645026]  ? ___slab_alloc+0x5c6/0x620
[ 1095.645027]  ? trace_hardirqs_on+0xd/0x10
[ 1095.645058]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645066]  __slab_alloc+0x46/0x7d
[ 1095.645069]  __kmalloc+0x301/0x3b0
[ 1095.645088]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.645089]  ? kfree+0x1fa/0x330
[ 1095.645110]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.645132]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.645151]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.645169]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.645188]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.645204]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.645221]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.645223]  process_one_work+0x22b/0x760
[ 1095.645224]  ? process_one_work+0x194/0x760
[ 1095.645228]  worker_thread+0x137/0x4b0
[ 1095.645231]  kthread+0x10f/0x150
[ 1095.645232]  ? process_one_work+0x760/0x760
[ 1095.645233]  ? kthread_create_on_node+0x70/0x70
[ 1095.645236]  ret_from_fork+0x31/0x40

[ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
[ 1095.633626] xfs-data/sda1   R  running task    12696   451      2 0x00000000
[ 1095.633663] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.633665] Call Trace:
[ 1095.633668]  __schedule+0x336/0xe00
[ 1095.633671]  schedule+0x3d/0x90
[ 1095.633672]  schedule_timeout+0x20d/0x510
[ 1095.633675]  ? lock_timer_base+0xa0/0xa0
[ 1095.633678]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.633680]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.633687]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.633699]  alloc_pages_current+0x97/0x1b0
[ 1095.633702]  new_slab+0x4cb/0x6b0
[ 1095.633706]  ___slab_alloc+0x3a3/0x620
[ 1095.633728]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633730]  ? ___slab_alloc+0x5c6/0x620
[ 1095.633732]  ? cpuacct_charge+0x38/0x1e0
[ 1095.633767]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633770]  __slab_alloc+0x46/0x7d
[ 1095.633773]  __kmalloc+0x301/0x3b0
[ 1095.633802]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.633804]  ? kfree+0x1fa/0x330
[ 1095.633842]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.633864]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.633883]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.633901]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.633936]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.633954]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.633971]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.633973]  process_one_work+0x22b/0x760
[ 1095.633975]  ? process_one_work+0x194/0x760
[ 1095.633997]  rescuer_thread+0x1f2/0x3d0
[ 1095.634002]  kthread+0x10f/0x150
[ 1095.634003]  ? worker_thread+0x4b0/0x4b0
[ 1095.634004]  ? kthread_create_on_node+0x70/0x70
[ 1095.634007]  ret_from_fork+0x31/0x40

[ 1095.806606] MemAlloc: kworker/3:0(10498) flags=0x4228060 switches=16222 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
[ 1095.806607] kworker/3:0     R  running task    11352 10498      2 0x00000080
[ 1095.806631] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.806633] Call Trace:
[ 1095.806636]  __schedule+0x336/0xe00
[ 1095.806640]  schedule+0x3d/0x90
[ 1095.806641]  schedule_timeout+0x20d/0x510
[ 1095.806644]  ? lock_timer_base+0xa0/0xa0
[ 1095.806649]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.806651]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.806652]  ? cpuacct_charge+0xf3/0x1e0
[ 1095.806654]  ? cpuacct_charge+0x38/0x1e0
[ 1095.806663]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.806667]  alloc_pages_current+0x97/0x1b0
[ 1095.806671]  new_slab+0x4cb/0x6b0
[ 1095.806675]  ___slab_alloc+0x3a3/0x620
[ 1095.806698]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.806701]  ? ___slab_alloc+0x5c6/0x620
[ 1095.806721]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.806724]  __slab_alloc+0x46/0x7d
[ 1095.806727]  __kmalloc+0x301/0x3b0
[ 1095.806747]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.806749]  ? kfree+0x1fa/0x330
[ 1095.806771]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.806794]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.806813]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.806832]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.806852]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.806868]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.806886]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.806889]  process_one_work+0x22b/0x760
[ 1095.806890]  ? process_one_work+0x194/0x760
[ 1095.806896]  worker_thread+0x137/0x4b0
[ 1095.806899]  kthread+0x10f/0x150
[ 1095.806901]  ? process_one_work+0x760/0x760
[ 1095.806902]  ? kthread_create_on_node+0x70/0x70
[ 1095.806904]  ? do_syscall_64+0x195/0x200
[ 1095.806906]  ret_from_fork+0x31/0x40

[ 1095.647293] MemAlloc: kworker/3:5(6386) flags=0x4228860 switches=43427 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
[ 1095.647294] kworker/3:5     R  running task    11048  6386      2 0x00000080
[ 1095.647754] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.648203] Call Trace:
[ 1095.648207]  __schedule+0x336/0xe00
[ 1095.648504]  preempt_schedule_common+0x1f/0x31
[ 1095.648806]  _cond_resched+0x1c/0x30
[ 1095.648952]  shrink_slab+0x339/0x6d0
[ 1095.649252]  shrink_node+0x2fa/0x310
[ 1095.649694]  do_try_to_free_pages+0xe1/0x300
[ 1095.649699]  try_to_free_pages+0x131/0x3f0
[ 1095.649996]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.650296]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.650743]  alloc_pages_current+0x97/0x1b0
[ 1095.650892]  new_slab+0x4cb/0x6b0
[ 1095.651193]  ___slab_alloc+0x3a3/0x620
[ 1095.652087]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.652400]  ? ___slab_alloc+0x5c6/0x620
[ 1095.653457]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.653901]  __slab_alloc+0x46/0x7d
[ 1095.654050]  __kmalloc+0x301/0x3b0
[ 1095.654948]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.655396]  ? kfree+0x1fa/0x330
[ 1095.656589]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.656919]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.656941]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.656961]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.658006]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.659053]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.660098]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.660846]  process_one_work+0x22b/0x760
[ 1095.660994]  ? process_one_work+0x194/0x760
[ 1095.661292]  worker_thread+0x137/0x4b0
[ 1095.661439]  kthread+0x10f/0x150
[ 1095.661741]  ? process_one_work+0x760/0x760
[ 1095.661743]  ? kthread_create_on_node+0x70/0x70
[ 1095.661891]  ret_from_fork+0x31/0x40

[ 1095.635223] MemAlloc: kworker/2:3(535) flags=0x4228860 switches=49285 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.635224] kworker/2:3     R  running task    12272   535      2 0x00000000
[ 1095.635246] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.635247] Call Trace:
[ 1095.635250]  __schedule+0x336/0xe00
[ 1095.635253]  preempt_schedule_common+0x1f/0x31
[ 1095.635254]  _cond_resched+0x1c/0x30
[ 1095.635256]  shrink_slab+0x339/0x6d0
[ 1095.635263]  shrink_node+0x2fa/0x310
[ 1095.635267]  do_try_to_free_pages+0xe1/0x300
[ 1095.635271]  try_to_free_pages+0x131/0x3f0
[ 1095.635293]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.635301]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.635305]  alloc_pages_current+0x97/0x1b0
[ 1095.635308]  new_slab+0x4cb/0x6b0
[ 1095.635312]  ___slab_alloc+0x3a3/0x620
[ 1095.635351]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.635355]  ? ___slab_alloc+0x5c6/0x620
[ 1095.635375]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.635377]  __slab_alloc+0x46/0x7d
[ 1095.635380]  __kmalloc+0x301/0x3b0
[ 1095.635399]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.635400]  ? kfree+0x1fa/0x330
[ 1095.635421]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.635460]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.635494]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.635529]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.635549]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.635566]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.635583]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.635586]  process_one_work+0x22b/0x760
[ 1095.635587]  ? process_one_work+0x194/0x760
[ 1095.635591]  worker_thread+0x137/0x4b0
[ 1095.635594]  kthread+0x10f/0x150
[ 1095.635595]  ? process_one_work+0x760/0x760
[ 1095.635596]  ? kthread_create_on_node+0x70/0x70
[ 1095.635598]  ret_from_fork+0x31/0x40

[ 1095.663708] MemAlloc: kworker/2:5(7416) flags=0x4228060 switches=22830 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652069 uninterruptible
[ 1095.663708] kworker/2:5     D12272  7416      2 0x00000080
[ 1095.663738] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.663740] Call Trace:
[ 1095.663743]  __schedule+0x336/0xe00
[ 1095.663746]  schedule+0x3d/0x90
[ 1095.663747]  schedule_timeout+0x20d/0x510
[ 1095.663749]  ? lock_timer_base+0xa0/0xa0
[ 1095.663753]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.663755]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.663761]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.663765]  alloc_pages_current+0x97/0x1b0
[ 1095.663768]  new_slab+0x4cb/0x6b0
[ 1095.663771]  ___slab_alloc+0x3a3/0x620
[ 1095.663793]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663795]  ? ___slab_alloc+0x5c6/0x620
[ 1095.663815]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663817]  __slab_alloc+0x46/0x7d
[ 1095.663820]  __kmalloc+0x301/0x3b0
[ 1095.663859]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.663861]  ? kfree+0x1fa/0x330
[ 1095.663883]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.663905]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.663924]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.663942]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.663961]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.663996]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.664015]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.664017]  process_one_work+0x22b/0x760
[ 1095.664018]  ? process_one_work+0x194/0x760
[ 1095.664022]  worker_thread+0x137/0x4b0
[ 1095.664025]  kthread+0x10f/0x150
[ 1095.664026]  ? process_one_work+0x760/0x760
[ 1095.664027]  ? kthread_create_on_node+0x70/0x70
[ 1095.664030]  ret_from_fork+0x31/0x40

[ 1095.663334] MemAlloc: kworker/2:4(7415) flags=0x4228060 switches=23221 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652078 uninterruptible
[ 1095.663335] kworker/2:4     D11496  7415      2 0x00000080
[ 1095.663361] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.663362] Call Trace:
[ 1095.663366]  __schedule+0x336/0xe00
[ 1095.663369]  schedule+0x3d/0x90
[ 1095.663371]  schedule_timeout+0x20d/0x510
[ 1095.663407]  ? lock_timer_base+0xa0/0xa0
[ 1095.663412]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.663414]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.663421]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.663425]  alloc_pages_current+0x97/0x1b0
[ 1095.663428]  new_slab+0x4cb/0x6b0
[ 1095.663431]  ___slab_alloc+0x3a3/0x620
[ 1095.663457]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663459]  ? ___slab_alloc+0x5c6/0x620
[ 1095.663481]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.663483]  __slab_alloc+0x46/0x7d
[ 1095.663485]  __kmalloc+0x301/0x3b0
[ 1095.663505]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.663507]  ? kfree+0x1fa/0x330
[ 1095.663548]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.663572]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.663591]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.663611]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.663630]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.663647]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.663664]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.663667]  process_one_work+0x22b/0x760
[ 1095.663668]  ? process_one_work+0x194/0x760
[ 1095.663672]  worker_thread+0x137/0x4b0
[ 1095.663674]  kthread+0x10f/0x150
[ 1095.663676]  ? process_one_work+0x760/0x760
[ 1095.663696]  ? kthread_create_on_node+0x70/0x70
[ 1095.663699]  ? do_syscall_64+0x195/0x200
[ 1095.663701]  ret_from_fork+0x31/0x40

[ 1095.632604] MemAlloc: kworker/2:1(65) flags=0x4228060 switches=22879 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652043
[ 1095.632604] kworker/2:1     R  running task    12184    65      2 0x00000000
[ 1095.632641] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.632642] Call Trace:
[ 1095.632646]  __schedule+0x336/0xe00
[ 1095.632649]  schedule+0x3d/0x90
[ 1095.632651]  schedule_timeout+0x20d/0x510
[ 1095.632654]  ? lock_timer_base+0xa0/0xa0
[ 1095.632658]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.632660]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.632668]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.632672]  alloc_pages_current+0x97/0x1b0
[ 1095.632675]  new_slab+0x4cb/0x6b0
[ 1095.632678]  ___slab_alloc+0x3a3/0x620
[ 1095.632700]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632702]  ? ___slab_alloc+0x5c6/0x620
[ 1095.632704]  ? free_debug_processing+0x27d/0x2af
[ 1095.632723]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632725]  __slab_alloc+0x46/0x7d
[ 1095.632728]  __kmalloc+0x301/0x3b0
[ 1095.632765]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.632767]  ? kfree+0x1fa/0x330
[ 1095.632804]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.632828]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.632847]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.632865]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.632884]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.632916]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.632935]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.632938]  process_one_work+0x22b/0x760
[ 1095.632939]  ? process_one_work+0x194/0x760
[ 1095.632944]  worker_thread+0x137/0x4b0
[ 1095.632947]  kthread+0x10f/0x150
[ 1095.632949]  ? process_one_work+0x760/0x760
[ 1095.632960]  ? kthread_create_on_node+0x70/0x70
[ 1095.632963]  ret_from_fork+0x31/0x40

[ 1095.645241] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=30893 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.645242] kworker/1:3     R  running task    12184  5357      2 0x00000080
[ 1095.645261] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.645262] Call Trace:
[ 1095.645263]  ? shrink_node+0x2fa/0x310
[ 1095.645267]  ? do_try_to_free_pages+0xe1/0x300
[ 1095.645270]  ? try_to_free_pages+0x131/0x3f0
[ 1095.645274]  ? __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.645280]  ? __alloc_pages_nodemask+0x3e4/0x460
[ 1095.645283]  ? alloc_pages_current+0x97/0x1b0
[ 1095.645286]  ? new_slab+0x4cb/0x6b0
[ 1095.645289]  ? ___slab_alloc+0x3a3/0x620
[ 1095.645310]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645312]  ? ___slab_alloc+0x5c6/0x620
[ 1095.645331]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645333]  ? __slab_alloc+0x46/0x7d
[ 1095.645335]  ? __kmalloc+0x301/0x3b0
[ 1095.645355]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645356]  ? kfree+0x1fa/0x330
[ 1095.645377]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.645398]  ? __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.645417]  ? xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.645435]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.645453]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.645470]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.645486]  ? xfs_end_io+0x81/0x110 [xfs]
[ 1095.645489]  ? process_one_work+0x22b/0x760
[ 1095.645490]  ? process_one_work+0x194/0x760
[ 1095.645493]  ? worker_thread+0x137/0x4b0
[ 1095.645496]  ? kthread+0x10f/0x150
[ 1095.645497]  ? process_one_work+0x760/0x760
[ 1095.645498]  ? kthread_create_on_node+0x70/0x70
[ 1095.645501]  ? ret_from_fork+0x31/0x40

[ 1095.633213] MemAlloc: kworker/1:2(193) flags=0x4228860 switches=28494 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.633213] kworker/1:2     R  running task    12760   193      2 0x00000000
[ 1095.633236] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.633237] Call Trace:
[ 1095.633240]  __schedule+0x336/0xe00
[ 1095.633260]  preempt_schedule_common+0x1f/0x31
[ 1095.633261]  _cond_resched+0x1c/0x30
[ 1095.633263]  shrink_slab+0x339/0x6d0
[ 1095.633270]  shrink_node+0x2fa/0x310
[ 1095.633275]  do_try_to_free_pages+0xe1/0x300
[ 1095.633278]  try_to_free_pages+0x131/0x3f0
[ 1095.633283]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.633292]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.633295]  alloc_pages_current+0x97/0x1b0
[ 1095.633298]  new_slab+0x4cb/0x6b0
[ 1095.633318]  ___slab_alloc+0x3a3/0x620
[ 1095.633349]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633353]  ? ___slab_alloc+0x5c6/0x620
[ 1095.633373]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.633375]  __slab_alloc+0x46/0x7d
[ 1095.633378]  __kmalloc+0x301/0x3b0
[ 1095.633397]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.633398]  ? kfree+0x1fa/0x330
[ 1095.633436]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.633464]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.633500]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.633520]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.633538]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.633555]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.633572]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.633575]  process_one_work+0x22b/0x760
[ 1095.633576]  ? process_one_work+0x194/0x760
[ 1095.633597]  worker_thread+0x137/0x4b0
[ 1095.633602]  kthread+0x10f/0x150
[ 1095.633603]  ? process_one_work+0x760/0x760
[ 1095.633604]  ? kthread_create_on_node+0x70/0x70
[ 1095.633607]  ret_from_fork+0x31/0x40

[ 1095.632186] MemAlloc: kworker/1:1(52) flags=0x4228860 switches=28036 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652083
[ 1095.632186] kworker/1:1     R  running task    12760    52      2 0x00000000
[ 1095.632209] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.632210] Call Trace:
[ 1095.632213]  ? _raw_spin_lock+0x69/0x80
[ 1095.632214]  ? __list_lru_count_one.isra.2+0x22/0x80
[ 1095.632216]  ? __list_lru_count_one.isra.2+0x22/0x80
[ 1095.632217]  ? list_lru_count_one+0x23/0x30
[ 1095.632219]  ? super_cache_count+0x6c/0xe0
[ 1095.632221]  ? shrink_slab+0x15c/0x6d0
[ 1095.632241]  ? mem_cgroup_iter+0x14d/0x720
[ 1095.632244]  ? css_next_child+0x17/0xd0
[ 1095.632247]  ? shrink_node+0x2fa/0x310
[ 1095.632251]  ? do_try_to_free_pages+0xe1/0x300
[ 1095.632254]  ? try_to_free_pages+0x131/0x3f0
[ 1095.632258]  ? __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.632265]  ? __alloc_pages_nodemask+0x3e4/0x460
[ 1095.632268]  ? alloc_pages_current+0x97/0x1b0
[ 1095.632271]  ? new_slab+0x4cb/0x6b0
[ 1095.632274]  ? ___slab_alloc+0x3a3/0x620
[ 1095.632313]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632317]  ? ___slab_alloc+0x5c6/0x620
[ 1095.632337]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632339]  ? __slab_alloc+0x46/0x7d
[ 1095.632341]  ? __kmalloc+0x301/0x3b0
[ 1095.632371]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.632373]  ? kfree+0x1fa/0x330
[ 1095.632412]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.632443]  ? __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.632480]  ? xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.632500]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.632519]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.632536]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.632553]  ? xfs_end_io+0x81/0x110 [xfs]
[ 1095.632556]  ? process_one_work+0x22b/0x760
[ 1095.632558]  ? process_one_work+0x194/0x760
[ 1095.632588]  ? worker_thread+0x137/0x4b0
[ 1095.632591]  ? kthread+0x10f/0x150
[ 1095.632592]  ? process_one_work+0x760/0x760
[ 1095.632593]  ? kthread_create_on_node+0x70/0x70
[ 1095.632596]  ? ret_from_fork+0x31/0x40

[ 1095.645506] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=27329 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
[ 1095.645506] kworker/1:4     R  running task    12272  5358      2 0x00000080
[ 1095.645525] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.645526] Call Trace:
[ 1095.645528]  __schedule+0x336/0xe00
[ 1095.645531]  schedule+0x3d/0x90
[ 1095.645532]  schedule_timeout+0x20d/0x510
[ 1095.645534]  ? lock_timer_base+0xa0/0xa0
[ 1095.645538]  schedule_timeout_uninterruptible+0x2a/0x30
[ 1095.645540]  __alloc_pages_slowpath+0x2b5/0xd95
[ 1095.645546]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.645550]  alloc_pages_current+0x97/0x1b0
[ 1095.645552]  new_slab+0x4cb/0x6b0
[ 1095.645555]  ___slab_alloc+0x3a3/0x620
[ 1095.645576]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645578]  ? ___slab_alloc+0x5c6/0x620
[ 1095.645597]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.645599]  __slab_alloc+0x46/0x7d
[ 1095.645601]  __kmalloc+0x301/0x3b0
[ 1095.645620]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.645621]  ? kfree+0x1fa/0x330
[ 1095.645642]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.645663]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.645681]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.645699]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.646636]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.647249]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.647269]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.647273]  process_one_work+0x22b/0x760
[ 1095.647274]  ? process_one_work+0x194/0x760
[ 1095.647277]  worker_thread+0x137/0x4b0
[ 1095.647280]  kthread+0x10f/0x150
[ 1095.647281]  ? process_one_work+0x760/0x760
[ 1095.647283]  ? kthread_create_on_node+0x70/0x70
[ 1095.647285]  ret_from_fork+0x31/0x40

[ 1095.638807] MemAlloc: kworker/0:3(2486) flags=0x4228860 switches=76240 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652064
[ 1095.638807] kworker/0:3     R  running task    11608  2486      2 0x00000080
[ 1095.638829] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
[ 1095.638830] Call Trace:
[ 1095.638833]  __schedule+0x336/0xe00
[ 1095.638837]  preempt_schedule_common+0x1f/0x31
[ 1095.638838]  _cond_resched+0x1c/0x30
[ 1095.638839]  shrink_slab+0x339/0x6d0
[ 1095.638846]  shrink_node+0x2fa/0x310
[ 1095.638850]  do_try_to_free_pages+0xe1/0x300
[ 1095.638854]  try_to_free_pages+0x131/0x3f0
[ 1095.638859]  __alloc_pages_slowpath+0x3ec/0xd95
[ 1095.638866]  __alloc_pages_nodemask+0x3e4/0x460
[ 1095.638871]  alloc_pages_current+0x97/0x1b0
[ 1095.638874]  new_slab+0x4cb/0x6b0
[ 1095.638878]  ___slab_alloc+0x3a3/0x620
[ 1095.638900]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.638903]  ? ___slab_alloc+0x5c6/0x620
[ 1095.638923]  ? kmem_alloc+0x96/0x120 [xfs]
[ 1095.638925]  __slab_alloc+0x46/0x7d
[ 1095.638928]  __kmalloc+0x301/0x3b0
[ 1095.638947]  kmem_alloc+0x96/0x120 [xfs]
[ 1095.638948]  ? kfree+0x1fa/0x330
[ 1095.638970]  xfs_log_commit_cil+0x489/0x710 [xfs]
[ 1095.638992]  __xfs_trans_commit+0x83/0x260 [xfs]
[ 1095.639011]  xfs_trans_commit+0x10/0x20 [xfs]
[ 1095.639030]  __xfs_setfilesize+0xdb/0x240 [xfs]
[ 1095.639066]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
[ 1095.639083]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
[ 1095.639100]  xfs_end_io+0x81/0x110 [xfs]
[ 1095.639103]  process_one_work+0x22b/0x760
[ 1095.639104]  ? process_one_work+0x194/0x760
[ 1095.639108]  worker_thread+0x137/0x4b0
[ 1095.639111]  kthread+0x10f/0x150
[ 1095.639112]  ? process_one_work+0x760/0x760
[ 1095.639113]  ? kthread_create_on_node+0x70/0x70
[ 1095.639116]  ? do_syscall_64+0x6c/0x200
[ 1095.639118]  ret_from_fork+0x31/0x40

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Tue, Mar 07, 2017 at 01:08:24AM +0900, Tetsuo Handa wrote:
> Brian Foster wrote:
> > As noted in my previous reply, I'm not sure there's enough here to point
> > at allocation failure as the root cause. E.g., kswapd is stuck here:
> > 
> > [ 1095.632985] kswapd0         D10776    69      2 0x00000000
> > [ 1095.632988] Call Trace:
> > [ 1095.632991]  __schedule+0x336/0xe00
> > [ 1095.632994]  schedule+0x3d/0x90
> > [ 1095.632996]  io_schedule+0x16/0x40
> > [ 1095.633017]  __xfs_iflock+0x129/0x140 [xfs]
> > [ 1095.633021]  ? autoremove_wake_function+0x60/0x60
> > [ 1095.633051]  xfs_reclaim_inode+0x162/0x440 [xfs]
> > [ 1095.633072]  xfs_reclaim_inodes_ag+0x2cf/0x4f0 [xfs]
> > [ 1095.633106]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
> > [ 1095.633114]  ? trace_hardirqs_on+0xd/0x10
> > [ 1095.633116]  ? try_to_wake_up+0x59/0x7a0
> > [ 1095.633120]  ? wake_up_process+0x15/0x20
> > [ 1095.633156]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> > [ 1095.633178]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
> > [ 1095.633180]  super_cache_scan+0x181/0x190
> > [ 1095.633183]  shrink_slab+0x29f/0x6d0
> > [ 1095.633189]  shrink_node+0x2fa/0x310
> > [ 1095.633193]  kswapd+0x362/0x9b0
> > [ 1095.633200]  kthread+0x10f/0x150
> > [ 1095.633201]  ? mem_cgroup_shrink_node+0x3b0/0x3b0
> > [ 1095.633202]  ? kthread_create_on_node+0x70/0x70
> > [ 1095.633205]  ret_from_fork+0x31/0x40
> > 
> > ... which is waiting on an inode flush lock. It can't get the lock
> > (presumably) because xfsaild has it:
> > 
> > [ 1041.772095] xfsaild/sda1    D13216   457      2 0x00000000
> > [ 1041.773726] Call Trace:
> > [ 1041.774734]  __schedule+0x336/0xe00
> > [ 1041.775956]  schedule+0x3d/0x90
> > [ 1041.777105]  schedule_timeout+0x26a/0x510
> > [ 1041.778426]  ? wait_for_completion+0x4c/0x190
> > [ 1041.779824]  wait_for_completion+0x12c/0x190
> > [ 1041.781273]  ? wake_up_q+0x80/0x80
> > [ 1041.782597]  ? _xfs_buf_read+0x44/0x90 [xfs]
> > [ 1041.784086]  xfs_buf_submit_wait+0xe9/0x5c0 [xfs]
> > [ 1041.785659]  _xfs_buf_read+0x44/0x90 [xfs]
> > [ 1041.787067]  xfs_buf_read_map+0xfa/0x400 [xfs]
> > [ 1041.788501]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
> > [ 1041.790103]  xfs_trans_read_buf_map+0x186/0x830 [xfs]
> > [ 1041.791672]  xfs_imap_to_bp+0x71/0x110 [xfs]
> > [ 1041.793090]  xfs_iflush+0x122/0x3b0 [xfs]
> > [ 1041.794444]  xfs_inode_item_push+0x108/0x1c0 [xfs]
> > [ 1041.795956]  xfsaild_push+0x1d8/0xb70 [xfs]
> > [ 1041.797344]  xfsaild+0x150/0x270 [xfs]
> > [ 1041.798623]  kthread+0x10f/0x150
> > [ 1041.799819]  ? xfsaild_push+0xb70/0xb70 [xfs]
> > [ 1041.801217]  ? kthread_create_on_node+0x70/0x70
> > [ 1041.802652]  ret_from_fork+0x31/0x40
> > 
> > xfsaild is flushing an inode, but is waiting on a read of the underlying
> > inode cluster buffer such that it can flush out the in-core inode data
> > structure. I cannot tell if the read had actually completed and is
> > blocked somewhere else before running the completion. As Dave notes
> > earlier, buffer I/O completion relies on the xfs-buf wq. What is evident
> > from the logs is that xfs-buf has a rescuer thread that is sitting idle:
> > 
> > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > [ 1041.556813] Call Trace:
> > [ 1041.557796]  __schedule+0x336/0xe00
> > [ 1041.558983]  schedule+0x3d/0x90
> > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > [ 1041.561333]  kthread+0x10f/0x150
> > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > [ 1041.565123]  ret_from_fork+0x31/0x40
> > 
> > So AFAICT if the buffer I/O completion would run, it would allow xfsaild
> > to progress, which would eventually flush the underlying buffer, write
> > it, release the flush lock and allow kswapd to continue. The question is
> > has the actually I/O completed? If so, is the xfs-buf workqueue stuck
> > (waiting on an allocation perhaps)? And if that is the case, why is the
> > xfs-buf rescuer thread not doing anything?
> 
> [ 1116.803780] workqueue xfs-data/sda1: flags=0xc
> [ 1116.805324]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=27/256 MAYDAY
> [ 1116.807272]     in-flight: 5356:xfs_end_io [xfs], 451(RESCUER):xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs], 10498:xfs_end_io [xfs], 6386:xfs_end_io [xfs]
> [ 1116.812145]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
>  [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
> [ 1116.820988]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=21/256 MAYDAY
> [ 1116.823105]     in-flight: 535:xfs_end_io [xfs], 7416:xfs_end_io [xfs], 7415:xfs_end_io [xfs], 65:xfs_end_io [xfs]
> [ 1116.826062]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
>  [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
> [ 1116.834549]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=4/256 MAYDAY
> [ 1116.837139]     in-flight: 5357:xfs_end_io [xfs], 193:xfs_end_io [xfs], 52:xfs_end_io [xfs], 5358:xfs_end_io [xfs]
> [ 1116.840182]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
> [ 1116.842297]     in-flight: 2486:xfs_end_io [xfs]
> [ 1116.844230] workqueue xfs-reclaim/sda1: flags=0xc
> [ 1116.846168]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
> [ 1116.848323]     pending: xfs_reclaim_worker [xfs]
> [ 1116.850280] workqueue xfs-eofblocks/sda1: flags=0xc
> [ 1116.852358]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
> [ 1116.854601]     in-flight: 456(RESCUER):xfs_eofblocks_worker [xfs]
> 
> As listed below, all in-flight workqueues for processing xfs_end_io (PID = 5356,
> 451(RESCUER), 10498, 6386, 535, 7416, 7415, 65, 5357, 193, 52, 5358, 2486) are
> stuck at memory allocation, and thus cannot call complete() when xfsaild/sda1
> (PID = 457) is waiting at wait_for_completion(&bp->b_iowait) in xfs_buf_submit_wait().
> 

These are all in xfs_end_io(), which means they are either the xfs-data
or xfs-conv workqueues. ->b_iowait is ultimately triggered via
xfs_buf_ioend() by the xfs-buf workqueue (or xfs-log for log buffers).
None of the following use xfs-buf.

Brian

> [ 1095.644936] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=29192 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652086
> [ 1095.644937] kworker/3:3     R  running task    12760  5356      2 0x00000080
> [ 1095.644959] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.644960] Call Trace:
> [ 1095.644962]  __schedule+0x336/0xe00
> [ 1095.644965]  preempt_schedule_common+0x1f/0x31
> [ 1095.644966]  _cond_resched+0x1c/0x30
> [ 1095.644968]  shrink_slab+0x339/0x6d0
> [ 1095.644973]  shrink_node+0x2fa/0x310
> [ 1095.644977]  do_try_to_free_pages+0xe1/0x300
> [ 1095.644979]  try_to_free_pages+0x131/0x3f0
> [ 1095.644984]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.644985]  ? cpuacct_charge+0xf3/0x1e0
> [ 1095.644986]  ? cpuacct_charge+0x38/0x1e0
> [ 1095.644992]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.644995]  alloc_pages_current+0x97/0x1b0
> [ 1095.644998]  new_slab+0x4cb/0x6b0
> [ 1095.645001]  ___slab_alloc+0x3a3/0x620
> [ 1095.645023]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645026]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.645027]  ? trace_hardirqs_on+0xd/0x10
> [ 1095.645058]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645066]  __slab_alloc+0x46/0x7d
> [ 1095.645069]  __kmalloc+0x301/0x3b0
> [ 1095.645088]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645089]  ? kfree+0x1fa/0x330
> [ 1095.645110]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.645132]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.645151]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.645169]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.645188]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.645204]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.645221]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.645223]  process_one_work+0x22b/0x760
> [ 1095.645224]  ? process_one_work+0x194/0x760
> [ 1095.645228]  worker_thread+0x137/0x4b0
> [ 1095.645231]  kthread+0x10f/0x150
> [ 1095.645232]  ? process_one_work+0x760/0x760
> [ 1095.645233]  ? kthread_create_on_node+0x70/0x70
> [ 1095.645236]  ret_from_fork+0x31/0x40
> 
> [ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
> [ 1095.633626] xfs-data/sda1   R  running task    12696   451      2 0x00000000
> [ 1095.633663] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.633665] Call Trace:
> [ 1095.633668]  __schedule+0x336/0xe00
> [ 1095.633671]  schedule+0x3d/0x90
> [ 1095.633672]  schedule_timeout+0x20d/0x510
> [ 1095.633675]  ? lock_timer_base+0xa0/0xa0
> [ 1095.633678]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.633680]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.633687]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.633699]  alloc_pages_current+0x97/0x1b0
> [ 1095.633702]  new_slab+0x4cb/0x6b0
> [ 1095.633706]  ___slab_alloc+0x3a3/0x620
> [ 1095.633728]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633730]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.633732]  ? cpuacct_charge+0x38/0x1e0
> [ 1095.633767]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633770]  __slab_alloc+0x46/0x7d
> [ 1095.633773]  __kmalloc+0x301/0x3b0
> [ 1095.633802]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633804]  ? kfree+0x1fa/0x330
> [ 1095.633842]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.633864]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.633883]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.633901]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.633936]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.633954]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.633971]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.633973]  process_one_work+0x22b/0x760
> [ 1095.633975]  ? process_one_work+0x194/0x760
> [ 1095.633997]  rescuer_thread+0x1f2/0x3d0
> [ 1095.634002]  kthread+0x10f/0x150
> [ 1095.634003]  ? worker_thread+0x4b0/0x4b0
> [ 1095.634004]  ? kthread_create_on_node+0x70/0x70
> [ 1095.634007]  ret_from_fork+0x31/0x40
> 
> [ 1095.806606] MemAlloc: kworker/3:0(10498) flags=0x4228060 switches=16222 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
> [ 1095.806607] kworker/3:0     R  running task    11352 10498      2 0x00000080
> [ 1095.806631] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.806633] Call Trace:
> [ 1095.806636]  __schedule+0x336/0xe00
> [ 1095.806640]  schedule+0x3d/0x90
> [ 1095.806641]  schedule_timeout+0x20d/0x510
> [ 1095.806644]  ? lock_timer_base+0xa0/0xa0
> [ 1095.806649]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.806651]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.806652]  ? cpuacct_charge+0xf3/0x1e0
> [ 1095.806654]  ? cpuacct_charge+0x38/0x1e0
> [ 1095.806663]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.806667]  alloc_pages_current+0x97/0x1b0
> [ 1095.806671]  new_slab+0x4cb/0x6b0
> [ 1095.806675]  ___slab_alloc+0x3a3/0x620
> [ 1095.806698]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.806701]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.806721]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.806724]  __slab_alloc+0x46/0x7d
> [ 1095.806727]  __kmalloc+0x301/0x3b0
> [ 1095.806747]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.806749]  ? kfree+0x1fa/0x330
> [ 1095.806771]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.806794]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.806813]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.806832]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.806852]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.806868]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.806886]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.806889]  process_one_work+0x22b/0x760
> [ 1095.806890]  ? process_one_work+0x194/0x760
> [ 1095.806896]  worker_thread+0x137/0x4b0
> [ 1095.806899]  kthread+0x10f/0x150
> [ 1095.806901]  ? process_one_work+0x760/0x760
> [ 1095.806902]  ? kthread_create_on_node+0x70/0x70
> [ 1095.806904]  ? do_syscall_64+0x195/0x200
> [ 1095.806906]  ret_from_fork+0x31/0x40
> 
> [ 1095.647293] MemAlloc: kworker/3:5(6386) flags=0x4228860 switches=43427 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
> [ 1095.647294] kworker/3:5     R  running task    11048  6386      2 0x00000080
> [ 1095.647754] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.648203] Call Trace:
> [ 1095.648207]  __schedule+0x336/0xe00
> [ 1095.648504]  preempt_schedule_common+0x1f/0x31
> [ 1095.648806]  _cond_resched+0x1c/0x30
> [ 1095.648952]  shrink_slab+0x339/0x6d0
> [ 1095.649252]  shrink_node+0x2fa/0x310
> [ 1095.649694]  do_try_to_free_pages+0xe1/0x300
> [ 1095.649699]  try_to_free_pages+0x131/0x3f0
> [ 1095.649996]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.650296]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.650743]  alloc_pages_current+0x97/0x1b0
> [ 1095.650892]  new_slab+0x4cb/0x6b0
> [ 1095.651193]  ___slab_alloc+0x3a3/0x620
> [ 1095.652087]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.652400]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.653457]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.653901]  __slab_alloc+0x46/0x7d
> [ 1095.654050]  __kmalloc+0x301/0x3b0
> [ 1095.654948]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.655396]  ? kfree+0x1fa/0x330
> [ 1095.656589]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.656919]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.656941]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.656961]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.658006]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.659053]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.660098]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.660846]  process_one_work+0x22b/0x760
> [ 1095.660994]  ? process_one_work+0x194/0x760
> [ 1095.661292]  worker_thread+0x137/0x4b0
> [ 1095.661439]  kthread+0x10f/0x150
> [ 1095.661741]  ? process_one_work+0x760/0x760
> [ 1095.661743]  ? kthread_create_on_node+0x70/0x70
> [ 1095.661891]  ret_from_fork+0x31/0x40
> 
> [ 1095.635223] MemAlloc: kworker/2:3(535) flags=0x4228860 switches=49285 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.635224] kworker/2:3     R  running task    12272   535      2 0x00000000
> [ 1095.635246] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.635247] Call Trace:
> [ 1095.635250]  __schedule+0x336/0xe00
> [ 1095.635253]  preempt_schedule_common+0x1f/0x31
> [ 1095.635254]  _cond_resched+0x1c/0x30
> [ 1095.635256]  shrink_slab+0x339/0x6d0
> [ 1095.635263]  shrink_node+0x2fa/0x310
> [ 1095.635267]  do_try_to_free_pages+0xe1/0x300
> [ 1095.635271]  try_to_free_pages+0x131/0x3f0
> [ 1095.635293]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.635301]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.635305]  alloc_pages_current+0x97/0x1b0
> [ 1095.635308]  new_slab+0x4cb/0x6b0
> [ 1095.635312]  ___slab_alloc+0x3a3/0x620
> [ 1095.635351]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.635355]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.635375]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.635377]  __slab_alloc+0x46/0x7d
> [ 1095.635380]  __kmalloc+0x301/0x3b0
> [ 1095.635399]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.635400]  ? kfree+0x1fa/0x330
> [ 1095.635421]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.635460]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.635494]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.635529]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.635549]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.635566]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.635583]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.635586]  process_one_work+0x22b/0x760
> [ 1095.635587]  ? process_one_work+0x194/0x760
> [ 1095.635591]  worker_thread+0x137/0x4b0
> [ 1095.635594]  kthread+0x10f/0x150
> [ 1095.635595]  ? process_one_work+0x760/0x760
> [ 1095.635596]  ? kthread_create_on_node+0x70/0x70
> [ 1095.635598]  ret_from_fork+0x31/0x40
> 
> [ 1095.663708] MemAlloc: kworker/2:5(7416) flags=0x4228060 switches=22830 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652069 uninterruptible
> [ 1095.663708] kworker/2:5     D12272  7416      2 0x00000080
> [ 1095.663738] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.663740] Call Trace:
> [ 1095.663743]  __schedule+0x336/0xe00
> [ 1095.663746]  schedule+0x3d/0x90
> [ 1095.663747]  schedule_timeout+0x20d/0x510
> [ 1095.663749]  ? lock_timer_base+0xa0/0xa0
> [ 1095.663753]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.663755]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.663761]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.663765]  alloc_pages_current+0x97/0x1b0
> [ 1095.663768]  new_slab+0x4cb/0x6b0
> [ 1095.663771]  ___slab_alloc+0x3a3/0x620
> [ 1095.663793]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663795]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.663815]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663817]  __slab_alloc+0x46/0x7d
> [ 1095.663820]  __kmalloc+0x301/0x3b0
> [ 1095.663859]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663861]  ? kfree+0x1fa/0x330
> [ 1095.663883]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.663905]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.663924]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.663942]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.663961]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.663996]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.664015]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.664017]  process_one_work+0x22b/0x760
> [ 1095.664018]  ? process_one_work+0x194/0x760
> [ 1095.664022]  worker_thread+0x137/0x4b0
> [ 1095.664025]  kthread+0x10f/0x150
> [ 1095.664026]  ? process_one_work+0x760/0x760
> [ 1095.664027]  ? kthread_create_on_node+0x70/0x70
> [ 1095.664030]  ret_from_fork+0x31/0x40
> 
> [ 1095.663334] MemAlloc: kworker/2:4(7415) flags=0x4228060 switches=23221 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652078 uninterruptible
> [ 1095.663335] kworker/2:4     D11496  7415      2 0x00000080
> [ 1095.663361] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.663362] Call Trace:
> [ 1095.663366]  __schedule+0x336/0xe00
> [ 1095.663369]  schedule+0x3d/0x90
> [ 1095.663371]  schedule_timeout+0x20d/0x510
> [ 1095.663407]  ? lock_timer_base+0xa0/0xa0
> [ 1095.663412]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.663414]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.663421]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.663425]  alloc_pages_current+0x97/0x1b0
> [ 1095.663428]  new_slab+0x4cb/0x6b0
> [ 1095.663431]  ___slab_alloc+0x3a3/0x620
> [ 1095.663457]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663459]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.663481]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663483]  __slab_alloc+0x46/0x7d
> [ 1095.663485]  __kmalloc+0x301/0x3b0
> [ 1095.663505]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663507]  ? kfree+0x1fa/0x330
> [ 1095.663548]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.663572]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.663591]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.663611]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.663630]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.663647]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.663664]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.663667]  process_one_work+0x22b/0x760
> [ 1095.663668]  ? process_one_work+0x194/0x760
> [ 1095.663672]  worker_thread+0x137/0x4b0
> [ 1095.663674]  kthread+0x10f/0x150
> [ 1095.663676]  ? process_one_work+0x760/0x760
> [ 1095.663696]  ? kthread_create_on_node+0x70/0x70
> [ 1095.663699]  ? do_syscall_64+0x195/0x200
> [ 1095.663701]  ret_from_fork+0x31/0x40
> 
> [ 1095.632604] MemAlloc: kworker/2:1(65) flags=0x4228060 switches=22879 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652043
> [ 1095.632604] kworker/2:1     R  running task    12184    65      2 0x00000000
> [ 1095.632641] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.632642] Call Trace:
> [ 1095.632646]  __schedule+0x336/0xe00
> [ 1095.632649]  schedule+0x3d/0x90
> [ 1095.632651]  schedule_timeout+0x20d/0x510
> [ 1095.632654]  ? lock_timer_base+0xa0/0xa0
> [ 1095.632658]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.632660]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.632668]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.632672]  alloc_pages_current+0x97/0x1b0
> [ 1095.632675]  new_slab+0x4cb/0x6b0
> [ 1095.632678]  ___slab_alloc+0x3a3/0x620
> [ 1095.632700]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632702]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.632704]  ? free_debug_processing+0x27d/0x2af
> [ 1095.632723]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632725]  __slab_alloc+0x46/0x7d
> [ 1095.632728]  __kmalloc+0x301/0x3b0
> [ 1095.632765]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632767]  ? kfree+0x1fa/0x330
> [ 1095.632804]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.632828]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.632847]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.632865]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.632884]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.632916]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.632935]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.632938]  process_one_work+0x22b/0x760
> [ 1095.632939]  ? process_one_work+0x194/0x760
> [ 1095.632944]  worker_thread+0x137/0x4b0
> [ 1095.632947]  kthread+0x10f/0x150
> [ 1095.632949]  ? process_one_work+0x760/0x760
> [ 1095.632960]  ? kthread_create_on_node+0x70/0x70
> [ 1095.632963]  ret_from_fork+0x31/0x40
> 
> [ 1095.645241] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=30893 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.645242] kworker/1:3     R  running task    12184  5357      2 0x00000080
> [ 1095.645261] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.645262] Call Trace:
> [ 1095.645263]  ? shrink_node+0x2fa/0x310
> [ 1095.645267]  ? do_try_to_free_pages+0xe1/0x300
> [ 1095.645270]  ? try_to_free_pages+0x131/0x3f0
> [ 1095.645274]  ? __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.645280]  ? __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.645283]  ? alloc_pages_current+0x97/0x1b0
> [ 1095.645286]  ? new_slab+0x4cb/0x6b0
> [ 1095.645289]  ? ___slab_alloc+0x3a3/0x620
> [ 1095.645310]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645312]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.645331]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645333]  ? __slab_alloc+0x46/0x7d
> [ 1095.645335]  ? __kmalloc+0x301/0x3b0
> [ 1095.645355]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645356]  ? kfree+0x1fa/0x330
> [ 1095.645377]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.645398]  ? __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.645417]  ? xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.645435]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.645453]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.645470]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.645486]  ? xfs_end_io+0x81/0x110 [xfs]
> [ 1095.645489]  ? process_one_work+0x22b/0x760
> [ 1095.645490]  ? process_one_work+0x194/0x760
> [ 1095.645493]  ? worker_thread+0x137/0x4b0
> [ 1095.645496]  ? kthread+0x10f/0x150
> [ 1095.645497]  ? process_one_work+0x760/0x760
> [ 1095.645498]  ? kthread_create_on_node+0x70/0x70
> [ 1095.645501]  ? ret_from_fork+0x31/0x40
> 
> [ 1095.633213] MemAlloc: kworker/1:2(193) flags=0x4228860 switches=28494 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.633213] kworker/1:2     R  running task    12760   193      2 0x00000000
> [ 1095.633236] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.633237] Call Trace:
> [ 1095.633240]  __schedule+0x336/0xe00
> [ 1095.633260]  preempt_schedule_common+0x1f/0x31
> [ 1095.633261]  _cond_resched+0x1c/0x30
> [ 1095.633263]  shrink_slab+0x339/0x6d0
> [ 1095.633270]  shrink_node+0x2fa/0x310
> [ 1095.633275]  do_try_to_free_pages+0xe1/0x300
> [ 1095.633278]  try_to_free_pages+0x131/0x3f0
> [ 1095.633283]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.633292]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.633295]  alloc_pages_current+0x97/0x1b0
> [ 1095.633298]  new_slab+0x4cb/0x6b0
> [ 1095.633318]  ___slab_alloc+0x3a3/0x620
> [ 1095.633349]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633353]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.633373]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633375]  __slab_alloc+0x46/0x7d
> [ 1095.633378]  __kmalloc+0x301/0x3b0
> [ 1095.633397]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633398]  ? kfree+0x1fa/0x330
> [ 1095.633436]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.633464]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.633500]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.633520]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.633538]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.633555]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.633572]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.633575]  process_one_work+0x22b/0x760
> [ 1095.633576]  ? process_one_work+0x194/0x760
> [ 1095.633597]  worker_thread+0x137/0x4b0
> [ 1095.633602]  kthread+0x10f/0x150
> [ 1095.633603]  ? process_one_work+0x760/0x760
> [ 1095.633604]  ? kthread_create_on_node+0x70/0x70
> [ 1095.633607]  ret_from_fork+0x31/0x40
> 
> [ 1095.632186] MemAlloc: kworker/1:1(52) flags=0x4228860 switches=28036 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652083
> [ 1095.632186] kworker/1:1     R  running task    12760    52      2 0x00000000
> [ 1095.632209] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.632210] Call Trace:
> [ 1095.632213]  ? _raw_spin_lock+0x69/0x80
> [ 1095.632214]  ? __list_lru_count_one.isra.2+0x22/0x80
> [ 1095.632216]  ? __list_lru_count_one.isra.2+0x22/0x80
> [ 1095.632217]  ? list_lru_count_one+0x23/0x30
> [ 1095.632219]  ? super_cache_count+0x6c/0xe0
> [ 1095.632221]  ? shrink_slab+0x15c/0x6d0
> [ 1095.632241]  ? mem_cgroup_iter+0x14d/0x720
> [ 1095.632244]  ? css_next_child+0x17/0xd0
> [ 1095.632247]  ? shrink_node+0x2fa/0x310
> [ 1095.632251]  ? do_try_to_free_pages+0xe1/0x300
> [ 1095.632254]  ? try_to_free_pages+0x131/0x3f0
> [ 1095.632258]  ? __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.632265]  ? __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.632268]  ? alloc_pages_current+0x97/0x1b0
> [ 1095.632271]  ? new_slab+0x4cb/0x6b0
> [ 1095.632274]  ? ___slab_alloc+0x3a3/0x620
> [ 1095.632313]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632317]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.632337]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632339]  ? __slab_alloc+0x46/0x7d
> [ 1095.632341]  ? __kmalloc+0x301/0x3b0
> [ 1095.632371]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632373]  ? kfree+0x1fa/0x330
> [ 1095.632412]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.632443]  ? __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.632480]  ? xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.632500]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.632519]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.632536]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.632553]  ? xfs_end_io+0x81/0x110 [xfs]
> [ 1095.632556]  ? process_one_work+0x22b/0x760
> [ 1095.632558]  ? process_one_work+0x194/0x760
> [ 1095.632588]  ? worker_thread+0x137/0x4b0
> [ 1095.632591]  ? kthread+0x10f/0x150
> [ 1095.632592]  ? process_one_work+0x760/0x760
> [ 1095.632593]  ? kthread_create_on_node+0x70/0x70
> [ 1095.632596]  ? ret_from_fork+0x31/0x40
> 
> [ 1095.645506] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=27329 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.645506] kworker/1:4     R  running task    12272  5358      2 0x00000080
> [ 1095.645525] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.645526] Call Trace:
> [ 1095.645528]  __schedule+0x336/0xe00
> [ 1095.645531]  schedule+0x3d/0x90
> [ 1095.645532]  schedule_timeout+0x20d/0x510
> [ 1095.645534]  ? lock_timer_base+0xa0/0xa0
> [ 1095.645538]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.645540]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.645546]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.645550]  alloc_pages_current+0x97/0x1b0
> [ 1095.645552]  new_slab+0x4cb/0x6b0
> [ 1095.645555]  ___slab_alloc+0x3a3/0x620
> [ 1095.645576]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645578]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.645597]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645599]  __slab_alloc+0x46/0x7d
> [ 1095.645601]  __kmalloc+0x301/0x3b0
> [ 1095.645620]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645621]  ? kfree+0x1fa/0x330
> [ 1095.645642]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.645663]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.645681]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.645699]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.646636]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.647249]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.647269]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.647273]  process_one_work+0x22b/0x760
> [ 1095.647274]  ? process_one_work+0x194/0x760
> [ 1095.647277]  worker_thread+0x137/0x4b0
> [ 1095.647280]  kthread+0x10f/0x150
> [ 1095.647281]  ? process_one_work+0x760/0x760
> [ 1095.647283]  ? kthread_create_on_node+0x70/0x70
> [ 1095.647285]  ret_from_fork+0x31/0x40
> 
> [ 1095.638807] MemAlloc: kworker/0:3(2486) flags=0x4228860 switches=76240 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652064
> [ 1095.638807] kworker/0:3     R  running task    11608  2486      2 0x00000080
> [ 1095.638829] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.638830] Call Trace:
> [ 1095.638833]  __schedule+0x336/0xe00
> [ 1095.638837]  preempt_schedule_common+0x1f/0x31
> [ 1095.638838]  _cond_resched+0x1c/0x30
> [ 1095.638839]  shrink_slab+0x339/0x6d0
> [ 1095.638846]  shrink_node+0x2fa/0x310
> [ 1095.638850]  do_try_to_free_pages+0xe1/0x300
> [ 1095.638854]  try_to_free_pages+0x131/0x3f0
> [ 1095.638859]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.638866]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.638871]  alloc_pages_current+0x97/0x1b0
> [ 1095.638874]  new_slab+0x4cb/0x6b0
> [ 1095.638878]  ___slab_alloc+0x3a3/0x620
> [ 1095.638900]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.638903]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.638923]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.638925]  __slab_alloc+0x46/0x7d
> [ 1095.638928]  __kmalloc+0x301/0x3b0
> [ 1095.638947]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.638948]  ? kfree+0x1fa/0x330
> [ 1095.638970]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.638992]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.639011]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.639030]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.639066]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.639083]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.639100]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.639103]  process_one_work+0x22b/0x760
> [ 1095.639104]  ? process_one_work+0x194/0x760
> [ 1095.639108]  worker_thread+0x137/0x4b0
> [ 1095.639111]  kthread+0x10f/0x150
> [ 1095.639112]  ? process_one_work+0x760/0x760
> [ 1095.639113]  ? kthread_create_on_node+0x70/0x70
> [ 1095.639116]  ? do_syscall_64+0x6c/0x200
> [ 1095.639118]  ret_from_fork+0x31/0x40

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Brian Foster]

On Tue, Mar 07, 2017 at 01:08:24AM +0900, Tetsuo Handa wrote:
> Brian Foster wrote:
> > As noted in my previous reply, I'm not sure there's enough here to point
> > at allocation failure as the root cause. E.g., kswapd is stuck here:
> > 
> > [ 1095.632985] kswapd0         D10776    69      2 0x00000000
> > [ 1095.632988] Call Trace:
> > [ 1095.632991]  __schedule+0x336/0xe00
> > [ 1095.632994]  schedule+0x3d/0x90
> > [ 1095.632996]  io_schedule+0x16/0x40
> > [ 1095.633017]  __xfs_iflock+0x129/0x140 [xfs]
> > [ 1095.633021]  ? autoremove_wake_function+0x60/0x60
> > [ 1095.633051]  xfs_reclaim_inode+0x162/0x440 [xfs]
> > [ 1095.633072]  xfs_reclaim_inodes_ag+0x2cf/0x4f0 [xfs]
> > [ 1095.633106]  ? xfs_reclaim_inodes_ag+0xf2/0x4f0 [xfs]
> > [ 1095.633114]  ? trace_hardirqs_on+0xd/0x10
> > [ 1095.633116]  ? try_to_wake_up+0x59/0x7a0
> > [ 1095.633120]  ? wake_up_process+0x15/0x20
> > [ 1095.633156]  xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> > [ 1095.633178]  xfs_fs_free_cached_objects+0x19/0x20 [xfs]
> > [ 1095.633180]  super_cache_scan+0x181/0x190
> > [ 1095.633183]  shrink_slab+0x29f/0x6d0
> > [ 1095.633189]  shrink_node+0x2fa/0x310
> > [ 1095.633193]  kswapd+0x362/0x9b0
> > [ 1095.633200]  kthread+0x10f/0x150
> > [ 1095.633201]  ? mem_cgroup_shrink_node+0x3b0/0x3b0
> > [ 1095.633202]  ? kthread_create_on_node+0x70/0x70
> > [ 1095.633205]  ret_from_fork+0x31/0x40
> > 
> > ... which is waiting on an inode flush lock. It can't get the lock
> > (presumably) because xfsaild has it:
> > 
> > [ 1041.772095] xfsaild/sda1    D13216   457      2 0x00000000
> > [ 1041.773726] Call Trace:
> > [ 1041.774734]  __schedule+0x336/0xe00
> > [ 1041.775956]  schedule+0x3d/0x90
> > [ 1041.777105]  schedule_timeout+0x26a/0x510
> > [ 1041.778426]  ? wait_for_completion+0x4c/0x190
> > [ 1041.779824]  wait_for_completion+0x12c/0x190
> > [ 1041.781273]  ? wake_up_q+0x80/0x80
> > [ 1041.782597]  ? _xfs_buf_read+0x44/0x90 [xfs]
> > [ 1041.784086]  xfs_buf_submit_wait+0xe9/0x5c0 [xfs]
> > [ 1041.785659]  _xfs_buf_read+0x44/0x90 [xfs]
> > [ 1041.787067]  xfs_buf_read_map+0xfa/0x400 [xfs]
> > [ 1041.788501]  ? xfs_trans_read_buf_map+0x186/0x830 [xfs]
> > [ 1041.790103]  xfs_trans_read_buf_map+0x186/0x830 [xfs]
> > [ 1041.791672]  xfs_imap_to_bp+0x71/0x110 [xfs]
> > [ 1041.793090]  xfs_iflush+0x122/0x3b0 [xfs]
> > [ 1041.794444]  xfs_inode_item_push+0x108/0x1c0 [xfs]
> > [ 1041.795956]  xfsaild_push+0x1d8/0xb70 [xfs]
> > [ 1041.797344]  xfsaild+0x150/0x270 [xfs]
> > [ 1041.798623]  kthread+0x10f/0x150
> > [ 1041.799819]  ? xfsaild_push+0xb70/0xb70 [xfs]
> > [ 1041.801217]  ? kthread_create_on_node+0x70/0x70
> > [ 1041.802652]  ret_from_fork+0x31/0x40
> > 
> > xfsaild is flushing an inode, but is waiting on a read of the underlying
> > inode cluster buffer such that it can flush out the in-core inode data
> > structure. I cannot tell if the read had actually completed and is
> > blocked somewhere else before running the completion. As Dave notes
> > earlier, buffer I/O completion relies on the xfs-buf wq. What is evident
> > from the logs is that xfs-buf has a rescuer thread that is sitting idle:
> > 
> > [ 1041.555227] xfs-buf/sda1    S14904   450      2 0x00000000
> > [ 1041.556813] Call Trace:
> > [ 1041.557796]  __schedule+0x336/0xe00
> > [ 1041.558983]  schedule+0x3d/0x90
> > [ 1041.560085]  rescuer_thread+0x322/0x3d0
> > [ 1041.561333]  kthread+0x10f/0x150
> > [ 1041.562464]  ? worker_thread+0x4b0/0x4b0
> > [ 1041.563732]  ? kthread_create_on_node+0x70/0x70
> > [ 1041.565123]  ret_from_fork+0x31/0x40
> > 
> > So AFAICT if the buffer I/O completion would run, it would allow xfsaild
> > to progress, which would eventually flush the underlying buffer, write
> > it, release the flush lock and allow kswapd to continue. The question is
> > has the actually I/O completed? If so, is the xfs-buf workqueue stuck
> > (waiting on an allocation perhaps)? And if that is the case, why is the
> > xfs-buf rescuer thread not doing anything?
> 
> [ 1116.803780] workqueue xfs-data/sda1: flags=0xc
> [ 1116.805324]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=27/256 MAYDAY
> [ 1116.807272]     in-flight: 5356:xfs_end_io [xfs], 451(RESCUER):xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs] xfs_end_io [xfs], 10498:xfs_end_io [xfs], 6386:xfs_end_io [xfs]
> [ 1116.812145]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
>  [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
> [ 1116.820988]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=21/256 MAYDAY
> [ 1116.823105]     in-flight: 535:xfs_end_io [xfs], 7416:xfs_end_io [xfs], 7415:xfs_end_io [xfs], 65:xfs_end_io [xfs]
> [ 1116.826062]     pending: xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io\
>  [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs], xfs_end_io [xfs]
> [ 1116.834549]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=4/256 MAYDAY
> [ 1116.837139]     in-flight: 5357:xfs_end_io [xfs], 193:xfs_end_io [xfs], 52:xfs_end_io [xfs], 5358:xfs_end_io [xfs]
> [ 1116.840182]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
> [ 1116.842297]     in-flight: 2486:xfs_end_io [xfs]
> [ 1116.844230] workqueue xfs-reclaim/sda1: flags=0xc
> [ 1116.846168]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
> [ 1116.848323]     pending: xfs_reclaim_worker [xfs]
> [ 1116.850280] workqueue xfs-eofblocks/sda1: flags=0xc
> [ 1116.852358]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
> [ 1116.854601]     in-flight: 456(RESCUER):xfs_eofblocks_worker [xfs]
> 
> As listed below, all in-flight workqueues for processing xfs_end_io (PID = 5356,
> 451(RESCUER), 10498, 6386, 535, 7416, 7415, 65, 5357, 193, 52, 5358, 2486) are
> stuck at memory allocation, and thus cannot call complete() when xfsaild/sda1
> (PID = 457) is waiting at wait_for_completion(&bp->b_iowait) in xfs_buf_submit_wait().
> 

These are all in xfs_end_io(), which means they are either the xfs-data
or xfs-conv workqueues. ->b_iowait is ultimately triggered via
xfs_buf_ioend() by the xfs-buf workqueue (or xfs-log for log buffers).
None of the following use xfs-buf.

Brian

> [ 1095.644936] MemAlloc: kworker/3:3(5356) flags=0x4228860 switches=29192 seq=6 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652086
> [ 1095.644937] kworker/3:3     R  running task    12760  5356      2 0x00000080
> [ 1095.644959] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.644960] Call Trace:
> [ 1095.644962]  __schedule+0x336/0xe00
> [ 1095.644965]  preempt_schedule_common+0x1f/0x31
> [ 1095.644966]  _cond_resched+0x1c/0x30
> [ 1095.644968]  shrink_slab+0x339/0x6d0
> [ 1095.644973]  shrink_node+0x2fa/0x310
> [ 1095.644977]  do_try_to_free_pages+0xe1/0x300
> [ 1095.644979]  try_to_free_pages+0x131/0x3f0
> [ 1095.644984]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.644985]  ? cpuacct_charge+0xf3/0x1e0
> [ 1095.644986]  ? cpuacct_charge+0x38/0x1e0
> [ 1095.644992]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.644995]  alloc_pages_current+0x97/0x1b0
> [ 1095.644998]  new_slab+0x4cb/0x6b0
> [ 1095.645001]  ___slab_alloc+0x3a3/0x620
> [ 1095.645023]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645026]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.645027]  ? trace_hardirqs_on+0xd/0x10
> [ 1095.645058]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645066]  __slab_alloc+0x46/0x7d
> [ 1095.645069]  __kmalloc+0x301/0x3b0
> [ 1095.645088]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645089]  ? kfree+0x1fa/0x330
> [ 1095.645110]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.645132]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.645151]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.645169]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.645188]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.645204]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.645221]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.645223]  process_one_work+0x22b/0x760
> [ 1095.645224]  ? process_one_work+0x194/0x760
> [ 1095.645228]  worker_thread+0x137/0x4b0
> [ 1095.645231]  kthread+0x10f/0x150
> [ 1095.645232]  ? process_one_work+0x760/0x760
> [ 1095.645233]  ? kthread_create_on_node+0x70/0x70
> [ 1095.645236]  ret_from_fork+0x31/0x40
> 
> [ 1095.633625] MemAlloc: xfs-data/sda1(451) flags=0x4228060 switches=45509 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652073
> [ 1095.633626] xfs-data/sda1   R  running task    12696   451      2 0x00000000
> [ 1095.633663] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.633665] Call Trace:
> [ 1095.633668]  __schedule+0x336/0xe00
> [ 1095.633671]  schedule+0x3d/0x90
> [ 1095.633672]  schedule_timeout+0x20d/0x510
> [ 1095.633675]  ? lock_timer_base+0xa0/0xa0
> [ 1095.633678]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.633680]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.633687]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.633699]  alloc_pages_current+0x97/0x1b0
> [ 1095.633702]  new_slab+0x4cb/0x6b0
> [ 1095.633706]  ___slab_alloc+0x3a3/0x620
> [ 1095.633728]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633730]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.633732]  ? cpuacct_charge+0x38/0x1e0
> [ 1095.633767]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633770]  __slab_alloc+0x46/0x7d
> [ 1095.633773]  __kmalloc+0x301/0x3b0
> [ 1095.633802]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633804]  ? kfree+0x1fa/0x330
> [ 1095.633842]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.633864]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.633883]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.633901]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.633936]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.633954]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.633971]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.633973]  process_one_work+0x22b/0x760
> [ 1095.633975]  ? process_one_work+0x194/0x760
> [ 1095.633997]  rescuer_thread+0x1f2/0x3d0
> [ 1095.634002]  kthread+0x10f/0x150
> [ 1095.634003]  ? worker_thread+0x4b0/0x4b0
> [ 1095.634004]  ? kthread_create_on_node+0x70/0x70
> [ 1095.634007]  ret_from_fork+0x31/0x40
> 
> [ 1095.806606] MemAlloc: kworker/3:0(10498) flags=0x4228060 switches=16222 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
> [ 1095.806607] kworker/3:0     R  running task    11352 10498      2 0x00000080
> [ 1095.806631] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.806633] Call Trace:
> [ 1095.806636]  __schedule+0x336/0xe00
> [ 1095.806640]  schedule+0x3d/0x90
> [ 1095.806641]  schedule_timeout+0x20d/0x510
> [ 1095.806644]  ? lock_timer_base+0xa0/0xa0
> [ 1095.806649]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.806651]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.806652]  ? cpuacct_charge+0xf3/0x1e0
> [ 1095.806654]  ? cpuacct_charge+0x38/0x1e0
> [ 1095.806663]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.806667]  alloc_pages_current+0x97/0x1b0
> [ 1095.806671]  new_slab+0x4cb/0x6b0
> [ 1095.806675]  ___slab_alloc+0x3a3/0x620
> [ 1095.806698]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.806701]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.806721]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.806724]  __slab_alloc+0x46/0x7d
> [ 1095.806727]  __kmalloc+0x301/0x3b0
> [ 1095.806747]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.806749]  ? kfree+0x1fa/0x330
> [ 1095.806771]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.806794]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.806813]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.806832]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.806852]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.806868]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.806886]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.806889]  process_one_work+0x22b/0x760
> [ 1095.806890]  ? process_one_work+0x194/0x760
> [ 1095.806896]  worker_thread+0x137/0x4b0
> [ 1095.806899]  kthread+0x10f/0x150
> [ 1095.806901]  ? process_one_work+0x760/0x760
> [ 1095.806902]  ? kthread_create_on_node+0x70/0x70
> [ 1095.806904]  ? do_syscall_64+0x195/0x200
> [ 1095.806906]  ret_from_fork+0x31/0x40
> 
> [ 1095.647293] MemAlloc: kworker/3:5(6386) flags=0x4228860 switches=43427 seq=15 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652087
> [ 1095.647294] kworker/3:5     R  running task    11048  6386      2 0x00000080
> [ 1095.647754] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.648203] Call Trace:
> [ 1095.648207]  __schedule+0x336/0xe00
> [ 1095.648504]  preempt_schedule_common+0x1f/0x31
> [ 1095.648806]  _cond_resched+0x1c/0x30
> [ 1095.648952]  shrink_slab+0x339/0x6d0
> [ 1095.649252]  shrink_node+0x2fa/0x310
> [ 1095.649694]  do_try_to_free_pages+0xe1/0x300
> [ 1095.649699]  try_to_free_pages+0x131/0x3f0
> [ 1095.649996]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.650296]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.650743]  alloc_pages_current+0x97/0x1b0
> [ 1095.650892]  new_slab+0x4cb/0x6b0
> [ 1095.651193]  ___slab_alloc+0x3a3/0x620
> [ 1095.652087]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.652400]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.653457]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.653901]  __slab_alloc+0x46/0x7d
> [ 1095.654050]  __kmalloc+0x301/0x3b0
> [ 1095.654948]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.655396]  ? kfree+0x1fa/0x330
> [ 1095.656589]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.656919]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.656941]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.656961]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.658006]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.659053]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.660098]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.660846]  process_one_work+0x22b/0x760
> [ 1095.660994]  ? process_one_work+0x194/0x760
> [ 1095.661292]  worker_thread+0x137/0x4b0
> [ 1095.661439]  kthread+0x10f/0x150
> [ 1095.661741]  ? process_one_work+0x760/0x760
> [ 1095.661743]  ? kthread_create_on_node+0x70/0x70
> [ 1095.661891]  ret_from_fork+0x31/0x40
> 
> [ 1095.635223] MemAlloc: kworker/2:3(535) flags=0x4228860 switches=49285 seq=57 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.635224] kworker/2:3     R  running task    12272   535      2 0x00000000
> [ 1095.635246] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.635247] Call Trace:
> [ 1095.635250]  __schedule+0x336/0xe00
> [ 1095.635253]  preempt_schedule_common+0x1f/0x31
> [ 1095.635254]  _cond_resched+0x1c/0x30
> [ 1095.635256]  shrink_slab+0x339/0x6d0
> [ 1095.635263]  shrink_node+0x2fa/0x310
> [ 1095.635267]  do_try_to_free_pages+0xe1/0x300
> [ 1095.635271]  try_to_free_pages+0x131/0x3f0
> [ 1095.635293]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.635301]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.635305]  alloc_pages_current+0x97/0x1b0
> [ 1095.635308]  new_slab+0x4cb/0x6b0
> [ 1095.635312]  ___slab_alloc+0x3a3/0x620
> [ 1095.635351]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.635355]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.635375]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.635377]  __slab_alloc+0x46/0x7d
> [ 1095.635380]  __kmalloc+0x301/0x3b0
> [ 1095.635399]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.635400]  ? kfree+0x1fa/0x330
> [ 1095.635421]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.635460]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.635494]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.635529]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.635549]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.635566]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.635583]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.635586]  process_one_work+0x22b/0x760
> [ 1095.635587]  ? process_one_work+0x194/0x760
> [ 1095.635591]  worker_thread+0x137/0x4b0
> [ 1095.635594]  kthread+0x10f/0x150
> [ 1095.635595]  ? process_one_work+0x760/0x760
> [ 1095.635596]  ? kthread_create_on_node+0x70/0x70
> [ 1095.635598]  ret_from_fork+0x31/0x40
> 
> [ 1095.663708] MemAlloc: kworker/2:5(7416) flags=0x4228060 switches=22830 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652069 uninterruptible
> [ 1095.663708] kworker/2:5     D12272  7416      2 0x00000080
> [ 1095.663738] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.663740] Call Trace:
> [ 1095.663743]  __schedule+0x336/0xe00
> [ 1095.663746]  schedule+0x3d/0x90
> [ 1095.663747]  schedule_timeout+0x20d/0x510
> [ 1095.663749]  ? lock_timer_base+0xa0/0xa0
> [ 1095.663753]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.663755]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.663761]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.663765]  alloc_pages_current+0x97/0x1b0
> [ 1095.663768]  new_slab+0x4cb/0x6b0
> [ 1095.663771]  ___slab_alloc+0x3a3/0x620
> [ 1095.663793]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663795]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.663815]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663817]  __slab_alloc+0x46/0x7d
> [ 1095.663820]  __kmalloc+0x301/0x3b0
> [ 1095.663859]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663861]  ? kfree+0x1fa/0x330
> [ 1095.663883]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.663905]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.663924]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.663942]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.663961]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.663996]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.664015]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.664017]  process_one_work+0x22b/0x760
> [ 1095.664018]  ? process_one_work+0x194/0x760
> [ 1095.664022]  worker_thread+0x137/0x4b0
> [ 1095.664025]  kthread+0x10f/0x150
> [ 1095.664026]  ? process_one_work+0x760/0x760
> [ 1095.664027]  ? kthread_create_on_node+0x70/0x70
> [ 1095.664030]  ret_from_fork+0x31/0x40
> 
> [ 1095.663334] MemAlloc: kworker/2:4(7415) flags=0x4228060 switches=23221 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652078 uninterruptible
> [ 1095.663335] kworker/2:4     D11496  7415      2 0x00000080
> [ 1095.663361] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.663362] Call Trace:
> [ 1095.663366]  __schedule+0x336/0xe00
> [ 1095.663369]  schedule+0x3d/0x90
> [ 1095.663371]  schedule_timeout+0x20d/0x510
> [ 1095.663407]  ? lock_timer_base+0xa0/0xa0
> [ 1095.663412]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.663414]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.663421]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.663425]  alloc_pages_current+0x97/0x1b0
> [ 1095.663428]  new_slab+0x4cb/0x6b0
> [ 1095.663431]  ___slab_alloc+0x3a3/0x620
> [ 1095.663457]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663459]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.663481]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663483]  __slab_alloc+0x46/0x7d
> [ 1095.663485]  __kmalloc+0x301/0x3b0
> [ 1095.663505]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.663507]  ? kfree+0x1fa/0x330
> [ 1095.663548]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.663572]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.663591]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.663611]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.663630]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.663647]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.663664]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.663667]  process_one_work+0x22b/0x760
> [ 1095.663668]  ? process_one_work+0x194/0x760
> [ 1095.663672]  worker_thread+0x137/0x4b0
> [ 1095.663674]  kthread+0x10f/0x150
> [ 1095.663676]  ? process_one_work+0x760/0x760
> [ 1095.663696]  ? kthread_create_on_node+0x70/0x70
> [ 1095.663699]  ? do_syscall_64+0x195/0x200
> [ 1095.663701]  ret_from_fork+0x31/0x40
> 
> [ 1095.632604] MemAlloc: kworker/2:1(65) flags=0x4228060 switches=22879 seq=2 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652043
> [ 1095.632604] kworker/2:1     R  running task    12184    65      2 0x00000000
> [ 1095.632641] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.632642] Call Trace:
> [ 1095.632646]  __schedule+0x336/0xe00
> [ 1095.632649]  schedule+0x3d/0x90
> [ 1095.632651]  schedule_timeout+0x20d/0x510
> [ 1095.632654]  ? lock_timer_base+0xa0/0xa0
> [ 1095.632658]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.632660]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.632668]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.632672]  alloc_pages_current+0x97/0x1b0
> [ 1095.632675]  new_slab+0x4cb/0x6b0
> [ 1095.632678]  ___slab_alloc+0x3a3/0x620
> [ 1095.632700]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632702]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.632704]  ? free_debug_processing+0x27d/0x2af
> [ 1095.632723]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632725]  __slab_alloc+0x46/0x7d
> [ 1095.632728]  __kmalloc+0x301/0x3b0
> [ 1095.632765]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632767]  ? kfree+0x1fa/0x330
> [ 1095.632804]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.632828]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.632847]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.632865]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.632884]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.632916]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.632935]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.632938]  process_one_work+0x22b/0x760
> [ 1095.632939]  ? process_one_work+0x194/0x760
> [ 1095.632944]  worker_thread+0x137/0x4b0
> [ 1095.632947]  kthread+0x10f/0x150
> [ 1095.632949]  ? process_one_work+0x760/0x760
> [ 1095.632960]  ? kthread_create_on_node+0x70/0x70
> [ 1095.632963]  ret_from_fork+0x31/0x40
> 
> [ 1095.645241] MemAlloc: kworker/1:3(5357) flags=0x4228860 switches=30893 seq=3 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.645242] kworker/1:3     R  running task    12184  5357      2 0x00000080
> [ 1095.645261] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.645262] Call Trace:
> [ 1095.645263]  ? shrink_node+0x2fa/0x310
> [ 1095.645267]  ? do_try_to_free_pages+0xe1/0x300
> [ 1095.645270]  ? try_to_free_pages+0x131/0x3f0
> [ 1095.645274]  ? __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.645280]  ? __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.645283]  ? alloc_pages_current+0x97/0x1b0
> [ 1095.645286]  ? new_slab+0x4cb/0x6b0
> [ 1095.645289]  ? ___slab_alloc+0x3a3/0x620
> [ 1095.645310]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645312]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.645331]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645333]  ? __slab_alloc+0x46/0x7d
> [ 1095.645335]  ? __kmalloc+0x301/0x3b0
> [ 1095.645355]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645356]  ? kfree+0x1fa/0x330
> [ 1095.645377]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.645398]  ? __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.645417]  ? xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.645435]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.645453]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.645470]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.645486]  ? xfs_end_io+0x81/0x110 [xfs]
> [ 1095.645489]  ? process_one_work+0x22b/0x760
> [ 1095.645490]  ? process_one_work+0x194/0x760
> [ 1095.645493]  ? worker_thread+0x137/0x4b0
> [ 1095.645496]  ? kthread+0x10f/0x150
> [ 1095.645497]  ? process_one_work+0x760/0x760
> [ 1095.645498]  ? kthread_create_on_node+0x70/0x70
> [ 1095.645501]  ? ret_from_fork+0x31/0x40
> 
> [ 1095.633213] MemAlloc: kworker/1:2(193) flags=0x4228860 switches=28494 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.633213] kworker/1:2     R  running task    12760   193      2 0x00000000
> [ 1095.633236] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.633237] Call Trace:
> [ 1095.633240]  __schedule+0x336/0xe00
> [ 1095.633260]  preempt_schedule_common+0x1f/0x31
> [ 1095.633261]  _cond_resched+0x1c/0x30
> [ 1095.633263]  shrink_slab+0x339/0x6d0
> [ 1095.633270]  shrink_node+0x2fa/0x310
> [ 1095.633275]  do_try_to_free_pages+0xe1/0x300
> [ 1095.633278]  try_to_free_pages+0x131/0x3f0
> [ 1095.633283]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.633292]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.633295]  alloc_pages_current+0x97/0x1b0
> [ 1095.633298]  new_slab+0x4cb/0x6b0
> [ 1095.633318]  ___slab_alloc+0x3a3/0x620
> [ 1095.633349]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633353]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.633373]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633375]  __slab_alloc+0x46/0x7d
> [ 1095.633378]  __kmalloc+0x301/0x3b0
> [ 1095.633397]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.633398]  ? kfree+0x1fa/0x330
> [ 1095.633436]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.633464]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.633500]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.633520]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.633538]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.633555]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.633572]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.633575]  process_one_work+0x22b/0x760
> [ 1095.633576]  ? process_one_work+0x194/0x760
> [ 1095.633597]  worker_thread+0x137/0x4b0
> [ 1095.633602]  kthread+0x10f/0x150
> [ 1095.633603]  ? process_one_work+0x760/0x760
> [ 1095.633604]  ? kthread_create_on_node+0x70/0x70
> [ 1095.633607]  ret_from_fork+0x31/0x40
> 
> [ 1095.632186] MemAlloc: kworker/1:1(52) flags=0x4228860 switches=28036 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652083
> [ 1095.632186] kworker/1:1     R  running task    12760    52      2 0x00000000
> [ 1095.632209] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.632210] Call Trace:
> [ 1095.632213]  ? _raw_spin_lock+0x69/0x80
> [ 1095.632214]  ? __list_lru_count_one.isra.2+0x22/0x80
> [ 1095.632216]  ? __list_lru_count_one.isra.2+0x22/0x80
> [ 1095.632217]  ? list_lru_count_one+0x23/0x30
> [ 1095.632219]  ? super_cache_count+0x6c/0xe0
> [ 1095.632221]  ? shrink_slab+0x15c/0x6d0
> [ 1095.632241]  ? mem_cgroup_iter+0x14d/0x720
> [ 1095.632244]  ? css_next_child+0x17/0xd0
> [ 1095.632247]  ? shrink_node+0x2fa/0x310
> [ 1095.632251]  ? do_try_to_free_pages+0xe1/0x300
> [ 1095.632254]  ? try_to_free_pages+0x131/0x3f0
> [ 1095.632258]  ? __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.632265]  ? __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.632268]  ? alloc_pages_current+0x97/0x1b0
> [ 1095.632271]  ? new_slab+0x4cb/0x6b0
> [ 1095.632274]  ? ___slab_alloc+0x3a3/0x620
> [ 1095.632313]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632317]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.632337]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632339]  ? __slab_alloc+0x46/0x7d
> [ 1095.632341]  ? __kmalloc+0x301/0x3b0
> [ 1095.632371]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.632373]  ? kfree+0x1fa/0x330
> [ 1095.632412]  ? xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.632443]  ? __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.632480]  ? xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.632500]  ? __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.632519]  ? xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.632536]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.632553]  ? xfs_end_io+0x81/0x110 [xfs]
> [ 1095.632556]  ? process_one_work+0x22b/0x760
> [ 1095.632558]  ? process_one_work+0x194/0x760
> [ 1095.632588]  ? worker_thread+0x137/0x4b0
> [ 1095.632591]  ? kthread+0x10f/0x150
> [ 1095.632592]  ? process_one_work+0x760/0x760
> [ 1095.632593]  ? kthread_create_on_node+0x70/0x70
> [ 1095.632596]  ? ret_from_fork+0x31/0x40
> 
> [ 1095.645506] MemAlloc: kworker/1:4(5358) flags=0x4228060 switches=27329 seq=1 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652084
> [ 1095.645506] kworker/1:4     R  running task    12272  5358      2 0x00000080
> [ 1095.645525] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.645526] Call Trace:
> [ 1095.645528]  __schedule+0x336/0xe00
> [ 1095.645531]  schedule+0x3d/0x90
> [ 1095.645532]  schedule_timeout+0x20d/0x510
> [ 1095.645534]  ? lock_timer_base+0xa0/0xa0
> [ 1095.645538]  schedule_timeout_uninterruptible+0x2a/0x30
> [ 1095.645540]  __alloc_pages_slowpath+0x2b5/0xd95
> [ 1095.645546]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.645550]  alloc_pages_current+0x97/0x1b0
> [ 1095.645552]  new_slab+0x4cb/0x6b0
> [ 1095.645555]  ___slab_alloc+0x3a3/0x620
> [ 1095.645576]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645578]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.645597]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645599]  __slab_alloc+0x46/0x7d
> [ 1095.645601]  __kmalloc+0x301/0x3b0
> [ 1095.645620]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.645621]  ? kfree+0x1fa/0x330
> [ 1095.645642]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.645663]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.645681]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.645699]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.646636]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.647249]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.647269]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.647273]  process_one_work+0x22b/0x760
> [ 1095.647274]  ? process_one_work+0x194/0x760
> [ 1095.647277]  worker_thread+0x137/0x4b0
> [ 1095.647280]  kthread+0x10f/0x150
> [ 1095.647281]  ? process_one_work+0x760/0x760
> [ 1095.647283]  ? kthread_create_on_node+0x70/0x70
> [ 1095.647285]  ret_from_fork+0x31/0x40
> 
> [ 1095.638807] MemAlloc: kworker/0:3(2486) flags=0x4228860 switches=76240 seq=10 gfp=0x1604240(GFP_NOFS|__GFP_NOWARN|__GFP_COMP|__GFP_NOTRACK) order=0 delay=652064
> [ 1095.638807] kworker/0:3     R  running task    11608  2486      2 0x00000080
> [ 1095.638829] Workqueue: xfs-data/sda1 xfs_end_io [xfs]
> [ 1095.638830] Call Trace:
> [ 1095.638833]  __schedule+0x336/0xe00
> [ 1095.638837]  preempt_schedule_common+0x1f/0x31
> [ 1095.638838]  _cond_resched+0x1c/0x30
> [ 1095.638839]  shrink_slab+0x339/0x6d0
> [ 1095.638846]  shrink_node+0x2fa/0x310
> [ 1095.638850]  do_try_to_free_pages+0xe1/0x300
> [ 1095.638854]  try_to_free_pages+0x131/0x3f0
> [ 1095.638859]  __alloc_pages_slowpath+0x3ec/0xd95
> [ 1095.638866]  __alloc_pages_nodemask+0x3e4/0x460
> [ 1095.638871]  alloc_pages_current+0x97/0x1b0
> [ 1095.638874]  new_slab+0x4cb/0x6b0
> [ 1095.638878]  ___slab_alloc+0x3a3/0x620
> [ 1095.638900]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.638903]  ? ___slab_alloc+0x5c6/0x620
> [ 1095.638923]  ? kmem_alloc+0x96/0x120 [xfs]
> [ 1095.638925]  __slab_alloc+0x46/0x7d
> [ 1095.638928]  __kmalloc+0x301/0x3b0
> [ 1095.638947]  kmem_alloc+0x96/0x120 [xfs]
> [ 1095.638948]  ? kfree+0x1fa/0x330
> [ 1095.638970]  xfs_log_commit_cil+0x489/0x710 [xfs]
> [ 1095.638992]  __xfs_trans_commit+0x83/0x260 [xfs]
> [ 1095.639011]  xfs_trans_commit+0x10/0x20 [xfs]
> [ 1095.639030]  __xfs_setfilesize+0xdb/0x240 [xfs]
> [ 1095.639066]  xfs_setfilesize_ioend+0x89/0xb0 [xfs]
> [ 1095.639083]  ? xfs_setfilesize_ioend+0x5/0xb0 [xfs]
> [ 1095.639100]  xfs_end_io+0x81/0x110 [xfs]
> [ 1095.639103]  process_one_work+0x22b/0x760
> [ 1095.639104]  ? process_one_work+0x194/0x760
> [ 1095.639108]  worker_thread+0x137/0x4b0
> [ 1095.639111]  kthread+0x10f/0x150
> [ 1095.639112]  ? process_one_work+0x760/0x760
> [ 1095.639113]  ? kthread_create_on_node+0x70/0x70
> [ 1095.639116]  ? do_syscall_64+0x6c/0x200
> [ 1095.639118]  ret_from_fork+0x31/0x40

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Michal Hocko]

[Let's add Tejun]

On Sat 04-03-17 10:25:12, Dave Chinner wrote:
> On Fri, Mar 03, 2017 at 02:39:51PM +0100, Michal Hocko wrote:
> > On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> > > Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> > > 
> > > While I was testing a patch which avoids infinite too_many_isolated() loop in
> > > shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> > > waiting for memory allocation. I guess that we overlooked a basic thing about
> > > WQ_MEM_RECLAIM.
> > > 
> > >   WQ_MEM_RECLAIM helps only when the cause of failing to complete
> > >   a work item is lack of "struct task_struct" to run that work item, for
> > >   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
> > >   will not be blocked waiting for memory allocation for "struct task_struct".
> > > 
> > >   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
> > >   item is blocked waiting for memory allocation (or is indirectly blocked
> > >   on a lock where the owner of that lock is blocked waiting for memory
> > >   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
> > >   if memory allocation (including indirect memory allocation via
> > >   locks/completions) is needed.
> > > 
> > > In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> > > "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> > > workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> > > 
> > > What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> > > I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> > > workqueues (which are RESCUER) got stuck waiting for memory allocation.
> > 
> > If those workers are really required for a further progress of the
> > memory reclaim then they shouldn't block on allocation at all and either
> > use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
> > that only very limited amount of memory is allocated from that context
> > and there will be at least the same amount of memory freed as a result
> > in a reasonable time.
> > 
> > This is something for xfs people to answer though. Please note that I
> > didn't really have time to look through the below traces so the above
> > note is rather generic. It would be really helpful if you could provide
> > a high level dependency chains to see why those rescuers are necessary
> > for the forward progress because it is really easy to get lost in so
> > many traces.
> 
> Data IO completion is required to make progress to free memory. IO
> completion is done via work queues, so they need rescuer threads to
> ensure work can be run.
> 
> IO completion can require transactions to run. Transactions require
> memory allocation. Freeing memory therefore requires IO completion
> to have access to memory reserves if it's occurring from rescuer
> threads to allow progress to be made.
> 
> That means metadata IO completion require rescuer threads, because
> data IO completion can be dependent on metadata buffers being
> available. e.g. reserving space in the log for the transaction can
> require waiting on metadata IO dispatch and completion. Hence the
> buffer IO completion workqueues need rescuer threads.
> 
> Transactions can also require log forces and flushes to occur, which
> means they require the log workqueues (both the CIL flush and IO
> completion workqueues) to make progress.  Log flushes also require
> both IO and memory allocation to make progress to complete. Again,
> this means the log workqueues need rescuer threads. It also needs
> the log workqueues to be high priority so that they can make
> progress before IO completion work that is dependent on
> transactions making progress are processed.
> 
> IOWs, pretty much all the XFS workqueues are involved in memory
> reclaim in one way or another.
> 
> The real problem here is that the XFS code has /no idea/ of what
> workqueue context it is operating in - the fact it is in a rescuer
> thread is completely hidden from the executing context. It seems to
> me that the workqueue infrastructure's responsibility to tell memory
> reclaim that the rescuer thread needs special access to the memory
> reserves to allow the work it is running to allow forwards progress
> to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
> something similar...

I am not sure an automatic access to memory reserves from the rescuer
context is safe. This sounds too easy to break (read consume all the
reserves) - note that we have almost 200 users of WQ_MEM_RECLAIM and
chances are some of them will not be careful with the memory
allocations. I agree it would be helpful to know that the current item
runs from the rescuer context, though. In such a case the implementation
can do what ever it takes to make a forward progress. If that is using
__GFP_MEMALLOC then be it but it would be at least explicit and well
thought through (I hope).

Tejun, would it be possible/reasonable to add current_is_wq_rescuer() API?
-- 
Michal Hocko
SUSE Labs

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Michal Hocko]

[Let's add Tejun]

On Sat 04-03-17 10:25:12, Dave Chinner wrote:
> On Fri, Mar 03, 2017 at 02:39:51PM +0100, Michal Hocko wrote:
> > On Fri 03-03-17 19:48:30, Tetsuo Handa wrote:
> > > Continued from http://lkml.kernel.org/r/201702261530.JDD56292.OFOLFHQtVMJSOF@I-love.SAKURA.ne.jp :
> > > 
> > > While I was testing a patch which avoids infinite too_many_isolated() loop in
> > > shrink_inactive_list(), I hit a lockup where WQ_MEM_RECLAIM threads got stuck
> > > waiting for memory allocation. I guess that we overlooked a basic thing about
> > > WQ_MEM_RECLAIM.
> > > 
> > >   WQ_MEM_RECLAIM helps only when the cause of failing to complete
> > >   a work item is lack of "struct task_struct" to run that work item, for
> > >   WQ_MEM_RECLAIM preallocates one "struct task_struct" so that the workqueue
> > >   will not be blocked waiting for memory allocation for "struct task_struct".
> > > 
> > >   WQ_MEM_RECLAIM does not help when "struct task_struct" running that work
> > >   item is blocked waiting for memory allocation (or is indirectly blocked
> > >   on a lock where the owner of that lock is blocked waiting for memory
> > >   allocation). That is, WQ_MEM_RECLAIM users must guarantee forward progress
> > >   if memory allocation (including indirect memory allocation via
> > >   locks/completions) is needed.
> > > 
> > > In XFS, "xfs_mru_cache", "xfs-buf/%s", "xfs-data/%s", "xfs-conv/%s", "xfs-cil/%s",
> > > "xfs-reclaim/%s", "xfs-log/%s", "xfs-eofblocks/%s", "xfsalloc" and "xfsdiscard"
> > > workqueues are used, and all but "xfsdiscard" are WQ_MEM_RECLAIM workqueues.
> > > 
> > > What I observed is at http://I-love.SAKURA.ne.jp/tmp/serial-20170226.txt.xz .
> > > I guess that the key of this lockup is that xfs-data/sda1 and xfs-eofblocks/s
> > > workqueues (which are RESCUER) got stuck waiting for memory allocation.
> > 
> > If those workers are really required for a further progress of the
> > memory reclaim then they shouldn't block on allocation at all and either
> > use pre allocated memory or use PF_MEMALLOC in case there is a guarantee
> > that only very limited amount of memory is allocated from that context
> > and there will be at least the same amount of memory freed as a result
> > in a reasonable time.
> > 
> > This is something for xfs people to answer though. Please note that I
> > didn't really have time to look through the below traces so the above
> > note is rather generic. It would be really helpful if you could provide
> > a high level dependency chains to see why those rescuers are necessary
> > for the forward progress because it is really easy to get lost in so
> > many traces.
> 
> Data IO completion is required to make progress to free memory. IO
> completion is done via work queues, so they need rescuer threads to
> ensure work can be run.
> 
> IO completion can require transactions to run. Transactions require
> memory allocation. Freeing memory therefore requires IO completion
> to have access to memory reserves if it's occurring from rescuer
> threads to allow progress to be made.
> 
> That means metadata IO completion require rescuer threads, because
> data IO completion can be dependent on metadata buffers being
> available. e.g. reserving space in the log for the transaction can
> require waiting on metadata IO dispatch and completion. Hence the
> buffer IO completion workqueues need rescuer threads.
> 
> Transactions can also require log forces and flushes to occur, which
> means they require the log workqueues (both the CIL flush and IO
> completion workqueues) to make progress.  Log flushes also require
> both IO and memory allocation to make progress to complete. Again,
> this means the log workqueues need rescuer threads. It also needs
> the log workqueues to be high priority so that they can make
> progress before IO completion work that is dependent on
> transactions making progress are processed.
> 
> IOWs, pretty much all the XFS workqueues are involved in memory
> reclaim in one way or another.
> 
> The real problem here is that the XFS code has /no idea/ of what
> workqueue context it is operating in - the fact it is in a rescuer
> thread is completely hidden from the executing context. It seems to
> me that the workqueue infrastructure's responsibility to tell memory
> reclaim that the rescuer thread needs special access to the memory
> reserves to allow the work it is running to allow forwards progress
> to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
> something similar...

I am not sure an automatic access to memory reserves from the rescuer
context is safe. This sounds too easy to break (read consume all the
reserves) - note that we have almost 200 users of WQ_MEM_RECLAIM and
chances are some of them will not be careful with the memory
allocations. I agree it would be helpful to know that the current item
runs from the rescuer context, though. In such a case the implementation
can do what ever it takes to make a forward progress. If that is using
__GFP_MEMALLOC then be it but it would be at least explicit and well
thought through (I hope).

Tejun, would it be possible/reasonable to add current_is_wq_rescuer() API?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tejun Heo]

Hello,

On Tue, Mar 07, 2017 at 01:15:04PM +0100, Michal Hocko wrote:
> > The real problem here is that the XFS code has /no idea/ of what
> > workqueue context it is operating in - the fact it is in a rescuer

I don't see how whether something is running off of a rescuer or not
matters here.  The only thing workqueue guarantees is that there's
gonna be at least one kworker thread executing work items from the
workqueue.  Running on a rescuer doesn't necessarily indicate memory
pressure condition.

> > thread is completely hidden from the executing context. It seems to
> > me that the workqueue infrastructure's responsibility to tell memory
> > reclaim that the rescuer thread needs special access to the memory
> > reserves to allow the work it is running to allow forwards progress
> > to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
> > something similar...
>
> I am not sure an automatic access to memory reserves from the rescuer
> context is safe. This sounds too easy to break (read consume all the
> reserves) - note that we have almost 200 users of WQ_MEM_RECLAIM and
> chances are some of them will not be careful with the memory
> allocations. I agree it would be helpful to know that the current item
> runs from the rescuer context, though. In such a case the implementation
> can do what ever it takes to make a forward progress. If that is using
> __GFP_MEMALLOC then be it but it would be at least explicit and well
> thought through (I hope).

I don't think doing this automatically is a good idea.  xfs work items
are free to mark itself PF_MEMALLOC while running tho.  It makes sense
to mark these cases explicitly anyway.  We can update workqueue code
so that it automatically clears the flag after each work item
completion to help.

> Tejun, would it be possible/reasonable to add current_is_wq_rescuer() API?

It's implementable for sure.  I'm just not sure how it'd help
anything.  It's not a relevant information on anything.

Thanks.

-- 
tejun

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tejun Heo]

Hello,

On Tue, Mar 07, 2017 at 01:15:04PM +0100, Michal Hocko wrote:
> > The real problem here is that the XFS code has /no idea/ of what
> > workqueue context it is operating in - the fact it is in a rescuer

I don't see how whether something is running off of a rescuer or not
matters here.  The only thing workqueue guarantees is that there's
gonna be at least one kworker thread executing work items from the
workqueue.  Running on a rescuer doesn't necessarily indicate memory
pressure condition.

> > thread is completely hidden from the executing context. It seems to
> > me that the workqueue infrastructure's responsibility to tell memory
> > reclaim that the rescuer thread needs special access to the memory
> > reserves to allow the work it is running to allow forwards progress
> > to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
> > something similar...
>
> I am not sure an automatic access to memory reserves from the rescuer
> context is safe. This sounds too easy to break (read consume all the
> reserves) - note that we have almost 200 users of WQ_MEM_RECLAIM and
> chances are some of them will not be careful with the memory
> allocations. I agree it would be helpful to know that the current item
> runs from the rescuer context, though. In such a case the implementation
> can do what ever it takes to make a forward progress. If that is using
> __GFP_MEMALLOC then be it but it would be at least explicit and well
> thought through (I hope).

I don't think doing this automatically is a good idea.  xfs work items
are free to mark itself PF_MEMALLOC while running tho.  It makes sense
to mark these cases explicitly anyway.  We can update workqueue code
so that it automatically clears the flag after each work item
completion to help.

> Tejun, would it be possible/reasonable to add current_is_wq_rescuer() API?

It's implementable for sure.  I'm just not sure how it'd help
anything.  It's not a relevant information on anything.

Thanks.

-- 
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Dave Chinner]

On Tue, Mar 07, 2017 at 02:36:59PM -0500, Tejun Heo wrote:
> Hello,
> 
> On Tue, Mar 07, 2017 at 01:15:04PM +0100, Michal Hocko wrote:
> > > The real problem here is that the XFS code has /no idea/ of what
> > > workqueue context it is operating in - the fact it is in a rescuer
> 
> I don't see how whether something is running off of a rescuer or not
> matters here.  The only thing workqueue guarantees is that there's
> gonna be at least one kworker thread executing work items from the
> workqueue.  Running on a rescuer doesn't necessarily indicate memory
> pressure condition.

That's news to me. In what situations do we run the rescuer thread
other than memory allocation failure when queuing work?

> > > thread is completely hidden from the executing context. It seems to
> > > me that the workqueue infrastructure's responsibility to tell memory
> > > reclaim that the rescuer thread needs special access to the memory
> > > reserves to allow the work it is running to allow forwards progress
> > > to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
> > > something similar...
> >
> > I am not sure an automatic access to memory reserves from the rescuer
> > context is safe. This sounds too easy to break (read consume all the
> > reserves) - note that we have almost 200 users of WQ_MEM_RECLAIM and
> > chances are some of them will not be careful with the memory
> > allocations. I agree it would be helpful to know that the current item
> > runs from the rescuer context, though. In such a case the implementation
> > can do what ever it takes to make a forward progress. If that is using
> > __GFP_MEMALLOC then be it but it would be at least explicit and well
> > thought through (I hope).
> 
> I don't think doing this automatically is a good idea.  xfs work items
> are free to mark itself PF_MEMALLOC while running tho.

I don't think that's a good idea to do unconditionally.It's quite
common to have IO intensive XFS workloads queue so much work that we
see several /thousand/ kworker threads running at once, even
on realtively small 16p systems.

> It makes sense
> to mark these cases explicitly anyway. 

Doing it on every work we queue will lead to immediate depletion of
memory reserves under heavy IO loads.

> W  can update workqueue code
> so that it automatically clears the flag after each work item
> completion to help.
> 
> > Tejun, would it be possible/reasonable to add current_is_wq_rescuer() API?
> 
> It's implementable for sure.  I'm just not sure how it'd help
> anything.  It's not a relevant information on anything.

Except to enable us to get closer to the "rescuer must make forwards
progress" guarantee. In this context, the rescuer is the only
context we should allow to dip into memory reserves. I'm happy if we
have to explicitly check for that and set PF_MEMALLOC ourselves 
(we do that for XFS kernel threads involved in memory reclaim),
but it's not something we should set automatically on every
IO completion work item we run....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Dave Chinner]

On Tue, Mar 07, 2017 at 02:36:59PM -0500, Tejun Heo wrote:
> Hello,
> 
> On Tue, Mar 07, 2017 at 01:15:04PM +0100, Michal Hocko wrote:
> > > The real problem here is that the XFS code has /no idea/ of what
> > > workqueue context it is operating in - the fact it is in a rescuer
> 
> I don't see how whether something is running off of a rescuer or not
> matters here.  The only thing workqueue guarantees is that there's
> gonna be at least one kworker thread executing work items from the
> workqueue.  Running on a rescuer doesn't necessarily indicate memory
> pressure condition.

That's news to me. In what situations do we run the rescuer thread
other than memory allocation failure when queuing work?

> > > thread is completely hidden from the executing context. It seems to
> > > me that the workqueue infrastructure's responsibility to tell memory
> > > reclaim that the rescuer thread needs special access to the memory
> > > reserves to allow the work it is running to allow forwards progress
> > > to be made. i.e.  setting PF_MEMALLOC on the rescuer thread or
> > > something similar...
> >
> > I am not sure an automatic access to memory reserves from the rescuer
> > context is safe. This sounds too easy to break (read consume all the
> > reserves) - note that we have almost 200 users of WQ_MEM_RECLAIM and
> > chances are some of them will not be careful with the memory
> > allocations. I agree it would be helpful to know that the current item
> > runs from the rescuer context, though. In such a case the implementation
> > can do what ever it takes to make a forward progress. If that is using
> > __GFP_MEMALLOC then be it but it would be at least explicit and well
> > thought through (I hope).
> 
> I don't think doing this automatically is a good idea.  xfs work items
> are free to mark itself PF_MEMALLOC while running tho.

I don't think that's a good idea to do unconditionally.It's quite
common to have IO intensive XFS workloads queue so much work that we
see several /thousand/ kworker threads running at once, even
on realtively small 16p systems.

> It makes sense
> to mark these cases explicitly anyway. 

Doing it on every work we queue will lead to immediate depletion of
memory reserves under heavy IO loads.

> W  can update workqueue code
> so that it automatically clears the flag after each work item
> completion to help.
> 
> > Tejun, would it be possible/reasonable to add current_is_wq_rescuer() API?
> 
> It's implementable for sure.  I'm just not sure how it'd help
> anything.  It's not a relevant information on anything.

Except to enable us to get closer to the "rescuer must make forwards
progress" guarantee. In this context, the rescuer is the only
context we should allow to dip into memory reserves. I'm happy if we
have to explicitly check for that and set PF_MEMALLOC ourselves 
(we do that for XFS kernel threads involved in memory reclaim),
but it's not something we should set automatically on every
IO completion work item we run....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tejun Heo]

Hello,

On Wed, Mar 08, 2017 at 08:21:32AM +1100, Dave Chinner wrote:
> > I don't see how whether something is running off of a rescuer or not
> > matters here.  The only thing workqueue guarantees is that there's
> > gonna be at least one kworker thread executing work items from the
> > workqueue.  Running on a rescuer doesn't necessarily indicate memory
> > pressure condition.
> 
> That's news to me. In what situations do we run the rescuer thread
> other than memory allocation failure when queuing work?

It's a timeout based mechanism.  Whevever the delay might be coming
from, the rescuer kicks in if the workqueue fails to make forward
progress for a while.  The only thing which can induce delay there is
kthread creation path, which usually gets blocked on memory pressure
but it could easily be something else - severe cpu contention,
somebody holding some mutex for too long, whatever.

> > It's implementable for sure.  I'm just not sure how it'd help
> > anything.  It's not a relevant information on anything.
> 
> Except to enable us to get closer to the "rescuer must make forwards
> progress" guarantee. In this context, the rescuer is the only
> context we should allow to dip into memory reserves. I'm happy if we
> have to explicitly check for that and set PF_MEMALLOC ourselves 
> (we do that for XFS kernel threads involved in memory reclaim),
> but it's not something we should set automatically on every
> IO completion work item we run....

Ah, okay, that does make sense to me.  Yeah, providing that test
shouldn't be difficult at all.  Lemme cook up a patch.

Thanks.

-- 
tejun

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tejun Heo]

Hello,

On Wed, Mar 08, 2017 at 08:21:32AM +1100, Dave Chinner wrote:
> > I don't see how whether something is running off of a rescuer or not
> > matters here.  The only thing workqueue guarantees is that there's
> > gonna be at least one kworker thread executing work items from the
> > workqueue.  Running on a rescuer doesn't necessarily indicate memory
> > pressure condition.
> 
> That's news to me. In what situations do we run the rescuer thread
> other than memory allocation failure when queuing work?

It's a timeout based mechanism.  Whevever the delay might be coming
from, the rescuer kicks in if the workqueue fails to make forward
progress for a while.  The only thing which can induce delay there is
kthread creation path, which usually gets blocked on memory pressure
but it could easily be something else - severe cpu contention,
somebody holding some mutex for too long, whatever.

> > It's implementable for sure.  I'm just not sure how it'd help
> > anything.  It's not a relevant information on anything.
> 
> Except to enable us to get closer to the "rescuer must make forwards
> progress" guarantee. In this context, the rescuer is the only
> context we should allow to dip into memory reserves. I'm happy if we
> have to explicitly check for that and set PF_MEMALLOC ourselves 
> (we do that for XFS kernel threads involved in memory reclaim),
> but it's not something we should set automatically on every
> IO completion work item we run....

Ah, okay, that does make sense to me.  Yeah, providing that test
shouldn't be difficult at all.  Lemme cook up a patch.

Thanks.

-- 
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tejun Heo]

Hello,

On Tue, Mar 07, 2017 at 04:48:42PM -0500, Tejun Heo wrote:
> > > It's implementable for sure.  I'm just not sure how it'd help
> > > anything.  It's not a relevant information on anything.
> > 
> > Except to enable us to get closer to the "rescuer must make forwards
> > progress" guarantee. In this context, the rescuer is the only
> > context we should allow to dip into memory reserves. I'm happy if we
> > have to explicitly check for that and set PF_MEMALLOC ourselves 
> > (we do that for XFS kernel threads involved in memory reclaim),
> > but it's not something we should set automatically on every
> > IO completion work item we run....
> 
> Ah, okay, that does make sense to me.  Yeah, providing that test
> shouldn't be difficult at all.  Lemme cook up a patch.

Turns out we already have this.  Writeback path already has a special
case handling for the rescuer.  You can just use
current_is_workqueue_rescuer().  The function can be called safely
from any task context.

Thanks.

-- 
tejun

Re: How to favor memory allocations for WQ_MEM_RECLAIM threads?

[Tejun Heo]

Hello,

On Tue, Mar 07, 2017 at 04:48:42PM -0500, Tejun Heo wrote:
> > > It's implementable for sure.  I'm just not sure how it'd help
> > > anything.  It's not a relevant information on anything.
> > 
> > Except to enable us to get closer to the "rescuer must make forwards
> > progress" guarantee. In this context, the rescuer is the only
> > context we should allow to dip into memory reserves. I'm happy if we
> > have to explicitly check for that and set PF_MEMALLOC ourselves 
> > (we do that for XFS kernel threads involved in memory reclaim),
> > but it's not something we should set automatically on every
> > IO completion work item we run....
> 
> Ah, okay, that does make sense to me.  Yeah, providing that test
> shouldn't be difficult at all.  Lemme cook up a patch.

Turns out we already have this.  Writeback path already has a special
case handling for the rescuer.  You can just use
current_is_workqueue_rescuer().  The function can be called safely
from any task context.

Thanks.

-- 
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>