linux-xfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* XFS: possible memory allocation deadlock in kmem_alloc
@ 2019-11-04 23:38 Chris Holcombe
  2019-11-05  0:01 ` Darrick J. Wong
  0 siblings, 1 reply; 18+ messages in thread
From: Chris Holcombe @ 2019-11-04 23:38 UTC (permalink / raw)
  To: linux-xfs

After upgrading from scientific linux 6 -> centos 7 i'm starting to
see a sharp uptick in dmesg lines about xfs having a possible memory
allocation deadlock.  All the searching I did through previous mailing
list archives and blog posts show all pointing to large files having
too many extents.
I don't think that is the case with these servers so I'm reaching out
in the hopes of getting an answer to what is going on.  The largest
file sizes I can find on the servers are roughly 15GB with maybe 9
extents total.  The vast majority small with only a few extents.
I've setup a cron job to drop the cache every 5 minutes which is
helping but not eliminating the problem.  These servers are dedicated
to storing data that is written through nginx webdav.  AFAIK nginx
webdav put does not use sparse files.

Some info about the servers this issue is occurring on:

nginx is writing to 82TB filesystems:
 xfs_info /dev/sdb1
meta-data=/dev/sdb1              isize=512    agcount=82, agsize=268435424 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=21973302784, imaxpct=1
         =                       sunit=16     swidth=144 blks
naming   =version 2              bsize=65536  ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_db -r /dev/sdb1
xfs_db> frag
actual 6565, ideal 5996, fragmentation factor 8.67%
Note, this number is largely meaningless.
Files on this filesystem average 1.09 extents per file

I see dmesg lines with various size numbers in the line:
[6262080.803537] XFS: nginx(2514) possible memory allocation deadlock
size 50184 in kmem_alloc (mode:0x250)

Typical extents for the largest files on the filesystem are:

find /mnt/jbod/ -type f -size +15G -printf '%s %p\n' -exec xfs_bmap
-vp {} \; | tee extents
17093242444 /mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc
/mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc:
 EXT: FILE-OFFSET           BLOCK-RANGE              AG AG-OFFSET
           TOTAL FLAGS
   0: [0..1919]:            51660187008..51660188927 24
(120585600..120587519)     1920 00010
   1: [1920..8063]:         51660189056..51660195199 24
(120587648..120593791)     6144 00011
   2: [8064..4194175]:      51660210816..51664396927 24
(120609408..124795519)  4186112 00001
   3: [4194176..11552759]:  51664560768..51671919351 24
(124959360..132317943)  7358584 00101
   4: [11552760..33385239]: 51678355840..51700188319 24
(138754432..160586911) 21832480 00111


Memory size:
 free -m
              total        used        free      shared  buff/cache   available
Mem:          64150        6338         421           2       57390       57123
Swap:          2047           6        2041

cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

cat /proc/buddyinfo
Node 0, zone      DMA      0      0      1      0      1      0      0
     0      0      1      3
Node 0, zone    DMA32  31577     88      2      0      0      0      0
     0      0      0      0
Node 0, zone   Normal  33331   3323    582     87      0      0      0
     0      0      0      0
Node 1, zone   Normal  51121   6343    822     77      1      0      0
     0      0      0      0

tuned-adm shows 'balanced' as the current tuning profile.

Thanks for your help!

^ permalink raw reply	[flat|nested] 18+ messages in thread
* XFS: possible memory allocation deadlock in kmem_alloc
@ 2016-05-30  4:45 baotiao
  2016-05-30  5:04 ` Dave Chinner
  0 siblings, 1 reply; 18+ messages in thread
From: baotiao @ 2016-05-30  4:45 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 2557 bytes --]

Hello. 

we have experience this problem many time, and we try many ways to solve this problem, but always failed.

such as sync && echo 3 > /proc/sys/vm/drop_caches to free more memory, however, after a while, the message come again.

the dmesg show:
May 30 12:35:51 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:53 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:55 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:57 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:59 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

our kernel version is 3.10.0-327.4.5.el7.x86_64

we think this is the kernel memory fragmentation problem, when dmesg show the error message, we check the buddyinfo

[root@w-openstack21 /home/xusiliang]# cat /proc/buddyinfo
Node 0, zone      DMA      1      0      1      0      2      1      1      0      1      1      3
Node 0, zone    DMA32   2190   1898   1367    583    284     81     21      8      5      0      4
Node 0, zone   Normal  85195  90572  56688  32379  16793   5568    930    230    184     11      0
Node 1, zone   Normal  95799 127034  94363  49321  21555   5239    225      2      2      0      0

and I think the kernel have enough free memory
[root@w-openstack21 /home/xusiliang]# free -m
              total        used        free      shared  buff/cache   available
Mem:          64269       50357       11802          65        2109       11757
Swap:         32255         347       31908

This machine is running for qemu,  the file is qemu qcow file type

[root@w-openstack20 /data/nova/instances/7898b630-c4ef-49ab-9ce9-3735e090c282]# file disk
disk: QEMU QCOW Image (v3), has backing file (path /data/nova/instances/_base/3f27393376152ac352b2d85703011e38517e), 429496729600 bytes

we also use xfs_db to check the fragmentation

[root@w-openstack20 /home/xusiliang]# xfs_db -r /dev/mapper/VolGroup00-LogVol04
xfs_db> frag

actual 48421430, ideal 20716, fragmentation factor 99.96%


How can I solve this problem, what do you suggest me to do?

thank you


----------------------------------------
 
Github: https://github.com/baotiao
Blog: http://baotiao.github.io
Stackoverflow: http://stackoverflow.com/users/634415/baotiao 
Linkedin: http://www.linkedin.com/profile/view?id=145231990


[-- Attachment #1.2: Type: text/html, Size: 6916 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2019-11-05 21:23 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-04 23:38 XFS: possible memory allocation deadlock in kmem_alloc Chris Holcombe
2019-11-05  0:01 ` Darrick J. Wong
2019-11-05  0:31   ` Eric Sandeen
     [not found]     ` <CAC752AmahECFry9x=pvqDkwQUj1PEJjoWGa2KFG1uaTzT1Bbnw@mail.gmail.com>
2019-11-05  4:21       ` Eric Sandeen
2019-11-05 16:25         ` Chris Holcombe
2019-11-05 17:11           ` Eric Sandeen
2019-11-05 19:53             ` Chris Holcombe
2019-11-05 20:08               ` Eric Sandeen
     [not found]                 ` <CAC752AnZ4biDGk6V17URQm5YVp=MwZBhiMH8=t733zaypxUsmA@mail.gmail.com>
2019-11-05 20:47                   ` Eric Sandeen
     [not found]                     ` <CAC752A=y9PMEQ1e4mXskha1GFeKXWi8PsdBW-nX40pgFCYp1Uw@mail.gmail.com>
2019-11-05 21:23                       ` Eric Sandeen
  -- strict thread matches above, loose matches on Subject: below --
2016-05-30  4:45 baotiao
2016-05-30  5:04 ` Dave Chinner
2016-05-30  8:48   ` baotiao
2016-05-30  9:20     ` Carlos Maiolino
2016-05-31  2:43     ` 陈宗志
2016-05-31  3:10       ` Dave Chinner
2016-05-31 11:00         ` 陈宗志
2016-05-31 12:14           ` Carlos Maiolino

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).