Re: XFS: possible memory allocation deadlock in kmem_alloc

[Eric Sandeen <sandeen@sandeen.net>]

On 11/4/19 6:01 PM, Darrick J. Wong wrote:
> On Mon, Nov 04, 2019 at 03:38:12PM -0800, Chris Holcombe wrote:
>> After upgrading from scientific linux 6 -> centos 7 i'm starting to
>> see a sharp uptick in dmesg lines about xfs having a possible memory
>> allocation deadlock.  All the searching I did through previous mailing
>> list archives and blog posts show all pointing to large files having
>> too many extents.
>> I don't think that is the case with these servers so I'm reaching out
>> in the hopes of getting an answer to what is going on.  The largest
>> file sizes I can find on the servers are roughly 15GB with maybe 9
>> extents total.  The vast majority small with only a few extents.
>> I've setup a cron job to drop the cache every 5 minutes which is
>> helping but not eliminating the problem.  These servers are dedicated
>> to storing data that is written through nginx webdav.  AFAIK nginx
>> webdav put does not use sparse files.
>>
>> Some info about the servers this issue is occurring on:
>>
>> nginx is writing to 82TB filesystems:
>>  xfs_info /dev/sdb1
>> meta-data=/dev/sdb1              isize=512    agcount=82, agsize=268435424 blks
>>          =                       sectsz=4096  attr=2, projid32bit=1
>>          =                       crc=1        finobt=0 spinodes=0
>> data     =                       bsize=4096   blocks=21973302784, imaxpct=1
>>          =                       sunit=16     swidth=144 blks
>> naming   =version 2              bsize=65536  ascii-ci=0 ftype=1
>> log      =internal               bsize=4096   blocks=521728, version=2
>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>
>> xfs_db -r /dev/sdb1
>> xfs_db> frag
>> actual 6565, ideal 5996, fragmentation factor 8.67%
>> Note, this number is largely meaningless.
>> Files on this filesystem average 1.09 extents per file
>>
>> I see dmesg lines with various size numbers in the line:
>> [6262080.803537] XFS: nginx(2514) possible memory allocation deadlock
>> size 50184 in kmem_alloc (mode:0x250)
> 
> Full kernel logs, please.  There's not enough info here to tell what's
> trying to grab a 50K memory buffer.

a kernel uname -a would be groovy too so we have some idea what you're running.

-Eric

> --D
>