XFS: possible memory allocation deadlock in kmem_alloc

XFS: possible memory allocation deadlock in kmem_alloc

[Chris Holcombe]

After upgrading from scientific linux 6 -> centos 7 i'm starting to
see a sharp uptick in dmesg lines about xfs having a possible memory
allocation deadlock.  All the searching I did through previous mailing
list archives and blog posts show all pointing to large files having
too many extents.
I don't think that is the case with these servers so I'm reaching out
in the hopes of getting an answer to what is going on.  The largest
file sizes I can find on the servers are roughly 15GB with maybe 9
extents total.  The vast majority small with only a few extents.
I've setup a cron job to drop the cache every 5 minutes which is
helping but not eliminating the problem.  These servers are dedicated
to storing data that is written through nginx webdav.  AFAIK nginx
webdav put does not use sparse files.

Some info about the servers this issue is occurring on:

nginx is writing to 82TB filesystems:
 xfs_info /dev/sdb1
meta-data=/dev/sdb1              isize=512    agcount=82, agsize=268435424 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=21973302784, imaxpct=1
         =                       sunit=16     swidth=144 blks
naming   =version 2              bsize=65536  ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

xfs_db -r /dev/sdb1
xfs_db> frag
actual 6565, ideal 5996, fragmentation factor 8.67%
Note, this number is largely meaningless.
Files on this filesystem average 1.09 extents per file

I see dmesg lines with various size numbers in the line:
[6262080.803537] XFS: nginx(2514) possible memory allocation deadlock
size 50184 in kmem_alloc (mode:0x250)

Typical extents for the largest files on the filesystem are:

find /mnt/jbod/ -type f -size +15G -printf '%s %p\n' -exec xfs_bmap
-vp {} \; | tee extents
17093242444 /mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc
/mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc:
 EXT: FILE-OFFSET           BLOCK-RANGE              AG AG-OFFSET
           TOTAL FLAGS
   0: [0..1919]:            51660187008..51660188927 24
(120585600..120587519)     1920 00010
   1: [1920..8063]:         51660189056..51660195199 24
(120587648..120593791)     6144 00011
   2: [8064..4194175]:      51660210816..51664396927 24
(120609408..124795519)  4186112 00001
   3: [4194176..11552759]:  51664560768..51671919351 24
(124959360..132317943)  7358584 00101
   4: [11552760..33385239]: 51678355840..51700188319 24
(138754432..160586911) 21832480 00111


Memory size:
 free -m
              total        used        free      shared  buff/cache   available
Mem:          64150        6338         421           2       57390       57123
Swap:          2047           6        2041

cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

cat /proc/buddyinfo
Node 0, zone      DMA      0      0      1      0      1      0      0
     0      0      1      3
Node 0, zone    DMA32  31577     88      2      0      0      0      0
     0      0      0      0
Node 0, zone   Normal  33331   3323    582     87      0      0      0
     0      0      0      0
Node 1, zone   Normal  51121   6343    822     77      1      0      0
     0      0      0      0

tuned-adm shows 'balanced' as the current tuning profile.

Thanks for your help!

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Darrick J. Wong]

On Mon, Nov 04, 2019 at 03:38:12PM -0800, Chris Holcombe wrote:
> After upgrading from scientific linux 6 -> centos 7 i'm starting to
> see a sharp uptick in dmesg lines about xfs having a possible memory
> allocation deadlock.  All the searching I did through previous mailing
> list archives and blog posts show all pointing to large files having
> too many extents.
> I don't think that is the case with these servers so I'm reaching out
> in the hopes of getting an answer to what is going on.  The largest
> file sizes I can find on the servers are roughly 15GB with maybe 9
> extents total.  The vast majority small with only a few extents.
> I've setup a cron job to drop the cache every 5 minutes which is
> helping but not eliminating the problem.  These servers are dedicated
> to storing data that is written through nginx webdav.  AFAIK nginx
> webdav put does not use sparse files.
> 
> Some info about the servers this issue is occurring on:
> 
> nginx is writing to 82TB filesystems:
>  xfs_info /dev/sdb1
> meta-data=/dev/sdb1              isize=512    agcount=82, agsize=268435424 blks
>          =                       sectsz=4096  attr=2, projid32bit=1
>          =                       crc=1        finobt=0 spinodes=0
> data     =                       bsize=4096   blocks=21973302784, imaxpct=1
>          =                       sunit=16     swidth=144 blks
> naming   =version 2              bsize=65536  ascii-ci=0 ftype=1
> log      =internal               bsize=4096   blocks=521728, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> xfs_db -r /dev/sdb1
> xfs_db> frag
> actual 6565, ideal 5996, fragmentation factor 8.67%
> Note, this number is largely meaningless.
> Files on this filesystem average 1.09 extents per file
> 
> I see dmesg lines with various size numbers in the line:
> [6262080.803537] XFS: nginx(2514) possible memory allocation deadlock
> size 50184 in kmem_alloc (mode:0x250)

Full kernel logs, please.  There's not enough info here to tell what's
trying to grab a 50K memory buffer.

--D

> Typical extents for the largest files on the filesystem are:
> 
> find /mnt/jbod/ -type f -size +15G -printf '%s %p\n' -exec xfs_bmap
> -vp {} \; | tee extents
> 17093242444 /mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc
> /mnt/jbod/boxfiler3038-sdb1/data/220190411/ephemeral/2019-08-12/18/0f6bee4d6ee0136af3b58eef611e2586.enc:
>  EXT: FILE-OFFSET           BLOCK-RANGE              AG AG-OFFSET
>            TOTAL FLAGS
>    0: [0..1919]:            51660187008..51660188927 24
> (120585600..120587519)     1920 00010
>    1: [1920..8063]:         51660189056..51660195199 24
> (120587648..120593791)     6144 00011
>    2: [8064..4194175]:      51660210816..51664396927 24
> (120609408..124795519)  4186112 00001
>    3: [4194176..11552759]:  51664560768..51671919351 24
> (124959360..132317943)  7358584 00101
>    4: [11552760..33385239]: 51678355840..51700188319 24
> (138754432..160586911) 21832480 00111
> 
> 
> Memory size:
>  free -m
>               total        used        free      shared  buff/cache   available
> Mem:          64150        6338         421           2       57390       57123
> Swap:          2047           6        2041
> 
> cat /etc/redhat-release
> CentOS Linux release 7.6.1810 (Core)
> 
> cat /proc/buddyinfo
> Node 0, zone      DMA      0      0      1      0      1      0      0
>      0      0      1      3
> Node 0, zone    DMA32  31577     88      2      0      0      0      0
>      0      0      0      0
> Node 0, zone   Normal  33331   3323    582     87      0      0      0
>      0      0      0      0
> Node 1, zone   Normal  51121   6343    822     77      1      0      0
>      0      0      0      0
> 
> tuned-adm shows 'balanced' as the current tuning profile.
> 
> Thanks for your help!

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Eric Sandeen]

On 11/4/19 6:01 PM, Darrick J. Wong wrote:
> On Mon, Nov 04, 2019 at 03:38:12PM -0800, Chris Holcombe wrote:
>> After upgrading from scientific linux 6 -> centos 7 i'm starting to
>> see a sharp uptick in dmesg lines about xfs having a possible memory
>> allocation deadlock.  All the searching I did through previous mailing
>> list archives and blog posts show all pointing to large files having
>> too many extents.
>> I don't think that is the case with these servers so I'm reaching out
>> in the hopes of getting an answer to what is going on.  The largest
>> file sizes I can find on the servers are roughly 15GB with maybe 9
>> extents total.  The vast majority small with only a few extents.
>> I've setup a cron job to drop the cache every 5 minutes which is
>> helping but not eliminating the problem.  These servers are dedicated
>> to storing data that is written through nginx webdav.  AFAIK nginx
>> webdav put does not use sparse files.
>>
>> Some info about the servers this issue is occurring on:
>>
>> nginx is writing to 82TB filesystems:
>>  xfs_info /dev/sdb1
>> meta-data=/dev/sdb1              isize=512    agcount=82, agsize=268435424 blks
>>          =                       sectsz=4096  attr=2, projid32bit=1
>>          =                       crc=1        finobt=0 spinodes=0
>> data     =                       bsize=4096   blocks=21973302784, imaxpct=1
>>          =                       sunit=16     swidth=144 blks
>> naming   =version 2              bsize=65536  ascii-ci=0 ftype=1
>> log      =internal               bsize=4096   blocks=521728, version=2
>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>
>> xfs_db -r /dev/sdb1
>> xfs_db> frag
>> actual 6565, ideal 5996, fragmentation factor 8.67%
>> Note, this number is largely meaningless.
>> Files on this filesystem average 1.09 extents per file
>>
>> I see dmesg lines with various size numbers in the line:
>> [6262080.803537] XFS: nginx(2514) possible memory allocation deadlock
>> size 50184 in kmem_alloc (mode:0x250)
> 
> Full kernel logs, please.  There's not enough info here to tell what's
> trying to grab a 50K memory buffer.

a kernel uname -a would be groovy too so we have some idea what you're running.

-Eric

> --D
> 

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Eric Sandeen]

On 11/4/19 8:28 PM, Blake Golliher wrote:
> Hi!  I work with Chris.
> 
> Linux boxfiler3038 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> Is the kernel release.  We're using the Centos7 release CentOS Linux release 7.6.1810 (Core) 
> 
> 
> The previous version that didn't have the problem is this kernel
> 
> Linux boxfiler3006 2.6.32-754.6.3.el6.x86_64 #1 SMP Tue Oct 9 11:31:50 CDT 2018 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> Scientific Linux release 6.7 (Carbon)
> 
> 
> So SL6, didn't have the problem on 2.6.32.  Centos7 does have the xfs deadlock issue, on 3.10.0-957.
> 

Thanks.

We need the full kernel log Darrick asked for, too.

-Eric

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Chris Holcombe]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]

Hi Eric,
I've attached both the dmesg command output and /var/log/dmesg.log
which have different values in them.

On Mon, Nov 4, 2019 at 8:21 PM Eric Sandeen <sandeen@sandeen.net> wrote:
>
>
>
> CAUTION: External Email
>
>
>
>
> On 11/4/19 8:28 PM, Blake Golliher wrote:
> > Hi!  I work with Chris.
> >
> > Linux boxfiler3038 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> > Is the kernel release.  We're using the Centos7 release CentOS Linux release 7.6.1810 (Core)
> >
> >
> > The previous version that didn't have the problem is this kernel
> >
> > Linux boxfiler3006 2.6.32-754.6.3.el6.x86_64 #1 SMP Tue Oct 9 11:31:50 CDT 2018 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> > Scientific Linux release 6.7 (Carbon)
> >
> >
> > So SL6, didn't have the problem on 2.6.32.  Centos7 does have the xfs deadlock issue, on 3.10.0-957.
> >
>
> Thanks.
>
> We need the full kernel log Darrick asked for, too.
>
> -Eric

[-- Attachment #2: logs.tar.lzma --]
[-- Type: application/octet-stream, Size: 74698 bytes --]

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Eric Sandeen]

On 11/5/19 10:25 AM, Chris Holcombe wrote:
> Hi Eric,
> I've attached both the dmesg command output and /var/log/dmesg.log
> which have different values in them.

Thanks.  Aaand .... I forgot that we don't dump a stack trace by default.

Can you please do:

# echo 11 > /proc/sys/fs/xfs/error_level

and hit it again, then send a few of the backtraces from dmesg?

# echo 3 > /proc/sys/fs/xfs/error_level

will quiet things down again.

-Eric

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Chris Holcombe]

I've got the stack traces and they're massive.
https://cloud.box.com/s/5grgnjwej5prmahl92v49937w6hgdnsv
https://cloud.box.com/s/wfrjg7yhiwufpgidoq1qos8mrwtu4ij3
https://cloud.box.com/s/3wcfgfdyvcsbslfdxc9ntfdt7yrhtcjt

On Tue, Nov 5, 2019 at 9:11 AM Eric Sandeen <sandeen@sandeen.net> wrote:
>
>
>
> CAUTION: External Email
>
>
>
>
> On 11/5/19 10:25 AM, Chris Holcombe wrote:
> > Hi Eric,
> > I've attached both the dmesg command output and /var/log/dmesg.log
> > which have different values in them.
>
> Thanks.  Aaand .... I forgot that we don't dump a stack trace by default.
>
> Can you please do:
>
> # echo 11 > /proc/sys/fs/xfs/error_level
>
> and hit it again, then send a few of the backtraces from dmesg?
>
> # echo 3 > /proc/sys/fs/xfs/error_level
>
> will quiet things down again.
>
> -Eric

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Eric Sandeen]

I don't think that's the information I requested:

>> Can you please do:
>>
>> # echo 11 > /proc/sys/fs/xfs/error_level
>>
>> and hit it again, then send a few of the backtraces from dmesg?

-Eric


On 11/5/19 1:53 PM, Chris Holcombe wrote:
> I've got the stack traces and they're massive.
> https://cloud.box.com/s/5grgnjwej5prmahl92v49937w6hgdnsv
> https://cloud.box.com/s/wfrjg7yhiwufpgidoq1qos8mrwtu4ij3
> https://cloud.box.com/s/3wcfgfdyvcsbslfdxc9ntfdt7yrhtcjt
> 
> On Tue, Nov 5, 2019 at 9:11 AM Eric Sandeen <sandeen@sandeen.net> wrote:
>>
>>
>>
>> CAUTION: External Email
>>
>>
>>
>>
>> On 11/5/19 10:25 AM, Chris Holcombe wrote:
>>> Hi Eric,
>>> I've attached both the dmesg command output and /var/log/dmesg.log
>>> which have different values in them.
>>
>> Thanks.  Aaand .... I forgot that we don't dump a stack trace by default.
>>
>> Can you please do:
>>
>> # echo 11 > /proc/sys/fs/xfs/error_level
>>
>> and hit it again, then send a few of the backtraces from dmesg?
>>
>> # echo 3 > /proc/sys/fs/xfs/error_level
>>
>> will quiet things down again.
>>
>> -Eric
> 

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Eric Sandeen]

On 11/5/19 2:36 PM, Blake Golliher wrote:
> We don't get anything more then this messages.
> 
> [Tue Nov  5 11:18:34 2019] XFS: nginx(2540) possible memory allocation deadlock size 63960 in kmem_alloc (mode:0x250)
> 
> [Tue Nov  5 11:18:34 2019] XFS: nginx(2517) possible memory allocation deadlock size 56880 in kmem_alloc (mode:0x250)
> 
> [Tue Nov  5 11:18:34 2019] XFS: nginx(2540) possible memory allocation deadlock size 63960 in kmem_alloc (mode:0x250)
> 
> [Tue Nov  5 11:18:35 2019] XFS: nginx(2517) possible memory allocation deadlock size 56880 in kmem_alloc (mode:0x250)
> 
> [Tue Nov  5 11:18:36 2019] XFS: nginx(2514) possible memory allocation deadlock size 63960 in kmem_alloc (mode:0x250)
> 
> [Tue Nov  5 11:18:36 2019] XFS: nginx(2540) possible memory allocation deadlock size 63960 in kmem_alloc (mode:0x250)
> 
> [Tue Nov  5 11:18:37 2019] XFS: nginx(2517) possible memory allocation deadlock size 56880 in kmem_alloc (mode:0x250)

no sure what to say.  In that kernel, when we print the message:

                        xfs_err(NULL,
        "%s(%u) possible memory allocation deadlock size %u in %s (mode:0x%x)",
                                current->comm, current->pid,
                                (unsigned int)size, __func__, lflags);

xfs_err() is:

define_xfs_printk_level(xfs_err, KERN_ERR);

which is the macro:

#define define_xfs_printk_level(func, kern_level)               \
void func(const struct xfs_mount *mp, const char *fmt, ...)     \
{                                                               \
        struct va_format        vaf;                            \
        va_list                 args;                           \
        int                     level;                          \
                                                                \
        va_start(args, fmt);                                    \
                                                                \
        vaf.fmt = fmt;                                          \
        vaf.va = &args;                                         \
                                                                \
        __xfs_printk(kern_level, mp, &vaf);                     \
        va_end(args);                                           \
                                                                \
        if (!kstrtoint(kern_level, 0, &level) &&                \ 
            level <= 3 /* LOGLEVEL_ERR */ &&                    \
            xfs_error_level >= XFS_ERRLEVEL_HIGH)               \
                xfs_stack_trace();                              \
}                                                               \

which should dump a stack if called w/ ERR priority and the xfs_error_level is at 11.

I suppose your general kernel ring buffer needs to be turned up high enough
as well (i.e. dmesg -n 8 to be sure?)

The resulting stack trace would tell us exactly how you got to the 
allocation message.

-Eric

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Eric Sandeen]

On 11/5/19 3:01 PM, Blake Golliher wrote:
> OK. Does the stacktrace show up in dmesg?  we have kernel.* in rsyslog going to /var/log/kern.log will it go there?  We capture 5 samples of 
> 
> /sys/kernel/debug/tracing/stack_trace
> 
> 
> While the kmem_alloc was being pushed into dmesg.  I set dmesg -n 8 and I'll set xfs_error_level to 11 again and see if we can get a stacktrace.  Where do I look?

dmesg

-Eric

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Carlos Maiolino]

On Tue, May 31, 2016 at 07:00:22PM +0800, 陈宗志 wrote:
>    can we change the existing file's extent size?
> 
No, you can only set extent hint size for empty files, or, more precisely for
files whose does not have allocated extents

>    On Tue, May 31, 2016 at 11:10 AM, Dave Chinner <[1]david@fromorbit.com>
>    wrote:
> 
>      On Tue, May 31, 2016 at 10:43:36AM +0800, 陈宗志 wrote:
>      > I have fint the way to change the extent size hint
>      > mkfs.xfs -r extsize=40960 /dev/sda1
>      Ah, no, that isn't the extsize I'm refering to. That's for realtime
>      device configuration at mkfs, not a per-inode extent size hint.
>      When you create the image file do this:
>      $ xfs_io -f -c "extsize 1m" /path/to/new/vm_image
>      $ qemu-image create -f qcow2 /path/to/new/vm_image 10g
>      $ xfs_io -c extsize /path/to/new/vm_image
>      [1048576] /path/to/new/vm_image
>      $
>      And now the qcow2 image file will have extents allocated in
>      multiples of 1MB.
> 
>    Cheers,
>    Dave.
>    --
>    Dave Chinner
>    [2]david@fromorbit.com
> 
>    --
>    ---
>    Blog: [3]http://www.chenzongzhi.info
>    Twitter: [4]https://twitter.com/baotiao
>    Git: [5]https://github.com/baotiao
> 
> References
> 
>    1. mailto:david@fromorbit.com
>    2. mailto:david@fromorbit.com
>    3. http://www.chenzongzhi.info/
>    4. https://twitter.com/#!/baotiao
>    5. https://github.com/baotiao

> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs


-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS: possible memory allocation deadlock in kmem_alloc

[陈宗志]

[-- Attachment #1.1: Type: text/plain, Size: 994 bytes --]

can we change the existing file's extent size?


On Tue, May 31, 2016 at 11:10 AM, Dave Chinner <david@fromorbit.com> wrote:

> On Tue, May 31, 2016 at 10:43:36AM +0800, 陈宗志 wrote:
> > I have fint the way to change the extent size hint
> > mkfs.xfs -r extsize=40960 /dev/sda1
>
> Ah, no, that isn't the extsize I'm refering to. That's for realtime
> device configuration at mkfs, not a per-inode extent size hint.
>
> When you create the image file do this:
>
> $ xfs_io -f -c "extsize 1m" /path/to/new/vm_image
> $ qemu-image create -f qcow2 /path/to/new/vm_image 10g
> $ xfs_io -c extsize /path/to/new/vm_image
> [1048576] /path/to/new/vm_image
> $
>
> And now the qcow2 image file will have extents allocated in
> multiples of 1MB.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 
---
Blog: http://www.chenzongzhi.info
Twitter: https://twitter.com/baotiao <https://twitter.com/#%21/baotiao>
Git: https://github.com/baotiao

[-- Attachment #1.2: Type: text/html, Size: 1753 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Dave Chinner]

On Tue, May 31, 2016 at 10:43:36AM +0800, 陈宗志 wrote:
> I have fint the way to change the extent size hint
> mkfs.xfs -r extsize=40960 /dev/sda1

Ah, no, that isn't the extsize I'm refering to. That's for realtime
device configuration at mkfs, not a per-inode extent size hint.

When you create the image file do this:

$ xfs_io -f -c "extsize 1m" /path/to/new/vm_image
$ qemu-image create -f qcow2 /path/to/new/vm_image 10g
$ xfs_io -c extsize /path/to/new/vm_image
[1048576] /path/to/new/vm_image
$

And now the qcow2 image file will have extents allocated in
multiples of 1MB.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS: possible memory allocation deadlock in kmem_alloc

[陈宗志]

[-- Attachment #1.1: Type: text/plain, Size: 1670 bytes --]

I have fint the way to change the extent size hint
mkfs.xfs -r extsize=40960 /dev/sda1

but I should rebuild the disk, do you have a way more smooth?


On Mon, May 30, 2016 at 4:48 PM, baotiao <baotiao@gmail.com> wrote:

> how can I set the extent size hint to a file with command line?
> I have google for a lot, but I can't find a answer
>
> ----------------------------------------
>
> Github: https://github.com/baotiao
> Blog: http://baotiao.github.io
> Stackoverflow: http://stackoverflow.com/users/634415/baotiao
> Linkedin: http://www.linkedin.com/profile/view?id=145231990
>
> On May 30, 2016, at 13:04, Dave Chinner <david@fromorbit.com> wrote:
>
> On Mon, May 30, 2016 at 12:45:07PM +0800, baotiao wrote:
>
> This machine is running for qemu,  the file is qemu qcow file type
>
> [root@w-openstack20
> /data/nova/instances/7898b630-c4ef-49ab-9ce9-3735e090c282]# file disk
> disk: QEMU QCOW Image (v3), has backing file (path
> /data/nova/instances/_base/3f27393376152ac352b2d85703011e38517e),
> 429496729600 bytes
>
>
> Oh, you're using delta/snapshot based qcow images. That, by it's
> very nature, generates fragmented image files as they are a delta
> over the backing file.
>
> ....
>
> actual 48421430, ideal 20716, fragmentation factor 99.96%
>
> How can I solve this problem, what do you suggest me to do?
>
>
> Use a extent size hint (say 1-8MB) for your qcow2 image files so that
> they don't fragment badly as they are written to.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
>
>


-- 
---
Blog: http://www.chenzongzhi.info
Twitter: https://twitter.com/baotiao <https://twitter.com/#%21/baotiao>
Git: https://github.com/baotiao

[-- Attachment #1.2: Type: text/html, Size: 3913 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Carlos Maiolino]

On Mon, May 30, 2016 at 04:48:36PM +0800, baotiao wrote:
>    how can I set the extent size hint to a file with command line?
> 
>    I have google for a lot, but I can't find a answer
> 

It can be set with xfs_io, like:

xfs_io -c "extsize 8m" <file>   # To set an extent size hint of 8m

Although, it only works if the file is empty, or more specific, the file must
not have any extents allocated
>    ----------------------------------------
> 
>    Github: [1]https://github.com/baotiao
>    Blog: [2]http://baotiao.github.io
>    Stackoverflow: [3]http://stackoverflow.com/users/634415/baotiao
>    Linkedin: [4]http://www.linkedin.com/profile/view?id=145231990
> 
>    On May 30, 2016, at 13:04, Dave Chinner <[5]david@fromorbit.com> wrote:
> 
>    On Mon, May 30, 2016 at 12:45:07PM +0800, baotiao wrote:
> 
>      This machine is running for qemu,  the file is qemu qcow file type
>      [root@w-openstack20
>      /data/nova/instances/7898b630-c4ef-49ab-9ce9-3735e090c282]# file
>      disk
>      disk: QEMU QCOW Image (v3), has backing file (path
>      /data/nova/instances/_base/3f27393376152ac352b2d85703011e38517e),
>      429496729600 bytes
> 
>    Oh, you're using delta/snapshot based qcow images. That, by it's
>    very nature, generates fragmented image files as they are a delta
>    over the backing file.
>    ....
> 
>      actual 48421430, ideal 20716, fragmentation factor 99.96%
>      How can I solve this problem, what do you suggest me to do?
> 
>    Use a extent size hint (say 1-8MB) for your qcow2 image files so that
>    they don't fragment badly as they are written to.
>    Cheers,
>    Dave.
>    --
>    Dave Chinner
>    [6]david@fromorbit.com
> 
> References
> 
>    1. https://github.com/baotiao
>    2. http://baotiao.github.io/
>    3. http://stackoverflow.com/users/634415/baotiao
>    4. http://www.linkedin.com/profile/view?id=145231990
>    5. mailto:david@fromorbit.com
>    6. mailto:david@fromorbit.com

> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs


-- 
Carlos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS: possible memory allocation deadlock in kmem_alloc

[baotiao]

[-- Attachment #1.1: Type: text/plain, Size: 1322 bytes --]

how can I set the extent size hint to a file with command line? 
I have google for a lot, but I can't find a answer

----------------------------------------
 
Github: https://github.com/baotiao
Blog: http://baotiao.github.io
Stackoverflow: http://stackoverflow.com/users/634415/baotiao 
Linkedin: http://www.linkedin.com/profile/view?id=145231990

> On May 30, 2016, at 13:04, Dave Chinner <david@fromorbit.com> wrote:
> 
> On Mon, May 30, 2016 at 12:45:07PM +0800, baotiao wrote:
>> This machine is running for qemu,  the file is qemu qcow file type
>> 
>> [root@w-openstack20 /data/nova/instances/7898b630-c4ef-49ab-9ce9-3735e090c282]# file disk
>> disk: QEMU QCOW Image (v3), has backing file (path /data/nova/instances/_base/3f27393376152ac352b2d85703011e38517e), 429496729600 bytes
> 
> Oh, you're using delta/snapshot based qcow images. That, by it's
> very nature, generates fragmented image files as they are a delta
> over the backing file.
> 
> ....
>> actual 48421430, ideal 20716, fragmentation factor 99.96%
>> 
>> How can I solve this problem, what do you suggest me to do?
> 
> Use a extent size hint (say 1-8MB) for your qcow2 image files so that
> they don't fragment badly as they are written to. 
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com


[-- Attachment #1.2: Type: text/html, Size: 4357 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS: possible memory allocation deadlock in kmem_alloc

[Dave Chinner]

On Mon, May 30, 2016 at 12:45:07PM +0800, baotiao wrote:
> This machine is running for qemu,  the file is qemu qcow file type
> 
> [root@w-openstack20 /data/nova/instances/7898b630-c4ef-49ab-9ce9-3735e090c282]# file disk
> disk: QEMU QCOW Image (v3), has backing file (path /data/nova/instances/_base/3f27393376152ac352b2d85703011e38517e), 429496729600 bytes

Oh, you're using delta/snapshot based qcow images. That, by it's
very nature, generates fragmented image files as they are a delta
over the backing file.

....
> actual 48421430, ideal 20716, fragmentation factor 99.96%
> 
> How can I solve this problem, what do you suggest me to do?

Use a extent size hint (say 1-8MB) for your qcow2 image files so that
they don't fragment badly as they are written to. 

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

XFS: possible memory allocation deadlock in kmem_alloc

[baotiao]

[-- Attachment #1.1: Type: text/plain, Size: 2557 bytes --]

Hello. 

we have experience this problem many time, and we try many ways to solve this problem, but always failed.

such as sync && echo 3 > /proc/sys/vm/drop_caches to free more memory, however, after a while, the message come again.

the dmesg show:
May 30 12:35:51 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:53 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:55 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:57 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
May 30 12:35:59 w-openstack21 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

our kernel version is 3.10.0-327.4.5.el7.x86_64

we think this is the kernel memory fragmentation problem, when dmesg show the error message, we check the buddyinfo

[root@w-openstack21 /home/xusiliang]# cat /proc/buddyinfo
Node 0, zone      DMA      1      0      1      0      2      1      1      0      1      1      3
Node 0, zone    DMA32   2190   1898   1367    583    284     81     21      8      5      0      4
Node 0, zone   Normal  85195  90572  56688  32379  16793   5568    930    230    184     11      0
Node 1, zone   Normal  95799 127034  94363  49321  21555   5239    225      2      2      0      0

and I think the kernel have enough free memory
[root@w-openstack21 /home/xusiliang]# free -m
              total        used        free      shared  buff/cache   available
Mem:          64269       50357       11802          65        2109       11757
Swap:         32255         347       31908

This machine is running for qemu,  the file is qemu qcow file type

[root@w-openstack20 /data/nova/instances/7898b630-c4ef-49ab-9ce9-3735e090c282]# file disk
disk: QEMU QCOW Image (v3), has backing file (path /data/nova/instances/_base/3f27393376152ac352b2d85703011e38517e), 429496729600 bytes

we also use xfs_db to check the fragmentation

[root@w-openstack20 /home/xusiliang]# xfs_db -r /dev/mapper/VolGroup00-LogVol04
xfs_db> frag

actual 48421430, ideal 20716, fragmentation factor 99.96%


How can I solve this problem, what do you suggest me to do?

thank you


----------------------------------------
 
Github: https://github.com/baotiao
Blog: http://baotiao.github.io
Stackoverflow: http://stackoverflow.com/users/634415/baotiao 
Linkedin: http://www.linkedin.com/profile/view?id=145231990


[-- Attachment #1.2: Type: text/html, Size: 6916 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs