mkfs.xfs -n size=65536

mkfs.xfs -n size=65536

[Al Lau (alau2)]

[-- Attachment #1.1: Type: text/plain, Size: 2426 bytes --]

I am looking for more details on the "-n size=65536" option in mkfs.xfs.  The question is the memory allocation this option generates.  The system is Redhat EL 7.0 (3.10.0-229.1.2.el7.x86_64).

We have been getting this memory allocation deadlock message in the /var/log/messages file.  The file system is used for ceph OSD and it has about 531894 files.

Oct  6 07:11:09 abc-ceph1-xyz kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)

Reading this post, https://access.redhat.com/solutions/532663 it discussed the "memory allocation deadlock" message.  It looks like the root cause is a very fragmented file.  Is setting the "-n size=65536" option on the file system causes (directly or indirectly) the "memory allocation deadlock" error?

Here is a note on the "-n size=65536".  The FAQ note is more on the CPU and IO performance.

http://xfs.org/index.php/XFS_FAQ
"
Q: Performance: mkfs.xfs -n size=64k option

Asking the implications of that mkfs option on the XFS mailing list,
Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name
and the inode number.  Hence the amount of space used by a directory entry is
determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of
one, to begin with, then there's the vmap overhead, etc), so for small
directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly
50% less CPU that 64k block size directories. The 4k block size directories
consume less CPU out to roughly 1.5 million entries where the two are roughly
equal. At directory sizes of 10 million entries, 64k directory block operations
are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more
CPU for a given lookup.  Hence it depends on your IO latency and whether
directory readahead can hide that latency as to which will be faster. e.g. For
SSDs, CPU usage might be the limiting factor, not the IO.  Right now I don't
have any numbers on what the difference might be - I'm getting 1 billion inode
population issues worked out first before I start on measuring cold cache
lookup times on 1 billion files....
"

Thanks,
Al Lau

[-- Attachment #1.2: Type: text/html, Size: 7203 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs -n size=65536

[Dave Chinner]

On Fri, Oct 09, 2015 at 10:40:00PM +0000, Al Lau (alau2) wrote:
> I am looking for more details on the "-n size=65536" option in
> mkfs.xfs.  The question is the memory allocation this option
> generates.  The system is Redhat EL 7.0
> (3.10.0-229.1.2.el7.x86_64).
> 
> We have been getting this memory allocation deadlock message in
> the /var/log/messages file.  The file system is used for ceph OSD
> and it has about 531894 files.

So, if you only have half a million files being stored, why would you
optimised the directory structure for tens of millions of files in a
single directory?

> Oct  6 07:11:09 abc-ceph1-xyz kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)

mode = ___GFP_WAIT | ___GFP_IO | ___GFP_NOWARN | ___GFP_ZERO
     = GFP_NOFS | GFP_ZERO | GFP_NOWARN

which means it's come through kmem_zalloc() and so is a heap
allocation and hence probably quite small.

Hence I doubt that has anything to do with the directory
block size, as the directory blocks are allocated as single pages
through a completely allocation different path and them virtually
mapped...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: mkfs.xfs -n size=65536

[Al Lau (alau2)]

Hi Dave,

The "-n size=65536" was introduced to our system before I arrived.  We can adjust the value with respect to the usage on our system.

I also did not think the kmme_alloc failure was related to the size=65536 setting.

I am attempting to reproduce the kmem_alloc error as requested in https://access.redhat.com/support/cases/#/case/01500239

Have a 3 TB file.  Logically divide into 1024 sections.  Each section has a process doing dd to a randomly selected 4K block in a loop.  Will this test case eventually cause the extent fragmentation that lead to the kmem_alloc message?

dd if=/var/kmem_alloc/junk of=/var/kmem_alloc/fragmented obs=4096 bs=4096 count=1 seek=604885543 conv=fsync,notrunc oflag=direct

Thanks,
-Al

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 12, 2015 5:23 PM
To: Al Lau (alau2)
Cc: xfs@oss.sgi.com
Subject: Re: mkfs.xfs -n size=65536

On Fri, Oct 09, 2015 at 10:40:00PM +0000, Al Lau (alau2) wrote:
> I am looking for more details on the "-n size=65536" option in 
> mkfs.xfs.  The question is the memory allocation this option 
> generates.  The system is Redhat EL 7.0 (3.10.0-229.1.2.el7.x86_64).
> 
> We have been getting this memory allocation deadlock message in the 
> /var/log/messages file.  The file system is used for ceph OSD and it 
> has about 531894 files.

So, if you only have half a million files being stored, why would you optimised the directory structure for tens of millions of files in a single directory?

> Oct  6 07:11:09 abc-ceph1-xyz kernel: XFS: possible memory allocation 
> deadlock in kmem_alloc (mode:0x8250)

mode = ___GFP_WAIT | ___GFP_IO | ___GFP_NOWARN | ___GFP_ZERO
     = GFP_NOFS | GFP_ZERO | GFP_NOWARN

which means it's come through kmem_zalloc() and so is a heap allocation and hence probably quite small.

Hence I doubt that has anything to do with the directory block size, as the directory blocks are allocated as single pages through a completely allocation different path and them virtually mapped...

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs -n size=65536

[Dave Chinner]

On Tue, Oct 13, 2015 at 01:39:13AM +0000, Al Lau (alau2) wrote:
> Have a 3 TB file.  Logically divide into 1024 sections.  Each
> section has a process doing dd to a randomly selected 4K block in
> a loop.  Will this test case eventually cause the extent
> fragmentation that lead to the kmem_alloc message?
> 
> dd if=/var/kmem_alloc/junk of=/var/kmem_alloc/fragmented obs=4096 bs=4096 count=1 seek=604885543 conv=fsync,notrunc oflag=direct

If you were loking for a recipe to massively fragment a file, then
you found it. And, yes, when you start to get millions of extents in
a file such as this workload will cause, you'll start having memory
allocation problems.

But I don't think that sets the GFP_ZERO flag anywhere, so that's
not necessarily where the memroy shortage is coming from. I just
committed some changes to the dev tree that allow for more detailed
information from this allocation error point to be obtained -
perhaps if woul dbe worthwhile trying a kernel build form the
current for-next tree and turning the error level up to 11?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: mkfs.xfs -n size=65536

[Al Lau (alau2)]

Hi Dave,

I can try the dev kernel with your change.  How do I go about getting the new bits?

# uname -a 
Linux abc.company.com 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 6 17:12:08 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

Thanks,
-Al

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 12, 2015 8:33 PM
To: Al Lau (alau2)
Cc: xfs@oss.sgi.com
Subject: Re: mkfs.xfs -n size=65536

On Tue, Oct 13, 2015 at 01:39:13AM +0000, Al Lau (alau2) wrote:
> Have a 3 TB file.  Logically divide into 1024 sections.  Each section 
> has a process doing dd to a randomly selected 4K block in a loop.  
> Will this test case eventually cause the extent fragmentation that 
> lead to the kmem_alloc message?
> 
> dd if=/var/kmem_alloc/junk of=/var/kmem_alloc/fragmented obs=4096 
> bs=4096 count=1 seek=604885543 conv=fsync,notrunc oflag=direct

If you were loking for a recipe to massively fragment a file, then you found it. And, yes, when you start to get millions of extents in a file such as this workload will cause, you'll start having memory allocation problems.

But I don't think that sets the GFP_ZERO flag anywhere, so that's not necessarily where the memroy shortage is coming from. I just committed some changes to the dev tree that allow for more detailed information from this allocation error point to be obtained - perhaps if woul dbe worthwhile trying a kernel build form the current for-next tree and turning the error level up to 11?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: mkfs.xfs -n size=65536

[Al Lau (alau2)]

To remedy the fragmented files in our production systems, can I run the xfs_fsr utility to de-fragment the files?

Thanks,
-Al

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 12, 2015 8:33 PM
To: Al Lau (alau2)
Cc: xfs@oss.sgi.com
Subject: Re: mkfs.xfs -n size=65536

On Tue, Oct 13, 2015 at 01:39:13AM +0000, Al Lau (alau2) wrote:
> Have a 3 TB file.  Logically divide into 1024 sections.  Each section 
> has a process doing dd to a randomly selected 4K block in a loop.  
> Will this test case eventually cause the extent fragmentation that 
> lead to the kmem_alloc message?
> 
> dd if=/var/kmem_alloc/junk of=/var/kmem_alloc/fragmented obs=4096 
> bs=4096 count=1 seek=604885543 conv=fsync,notrunc oflag=direct

If you were loking for a recipe to massively fragment a file, then you found it. And, yes, when you start to get millions of extents in a file such as this workload will cause, you'll start having memory allocation problems.

But I don't think that sets the GFP_ZERO flag anywhere, so that's not necessarily where the memroy shortage is coming from. I just committed some changes to the dev tree that allow for more detailed information from this allocation error point to be obtained - perhaps if woul dbe worthwhile trying a kernel build form the current for-next tree and turning the error level up to 11?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: mkfs.xfs -n size=65536

[Al Lau (alau2)]

I have sytemTap installed on the system.

This script is running in the background and hoping to catch the call-stack once the kemem_alloc message appears.

# stap -o /var/kmem_alloc/kmem_alloc_bt.out backtrace.stp

# cat backtrace.stp
#! /usr/bin/env stap
      2 
      3 probe module("xfs").function("xfs_err").call
      4 {
      5   print_backtrace();
      6 }
      9 
     10 


-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 12, 2015 8:33 PM
To: Al Lau (alau2)
Cc: xfs@oss.sgi.com
Subject: Re: mkfs.xfs -n size=65536

On Tue, Oct 13, 2015 at 01:39:13AM +0000, Al Lau (alau2) wrote:
> Have a 3 TB file.  Logically divide into 1024 sections.  Each section 
> has a process doing dd to a randomly selected 4K block in a loop.  
> Will this test case eventually cause the extent fragmentation that 
> lead to the kmem_alloc message?
> 
> dd if=/var/kmem_alloc/junk of=/var/kmem_alloc/fragmented obs=4096 
> bs=4096 count=1 seek=604885543 conv=fsync,notrunc oflag=direct

If you were loking for a recipe to massively fragment a file, then you found it. And, yes, when you start to get millions of extents in a file such as this workload will cause, you'll start having memory allocation problems.

But I don't think that sets the GFP_ZERO flag anywhere, so that's not necessarily where the memroy shortage is coming from. I just committed some changes to the dev tree that allow for more detailed information from this allocation error point to be obtained - perhaps if woul dbe worthwhile trying a kernel build form the current for-next tree and turning the error level up to 11?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs -n size=65536

[Dave Chinner]

On Tue, Oct 13, 2015 at 03:55:26AM +0000, Al Lau (alau2) wrote:
> To remedy the fragmented files in our production systems, can I
> run the xfs_fsr utility to de-fragment the files?

Well, first you need to determine that the files are fragmented
and that it is affecting performance....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: mkfs.xfs -n size=65536

[Al Lau (alau2)]

Are the xfs_db and filefrag the utilities to use to determine file fragmentation?

# df -k /var/kmem_alloc
Filesystem      1K-blocks       Used Available Use% Mounted on
/dev/sdf1      3905109820 3359385616 545724204  87% /var/kmem_alloc
# xfs_db -r -c frag /dev/sdf1 
actual 438970, ideal 388168, fragmentation factor 11.57%

# ls -l fragmented
-rw-r--r--. 1 root root 3360239878657 Oct 13 07:25 fragmented
# filefrag fragmented 
fragmented: 385533 extents found

Thanks,
-Al
-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 12, 2015 9:30 PM
To: Al Lau (alau2)
Cc: xfs@oss.sgi.com
Subject: Re: mkfs.xfs -n size=65536

On Tue, Oct 13, 2015 at 03:55:26AM +0000, Al Lau (alau2) wrote:
> To remedy the fragmented files in our production systems, can I run 
> the xfs_fsr utility to de-fragment the files?

Well, first you need to determine that the files are fragmented and that it is affecting performance....

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs -n size=65536

[Dave Chinner]

On Tue, Oct 13, 2015 at 07:28:48AM +0000, Al Lau (alau2) wrote:
> Are the xfs_db and filefrag the utilities to use to determine file fragmentation?
> 
> # df -k /var/kmem_alloc
> Filesystem      1K-blocks       Used Available Use% Mounted on
> /dev/sdf1      3905109820 3359385616 545724204  87% /var/kmem_alloc
> # xfs_db -r -c frag /dev/sdf1 
> actual 438970, ideal 388168, fragmentation factor 11.57%

http://xfs.org/index.php/XFS_FAQ#Q:_The_xfs_db_.22frag.22_command_says_I.27m_over_50.25._Is_that_bad.3F

> # ls -l fragmented
> -rw-r--r--. 1 root root 3360239878657 Oct 13 07:25 fragmented
> # filefrag fragmented 
> fragmented: 385533 extents found

That's a lot of extents, but for a 3TB sparse file that is being
written in random 4k blocks, that's expected and there's little you
can do about it. Preallocation of the file or use of extent size
hints will reduce physical fragmentation, but you only want to use
those if the file will eventually become non-sparse and sequential
read IO performance is required...

i.e. the definition of "fragmented" really depends on the
application, IO patterns and whether the current physical layout is
acheiving the desired performance attributes of the file in
question....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: mkfs.xfs -n size=65536

[Al Lau (alau2)]

With xfs file system that has about 1 million files, would the default value for the directory structure be sufficient?  We can remove the "-n size=<value>'" option and just use the default.

Thanks,
-Al

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, October 12, 2015 5:23 PM
To: Al Lau (alau2)
Cc: xfs@oss.sgi.com
Subject: Re: mkfs.xfs -n size=65536

On Fri, Oct 09, 2015 at 10:40:00PM +0000, Al Lau (alau2) wrote:
> I am looking for more details on the "-n size=65536" option in 
> mkfs.xfs.  The question is the memory allocation this option 
> generates.  The system is Redhat EL 7.0 (3.10.0-229.1.2.el7.x86_64).
> 
> We have been getting this memory allocation deadlock message in the 
> /var/log/messages file.  The file system is used for ceph OSD and it 
> has about 531894 files.

So, if you only have half a million files being stored, why would you optimised the directory structure for tens of millions of files in a single directory?

> Oct  6 07:11:09 abc-ceph1-xyz kernel: XFS: possible memory allocation 
> deadlock in kmem_alloc (mode:0x8250)

mode = ___GFP_WAIT | ___GFP_IO | ___GFP_NOWARN | ___GFP_ZERO
     = GFP_NOFS | GFP_ZERO | GFP_NOWARN

which means it's come through kmem_zalloc() and so is a heap allocation and hence probably quite small.

Hence I doubt that has anything to do with the directory block size, as the directory blocks are allocated as single pages through a completely allocation different path and them virtually mapped...

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: mkfs.xfs -n size=65536

[Al Lau (alau2)]

We run the systemtap script in the background.  The kmem_alloc message appeared in the /var/log/messages.  The call stack is captured by the systemtap is follow.

 0xffffffffa02a0030 : xfs_err+0x0/0x90 [xfs]
 0xffffffffa02a853d : kmem_alloc+0xbd/0xf0 [xfs]
 0xffffffffa02eeb6e : xfs_log_commit_cil+0x39e/0x4c0 [xfs]
 0xffffffffa02a7d3c : xfs_trans_commit+0x11c/0x240 [xfs]
 0xffffffffa02dd353 : xfs_remove+0x343/0x380 [xfs]
 0xffffffffa029d93a : xfs_vn_unlink+0x5a/0xc0 [xfs]
 0xffffffff811bde51
 0xffffffff811be056 (inexact)
 0xffffffff811b166e (inexact)
 0xffffffff810823cc (inexact)
 0xffffffff811c11b6 (inexact)
 0xffffffff815f3219 (inexact)

Nov  5 18:13:21 csx-ceph1-003 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)

#! /usr/bin/env stap

probe module("xfs").function("xfs_err").call
{
  print_backtrace();
}

# nohup stap -o /var/systemtap/kmem_alloc_bt.out backtrace.stp &

Redhat support case is https://access.redhat.com/support/cases/#/case/01500239

Thanks,
-Al
-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Tuesday, October 13, 2015 1:26 AM
To: Al Lau (alau2)
Cc: xfs@oss.sgi.com
Subject: Re: mkfs.xfs -n size=65536

On Tue, Oct 13, 2015 at 07:28:48AM +0000, Al Lau (alau2) wrote:
> Are the xfs_db and filefrag the utilities to use to determine file fragmentation?
> 
> # df -k /var/kmem_alloc
> Filesystem      1K-blocks       Used Available Use% Mounted on
> /dev/sdf1      3905109820 3359385616 545724204  87% /var/kmem_alloc
> # xfs_db -r -c frag /dev/sdf1
> actual 438970, ideal 388168, fragmentation factor 11.57%

http://xfs.org/index.php/XFS_FAQ#Q:_The_xfs_db_.22frag.22_command_says_I.27m_over_50.25._Is_that_bad.3F

> # ls -l fragmented
> -rw-r--r--. 1 root root 3360239878657 Oct 13 07:25 fragmented # 
> filefrag fragmented
> fragmented: 385533 extents found

That's a lot of extents, but for a 3TB sparse file that is being written in random 4k blocks, that's expected and there's little you can do about it. Preallocation of the file or use of extent size hints will reduce physical fragmentation, but you only want to use those if the file will eventually become non-sparse and sequential read IO performance is required...

i.e. the definition of "fragmented" really depends on the application, IO patterns and whether the current physical layout is acheiving the desired performance attributes of the file in question....

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs