vmap allocation for size 1048576 failed

vmap allocation for size 1048576 failed

[Michael Weissenbacher]

Dear List!

On one of my machines i started getting errors like this after upgrading
from Kernel 2.6.35 to 2.6.36.
--- --- snip --- ---
Nov  9 11:25:57 xxxxx kernel: [591880.880679] vmap allocation for size
1048576 failed: use vmalloc=<size> to increase size.
... (repeats a few times, with differing sizes)
Nov  9 11:25:57 xxxxx kernel: [591880.881651] xfs_buf_get: failed to map
pages
... (repeats a few times)
Nov  9 11:25:57 xxxxx kernel: [591881.111625] Filesystem "loop0": XFS
internal error xfs_trans_cancel at line 1796 of file fs/xfs/xfs_trans.c.
 Caller 0xc0270288
Nov  9 11:25:57 xxxxx kernel: [591881.111629]
Nov  9 11:25:57 xxxxx kernel: [591881.111636] Pid: 24899, comm: rsync
Not tainted 2.6.36 #1
Nov  9 11:25:57 xxxxx kernel: [591881.111640] Call Trace:
Nov  9 11:25:57 xxxxx kernel: [591881.111653]  [<c0254756>]
xfs_error_report+0x2c/0x2e
Nov  9 11:25:57 xxxxx kernel: [591881.111663]  [<c026c691>]
xfs_trans_cancel+0x4b/0xc9
Nov  9 11:25:57 xxxxx kernel: [591881.111674]  [<c0270288>] ?
xfs_create+0x473/0x4ef
Nov  9 11:25:57 xxxxx kernel: [591881.111683]  [<c0270288>]
xfs_create+0x473/0x4ef
Nov  9 11:25:57 xxxxx kernel: [591881.111693]  [<c02791c7>]
xfs_vn_mknod+0xca/0x156
Nov  9 11:25:57 xxxxx kernel: [591881.111700]  [<c027926c>]
xfs_vn_create+0xa/0xc
Nov  9 11:25:57 xxxxx kernel: [591881.111709]  [<c01bcedd>]
vfs_create+0x85/0xcf
Nov  9 11:25:57 xxxxx kernel: [591881.111716]  [<c01bd56e>]
do_last+0x277/0x4f8
Nov  9 11:25:57 xxxxx kernel: [591881.111725]  [<c01bed11>]
do_filp_open+0x197/0x458
Nov  9 11:25:57 xxxxx kernel: [591881.111733]  [<c01bd0fb>] ?
getname+0x1b/0xb9
Nov  9 11:25:57 xxxxx kernel: [591881.111741]  [<c01b3766>]
do_sys_open+0x48/0xc9
Nov  9 11:25:57 xxxxx kernel: [591881.111746]  [<c01b3829>]
sys_open+0x1e/0x26
Nov  9 11:25:57 xxxxx kernel: [591881.111752]  [<c0102798>]
sysenter_do_call+0x12/0x28
Nov  9 11:25:57 xxxxx kernel: [591881.111759]
xfs_force_shutdown(loop0,0x8) called from line 1797 of file
fs/xfs/xfs_trans.c.  Return address = 0xc026c6a7
Nov  9 11:25:57 xxxxx kernel: [591881.214442] Filesystem "loop0":
Corruption of in-memory data detected.  Shutting down filesystem: loop0
Nov  9 11:25:57 xxxxx kernel: [591881.214449] Please umount the
filesystem, and rectify the problem(s)
Nov  9 11:26:11 xxxxx kernel: [591894.827022] Filesystem "loop0":
xfs_log_force: error 5 returned.
Nov  9 11:26:41 xxxxx kernel: [591924.827024] Filesystem "loop0":
xfs_log_force: error 5 returned.
Nov  9 11:27:11 xxxxx kernel: [591954.827023] Filesystem "loop0":
xfs_log_force: error 5 returned.
Nov  9 11:27:18 xxxxx kernel: [591961.412716]
xfs_force_shutdown(loop0,0x1) called from line 994 of file
fs/xfs/linux-2.6/xfs_buf.c.  Return address = 0xc0275738
Nov  9 11:27:18 xxxxx kernel: [591961.660660] Filesystem "loop0":
xfs_log_force: error 5 returned.
--- --- snip --- ---

A few thoughts:
 * It always happens when i run "rsync" on the machine.
 * I've already tried setting vmalloc=256M and vmalloc=512M which both
delayed the problem but didn't cure it.
 * It's a 32-bit machine and the problem only occurs on a loop'ed XFS,
never on the root XFS.
 * I've used the experimental "delaylog" mount option, which may be
important.
 * I'm not sure if the problem is 100% related to the kernel upgrade
because i didn't use a loop XFS on this machine before the kernel upgrade.

Any ideas how to narrow down the problem?

tia,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: vmap allocation for size 1048576 failed

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 703 bytes --]

On Dienstag, 9. November 2010 Michael Weissenbacher wrote:
> * I'm not sure if the problem is 100% related to the kernel upgrade
> because i didn't use a loop XFS on this machine before the kernel
> upgrade.

You should describe the loop setup and all devices down to the disk. 
Might be easier for the devs to help then.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Radiointerview zum Thema Spam ******
http://www.it-podcast.at/archiv.html#podcast-100716

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: vmap allocation for size 1048576 failed

[Michael Weissenbacher]

> You should describe the loop setup and all devices down to the disk. 
> Might be easier for the devs to help then.

-rw-r--r--   1 root root 2.0G Nov  9 13:01 portage-loopback

/dev/loop0 /mnt/portage xfs rw,noatime,attr2,delaylog,logbsize=256k,noquota
meta-data=/dev/loop0             isize=256    agcount=4, agsize=1048576 blks
         =                       sectsz=512   attr=2
data     =                       bsize=512    blocks=4194304, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=512    blocks=65536, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

/dev/sda3 / xfs
rw,swalloc,attr2,delaylog,logbsize=256k,sunit=128,swidth=384,usrquota,grpquota
meta-data=/dev/sda3              isize=256    agcount=4, agsize=90929920
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=363719632, imaxpct=10
         =                       sunit=16     swidth=48 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

/dev/sda is a 4-disk RAID-5 on a 3ware 8506-4LP Controller.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: vmap allocation for size 1048576 failed

[Dave Chinner]

On Tue, Nov 09, 2010 at 11:38:30AM +0100, Michael Weissenbacher wrote:
> Dear List!
> 
> On one of my machines i started getting errors like this after upgrading
> from Kernel 2.6.35 to 2.6.36.
> --- --- snip --- ---
> Nov  9 11:25:57 xxxxx kernel: [591880.880679] vmap allocation for size
> 1048576 failed: use vmalloc=<size> to increase size.
> ... (repeats a few times, with differing sizes)
> Nov  9 11:25:57 xxxxx kernel: [591880.881651] xfs_buf_get: failed to map
> pages
> ... (repeats a few times)
> Nov  9 11:25:57 xxxxx kernel: [591881.111625] Filesystem "loop0": XFS
> internal error xfs_trans_cancel at line 1796 of file fs/xfs/xfs_trans.c.
>  Caller 0xc0270288
> Nov  9 11:25:57 xxxxx kernel: [591881.111629]
> Nov  9 11:25:57 xxxxx kernel: [591881.111636] Pid: 24899, comm: rsync
> Not tainted 2.6.36 #1
> Nov  9 11:25:57 xxxxx kernel: [591881.111640] Call Trace:
> Nov  9 11:25:57 xxxxx kernel: [591881.111653]  [<c0254756>]
> xfs_error_report+0x2c/0x2e
> Nov  9 11:25:57 xxxxx kernel: [591881.111663]  [<c026c691>]
> xfs_trans_cancel+0x4b/0xc9
> Nov  9 11:25:57 xxxxx kernel: [591881.111674]  [<c0270288>] ?
> xfs_create+0x473/0x4ef

I didn't think anything other than log recovery tries to vmap
buffers. This is clearly not in log recovery. Can you post an
unedited error log, how much data you are rsyncing, the
configuration of your filesystem (xfs_info, mount options, loop dev
config, etc) to give us an idea of what you are doing to trigger
this?

> A few thoughts:
>  * It always happens when i run "rsync" on the machine.
>  * I've already tried setting vmalloc=256M and vmalloc=512M which both
> delayed the problem but didn't cure it.
>  * It's a 32-bit machine and the problem only occurs on a loop'ed XFS,
> never on the root XFS.

Can't you run on a 64-bit machine?

>  * I've used the experimental "delaylog" mount option, which may be
> important.

Should be completely irrelevant.

>  * I'm not sure if the problem is 100% related to the kernel upgrade
> because i didn't use a loop XFS on this machine before the kernel upgrade.

Can you downgrade your kernel and run the loop device there to tell
us whether this is actually a regression or not? If it is a
regression, then if you could run a bisect to find the exact patch
that causes it woul dbe very helpful....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: vmap allocation for size 1048576 failed

[Michael Weissenbacher]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

Hi Dave!

> I didn't think anything other than log recovery tries to vmap
> buffers. This is clearly not in log recovery. Can you post an
> unedited error log, how much data you are rsyncing, the
> configuration of your filesystem (xfs_info, mount options, loop dev
> config, etc) to give us an idea of what you are doing to trigger
> this?
 * I'm attaching the latest kern.log, unedited
 * I am syncing the gentoo portage tree, not much data but many small
files (currently 228MiB in 117880 files). This sync is done twice per hour.
 * I already posted my xfs_info and mount options in another post. Maybe
i should note that the loop file system was deliberately created with
blocksize=512 to accommodate the fs to the nature of the portage tree
(many small files...)
 * TBH i don't know about any special configuration of the loop device.
I just created an empty file with "dd if=/dev/zero ..." and then did
mkfs.xfs on it.

> Can't you run on a 64-bit machine?
80% of my machines are 64-bit and i never saw anything like that on
them. But otoh i dont' use loop devices very much. Unfortunately this is
machine is old hardware (P4 class) which can't run a 64-bit kernel.

> Can you downgrade your kernel and run the loop device there to tell
> us whether this is actually a regression or not? If it is a
> regression, then if you could run a bisect to find the exact patch
> that causes it woul dbe very helpful....
Already did that yesterday and i can confirm it has the same problem -
so no regression. The kern.log i attached to this mail is from Kernel
2.6.35.8

cheers,
Michael


[-- Attachment #2: kern.log.gz --]
[-- Type: application/x-gzip, Size: 72580 bytes --]

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: vmap allocation for size 1048576 failed

[Dave Chinner]

On Wed, Nov 10, 2010 at 09:16:35AM +0100, Michael Weissenbacher wrote:
> Hi Dave!
> 
> > I didn't think anything other than log recovery tries to vmap
> > buffers. This is clearly not in log recovery. Can you post an
> > unedited error log, how much data you are rsyncing, the
> > configuration of your filesystem (xfs_info, mount options, loop dev
> > config, etc) to give us an idea of what you are doing to trigger
> > this?
>  * I'm attaching the latest kern.log, unedited
>  * I am syncing the gentoo portage tree, not much data but many small
> files (currently 228MiB in 117880 files). This sync is done twice per hour.
>  * I already posted my xfs_info and mount options in another post. Maybe
> i should note that the loop file system was deliberately created with
> blocksize=512 to accommodate the fs to the nature of the portage tree
> (many small files...)
>  * TBH i don't know about any special configuration of the loop device.
> I just created an empty file with "dd if=/dev/zero ..." and then did
> mkfs.xfs on it.

Ok, thanks for the info. I'm struggling to work out what is actually
consuming vmap space given what you are doing and the storage
config. None of the operations you are doing should be triggering
mapping of multipage metadata buffers given the filesystem
configuration....

> > Can't you run on a 64-bit machine?
> 80% of my machines are 64-bit and i never saw anything like that on
> them. But otoh i dont' use loop devices very much. Unfortunately this is
> machine is old hardware (P4 class) which can't run a 64-bit kernel.
> 
> > Can you downgrade your kernel and run the loop device there to tell
> > us whether this is actually a regression or not? If it is a
> > regression, then if you could run a bisect to find the exact patch
> > that causes it woul dbe very helpful....
> Already did that yesterday and i can confirm it has the same problem -
> so no regression. The kern.log i attached to this mail is from Kernel
> 2.6.35.8

Ok, I havent looked at this yet, but having looked at the code I
don't think it will tell me anything useful. However, if you can
reproduce the problem easily, the tracing output (google for
trace-cmd and/or kernelshark) of the xfs trace events for the short
period across the error condition (just a couple of seconds before
and after the error occurs) would tell me a lot more about what is
going wrong...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: vmap allocation for size 1048576 failed

[Michael Weissenbacher]

Hi Dave!
> 
> Ok, thanks for the info. I'm struggling to work out what is actually
> consuming vmap space given what you are doing and the storage
> config. None of the operations you are doing should be triggering
> mapping of multipage metadata buffers given the filesystem
> configuration....
> 
I've set vmalloc=768M and haven't seen the problem since. Maybe the
vmalloc/vmap problem has to do with the fact that i'm running
iscsi_target on the same machine? As long as the problem doesn't occur
again i'm ok with setting the vmalloc parameter.

Thanks for your help,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: vmap allocation for size 1048576 failed

[Michael Weissenbacher]

Hi Dave!
On Wed Nov 10 2010 13:30:52 GMT+0100 (CET), Dave Chinner
<david@fromorbit.com> wrote:
> Ok, thanks for the info. I'm struggling to work out what is actually
> consuming vmap space given what you are doing and the storage
> config. None of the operations you are doing should be triggering
> mapping of multipage metadata buffers given the filesystem
> configuration....
Coming back to this, i can confirm that your patch "[PATCH 2/6] vmap:
flush vmap aliases when mapping fails" posted on March 23 2011 did cure
my problems.

with kind regards,
Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs