* vmap allocation for size 1048576 failed @ 2010-11-09 10:38 Michael Weissenbacher 2010-11-09 11:13 ` Michael Monnerie 2010-11-09 12:07 ` Dave Chinner 0 siblings, 2 replies; 8+ messages in thread From: Michael Weissenbacher @ 2010-11-09 10:38 UTC (permalink / raw) To: xfs Dear List! On one of my machines i started getting errors like this after upgrading from Kernel 2.6.35 to 2.6.36. --- --- snip --- --- Nov 9 11:25:57 xxxxx kernel: [591880.880679] vmap allocation for size 1048576 failed: use vmalloc=<size> to increase size. ... (repeats a few times, with differing sizes) Nov 9 11:25:57 xxxxx kernel: [591880.881651] xfs_buf_get: failed to map pages ... (repeats a few times) Nov 9 11:25:57 xxxxx kernel: [591881.111625] Filesystem "loop0": XFS internal error xfs_trans_cancel at line 1796 of file fs/xfs/xfs_trans.c. Caller 0xc0270288 Nov 9 11:25:57 xxxxx kernel: [591881.111629] Nov 9 11:25:57 xxxxx kernel: [591881.111636] Pid: 24899, comm: rsync Not tainted 2.6.36 #1 Nov 9 11:25:57 xxxxx kernel: [591881.111640] Call Trace: Nov 9 11:25:57 xxxxx kernel: [591881.111653] [<c0254756>] xfs_error_report+0x2c/0x2e Nov 9 11:25:57 xxxxx kernel: [591881.111663] [<c026c691>] xfs_trans_cancel+0x4b/0xc9 Nov 9 11:25:57 xxxxx kernel: [591881.111674] [<c0270288>] ? xfs_create+0x473/0x4ef Nov 9 11:25:57 xxxxx kernel: [591881.111683] [<c0270288>] xfs_create+0x473/0x4ef Nov 9 11:25:57 xxxxx kernel: [591881.111693] [<c02791c7>] xfs_vn_mknod+0xca/0x156 Nov 9 11:25:57 xxxxx kernel: [591881.111700] [<c027926c>] xfs_vn_create+0xa/0xc Nov 9 11:25:57 xxxxx kernel: [591881.111709] [<c01bcedd>] vfs_create+0x85/0xcf Nov 9 11:25:57 xxxxx kernel: [591881.111716] [<c01bd56e>] do_last+0x277/0x4f8 Nov 9 11:25:57 xxxxx kernel: [591881.111725] [<c01bed11>] do_filp_open+0x197/0x458 Nov 9 11:25:57 xxxxx kernel: [591881.111733] [<c01bd0fb>] ? getname+0x1b/0xb9 Nov 9 11:25:57 xxxxx kernel: [591881.111741] [<c01b3766>] do_sys_open+0x48/0xc9 Nov 9 11:25:57 xxxxx kernel: [591881.111746] [<c01b3829>] sys_open+0x1e/0x26 Nov 9 11:25:57 xxxxx kernel: [591881.111752] [<c0102798>] sysenter_do_call+0x12/0x28 Nov 9 11:25:57 xxxxx kernel: [591881.111759] xfs_force_shutdown(loop0,0x8) called from line 1797 of file fs/xfs/xfs_trans.c. Return address = 0xc026c6a7 Nov 9 11:25:57 xxxxx kernel: [591881.214442] Filesystem "loop0": Corruption of in-memory data detected. Shutting down filesystem: loop0 Nov 9 11:25:57 xxxxx kernel: [591881.214449] Please umount the filesystem, and rectify the problem(s) Nov 9 11:26:11 xxxxx kernel: [591894.827022] Filesystem "loop0": xfs_log_force: error 5 returned. Nov 9 11:26:41 xxxxx kernel: [591924.827024] Filesystem "loop0": xfs_log_force: error 5 returned. Nov 9 11:27:11 xxxxx kernel: [591954.827023] Filesystem "loop0": xfs_log_force: error 5 returned. Nov 9 11:27:18 xxxxx kernel: [591961.412716] xfs_force_shutdown(loop0,0x1) called from line 994 of file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xc0275738 Nov 9 11:27:18 xxxxx kernel: [591961.660660] Filesystem "loop0": xfs_log_force: error 5 returned. --- --- snip --- --- A few thoughts: * It always happens when i run "rsync" on the machine. * I've already tried setting vmalloc=256M and vmalloc=512M which both delayed the problem but didn't cure it. * It's a 32-bit machine and the problem only occurs on a loop'ed XFS, never on the root XFS. * I've used the experimental "delaylog" mount option, which may be important. * I'm not sure if the problem is 100% related to the kernel upgrade because i didn't use a loop XFS on this machine before the kernel upgrade. Any ideas how to narrow down the problem? tia, Michael _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vmap allocation for size 1048576 failed 2010-11-09 10:38 vmap allocation for size 1048576 failed Michael Weissenbacher @ 2010-11-09 11:13 ` Michael Monnerie 2010-11-09 12:02 ` Michael Weissenbacher 2010-11-09 12:07 ` Dave Chinner 1 sibling, 1 reply; 8+ messages in thread From: Michael Monnerie @ 2010-11-09 11:13 UTC (permalink / raw) To: xfs; +Cc: Michael Weissenbacher [-- Attachment #1.1: Type: Text/Plain, Size: 703 bytes --] On Dienstag, 9. November 2010 Michael Weissenbacher wrote: > * I'm not sure if the problem is 100% related to the kernel upgrade > because i didn't use a loop XFS on this machine before the kernel > upgrade. You should describe the loop setup and all devices down to the disk. Might be easier for the devs to help then. -- mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Radiointerview zum Thema Spam ****** http://www.it-podcast.at/archiv.html#podcast-100716 // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/ [-- Attachment #1.2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 198 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vmap allocation for size 1048576 failed 2010-11-09 11:13 ` Michael Monnerie @ 2010-11-09 12:02 ` Michael Weissenbacher 0 siblings, 0 replies; 8+ messages in thread From: Michael Weissenbacher @ 2010-11-09 12:02 UTC (permalink / raw) To: xfs > You should describe the loop setup and all devices down to the disk. > Might be easier for the devs to help then. -rw-r--r-- 1 root root 2.0G Nov 9 13:01 portage-loopback /dev/loop0 /mnt/portage xfs rw,noatime,attr2,delaylog,logbsize=256k,noquota meta-data=/dev/loop0 isize=256 agcount=4, agsize=1048576 blks = sectsz=512 attr=2 data = bsize=512 blocks=4194304, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=512 blocks=65536, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 /dev/sda3 / xfs rw,swalloc,attr2,delaylog,logbsize=256k,sunit=128,swidth=384,usrquota,grpquota meta-data=/dev/sda3 isize=256 agcount=4, agsize=90929920 blks = sectsz=512 attr=2 data = bsize=4096 blocks=363719632, imaxpct=10 = sunit=16 swidth=48 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=16 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 /dev/sda is a 4-disk RAID-5 on a 3ware 8506-4LP Controller. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vmap allocation for size 1048576 failed 2010-11-09 10:38 vmap allocation for size 1048576 failed Michael Weissenbacher 2010-11-09 11:13 ` Michael Monnerie @ 2010-11-09 12:07 ` Dave Chinner 2010-11-10 8:16 ` Michael Weissenbacher 1 sibling, 1 reply; 8+ messages in thread From: Dave Chinner @ 2010-11-09 12:07 UTC (permalink / raw) To: Michael Weissenbacher; +Cc: xfs On Tue, Nov 09, 2010 at 11:38:30AM +0100, Michael Weissenbacher wrote: > Dear List! > > On one of my machines i started getting errors like this after upgrading > from Kernel 2.6.35 to 2.6.36. > --- --- snip --- --- > Nov 9 11:25:57 xxxxx kernel: [591880.880679] vmap allocation for size > 1048576 failed: use vmalloc=<size> to increase size. > ... (repeats a few times, with differing sizes) > Nov 9 11:25:57 xxxxx kernel: [591880.881651] xfs_buf_get: failed to map > pages > ... (repeats a few times) > Nov 9 11:25:57 xxxxx kernel: [591881.111625] Filesystem "loop0": XFS > internal error xfs_trans_cancel at line 1796 of file fs/xfs/xfs_trans.c. > Caller 0xc0270288 > Nov 9 11:25:57 xxxxx kernel: [591881.111629] > Nov 9 11:25:57 xxxxx kernel: [591881.111636] Pid: 24899, comm: rsync > Not tainted 2.6.36 #1 > Nov 9 11:25:57 xxxxx kernel: [591881.111640] Call Trace: > Nov 9 11:25:57 xxxxx kernel: [591881.111653] [<c0254756>] > xfs_error_report+0x2c/0x2e > Nov 9 11:25:57 xxxxx kernel: [591881.111663] [<c026c691>] > xfs_trans_cancel+0x4b/0xc9 > Nov 9 11:25:57 xxxxx kernel: [591881.111674] [<c0270288>] ? > xfs_create+0x473/0x4ef I didn't think anything other than log recovery tries to vmap buffers. This is clearly not in log recovery. Can you post an unedited error log, how much data you are rsyncing, the configuration of your filesystem (xfs_info, mount options, loop dev config, etc) to give us an idea of what you are doing to trigger this? > A few thoughts: > * It always happens when i run "rsync" on the machine. > * I've already tried setting vmalloc=256M and vmalloc=512M which both > delayed the problem but didn't cure it. > * It's a 32-bit machine and the problem only occurs on a loop'ed XFS, > never on the root XFS. Can't you run on a 64-bit machine? > * I've used the experimental "delaylog" mount option, which may be > important. Should be completely irrelevant. > * I'm not sure if the problem is 100% related to the kernel upgrade > because i didn't use a loop XFS on this machine before the kernel upgrade. Can you downgrade your kernel and run the loop device there to tell us whether this is actually a regression or not? If it is a regression, then if you could run a bisect to find the exact patch that causes it woul dbe very helpful.... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vmap allocation for size 1048576 failed 2010-11-09 12:07 ` Dave Chinner @ 2010-11-10 8:16 ` Michael Weissenbacher 2010-11-10 12:30 ` Dave Chinner 0 siblings, 1 reply; 8+ messages in thread From: Michael Weissenbacher @ 2010-11-10 8:16 UTC (permalink / raw) To: xfs [-- Attachment #1: Type: text/plain, Size: 1586 bytes --] Hi Dave! > I didn't think anything other than log recovery tries to vmap > buffers. This is clearly not in log recovery. Can you post an > unedited error log, how much data you are rsyncing, the > configuration of your filesystem (xfs_info, mount options, loop dev > config, etc) to give us an idea of what you are doing to trigger > this? * I'm attaching the latest kern.log, unedited * I am syncing the gentoo portage tree, not much data but many small files (currently 228MiB in 117880 files). This sync is done twice per hour. * I already posted my xfs_info and mount options in another post. Maybe i should note that the loop file system was deliberately created with blocksize=512 to accommodate the fs to the nature of the portage tree (many small files...) * TBH i don't know about any special configuration of the loop device. I just created an empty file with "dd if=/dev/zero ..." and then did mkfs.xfs on it. > Can't you run on a 64-bit machine? 80% of my machines are 64-bit and i never saw anything like that on them. But otoh i dont' use loop devices very much. Unfortunately this is machine is old hardware (P4 class) which can't run a 64-bit kernel. > Can you downgrade your kernel and run the loop device there to tell > us whether this is actually a regression or not? If it is a > regression, then if you could run a bisect to find the exact patch > that causes it woul dbe very helpful.... Already did that yesterday and i can confirm it has the same problem - so no regression. The kern.log i attached to this mail is from Kernel 2.6.35.8 cheers, Michael [-- Attachment #2: kern.log.gz --] [-- Type: application/x-gzip, Size: 72580 bytes --] [-- Attachment #3: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vmap allocation for size 1048576 failed 2010-11-10 8:16 ` Michael Weissenbacher @ 2010-11-10 12:30 ` Dave Chinner 2010-11-18 10:50 ` Michael Weissenbacher 2011-04-29 8:51 ` Michael Weissenbacher 0 siblings, 2 replies; 8+ messages in thread From: Dave Chinner @ 2010-11-10 12:30 UTC (permalink / raw) To: Michael Weissenbacher; +Cc: xfs On Wed, Nov 10, 2010 at 09:16:35AM +0100, Michael Weissenbacher wrote: > Hi Dave! > > > I didn't think anything other than log recovery tries to vmap > > buffers. This is clearly not in log recovery. Can you post an > > unedited error log, how much data you are rsyncing, the > > configuration of your filesystem (xfs_info, mount options, loop dev > > config, etc) to give us an idea of what you are doing to trigger > > this? > * I'm attaching the latest kern.log, unedited > * I am syncing the gentoo portage tree, not much data but many small > files (currently 228MiB in 117880 files). This sync is done twice per hour. > * I already posted my xfs_info and mount options in another post. Maybe > i should note that the loop file system was deliberately created with > blocksize=512 to accommodate the fs to the nature of the portage tree > (many small files...) > * TBH i don't know about any special configuration of the loop device. > I just created an empty file with "dd if=/dev/zero ..." and then did > mkfs.xfs on it. Ok, thanks for the info. I'm struggling to work out what is actually consuming vmap space given what you are doing and the storage config. None of the operations you are doing should be triggering mapping of multipage metadata buffers given the filesystem configuration.... > > Can't you run on a 64-bit machine? > 80% of my machines are 64-bit and i never saw anything like that on > them. But otoh i dont' use loop devices very much. Unfortunately this is > machine is old hardware (P4 class) which can't run a 64-bit kernel. > > > Can you downgrade your kernel and run the loop device there to tell > > us whether this is actually a regression or not? If it is a > > regression, then if you could run a bisect to find the exact patch > > that causes it woul dbe very helpful.... > Already did that yesterday and i can confirm it has the same problem - > so no regression. The kern.log i attached to this mail is from Kernel > 2.6.35.8 Ok, I havent looked at this yet, but having looked at the code I don't think it will tell me anything useful. However, if you can reproduce the problem easily, the tracing output (google for trace-cmd and/or kernelshark) of the xfs trace events for the short period across the error condition (just a couple of seconds before and after the error occurs) would tell me a lot more about what is going wrong... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vmap allocation for size 1048576 failed 2010-11-10 12:30 ` Dave Chinner @ 2010-11-18 10:50 ` Michael Weissenbacher 2011-04-29 8:51 ` Michael Weissenbacher 1 sibling, 0 replies; 8+ messages in thread From: Michael Weissenbacher @ 2010-11-18 10:50 UTC (permalink / raw) To: xfs Hi Dave! > > Ok, thanks for the info. I'm struggling to work out what is actually > consuming vmap space given what you are doing and the storage > config. None of the operations you are doing should be triggering > mapping of multipage metadata buffers given the filesystem > configuration.... > I've set vmalloc=768M and haven't seen the problem since. Maybe the vmalloc/vmap problem has to do with the fact that i'm running iscsi_target on the same machine? As long as the problem doesn't occur again i'm ok with setting the vmalloc parameter. Thanks for your help, Michael _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: vmap allocation for size 1048576 failed 2010-11-10 12:30 ` Dave Chinner 2010-11-18 10:50 ` Michael Weissenbacher @ 2011-04-29 8:51 ` Michael Weissenbacher 1 sibling, 0 replies; 8+ messages in thread From: Michael Weissenbacher @ 2011-04-29 8:51 UTC (permalink / raw) To: xfs Hi Dave! On Wed Nov 10 2010 13:30:52 GMT+0100 (CET), Dave Chinner <david@fromorbit.com> wrote: > Ok, thanks for the info. I'm struggling to work out what is actually > consuming vmap space given what you are doing and the storage > config. None of the operations you are doing should be triggering > mapping of multipage metadata buffers given the filesystem > configuration.... Coming back to this, i can confirm that your patch "[PATCH 2/6] vmap: flush vmap aliases when mapping fails" posted on March 23 2011 did cure my problems. with kind regards, Michael _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-04-29 8:47 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-11-09 10:38 vmap allocation for size 1048576 failed Michael Weissenbacher 2010-11-09 11:13 ` Michael Monnerie 2010-11-09 12:02 ` Michael Weissenbacher 2010-11-09 12:07 ` Dave Chinner 2010-11-10 8:16 ` Michael Weissenbacher 2010-11-10 12:30 ` Dave Chinner 2010-11-18 10:50 ` Michael Weissenbacher 2011-04-29 8:51 ` Michael Weissenbacher
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.