* XFS problem
@ 2012-01-20 17:55 Kelbel Junior
2012-01-24 14:05 ` Jan Kara
2012-01-24 21:39 ` Christoph Hellwig
0 siblings, 2 replies; 8+ messages in thread
From: Kelbel Junior @ 2012-01-20 17:55 UTC (permalink / raw)
To: linux-kernel
Hello again ...
So ... after changing the memory allocator SLUB to SLAB in the Linux
kernel and apply the patch mentioned below, the system was stable for
3 days and today it happened again delay to deliver content on the
disc.
When I went to see if there was something wrong in /var/log/messages
appear several times this block:
server012 kernel: XFS: possible memory allocation deadlock in
kmem_alloc (mode:0x250)
server012 kernel: Pid: 24885, comm: kworker/0:3 Tainted: G W 2.6.38.8-cve #2
server012 kernel: Call Trace:
server012 kernel: [<ffffffff811cfc65>] ? kmem_alloc+0xa6/0xb4
server012 kernel: [<ffffffff811cfc91>] ? kmem_realloc+0x1e/0x51
server012 kernel: [<ffffffff811b7b4e>] ?
xfs_iext_realloc_indirect+0x3e/0x46
server012 kernel: [<ffffffff811b7cf6>] ? xfs_iext_irec_new+0x36/0xcf
server012 kernel: [<ffffffff811cfc26>] ? kmem_alloc+0x67/0xb4
server012 kernel: [<ffffffff811b8014>] ?
xfs_iext_add_indirect_multi+0x195/0x1de
server012 kernel: [<ffffffff811b8a66>] ? xfs_iext_add+0x1a6/0x1e2
server012 kernel: [<ffffffff811a4523>] ? xfs_btree_update+0x68/0xb5
server012 kernel: [<ffffffff811b8b36>] ? xfs_iext_insert+0x34/0x5f
server012 kernel: [<ffffffff8119de57>] ?
xfs_bmap_add_extent_unwritten_real+0xace/0xc47
server012 kernel: [<ffffffff811b80e2>] ? xfs_iext_bno_to_irec+0x85/0xae
server012 kernel: [<ffffffff8119f367>] ? xfs_bmap_add_extent+0x1fd/0x326
server012 kernel: [<ffffffff811cfb13>] ? kmem_zone_alloc+0x67/0xb4
server012 kernel: [<ffffffff811a30b7>] ? xfs_bmbt_init_cursor+0x3a/0x110
server012 kernel: [<ffffffff811a1d9b>] ? xfs_bmapi+0x9ff/0xd3e
server012 kernel: [<ffffffff811bc4a1>] ?
xfs_iomap_write_unwritten+0x1d8/0x27d
server012 kernel: [<ffffffff8120a7c6>] ? kobject_put+0x47/0x4c
server012 kernel: [<ffffffff812dda57>] ? put_device+0x12/0x14
server012 kernel: [<ffffffff811d02b0>] ? xfs_end_io+0x42/0x91
server012 kernel: [<ffffffff81048789>] ? process_one_work+0x189/0x25e
server012 kernel: [<ffffffff811d026e>] ? xfs_end_io+0x0/0x91
server012 kernel: [<ffffffff81048b14>] ? worker_thread+0x132/0x24f
server012 kernel: [<ffffffff810489e2>] ? worker_thread+0x0/0x24f
server012 kernel: [<ffffffff8104d55f>] ? kthread+0x81/0x89
server012 kernel: [<ffffffff81003754>] ? kernel_thread_helper+0x4/0x10
server012 kernel: [<ffffffff8104d4de>] ? kthread+0x0/0x89
server012 kernel: [<ffffffff81003750>] ? kernel_thread_helper+0x0/0x10
Has anyone seen or had this problem?
below "uname" on host:
Linux server012 2.6.38.8 #2 SMP Tue Jan 17 10:17:05 BRST 2012 x86_64
Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz GenuineIntel GNU/Linux
Thanks.
Kelbel Junior
2012/1/16 Kelbel Junior <jymmyjr@gmail.com>:
> Hello, everybody.
> It appeared that same error message when using kernel version 2.6.38.8 ...
>
> Several lines in /var/log/messages with:
> kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
>
> Does anyone know know if this bug has been fixed in newer versions of Linux
> kernel? 3.x maybe?
>
>
> 2011/5/11 Christoph Hellwig <>
>>
>> And here's the actual patch, sorry:
>>
>>
>> Index: linux-2.6/fs/xfs/linux-2.6/kmem.c
>> ===================================================================
>> --- linux-2.6.orig/fs/xfs/linux-2.6/kmem.c 2011-05-11 17:29:51.729191621
>> +0200
>>
>> +++ linux-2.6/fs/xfs/linux-2.6/kmem.c 2011-05-11 17:30:22.915689382 +0200
>> @@ -56,10 +56,12 @@ kmem_alloc(size_t size, unsigned int __n
>> ptr = kmalloc(size, lflags);
>> if (ptr || (flags & (KM_MAYFAIL|KM_NOSLEEP)))
>>
>> return ptr;
>> - if (!(++retries % 100))
>> + if (!(++retries % 100)) {
>> xfs_err(NULL,
>> "possible memory allocation deadlock in %s (mode:0x%x)",
>> __func__, lflags);
>> + dump_stack();
>>
>> + }
>> congestion_wait(BLK_RW_ASYNC, HZ/50);
>> } while (1);
>> }
>> @@ -112,10 +114,12 @@ kmem_zone_alloc(kmem_zone_t *zone, unsig
>> ptr = kmem_cache_alloc(zone, lflags);
>> if (ptr || (flags & (KM_MAYFAIL|KM_NOSLEEP)))
>>
>> return ptr;
>> - if (!(++retries % 100))
>> + if (!(++retries % 100)) {
>> xfs_err(NULL,
>> "possible memory allocation deadlock in %s (mode:0x%x)",
>> __func__, lflags);
>> + dump_stack();
>>
>> + }
>> congestion_wait(BLK_RW_ASYNC, HZ/50);
>> } while (1);
>> }
>>
>>
>> 2011/5/11 Christoph Hellwig <>
>>>
>>> You're probably running into a different issue. Any chance you can try
>>> the attached patch to see where you're stuck exactly?
>>>
>>> Also in case you're using slub (CONFIG_SLUB) can you check if it still
>>>
>>> happens when using slab (CONFIG_SLAB) instead?
>>>
>>>
>>> 2011/5/10 Felipe Wilhelms Damasio - Taghos <>
>>>>
>>>> Hi Mr. Hellwig,
>>>>
>>>> I saw the XFS deadlock thread regarding 2.6.38:
>>>>
>>>> http://www.linux.sgi.com/archives/xfs/2011-03/msg00185.html
>>>>
>>>>
>>>> I'm seeing too in 2.6.38.4. A lot of
>>>>
>>>> XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
>>>>
>>>> I checked the changelog for 2.6.38.6 and can't be sure if the fix
>>>> got
>>>> in...could you tell me if it did?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>>
>>>> Felipe Wilhelms Damasio
>>>>
>>>>
>>>> TAGHOS - Tecnologia
>>>> Rua Prof. Alvaro Alvim, 211
>>>> Porto Alegre - RS - (51) 3239-3180
>>>> www.taghos.com.br <http://www.taghos.com.br/>
>>>>
>>>
>>
>
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: XFS problem 2012-01-20 17:55 XFS problem Kelbel Junior @ 2012-01-24 14:05 ` Jan Kara 2012-01-24 21:39 ` Christoph Hellwig 1 sibling, 0 replies; 8+ messages in thread From: Jan Kara @ 2012-01-24 14:05 UTC (permalink / raw) To: Kelbel Junior; +Cc: linux-kernel, xfs Just adding XFS list to catch more attention... On Fri 20-01-12 15:55:18, Kelbel Junior wrote: > Hello again ... > > So ... after changing the memory allocator SLUB to SLAB in the Linux > kernel and apply the patch mentioned below, the system was stable for > 3 days and today it happened again delay to deliver content on the > disc. > > When I went to see if there was something wrong in /var/log/messages > appear several times this block: > > server012 kernel: XFS: possible memory allocation deadlock in > kmem_alloc (mode:0x250) > server012 kernel: Pid: 24885, comm: kworker/0:3 Tainted: G W 2.6.38.8-cve #2 > server012 kernel: Call Trace: > server012 kernel: [<ffffffff811cfc65>] ? kmem_alloc+0xa6/0xb4 > server012 kernel: [<ffffffff811cfc91>] ? kmem_realloc+0x1e/0x51 > server012 kernel: [<ffffffff811b7b4e>] ? > xfs_iext_realloc_indirect+0x3e/0x46 > server012 kernel: [<ffffffff811b7cf6>] ? xfs_iext_irec_new+0x36/0xcf > server012 kernel: [<ffffffff811cfc26>] ? kmem_alloc+0x67/0xb4 > server012 kernel: [<ffffffff811b8014>] ? > xfs_iext_add_indirect_multi+0x195/0x1de > server012 kernel: [<ffffffff811b8a66>] ? xfs_iext_add+0x1a6/0x1e2 > server012 kernel: [<ffffffff811a4523>] ? xfs_btree_update+0x68/0xb5 > server012 kernel: [<ffffffff811b8b36>] ? xfs_iext_insert+0x34/0x5f > server012 kernel: [<ffffffff8119de57>] ? > xfs_bmap_add_extent_unwritten_real+0xace/0xc47 > server012 kernel: [<ffffffff811b80e2>] ? xfs_iext_bno_to_irec+0x85/0xae > server012 kernel: [<ffffffff8119f367>] ? xfs_bmap_add_extent+0x1fd/0x326 > server012 kernel: [<ffffffff811cfb13>] ? kmem_zone_alloc+0x67/0xb4 > server012 kernel: [<ffffffff811a30b7>] ? xfs_bmbt_init_cursor+0x3a/0x110 > server012 kernel: [<ffffffff811a1d9b>] ? xfs_bmapi+0x9ff/0xd3e > server012 kernel: [<ffffffff811bc4a1>] ? > xfs_iomap_write_unwritten+0x1d8/0x27d > server012 kernel: [<ffffffff8120a7c6>] ? kobject_put+0x47/0x4c > server012 kernel: [<ffffffff812dda57>] ? put_device+0x12/0x14 > server012 kernel: [<ffffffff811d02b0>] ? xfs_end_io+0x42/0x91 > server012 kernel: [<ffffffff81048789>] ? process_one_work+0x189/0x25e > server012 kernel: [<ffffffff811d026e>] ? xfs_end_io+0x0/0x91 > server012 kernel: [<ffffffff81048b14>] ? worker_thread+0x132/0x24f > server012 kernel: [<ffffffff810489e2>] ? worker_thread+0x0/0x24f > server012 kernel: [<ffffffff8104d55f>] ? kthread+0x81/0x89 > server012 kernel: [<ffffffff81003754>] ? kernel_thread_helper+0x4/0x10 > server012 kernel: [<ffffffff8104d4de>] ? kthread+0x0/0x89 > server012 kernel: [<ffffffff81003750>] ? kernel_thread_helper+0x0/0x10 > > Has anyone seen or had this problem? > > below "uname" on host: > Linux server012 2.6.38.8 #2 SMP Tue Jan 17 10:17:05 BRST 2012 x86_64 > Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz GenuineIntel GNU/Linux > > > > Thanks. > Kelbel Junior > > > > > 2012/1/16 Kelbel Junior <jymmyjr@gmail.com>: > > Hello, everybody. > > It appeared that same error message when using kernel version 2.6.38.8 ... > > > > Several lines in /var/log/messages with: > > kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250) > > > > Does anyone know know if this bug has been fixed in newer versions of Linux > > kernel? 3.x maybe? > > > > > > 2011/5/11 Christoph Hellwig <> > >> > >> And here's the actual patch, sorry: > >> > >> > >> Index: linux-2.6/fs/xfs/linux-2.6/kmem.c > >> =================================================================== > >> --- linux-2.6.orig/fs/xfs/linux-2.6/kmem.c 2011-05-11 17:29:51.729191621 > >> +0200 > >> > >> +++ linux-2.6/fs/xfs/linux-2.6/kmem.c 2011-05-11 17:30:22.915689382 +0200 > >> @@ -56,10 +56,12 @@ kmem_alloc(size_t size, unsigned int __n > >> ptr = kmalloc(size, lflags); > >> if (ptr || (flags & (KM_MAYFAIL|KM_NOSLEEP))) > >> > >> return ptr; > >> - if (!(++retries % 100)) > >> + if (!(++retries % 100)) { > >> xfs_err(NULL, > >> "possible memory allocation deadlock in %s (mode:0x%x)", > >> __func__, lflags); > >> + dump_stack(); > >> > >> + } > >> congestion_wait(BLK_RW_ASYNC, HZ/50); > >> } while (1); > >> } > >> @@ -112,10 +114,12 @@ kmem_zone_alloc(kmem_zone_t *zone, unsig > >> ptr = kmem_cache_alloc(zone, lflags); > >> if (ptr || (flags & (KM_MAYFAIL|KM_NOSLEEP))) > >> > >> return ptr; > >> - if (!(++retries % 100)) > >> + if (!(++retries % 100)) { > >> xfs_err(NULL, > >> "possible memory allocation deadlock in %s (mode:0x%x)", > >> __func__, lflags); > >> + dump_stack(); > >> > >> + } > >> congestion_wait(BLK_RW_ASYNC, HZ/50); > >> } while (1); > >> } > >> > >> > >> 2011/5/11 Christoph Hellwig <> > >>> > >>> You're probably running into a different issue. Any chance you can try > >>> the attached patch to see where you're stuck exactly? > >>> > >>> Also in case you're using slub (CONFIG_SLUB) can you check if it still > >>> > >>> happens when using slab (CONFIG_SLAB) instead? > >>> > >>> > >>> 2011/5/10 Felipe Wilhelms Damasio - Taghos <> > >>>> > >>>> Hi Mr. Hellwig, > >>>> > >>>> I saw the XFS deadlock thread regarding 2.6.38: > >>>> > >>>> http://www.linux.sgi.com/archives/xfs/2011-03/msg00185.html > >>>> > >>>> > >>>> I'm seeing too in 2.6.38.4. A lot of > >>>> > >>>> XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) > >>>> > >>>> I checked the changelog for 2.6.38.6 and can't be sure if the fix > >>>> got > >>>> in...could you tell me if it did? > >>>> > >>>> > >>>> Thanks, > >>>> > >>>> -- > >>>> > >>>> Felipe Wilhelms Damasio > >>>> > >>>> > >>>> TAGHOS - Tecnologia > >>>> Rua Prof. Alvaro Alvim, 211 > >>>> Porto Alegre - RS - (51) 3239-3180 > >>>> www.taghos.com.br <http://www.taghos.com.br/> > >>>> > >>> > >> > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- Jan Kara <jack@suse.cz> SUSE Labs, CR ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: XFS problem 2012-01-20 17:55 XFS problem Kelbel Junior 2012-01-24 14:05 ` Jan Kara @ 2012-01-24 21:39 ` Christoph Hellwig 2012-01-26 15:57 ` Kelbel Junior 1 sibling, 1 reply; 8+ messages in thread From: Christoph Hellwig @ 2012-01-24 21:39 UTC (permalink / raw) To: Kelbel Junior; +Cc: linux-kernel, xfs On Fri, Jan 20, 2012 at 03:55:18PM -0200, Kelbel Junior wrote: > Hello again ... > > So ... after changing the memory allocator SLUB to SLAB in the Linux > kernel and apply the patch mentioned below, the system was stable for > 3 days and today it happened again delay to deliver content on the > disc. > > When I went to see if there was something wrong in /var/log/messages > appear several times this block: Can you try the patch below to make the extent indirection array reallocation more efficient? It's still working around the symptoms, though. What kind of workload are you running to get this extremely high number of extents? Index: xfs/fs/xfs/kmem.c =================================================================== --- xfs.orig/fs/xfs/kmem.c 2011-10-17 09:28:57.159149025 +0200 +++ xfs/fs/xfs/kmem.c 2012-01-24 21:04:50.405954910 +0100 @@ -86,19 +86,22 @@ kmem_free(const void *ptr) } void * -kmem_realloc(const void *ptr, size_t newsize, size_t oldsize, - unsigned int __nocast flags) +kmem_realloc(const void *old, size_t newsize, unsigned int __nocast flags) { + gfp_t lflags = kmem_flags_convert(flags); + int retries = 0; void *new; - new = kmem_alloc(newsize, flags); - if (ptr) { - if (new) - memcpy(new, ptr, - ((oldsize < newsize) ? oldsize : newsize)); - kmem_free(ptr); - } - return new; + do { + new = krealloc(old, newsize, lflags); + if (new || (flags & (KM_MAYFAIL|KM_NOSLEEP))) + return new; + if (!(++retries % 100)) + xfs_err(NULL, + "possible memory allocation deadlock in %s (mode:0x%x)", + __func__, lflags); + congestion_wait(BLK_RW_ASYNC, HZ/50); + } while (1); } void * Index: xfs/fs/xfs/kmem.h =================================================================== --- xfs.orig/fs/xfs/kmem.h 2011-12-18 23:35:30.893167926 +0100 +++ xfs/fs/xfs/kmem.h 2012-01-24 21:00:07.865956600 +0100 @@ -56,7 +56,7 @@ kmem_flags_convert(unsigned int __nocast extern void *kmem_alloc(size_t, unsigned int __nocast); extern void *kmem_zalloc(size_t, unsigned int __nocast); -extern void *kmem_realloc(const void *, size_t, size_t, unsigned int __nocast); +extern void *kmem_realloc(const void *, size_t, unsigned int __nocast); extern void kmem_free(const void *); static inline void *kmem_zalloc_large(size_t size) Index: xfs/fs/xfs/xfs_inode.c =================================================================== --- xfs.orig/fs/xfs/xfs_inode.c 2012-01-24 20:57:27.079290895 +0100 +++ xfs/fs/xfs/xfs_inode.c 2012-01-24 21:02:48.145955643 +0100 @@ -1826,7 +1826,6 @@ xfs_iroot_realloc( new_max = cur_max + rec_diff; new_size = (size_t)XFS_BMAP_BROOT_SPACE_CALC(new_max); ifp->if_broot = kmem_realloc(ifp->if_broot, new_size, - (size_t)XFS_BMAP_BROOT_SPACE_CALC(cur_max), /* old size */ KM_SLEEP | KM_NOFS); op = (char *)XFS_BMAP_BROOT_PTR_ADDR(mp, ifp->if_broot, 1, ifp->if_broot_bytes); @@ -1968,7 +1967,6 @@ xfs_idata_realloc( ifp->if_u1.if_data = kmem_realloc(ifp->if_u1.if_data, real_size, - ifp->if_real_bytes, KM_SLEEP | KM_NOFS); } } else { @@ -3254,8 +3252,7 @@ xfs_iext_realloc_direct( if (rnew_size != ifp->if_real_bytes) { ifp->if_u1.if_extents = kmem_realloc(ifp->if_u1.if_extents, - rnew_size, - ifp->if_real_bytes, KM_NOFS); + rnew_size, KM_NOFS); } if (rnew_size > ifp->if_real_bytes) { memset(&ifp->if_u1.if_extents[ifp->if_bytes / @@ -3333,20 +3330,15 @@ xfs_iext_realloc_indirect( xfs_ifork_t *ifp, /* inode fork pointer */ int new_size) /* new indirection array size */ { - int nlists; /* number of irec's (ex lists) */ - int size; /* current indirection array size */ - ASSERT(ifp->if_flags & XFS_IFEXTIREC); - nlists = ifp->if_real_bytes / XFS_IEXT_BUFSZ; - size = nlists * sizeof(xfs_ext_irec_t); ASSERT(ifp->if_real_bytes); - ASSERT((new_size >= 0) && (new_size != size)); + ASSERT(new_size >= 0); + if (new_size == 0) { xfs_iext_destroy(ifp); } else { - ifp->if_u1.if_ext_irec = (xfs_ext_irec_t *) - kmem_realloc(ifp->if_u1.if_ext_irec, - new_size, size, KM_NOFS); + ifp->if_u1.if_ext_irec = + kmem_realloc(ifp->if_u1.if_ext_irec, new_size, KM_NOFS); } } Index: xfs/fs/xfs/xfs_log_recover.c =================================================================== --- xfs.orig/fs/xfs/xfs_log_recover.c 2011-12-18 23:35:30.906501259 +0100 +++ xfs/fs/xfs/xfs_log_recover.c 2012-01-24 21:02:20.352622478 +0100 @@ -1489,7 +1489,7 @@ xlog_recover_add_to_cont_trans( old_ptr = item->ri_buf[item->ri_cnt-1].i_addr; old_len = item->ri_buf[item->ri_cnt-1].i_len; - ptr = kmem_realloc(old_ptr, len+old_len, old_len, 0u); + ptr = kmem_realloc(old_ptr, len+old_len, KM_SLEEP); memcpy(&ptr[old_len], dp, len); /* d, s, l */ item->ri_buf[item->ri_cnt-1].i_len += len; item->ri_buf[item->ri_cnt-1].i_addr = ptr; Index: xfs/fs/xfs/xfs_mount.c =================================================================== --- xfs.orig/fs/xfs/xfs_mount.c 2011-12-18 23:35:30.906501259 +0100 +++ xfs/fs/xfs/xfs_mount.c 2012-01-24 21:05:29.699288013 +0100 @@ -147,7 +147,6 @@ xfs_uuid_mount( if (hole < 0) { xfs_uuid_table = kmem_realloc(xfs_uuid_table, (xfs_uuid_table_size + 1) * sizeof(*xfs_uuid_table), - xfs_uuid_table_size * sizeof(*xfs_uuid_table), KM_SLEEP); hole = xfs_uuid_table_size++; } ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: XFS problem 2012-01-24 21:39 ` Christoph Hellwig @ 2012-01-26 15:57 ` Kelbel Junior 2012-01-27 10:58 ` Christoph Hellwig 0 siblings, 1 reply; 8+ messages in thread From: Kelbel Junior @ 2012-01-26 15:57 UTC (permalink / raw) To: Christoph Hellwig; +Cc: linux-kernel, xfs Well, in 24/01 i upgrade to the kernel 3.2.1 in one server and forgot read my emails...until now it's running without problems. Then, yesterday i put kernel 3.2.1 on another server and this morning a several same messages "server013 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)" and delays. (without call trace because i forgot to apply that patch with dump_stack) Now i applied the dump_stack and the latest patches that you sent. I'm trying to discover what triggers this bug, to help in a solution... I tell you any news *** Some useful information: # mount /dev/sdb1 on MOUNTPOINT type xfs (rw,noatime) But in some cases i use RAID and mount the partition with the following settings (and happen the same problem): /dev/mapper/server012_vg-server012_vol on MOUNTPOINT type xfs (rw,noatime,nodiratime,inode64,nobarrier) HD model: Seagate ST31000528AS -- Barracuda 7200.12 SATA 3Gb/s 1TB Hard Drive I work in a cache solution company, so I/O performance is very important in our context. Thanks. Kelbel Junior ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: XFS problem 2012-01-26 15:57 ` Kelbel Junior @ 2012-01-27 10:58 ` Christoph Hellwig [not found] ` <CAAA8XhOoF_AN_-1PsYQ+hp7adc08YuYqvA_3Bv9nqtuOzO1kOg@mail.gmail.com> 0 siblings, 1 reply; 8+ messages in thread From: Christoph Hellwig @ 2012-01-27 10:58 UTC (permalink / raw) To: Kelbel Junior; +Cc: Christoph Hellwig, linux-kernel, xfs On Thu, Jan 26, 2012 at 01:57:34PM -0200, Kelbel Junior wrote: > Well, in 24/01 i upgrade to the kernel 3.2.1 in one server and forgot > read my emails...until now it's running without problems. > Then, yesterday i put kernel 3.2.1 on another server and this morning > a several same messages "server013 kernel: XFS: possible memory > allocation deadlock in kmem_alloc (mode:0x250)" and delays. > > (without call trace because i forgot to apply that patch with dump_stack) Not a problem, we know where it comes from now. > I work in a cache solution company, so I/O performance is very > important in our context. What I'm really curious about is what kind of workloads you have. We should only run into problems here if we have a huge extent indirection array, which points to a massively fragmented file. Right now the handling of that isn't optimal, and we need to improve on that. But you'd probably get better results by avoiding that massive fragmentaion in the first place, e.g. try to preallocate or set extent size hints if you do random writes to a sparse file. ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <CAAA8XhOoF_AN_-1PsYQ+hp7adc08YuYqvA_3Bv9nqtuOzO1kOg@mail.gmail.com>]
[parent not found: <20120127191532.GA25529@infradead.org>]
* Re: XFS problem [not found] ` <20120127191532.GA25529@infradead.org> @ 2012-01-30 14:31 ` Kelbel Junior 2012-01-30 20:34 ` Kelbel Junior 0 siblings, 1 reply; 8+ messages in thread From: Kelbel Junior @ 2012-01-30 14:31 UTC (permalink / raw) To: Christoph Hellwig; +Cc: linux-kernel, xfs Hi, Christoph. Yes, we work on 1TB file, doing lots of random I/O over a single file. We are using 'pwrite' ofr each write operation. Output of 'filefrag' in this file: file: 1364 extents found The 'mount' options are the same as mentioned earlier. About RAID, we use RAID 0. In a file with this size and this extents amount could happen the problem that we're seeing? Thanks. Kelbel Junior ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: XFS problem 2012-01-30 14:31 ` Kelbel Junior @ 2012-01-30 20:34 ` Kelbel Junior 2012-01-31 9:27 ` Christoph Hellwig 0 siblings, 1 reply; 8+ messages in thread From: Kelbel Junior @ 2012-01-30 20:34 UTC (permalink / raw) To: Christoph Hellwig; +Cc: linux-kernel, xfs Hi again, Christoph. How you prefer to pre allocate space? xfs I/O or by 'dd' to do it? What other information you would like to better understand and manage to solve this problem? Below sending the first lines of the output 'filefrag -v cachefile' to check if there is some useful information ... if you want, I can make available for download a file with the output of the 'filefrag-v' #### Filesystem type is: 58465342 File size of cachefile is 885075181568 (216082808 blocks, blocksize 4096) ext logical physical expected length flags 0 0 37651 1 1 8 8356011 37651 2 2 10 42985973 8356012 5 3 16 8356008 42985977 1 4 24 14384348 8356008 3 5 27 42978781 14384350 4 6 31 49416416 42978784 1 7 32 14384398 49416416 3 8 35 42985981 14384400 3 9 40 14384407 42985983 2 10 42 42990181 14384408 4 11 48 14384433 42990184 3 12 56 14384445 14384435 2 13 64 14384470 14384446 2 14 66 43029387 14384471 2 15 72 14384492 43029388 2 16 80 14384556 14384493 2 17 82 43108186 14384557 2 18 88 14388688 43108187 3 19 96 14391631 14388690 3 20 104 14391634 1 21 112 14391635 1 22 113 41118314 14391635 1 23 120 14391625 41118314 1 24 121 43107765 14391625 5 25 128 14391888 43107769 7 26 136 14391793 14391894 6 27 142 14391806 14391798 1 28 144 14391802 14391806 4 29 152 14391810 14391805 4 30 160 14391818 14391813 3 31 168 14391826 14391820 4 32 176 14391830 1 33 184 30960375 14391830 13 34 200 30960391 30960387 2 35 202 43111672 30960392 5 36 208 30960399 43111676 2 37 210 42949345 30960400 3 38 216 30960407 42949347 3 39 224 30960415 30960409 3 40 232 30960423 30960417 3 41 240 30960431 30960425 3 42 248 30960439 30960433 6 43 256 30960447 30960444 3 44 264 30960455 30960449 2 45 266 41079292 30960456 1 46 272 30960463 41079292 3 47 275 40779450 30960465 1 #### Another fact, tried to run a 'xfs_bmap' file on my 1TB (not sure if the output of this command would be relevant to you) and got the following message: # xfs_bmap -pv cachefile xfs_bmap: xfsctl(XFS_IOC_GETBMAPX) iflags=0x4 ["file"]: Unable to allocate memory Thanks. Kelbel Junior ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: XFS problem 2012-01-30 20:34 ` Kelbel Junior @ 2012-01-31 9:27 ` Christoph Hellwig 0 siblings, 0 replies; 8+ messages in thread From: Christoph Hellwig @ 2012-01-31 9:27 UTC (permalink / raw) To: Kelbel Junior; +Cc: Christoph Hellwig, linux-kernel, xfs On Mon, Jan 30, 2012 at 06:34:52PM -0200, Kelbel Junior wrote: > How you prefer to pre allocate space? > xfs I/O or by 'dd' to do it? > > What other information you would like to better understand and manage > to solve this problem? Do you expect to fill most of the file with data? In that case preallocating the file before starting I/O is the best way to do it. >From the shell you can do it using the xfs_io 'resvsp' command, and from C code using the fallocate system call, or if it's not present the xfs RESVSP ioctl, as documented by the xfsctl manpage when you have the xfsprogs development libraries installed. If you actually want to keep the file sparse it's much harder. You could try larger I/O sized, or trying using the xfs extent size hint which is set using the XFS_IOC_FSSETXATTR ioctl, again documented in the xfsctl man page. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-01-31 9:27 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-01-20 17:55 XFS problem Kelbel Junior
2012-01-24 14:05 ` Jan Kara
2012-01-24 21:39 ` Christoph Hellwig
2012-01-26 15:57 ` Kelbel Junior
2012-01-27 10:58 ` Christoph Hellwig
[not found] ` <CAAA8XhOoF_AN_-1PsYQ+hp7adc08YuYqvA_3Bv9nqtuOzO1kOg@mail.gmail.com>
[not found] ` <20120127191532.GA25529@infradead.org>
2012-01-30 14:31 ` Kelbel Junior
2012-01-30 20:34 ` Kelbel Junior
2012-01-31 9:27 ` Christoph Hellwig
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).