All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: UBI memory usage on large page nand
       [not found] <AANLkTi=GsmJOjurETG930t-TW7uXLR1M6yKXWBSk7ebV@mail.gmail.com>
@ 2011-03-31  3:07 ` Nicholas
  2011-04-01 15:31   ` Artem Bityutskiy
  0 siblings, 1 reply; 14+ messages in thread
From: Nicholas @ 2011-03-31  3:07 UTC (permalink / raw)
  To: linux-mtd

Hi there,

I've a short of memory usage issue while using large page size of nand flash.
Before, i was using 2KB page size of nand flash, i found UBI/UBIFS memory
consumption on normal zone are :

7 0xC0180858:   1024K, ubi_attach_mtd_de,
8 0xC0180834:   1024K, ubi_attach_mtd_de,
9 0xC00ED300:   1008K,      ubifs_get_sb,
10 0xC00ED31C:   1008K,      ubifs_get_sb,
11 0xC00FE7EC:   1008K,    ubifs_lpt_init,
11 0xC00F95AC:   1008K, ubifs_mount_orpha,

Then, i'm using 8KB page size of nand flash, and found the result as below :
4 0xC0180858:   4096K, ubi_attach_mtd_de,
5 0xC0180834:   4096K, ubi_attach_mtd_de,
6 0xC00FE7EC:   4064K,    ubifs_lpt_init,
7 0xC00ED31C:   4064K,      ubifs_get_sb,
8 0xC00F95AC:   4064K, ubifs_mount_orpha,
9 0xC00ED300:   4064K,      ubifs_get_sb,

My environment contains 2 mtd partitions for UBI.

Before, it took 6MB of memory consumption and now it takes 24MB.
Do you have any suggestion how to reduce the memory consumption since dram size
is pretty limited for other usage.

Thank you,
Alex

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-03-31  3:07 ` UBI memory usage on large page nand Nicholas
@ 2011-04-01 15:31   ` Artem Bityutskiy
  2011-04-15  9:23     ` Nicholas
  0 siblings, 1 reply; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-01 15:31 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

Hi,

I agree UBI/UBIFS uses more memory than it could. There is some room for
optimizations.

On Thu, 2011-03-31 at 11:07 +0800, Nicholas wrote:
> Hi there,
> 
> I've a short of memory usage issue while using large page size of nand flash.
> Before, i was using 2KB page size of nand flash, i found UBI/UBIFS memory
> consumption on normal zone are :
> 
> 7 0xC0180858:   1024K, ubi_attach_mtd_de,
> 8 0xC0180834:   1024K, ubi_attach_mtd_de,
> 9 0xC00ED300:   1008K,      ubifs_get_sb,
> 10 0xC00ED31C:   1008K,      ubifs_get_sb,
> 11 0xC00FE7EC:   1008K,    ubifs_lpt_init,
> 11 0xC00F95AC:   1008K, ubifs_mount_orpha,
> 
> Then, i'm using 8KB page size of nand flash, and found the result as below :
> 4 0xC0180858:   4096K, ubi_attach_mtd_de,
> 5 0xC0180834:   4096K, ubi_attach_mtd_de,
> 6 0xC00FE7EC:   4064K,    ubifs_lpt_init,
> 7 0xC00ED31C:   4064K,      ubifs_get_sb,
> 8 0xC00F95AC:   4064K, ubifs_mount_orpha,
> 9 0xC00ED300:   4064K,      ubifs_get_sb,
> 
> My environment contains 2 mtd partitions for UBI.
> 
> Before, it took 6MB of memory consumption and now it takes 24MB.
> Do you have any suggestion how to reduce the memory consumption since dram size
> is pretty limited for other usage.

I think this is mainly because your eraseblock size became larger. UBI
and UBIFS allocate many buffers of eraseblock size for various purposes.
The buffers are often allocated once on attach/mount and released only
on detach/unmount. All of these buffers are allocated with vmalloc()
call, so you can find them by grepping.

The simplest thing you can try is to allocate these buffers when they
are actually needed, and then free them when they are not needed. This
should help.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-01 15:31   ` Artem Bityutskiy
@ 2011-04-15  9:23     ` Nicholas
  2011-04-15 13:13       ` Artem Bityutskiy
  0 siblings, 1 reply; 14+ messages in thread
From: Nicholas @ 2011-04-15  9:23 UTC (permalink / raw)
  To: dedekind1; +Cc: linux-mtd

Dear Artem,

Thanks for the information. Indeed it has flexibility to be freed or consumed.
Due to my case, i need to use maximum memory space while in the
meantime i also need to write files into nand via ubi.
8KB page of nand flash will consume 16MB for a single volume on a
single UBI MTD.

1. Is there any idea to reduce the memory consumption ?
2. For smaller buffer allocation, can it be made for several attempts
of read / write ?

Regards,
Alex

On Fri, Apr 1, 2011 at 11:31 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> Hi,
>
> I agree UBI/UBIFS uses more memory than it could. There is some room for
> optimizations.
>
> On Thu, 2011-03-31 at 11:07 +0800, Nicholas wrote:
>> Hi there,
>>
>> I've a short of memory usage issue while using large page size of nand flash.
>> Before, i was using 2KB page size of nand flash, i found UBI/UBIFS memory
>> consumption on normal zone are :
>>
>> 7 0xC0180858:   1024K, ubi_attach_mtd_de,
>> 8 0xC0180834:   1024K, ubi_attach_mtd_de,
>> 9 0xC00ED300:   1008K,      ubifs_get_sb,
>> 10 0xC00ED31C:   1008K,      ubifs_get_sb,
>> 11 0xC00FE7EC:   1008K,    ubifs_lpt_init,
>> 11 0xC00F95AC:   1008K, ubifs_mount_orpha,
>>
>> Then, i'm using 8KB page size of nand flash, and found the result as below :
>> 4 0xC0180858:   4096K, ubi_attach_mtd_de,
>> 5 0xC0180834:   4096K, ubi_attach_mtd_de,
>> 6 0xC00FE7EC:   4064K,    ubifs_lpt_init,
>> 7 0xC00ED31C:   4064K,      ubifs_get_sb,
>> 8 0xC00F95AC:   4064K, ubifs_mount_orpha,
>> 9 0xC00ED300:   4064K,      ubifs_get_sb,
>>
>> My environment contains 2 mtd partitions for UBI.
>>
>> Before, it took 6MB of memory consumption and now it takes 24MB.
>> Do you have any suggestion how to reduce the memory consumption since dram size
>> is pretty limited for other usage.
>
> I think this is mainly because your eraseblock size became larger. UBI
> and UBIFS allocate many buffers of eraseblock size for various purposes.
> The buffers are often allocated once on attach/mount and released only
> on detach/unmount. All of these buffers are allocated with vmalloc()
> call, so you can find them by grepping.
>
> The simplest thing you can try is to allocate these buffers when they
> are actually needed, and then free them when they are not needed. This
> should help.
>
> --
> Best Regards,
> Artem Bityutskiy (Артём Битюцкий)
>
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-15  9:23     ` Nicholas
@ 2011-04-15 13:13       ` Artem Bityutskiy
  2011-04-15 13:17         ` Artem Bityutskiy
  0 siblings, 1 reply; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-15 13:13 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

On Fri, 2011-04-15 at 17:23 +0800, Nicholas wrote:
> Dear Artem,
> 
> Thanks for the information. Indeed it has flexibility to be freed or consumed.
> Due to my case, i need to use maximum memory space while in the
> meantime i also need to write files into nand via ubi.
> 8KB page of nand flash will consume 16MB for a single volume on a
> single UBI MTD.
> 
> 1. Is there any idea to reduce the memory consumption ?

Yes. The main idea is as I described: both UBI and UBIFS have several
multiple PEB-size buffers at attach/mount time. I did not count them,
but I think there may be about 6 of them. All these buffers are
allocated using vmalloc(), so you can easily fine all of them, because
'vmalloc()' is used only for those buffers.

And the basic idea is to not pre-allocate them, but allocate only when
they are needed. This is the first step you can try to do to see how
much you can gain.

E.g., in ubi.h I see the following buffers to optimize:

 * @peb_buf1: a buffer of PEB size used for different purposes
 * @peb_buf2: another buffer of PEB size used for different purposes

In UBIFS

 * @ileb_buf: buffer for commit in-the-gaps method
 * @orph_buf: buffer for orphan nodes
 * @sbuf: a buffer of LEB size used by GC and replay for scanning
 * @lpt_buf: buffer of LEB size used by LPT

May be there are other buffers. I'm sure these all buffers are not
needed simultaneously, may be max. 2-3 of them may be needed
simultaneously sometimes.

And there are many smaller buffers which are pre-allocated with
kmalloc() - you can also try to allocate them on the need only.

But if the allocation fails - you are screwed :-) But this can be
mitigated - there are several ways.

> 2. For smaller buffer allocation, can it be made for several attempts
> of read / write ?

Yes, I believe in many places this is possible. But this is a bit more
difficult to do.

We have another problem - some drivers do not like vmalloc()'ed buffers
because many arm platforms cannot do dma if the buffer was vmalloc()'ed.

So the idea is to switch from vmalloc(PEB_size) to kamalloc() an array
of pages and to vectored I/O (iovec). And in many places you do not
really need RAM for whole PEB, you man allocate smaller buffer and then
iterate and do one small piece of job at the time.

Anyway, you can certainly imporve UBI/UBIFS. It'd be interesting to help
you, but I have no time, and my priority is to solve the unstable bits
problem now. But I can help you by advising and reviewing your work if
you are serious about this.

So, again, if you are serious, start with UBI because it is simpler and
try to see where you can save RAM. Make some patches and send.

Thanks!

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-15 13:13       ` Artem Bityutskiy
@ 2011-04-15 13:17         ` Artem Bityutskiy
  2011-04-15 13:24           ` Artem Bityutskiy
  2011-04-17  9:03           ` Nicholas
  0 siblings, 2 replies; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-15 13:17 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

On Fri, 2011-04-15 at 16:13 +0300, Artem Bityutskiy wrote:
> On Fri, 2011-04-15 at 17:23 +0800, Nicholas wrote:
> > Dear Artem,
> > 
> > Thanks for the information. Indeed it has flexibility to be freed or consumed.
> > Due to my case, i need to use maximum memory space while in the
> > meantime i also need to write files into nand via ubi.
> > 8KB page of nand flash will consume 16MB for a single volume on a
> > single UBI MTD.
> > 
> > 1. Is there any idea to reduce the memory consumption ?
> 
> Yes. The main idea is as I described: both UBI and UBIFS have several
> multiple PEB-size buffers at attach/mount time. I did not count them,
> but I think there may be about 6 of them. All these buffers are
> allocated using vmalloc(), so you can easily fine all of them, because
> 'vmalloc()' is used only for those buffers.

And note, this is all off the top of my head, I might be mistaken
somewhere. If you need this - you need to spend time with the code and
try. I'm all yours helping you by looking at your patches and advising.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-15 13:17         ` Artem Bityutskiy
@ 2011-04-15 13:24           ` Artem Bityutskiy
  2011-04-17  9:03           ` Nicholas
  1 sibling, 0 replies; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-15 13:24 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

On Fri, 2011-04-15 at 16:17 +0300, Artem Bityutskiy wrote:
> On Fri, 2011-04-15 at 16:13 +0300, Artem Bityutskiy wrote:
> > On Fri, 2011-04-15 at 17:23 +0800, Nicholas wrote:
> > > Dear Artem,
> > > 
> > > Thanks for the information. Indeed it has flexibility to be freed or consumed.
> > > Due to my case, i need to use maximum memory space while in the
> > > meantime i also need to write files into nand via ubi.
> > > 8KB page of nand flash will consume 16MB for a single volume on a
> > > single UBI MTD.
> > > 
> > > 1. Is there any idea to reduce the memory consumption ?
> > 
> > Yes. The main idea is as I described: both UBI and UBIFS have several
> > multiple PEB-size buffers at attach/mount time. I did not count them,
> > but I think there may be about 6 of them. All these buffers are
> > allocated using vmalloc(), so you can easily fine all of them, because
> > 'vmalloc()' is used only for those buffers.
> 
> And note, this is all off the top of my head, I might be mistaken
> somewhere. If you need this - you need to spend time with the code and
> try. I'm all yours helping you by looking at your patches and advising.

Hmm, you can also optimize the UBIFS TNC - the cache where we store the
FS indexing information by shrinking the size of 'struct ubifs_znode'.
First of all, use slab cache for znodes, not kmalloc. Second of all,
many fields there do not have to be int.  
    
-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-15 13:17         ` Artem Bityutskiy
  2011-04-15 13:24           ` Artem Bityutskiy
@ 2011-04-17  9:03           ` Nicholas
  2011-04-17  9:16             ` Artem Bityutskiy
  1 sibling, 1 reply; 14+ messages in thread
From: Nicholas @ 2011-04-17  9:03 UTC (permalink / raw)
  To: dedekind1; +Cc: linux-mtd

Dear Artem,

Thanks for the advices. I'll try start for UBI first.

Reducing peb_buf would mean i need to modify the ubi IO for read/write.
As i know, ubi did an erase for a whole block before start writing,

Please correct me if i've written in a wrong way.

Alex

On Fri, Apr 15, 2011 at 9:17 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Fri, 2011-04-15 at 16:13 +0300, Artem Bityutskiy wrote:
>> On Fri, 2011-04-15 at 17:23 +0800, Nicholas wrote:
>> > Dear Artem,
>> >
>> > Thanks for the information. Indeed it has flexibility to be freed or consumed.
>> > Due to my case, i need to use maximum memory space while in the
>> > meantime i also need to write files into nand via ubi.
>> > 8KB page of nand flash will consume 16MB for a single volume on a
>> > single UBI MTD.
>> >
>> > 1. Is there any idea to reduce the memory consumption ?
>>
>> Yes. The main idea is as I described: both UBI and UBIFS have several
>> multiple PEB-size buffers at attach/mount time. I did not count them,
>> but I think there may be about 6 of them. All these buffers are
>> allocated using vmalloc(), so you can easily fine all of them, because
>> 'vmalloc()' is used only for those buffers.
>
> And note, this is all off the top of my head, I might be mistaken
> somewhere. If you need this - you need to spend time with the code and
> try. I'm all yours helping you by looking at your patches and advising.
>
> --
> Best Regards,
> Artem Bityutskiy (Артём Битюцкий)
>
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-17  9:03           ` Nicholas
@ 2011-04-17  9:16             ` Artem Bityutskiy
  2011-04-18  8:29               ` Nicholas
  0 siblings, 1 reply; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-17  9:16 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

On Sun, 2011-04-17 at 17:03 +0800, Nicholas wrote:
> Dear Artem,
> 
> Thanks for the advices. I'll try start for UBI first.
> 
> Reducing peb_buf would mean i need to modify the ubi IO for read/write.
> As i know, ubi did an erase for a whole block before start writing,
> 
> Please correct me if i've written in a wrong way.

Yes, when you investigate the code, you'll even find that if extra self
checks are enabled, UBI will verify that the area it writes to contains
only 0xFFs - it will read the area, compare with 0xFFs, and if all bytes
are 0xFFs it will proceed, otherwise it will shout loudly. See:

ubi_io_write() - all writes go through this function, and
ubi_dbg_check_all_ff - does the verification I talked about, but only in
debugging mode, because it is too costly to do it in production mode.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-17  9:16             ` Artem Bityutskiy
@ 2011-04-18  8:29               ` Nicholas
  2011-04-18  8:39                 ` Artem Bityutskiy
  0 siblings, 1 reply; 14+ messages in thread
From: Nicholas @ 2011-04-18  8:29 UTC (permalink / raw)
  To: dedekind1; +Cc: linux-mtd

Dear Artem,

If I reduce ubi->peb_buf1 and ubi->peb_buf2 with allocation of smaller
memory spaces, will they affect to IO ( read / write ) ?
Will it cause block data wasted ? Since buffer is allocated less than
block size .

Regards,
Alex

On Sun, Apr 17, 2011 at 5:16 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Sun, 2011-04-17 at 17:03 +0800, Nicholas wrote:
>> Dear Artem,
>>
>> Thanks for the advices. I'll try start for UBI first.
>>
>> Reducing peb_buf would mean i need to modify the ubi IO for read/write.
>> As i know, ubi did an erase for a whole block before start writing,
>>
>> Please correct me if i've written in a wrong way.
>
> Yes, when you investigate the code, you'll even find that if extra self
> checks are enabled, UBI will verify that the area it writes to contains
> only 0xFFs - it will read the area, compare with 0xFFs, and if all bytes
> are 0xFFs it will proceed, otherwise it will shout loudly. See:
>
> ubi_io_write() - all writes go through this function, and
> ubi_dbg_check_all_ff - does the verification I talked about, but only in
> debugging mode, because it is too costly to do it in production mode.
>
> --
> Best Regards,
> Artem Bityutskiy (Битюцкий Артём)
>
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-18  8:29               ` Nicholas
@ 2011-04-18  8:39                 ` Artem Bityutskiy
  2011-04-20  3:19                   ` Nicholas
  0 siblings, 1 reply; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-18  8:39 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

On Mon, 2011-04-18 at 16:29 +0800, Nicholas wrote:
> Dear Artem,
> 
> If I reduce ubi->peb_buf1 and ubi->peb_buf2 with allocation of smaller
> memory spaces, will they affect to IO ( read / write ) ?
> Will it cause block data wasted ? Since buffer is allocated less than
> block size .

Yes,  please, make the buffers smaller. Yes, on some setups this will
make I/O a bit slower. But I have some ideas how to deal with that.
Please, do your changes, I'll take a look and then suggest how to
improve that - but I first want to see how you will do a simple
thing :-)

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-18  8:39                 ` Artem Bityutskiy
@ 2011-04-20  3:19                   ` Nicholas
  2011-04-20  7:30                     ` Artem Bityutskiy
  0 siblings, 1 reply; 14+ messages in thread
From: Nicholas @ 2011-04-20  3:19 UTC (permalink / raw)
  To: dedekind1; +Cc: linux-mtd

Dear Artem,

Sorry for my late reply .

My modification :

build.c

#define MTD_SMALL_SIZE_PAGE_BLOCK_SIZE	0x40000 // block size for 2KB
page of nand flash which we set as an offset.

	if (ubi->peb_size > MTD_SMALL_SIZE_PAGE_BLOCK_SIZE)
		ubi->peb_buf1 = vmalloc(MTD_SMALL_SIZE_PAGE_BLOCK_SIZE);
	else
		ubi->peb_buf1 = vmalloc(ubi->peb_size);

	if (!ubi->peb_buf1)
		goto out_free;

  if (ubi->peb_size > MTD_SMALL_SIZE_PAGE_BLOCK_SIZE)
		ubi->peb_buf2 = vmalloc(MTD_SMALL_SIZE_PAGE_BLOCK_SIZE);
	else
		ubi->peb_buf2 = vmalloc(ubi->peb_size);

	if (!ubi->peb_buf2)
		goto out_free;

orphan.c
In ubifs_mount_orphans()
c->orph_buf = vmalloc((c->leb_size)/4); // for temporary

lpt.c
In lpt_init_wr()
c->lpt_buf = vmalloc((c->leb_size)/4); // for temporary

These modification can work properly, and reduce a lot of memory usage .

I also made modification on super.c in mount_ubifs(), seems more
complicated to handle sbuf & ileb_buf allocation. But i recovered them
to original code since it couldn't boot up properly.

Thanks in advance.

Alex

On Mon, Apr 18, 2011 at 4:39 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Mon, 2011-04-18 at 16:29 +0800, Nicholas wrote:
>> Dear Artem,
>>
>> If I reduce ubi->peb_buf1 and ubi->peb_buf2 with allocation of smaller
>> memory spaces, will they affect to IO ( read / write ) ?
>> Will it cause block data wasted ? Since buffer is allocated less than
>> block size .
>
> Yes,  please, make the buffers smaller. Yes, on some setups this will
> make I/O a bit slower. But I have some ideas how to deal with that.
> Please, do your changes, I'll take a look and then suggest how to
> improve that - but I first want to see how you will do a simple
> thing :-)
>
> --
> Best Regards,
> Artem Bityutskiy (Артём Битюцкий)
>
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-20  3:19                   ` Nicholas
@ 2011-04-20  7:30                     ` Artem Bityutskiy
  2011-08-17  6:35                       ` Nicholas
  0 siblings, 1 reply; 14+ messages in thread
From: Artem Bityutskiy @ 2011-04-20  7:30 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

Hi,

On Wed, 2011-04-20 at 11:19 +0800, Nicholas wrote:
> Dear Artem,
> 
> Sorry for my late reply .

Please, send proper Linux kernel patches. Hints:

1. Make a separate patch pere change, so that on modification is done by
one patch. In your case, one buffer improvement pere patch.
2. Write good commit message.
3. Use git format-patch to create patches.

> 
> #define MTD_SMALL_SIZE_PAGE_BLOCK_SIZE	0x40000 // block size for 2KB

No, please just make buffer size to be ubi->max_write_size. Kill this
constant.

> orphan.c
> In ubifs_mount_orphans()
> c->orph_buf = vmalloc((c->leb_size)/4); // for temporary

Similarly, just use ubi->max_write_size instead.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-04-20  7:30                     ` Artem Bityutskiy
@ 2011-08-17  6:35                       ` Nicholas
  2011-08-20  5:17                         ` Artem Bityutskiy
  0 siblings, 1 reply; 14+ messages in thread
From: Nicholas @ 2011-08-17  6:35 UTC (permalink / raw)
  To: dedekind1; +Cc: linux-mtd

Dear Artem,

1. drivers/mtd/ubi/build.c

attach_by_scanning(struct ubi_device *ubi)

        ubi->min_io_size = ubi->mtd->writesize;
        ubi->hdrs_min_io_size = ubi->mtd->writesize >> ubi->mtd->subpage_sft;
+       ubi->max_write_size = ubi->mtd->writesize;

-      ubi->peb_buf1 = vmalloc(ubi->peb_size);
+      ubi->peb_buf1 = vmalloc(ubi->max_write_size);
        if (!ubi->peb_buf1)
                goto out_free;

-       ubi->peb_buf2 = vmalloc(ubi->peb_size);
+      ubi->peb_buf2 = vmalloc(ubi->max_write_size);
        if (!ubi->peb_buf2)
                goto out_free;

2. drivers/mtd/ubi/kapi.c

ubi_do_get_device_info(struct ubi_device *ubi, struct ubi_device_info *di)

        di->min_io_size = ubi->min_io_size;
+      di->max_write_size = ubi->max_write_size;
        di->ro_mode = ubi->ro_mode;

3. drivers/mtd/ubi/ubi.h

    * @dbg_buf_mutex: protects @dbg_peb_buf
+  * @max_write_size : maximum write buffer size

struct ubi_device {

        unsigned int bad_allowed:1;
        unsigned int nor_flash:1;
+      int max_write_size;

4. fs/ubifs/ubifs.h
     * @min_io_shift: number of bits in @min_io_size minus one
+   * @max_write_size: maximum amount of bytes the underlying flash
can write at a
+   *                  time (MTD write buffer size)

        int min_io_shift;
+     int max_write_size;

5. fs/ubifs/super.c

int init_constants_early(struct ubifs_info *c)

        c->min_io_shift = fls(c->min_io_size) - 1;
+      c->max_write_size = c->di.max_write_size;

int mount_ubifs(struct ubifs_info *c)

	dbg_msg("min. I/O unit size:  %d bytes", c->min_io_size);
+	dbg_msg("max. write size:     %d bytes", c->max_write_size);

-        if (!mounted_read_only) {
-                c->ileb_buf = vmalloc(c->leb_size);
-                //c->ileb_buf = vmalloc(c->max_write_size);
-                if (!c->ileb_buf)
-                        goto out_free;
-        }

    out_free:
	kfree(c->bu.buf);
-	vfree(c->ileb_buf);
	vfree(c->sbuf);

ubifs_umount(struct ubifs_info *c)

	kfree(c->bu.buf);
-	vfree(c->ileb_buf);
	vfree(c->sbuf);

6. fs/ubifs/tnc_commit.c

fill_gap(struct ubifs_info *c, int lnum, int gap_start, int gap_end, int *dirt)

			ubifs_assert(alen <= gap_remains);
-			err = make_idx_node(c, c->ileb_buf + gap_pos, znode,
-					    lnum, gap_pos, len);
+			err = make_idx_node(c, c->sbuf + gap_pos, znode,
+					    lnum, gap_pos, len);

-  ubifs_pad(c, c->ileb_buf + gap_pos, pad_len);
+ ubifs_pad(c, c->sbuf + gap_pos, pad_len);

layout_leb_in_gaps(struct ubifs_info *c, int *p)

/*
	 * Scan the index LEB.  We use the generic scan for this even though
	 * it is more comprehensive and less efficient than is needed for this
	 * purpose.
	 */
-	sleb = ubifs_scan(c, lnum, 0, c->ileb_buf, 0);
+	sleb = ubifs_scan(c, lnum, 0, c->sbuf, 0);

        if (err)
		return err;
-	err = ubifs_leb_change(c, lnum, c->ileb_buf, c->ileb_len,
-			       UBI_SHORTTERM);
+	err = ubifs_leb_change(c, lnum, c->sbuf, c->ileb_len,
+			       UBI_SHORTTERM);

7. fs/ubifs/orphan.c

ubifs_mount_orphans(struct ubifs_inf *c, int unclean, int read_only)

        if (!read_only) {
-               c->orph_buf = vmalloc(c->leb_size);
+              c->orph_buf = vmalloc(c->max_write_size);
                if (!c->orph_buf)
                        return -ENOMEM;
        }

8. fs/ubifs/lpt.c

lpt_init_wr(struct ubifs_info *c)

-	c->lpt_buf = vmalloc(c->leb_size);
-	if (!c->lpt_buf)
-		return -ENOMEM;

9. fs/ubifs/lpt_commit.c

int write_cnodes(struct ubifs_info *c)

	struct ubifs_cnode *cnode;
-	void *buf = c->lpt_buf;
+	void *buf = c->sbuf;

int lpt_gc_lnum(struct ubifs_info *c, int lnum)

	int err, len = c->leb_size, node_type, node_num, node_len, offs;
-	void *buf = c->lpt_buf;
+	void *buf = c->sbuf;

ubifs_lpt_free(struct ubifs_info *c, int wr_only)

-	vfree(c->lpt_buf);
-	c->lpt_buf = NULL;
+	vfree(c->sbuf);
+	c->sbuf = NULL;

Since lpt & orphan's buffer is only used while doing commit, i think
there's a chance to use sbuf for commit activity.
Is there any chance to shrink the "c->sbuf" from fs/ubifs/super.c ?
>From this modification, the memory is shorten to 2~3MB for 8KB page
size of NAND.

Please have a correction or any idea will be highly recommended.

Thanks,
Alex

On Wed, Apr 20, 2011 at 3:30 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> Hi,
>
> On Wed, 2011-04-20 at 11:19 +0800, Nicholas wrote:
>> Dear Artem,
>>
>> Sorry for my late reply .
>
> Please, send proper Linux kernel patches. Hints:
>
> 1. Make a separate patch pere change, so that on modification is done by
> one patch. In your case, one buffer improvement pere patch.
> 2. Write good commit message.
> 3. Use git format-patch to create patches.
>
>>
>> #define MTD_SMALL_SIZE_PAGE_BLOCK_SIZE        0x40000 // block size for 2KB
>
> No, please just make buffer size to be ubi->max_write_size. Kill this
> constant.
>
>> orphan.c
>> In ubifs_mount_orphans()
>> c->orph_buf = vmalloc((c->leb_size)/4); // for temporary
>
> Similarly, just use ubi->max_write_size instead.
>
> --
> Best Regards,
> Artem Bityutskiy (Артём Битюцкий)
>
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: UBI memory usage on large page nand
  2011-08-17  6:35                       ` Nicholas
@ 2011-08-20  5:17                         ` Artem Bityutskiy
  0 siblings, 0 replies; 14+ messages in thread
From: Artem Bityutskiy @ 2011-08-20  5:17 UTC (permalink / raw)
  To: Nicholas; +Cc: linux-mtd

On Wed, 2011-08-17 at 14:35 +0800, Nicholas wrote:
> Dear Artem,
> 
> 1. drivers/mtd/ubi/build.c
> 
> attach_by_scanning(struct ubi_device *ubi)
> 
>         ubi->min_io_size = ubi->mtd->writesize;
>         ubi->hdrs_min_io_size = ubi->mtd->writesize >> ubi->mtd->subpage_sft;
> +       ubi->max_write_size = ubi->mtd->writesize;

Could you please send a normal kernel patch with a good commit message,
e.g., produced with 'git format-patch'. Describe how you tested it as
well.

There are many internet resources describing how to send good kernel
patches.

Thanks!

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2011-08-20  5:17 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <AANLkTi=GsmJOjurETG930t-TW7uXLR1M6yKXWBSk7ebV@mail.gmail.com>
2011-03-31  3:07 ` UBI memory usage on large page nand Nicholas
2011-04-01 15:31   ` Artem Bityutskiy
2011-04-15  9:23     ` Nicholas
2011-04-15 13:13       ` Artem Bityutskiy
2011-04-15 13:17         ` Artem Bityutskiy
2011-04-15 13:24           ` Artem Bityutskiy
2011-04-17  9:03           ` Nicholas
2011-04-17  9:16             ` Artem Bityutskiy
2011-04-18  8:29               ` Nicholas
2011-04-18  8:39                 ` Artem Bityutskiy
2011-04-20  3:19                   ` Nicholas
2011-04-20  7:30                     ` Artem Bityutskiy
2011-08-17  6:35                       ` Nicholas
2011-08-20  5:17                         ` Artem Bityutskiy

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.