Linux-mtd Archive on lore.kernel.org
 help / color / Atom feed
* A big issue of NAND fragmentation
@ 2019-09-05  1:27 JH
  2019-09-05 11:54 ` Richard Weinberger
  0 siblings, 1 reply; 5+ messages in thread
From: JH @ 2019-09-05  1:27 UTC (permalink / raw)
  To: linux-mtd

Hi,

I am running kernel 5.1.0 on iMX6 using NAND flash, I write small data
files about 250 bytes each every 5 minutes to a backup storage, the
total size of all data files is about 600 KB, but du shown me 9.7M is
used in that directory. I know NAND using page to flush files, how
does the MTD handler NAND fragmentation?

Does the latest version of MTD (hence the latest kernel version)
resolve this problem?

Thank you.

Kind regards,

- jupiter

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: A big issue of NAND fragmentation
  2019-09-05  1:27 A big issue of NAND fragmentation JH
@ 2019-09-05 11:54 ` Richard Weinberger
  2019-09-06  6:03   ` JH
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Weinberger @ 2019-09-05 11:54 UTC (permalink / raw)
  To: JH; +Cc: linux-mtd

On Thu, Sep 5, 2019 at 3:27 AM JH <jupiter.hce@gmail.com> wrote:
>
> Hi,
>
> I am running kernel 5.1.0 on iMX6 using NAND flash, I write small data
> files about 250 bytes each every 5 minutes to a backup storage, the
> total size of all data files is about 600 KB, but du shown me 9.7M is
> used in that directory. I know NAND using page to flush files, how
> does the MTD handler NAND fragmentation?

If you force UBIFS (I assume you use it) to persist 250 bytes,
it has to waste a full NAND page. This is how NAND works.
But UBIFS can pack such data chunks upon garbage collect when
it runs out of space.

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: A big issue of NAND fragmentation
  2019-09-05 11:54 ` Richard Weinberger
@ 2019-09-06  6:03   ` JH
  2019-09-06  7:02     ` Richard Weinberger
  0 siblings, 1 reply; 5+ messages in thread
From: JH @ 2019-09-06  6:03 UTC (permalink / raw)
  To: Richard Weinberger; +Cc: linux-mtd

Hi Richard,

On 9/5/19, Richard Weinberger <richard.weinberger@gmail.com> wrote:
> On Thu, Sep 5, 2019 at 3:27 AM JH <jupiter.hce@gmail.com> wrote:
>>
>> Hi,
>>
>> I am running kernel 5.1.0 on iMX6 using NAND flash, I write small data
>> files about 250 bytes each every 5 minutes to a backup storage, the
>> total size of all data files is about 600 KB, but du shown me 9.7M is
>> used in that directory. I know NAND using page to flush files, how
>> does the MTD handler NAND fragmentation?
>
> If you force UBIFS (I assume you use it) to persist 250 bytes,
> it has to waste a full NAND page. This is how NAND works.
> But UBIFS can pack such data chunks upon garbage collect when
> it runs out of space.

Yes, I use UBIFS, was your word "force" means "write" or did you
allude there would be an alternative to avoid forcing UBIFS persist
250 bytes?

Waiting for out of space would be too risk, alternatively, I should
have a UBIFS partition for the data storage, if it runs out of space,
it won't impact the root file system.

Thank you Richard.

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: A big issue of NAND fragmentation
  2019-09-06  6:03   ` JH
@ 2019-09-06  7:02     ` Richard Weinberger
  2019-09-06 11:18       ` JH
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Weinberger @ 2019-09-06  7:02 UTC (permalink / raw)
  To: JH; +Cc: Richard Weinberger, linux-mtd

----- Ursprüngliche Mail -----
> Von: "JH" <jupiter.hce@gmail.com>
> An: "Richard Weinberger" <richard.weinberger@gmail.com>
> CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> Gesendet: Freitag, 6. September 2019 08:03:55
> Betreff: Re: A big issue of NAND fragmentation
> 
> Yes, I use UBIFS, was your word "force" means "write" or did you
> allude there would be an alternative to avoid forcing UBIFS persist
> 250 bytes?
> 
> Waiting for out of space would be too risk, alternatively, I should
> have a UBIFS partition for the data storage, if it runs out of space,
> it won't impact the root file system.
> 

No, by force I mean forcing the filesystem to persist the data.
For example by using fsync(),fdatasync(), O_SYNC or a sync mounted
filesystem.
If you don't do this, data will be cached and can be packed
later upon write-back.

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: A big issue of NAND fragmentation
  2019-09-06  7:02     ` Richard Weinberger
@ 2019-09-06 11:18       ` JH
  0 siblings, 0 replies; 5+ messages in thread
From: JH @ 2019-09-06 11:18 UTC (permalink / raw)
  To: Richard Weinberger; +Cc: Richard Weinberger, linux-mtd

On 9/6/19, Richard Weinberger <richard@nod.at> wrote:
> ----- Ursprüngliche Mail -----
>> Von: "JH" <jupiter.hce@gmail.com>
>> An: "Richard Weinberger" <richard.weinberger@gmail.com>
>> CC: "linux-mtd" <linux-mtd@lists.infradead.org>
>> Gesendet: Freitag, 6. September 2019 08:03:55
>> Betreff: Re: A big issue of NAND fragmentation
>>
>> Yes, I use UBIFS, was your word "force" means "write" or did you
>> allude there would be an alternative to avoid forcing UBIFS persist
>> 250 bytes?
>>
>> Waiting for out of space would be too risk, alternatively, I should
>> have a UBIFS partition for the data storage, if it runs out of space,
>> it won't impact the root file system.
>>
>
> No, by force I mean forcing the filesystem to persist the data.
> For example by using fsync(),fdatasync(), O_SYNC or a sync mounted
> filesystem.
> If you don't do this, data will be cached and can be packed
> later upon write-back.

I use std::ofstream and operator<<, it does have a flush but I don't
know if the flush is a default setting or not, but obviously it is
default which caused fragmentation. More work to do, might be better
to use an old simple C write.

Thanks for the explanation.

- jupiter

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-05  1:27 A big issue of NAND fragmentation JH
2019-09-05 11:54 ` Richard Weinberger
2019-09-06  6:03   ` JH
2019-09-06  7:02     ` Richard Weinberger
2019-09-06 11:18       ` JH

Linux-mtd Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-mtd/0 linux-mtd/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-mtd linux-mtd/ https://lore.kernel.org/linux-mtd \
		linux-mtd@lists.infradead.org linux-mtd@archiver.kernel.org
	public-inbox-index linux-mtd


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.infradead.lists.linux-mtd


AGPL code for this site: git clone https://public-inbox.org/ public-inbox