A big issue of NAND fragmentation

A big issue of NAND fragmentation

[JH]

Hi,

I am running kernel 5.1.0 on iMX6 using NAND flash, I write small data
files about 250 bytes each every 5 minutes to a backup storage, the
total size of all data files is about 600 KB, but du shown me 9.7M is
used in that directory. I know NAND using page to flush files, how
does the MTD handler NAND fragmentation?

Does the latest version of MTD (hence the latest kernel version)
resolve this problem?

Thank you.

Kind regards,

- jupiter

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: A big issue of NAND fragmentation

[Richard Weinberger]

On Thu, Sep 5, 2019 at 3:27 AM JH <jupiter.hce@gmail.com> wrote:
>
> Hi,
>
> I am running kernel 5.1.0 on iMX6 using NAND flash, I write small data
> files about 250 bytes each every 5 minutes to a backup storage, the
> total size of all data files is about 600 KB, but du shown me 9.7M is
> used in that directory. I know NAND using page to flush files, how
> does the MTD handler NAND fragmentation?

If you force UBIFS (I assume you use it) to persist 250 bytes,
it has to waste a full NAND page. This is how NAND works.
But UBIFS can pack such data chunks upon garbage collect when
it runs out of space.

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: A big issue of NAND fragmentation

[JH]

Hi Richard,

On 9/5/19, Richard Weinberger <richard.weinberger@gmail.com> wrote:
> On Thu, Sep 5, 2019 at 3:27 AM JH <jupiter.hce@gmail.com> wrote:
>>
>> Hi,
>>
>> I am running kernel 5.1.0 on iMX6 using NAND flash, I write small data
>> files about 250 bytes each every 5 minutes to a backup storage, the
>> total size of all data files is about 600 KB, but du shown me 9.7M is
>> used in that directory. I know NAND using page to flush files, how
>> does the MTD handler NAND fragmentation?
>
> If you force UBIFS (I assume you use it) to persist 250 bytes,
> it has to waste a full NAND page. This is how NAND works.
> But UBIFS can pack such data chunks upon garbage collect when
> it runs out of space.

Yes, I use UBIFS, was your word "force" means "write" or did you
allude there would be an alternative to avoid forcing UBIFS persist
250 bytes?

Waiting for out of space would be too risk, alternatively, I should
have a UBIFS partition for the data storage, if it runs out of space,
it won't impact the root file system.

Thank you Richard.

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: A big issue of NAND fragmentation

[Richard Weinberger]

----- Ursprüngliche Mail -----
> Von: "JH" <jupiter.hce@gmail.com>
> An: "Richard Weinberger" <richard.weinberger@gmail.com>
> CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> Gesendet: Freitag, 6. September 2019 08:03:55
> Betreff: Re: A big issue of NAND fragmentation
> 
> Yes, I use UBIFS, was your word "force" means "write" or did you
> allude there would be an alternative to avoid forcing UBIFS persist
> 250 bytes?
> 
> Waiting for out of space would be too risk, alternatively, I should
> have a UBIFS partition for the data storage, if it runs out of space,
> it won't impact the root file system.
> 

No, by force I mean forcing the filesystem to persist the data.
For example by using fsync(),fdatasync(), O_SYNC or a sync mounted
filesystem.
If you don't do this, data will be cached and can be packed
later upon write-back.

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: A big issue of NAND fragmentation

[JH]

On 9/6/19, Richard Weinberger <richard@nod.at> wrote:
> ----- Ursprüngliche Mail -----
>> Von: "JH" <jupiter.hce@gmail.com>
>> An: "Richard Weinberger" <richard.weinberger@gmail.com>
>> CC: "linux-mtd" <linux-mtd@lists.infradead.org>
>> Gesendet: Freitag, 6. September 2019 08:03:55
>> Betreff: Re: A big issue of NAND fragmentation
>>
>> Yes, I use UBIFS, was your word "force" means "write" or did you
>> allude there would be an alternative to avoid forcing UBIFS persist
>> 250 bytes?
>>
>> Waiting for out of space would be too risk, alternatively, I should
>> have a UBIFS partition for the data storage, if it runs out of space,
>> it won't impact the root file system.
>>
>
> No, by force I mean forcing the filesystem to persist the data.
> For example by using fsync(),fdatasync(), O_SYNC or a sync mounted
> filesystem.
> If you don't do this, data will be cached and can be packed
> later upon write-back.

I use std::ofstream and operator<<, it does have a flush but I don't
know if the flush is a default setting or not, but obviously it is
default which caused fragmentation. More work to do, might be better
to use an old simple C write.

Thanks for the explanation.

- jupiter

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/