Available space loss due to fragmentation?

Available space loss due to fragmentation?

[Ben Schroeder]

Hi everyone,
I have an issue of loss of available space after file rewriting -

Setup:
3 UBI Volumes (25mb static, 25mb static, 15mb dynamic) on a single MTD device.
A/B partition scheme for upgrades with the 25mb volumes.
A/B partitions have around 400kb space available each, and cannot
expand beyond it.

Issue:
Partition 'A' is main RootFs, 'B' partition mounted on /mnt/ to be upgraded.
When attempting to upgrade the backup partition, by rewriting the files
according to a binary diff, I run out of available space on the volume.

The issue can be simplified:
Copying a 2mb file out of the partition, and then back to overwrite the original
may cause loss of 100kb of space.
Rewriting files with minor changes, even if the new file is smaller than the
original, can result in loss of available disk space.

I base this not only on the info from: df -h
But also run sync, unmount and mount again, and reboot.
All show a loss of available space.
I am aware that df -h does not show an accurate value, however, copying
and modifying files will fail due to no available disk space error!

Some writes will continue to succeed even when available space is 4kb.
But many writes will fail when available space is low.

I am working on a new format of a UBIFS image.
I suspect that the original UBI volume files are alligned perfectly,
and once I rewrite files wit binary diffs, the files become fragmented,
and a loss of available space occurs, even though the files remain the
same or smaller.

However, I am not familiar enough with the UBI internals to be certain.

Why do I see a loss of space when rewriting the same file?
Can I use an upgrade scheme with file binary diff as mentioned above -
One that would run correctly with low available space?
Can I use an upgrade scheme with UBI volume binary diff?

Sorry for the long mail, I have not found much information about fragmentation
and space loss in UBIFS. Let me know if I forgot any relevant details.

Thanks in advance.

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Available space loss due to fragmentation?

[Richard Weinberger]

Ben,

On Wed, Jul 10, 2019 at 5:32 AM Ben Schroeder <klowd92@gmail.com> wrote:
> Why do I see a loss of space when rewriting the same file?

Please see my answer below.

> Can I use an upgrade scheme with file binary diff as mentioned above -
> One that would run correctly with low available space?

If the filesystem is full and all nodes are already packed, it can be
a challenge.

> Can I use an upgrade scheme with UBI volume binary diff?

Yes, you can alter a dynamic volume as you wish. But keep NAND odds on mind.
So you need to replace whole LEBs.

> Sorry for the long mail, I have not found much information about fragmentation
> and space loss in UBIFS. Let me know if I forgot any relevant details.

I think the root cause of the problem you see is how NAND works.
On NAND we write always full pages. So if you ask UBIFS to change one byte
of a file or change meta data, it has to waste a full page.

Luckily Linux is a modern operating system with a write-cache and upon
write-back UBIFS can pack nodes (UBIFS data nodes, inode nodes, etc...) together
and wastes less space.
But it wastes still a significant amount of space if userspace forces
it to persist data.
i.e. by using fsync()/fdatasync().
If UBIFS runs out of space the garbage collector will rewrite nodes
and pack them tightly
together.

So, if you have a pre-created UBIFS, nodes are already packed and your
update mechanism
might force UBIFS to faster than the garbage collector can pack nodes.

With that information in mind, do your other questions resolve?

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Available space loss due to fragmentation?

[Ben Schroeder]

On Wed, Jul 10, 2019 at 12:34 PM Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> Ben,
>
> On Wed, Jul 10, 2019 at 5:32 AM Ben Schroeder <klowd92@gmail.com> wrote:
> > Why do I see a loss of space when rewriting the same file?
>
> Please see my answer below.
>
> > Can I use an upgrade scheme with file binary diff as mentioned above -
> > One that would run correctly with low available space?
>
> If the filesystem is full and all nodes are already packed, it can be
> a challenge.
>
> > Can I use an upgrade scheme with UBI volume binary diff?
>
> Yes, you can alter a dynamic volume as you wish. But keep NAND odds on mind.
> So you need to replace whole LEBs.
>
> > Sorry for the long mail, I have not found much information about fragmentation
> > and space loss in UBIFS. Let me know if I forgot any relevant details.
>
> I think the root cause of the problem you see is how NAND works.
> On NAND we write always full pages. So if you ask UBIFS to change one byte
> of a file or change meta data, it has to waste a full page.
>
> Luckily Linux is a modern operating system with a write-cache and upon
> write-back UBIFS can pack nodes (UBIFS data nodes, inode nodes, etc...) together
> and wastes less space.
> But it wastes still a significant amount of space if userspace forces
> it to persist data.
> i.e. by using fsync()/fdatasync().
> If UBIFS runs out of space the garbage collector will rewrite nodes
> and pack them tightly
> together.
>
> So, if you have a pre-created UBIFS, nodes are already packed and your
> update mechanism
> might force UBIFS to faster than the garbage collector can pack nodes.
>
> With that information in mind, do your other questions resolve?
>

Thanks for the reply Richard.
I just wanted to reiterate that i am using SPI NOR Flash, partitioned
in an A/B scheme as so:

mtd7
Name:                           rootfs
Type:                           nor
Eraseblock size:                65536 bytes, 64.0 KiB
Amount of eraseblocks:          880 (57671680 bytes, 55.0 MiB)
Minimum input/output unit size: 1 byte
Sub-page size:                  1 byte
Character device major/minor:   90:14
Bad blocks are allowed:         false
Device is writable:             true

mtd8
Name:                           rootfs1
Type:                           ubi
Eraseblock size:                65408 bytes, 63.9 KiB
Amount of eraseblocks:          353 (23089024 bytes, 22.0 MiB)
Minimum input/output unit size: 1 byte
Sub-page size:                  1 byte
Character device major/minor:   90:16
Bad blocks are allowed:         false
Device is writable:             true

mtd9
Name:                           rootfs2
Type:                           ubi
Eraseblock size:                65408 bytes, 63.9 KiB
Amount of eraseblocks:          353 (23089024 bytes, 22.0 MiB)
Minimum input/output unit size: 1 byte
Sub-page size:                  1 byte
Character device major/minor:   90:18
Bad blocks are allowed:         false
Device is writable:             true

I am not sure the garbage collector will improve the available space issue.
Regardless of the UBI being mounted with sync option enabled or disabled,
the issue persists. Even if i allow time for the background thread to run.
The issue seems very problematic when considering the fact that i am
downgrading the filesystem, patching files to be slightly smaller size
than before,
and i am still running out of disk space, regardless of how long i
wait for garbage collection.
On this regard, i will stick with your answer that it can be a serious
challenge if all nodes are packed,
and there is little available free space.

Could you please clarify your answer regarding binary patching UBI Volumes:
> Yes, you can alter a dynamic volume as you wish. But keep NAND odds on mind.
> So you need to replace whole LEBs.

It was my understanding that because UBI keeps tracks of bad blocks
and erase counters,
so that overwriting an existing and running UBI partition using a
binary diff against a newer UBI partition,
might cause loss of that metadata, or even corruption.

> --
> Thanks,
> //richard

Thanks,
Ben.

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Available space loss due to fragmentation?

[Richard Weinberger]

On Wed, Jul 10, 2019 at 5:18 PM Ben Schroeder <klowd92@gmail.com> wrote:
> Thanks for the reply Richard.
> I just wanted to reiterate that i am using SPI NOR Flash, partitioned
> in an A/B scheme as so:

Hmmm.
Did you create the rootfs using mkfs.ubifs with a different
compression than used
by the kernel?

> I am not sure the garbage collector will improve the available space issue.
> Regardless of the UBI being mounted with sync option enabled or disabled,
> the issue persists. Even if i allow time for the background thread to run.
> The issue seems very problematic when considering the fact that i am
> downgrading the filesystem, patching files to be slightly smaller size
> than before,
> and i am still running out of disk space, regardless of how long i
> wait for garbage collection.
> On this regard, i will stick with your answer that it can be a serious
> challenge if all nodes are packed,
> and there is little available free space.

How full is the filesystem btw?

> Could you please clarify your answer regarding binary patching UBI Volumes:
> > Yes, you can alter a dynamic volume as you wish. But keep NAND odds on mind.
> > So you need to replace whole LEBs.
>
> It was my understanding that because UBI keeps tracks of bad blocks
> and erase counters,
> so that overwriting an existing and running UBI partition using a
> binary diff against a newer UBI partition,
> might cause loss of that metadata, or even corruption.

You need to operate on UBI level, not mtd.
You can open /dev/ubiX_Y and ask it to replace whole LEBs.
UBI will take care of erase-counters and stuff.

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Available space loss due to fragmentation?

[Ben Schroeder]

On Thu, Jul 11, 2019 at 1:16 PM Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> On Wed, Jul 10, 2019 at 5:18 PM Ben Schroeder <klowd92@gmail.com> wrote:
> > Thanks for the reply Richard.
> > I just wanted to reiterate that i am using SPI NOR Flash, partitioned
> > in an A/B scheme as so:
>
> Hmmm.
> Did you create the rootfs using mkfs.ubifs with a different
> compression than used
> by the kernel?
mkfs.ubifs -r /tmp/rootfs -m 1 -e 0xFF80 -c 1024 -o data.ubifs
(I believe standard compression is used, lzo?)
>
> > I am not sure the garbage collector will improve the available space issue.
> > Regardless of the UBI being mounted with sync option enabled or disabled,
> > the issue persists. Even if i allow time for the background thread to run.
> > The issue seems very problematic when considering the fact that i am
> > downgrading the filesystem, patching files to be slightly smaller size
> > than before,
> > and i am still running out of disk space, regardless of how long i
> > wait for garbage collection.
> > On this regard, i will stick with your answer that it can be a serious
> > challenge if all nodes are packed,
> > and there is little available free space.
>
> How full is the filesystem btw?
250kb free out of 22mb ~ Approxy 1% free

>
> > Could you please clarify your answer regarding binary patching UBI Volumes:
> > > Yes, you can alter a dynamic volume as you wish. But keep NAND odds on mind.
> > > So you need to replace whole LEBs.
> >
> > It was my understanding that because UBI keeps tracks of bad blocks
> > and erase counters,
> > so that overwriting an existing and running UBI partition using a
> > binary diff against a newer UBI partition,
> > might cause loss of that metadata, or even corruption.
>
> You need to operate on UBI level, not mtd.
> You can open /dev/ubiX_Y and ask it to replace whole LEBs.
> UBI will take care of erase-counters and stuff.

I see, thanks!

>
> --
> Thanks,
> //richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Available space loss due to fragmentation?

[Richard Weinberger]

Ben,

----- Ursprüngliche Mail -----
> Von: "Ben Schroeder" <klowd92@gmail.com>
> An: "Richard Weinberger" <richard.weinberger@gmail.com>
> CC: "linux-mtd" <linux-mtd@lists.infradead.org>
> Gesendet: Donnerstag, 11. Juli 2019 17:53:53
> Betreff: Re: Available space loss due to fragmentation?

> On Thu, Jul 11, 2019 at 1:16 PM Richard Weinberger
> <richard.weinberger@gmail.com> wrote:
>>
>> On Wed, Jul 10, 2019 at 5:18 PM Ben Schroeder <klowd92@gmail.com> wrote:
>> > Thanks for the reply Richard.
>> > I just wanted to reiterate that i am using SPI NOR Flash, partitioned
>> > in an A/B scheme as so:
>>
>> Hmmm.
>> Did you create the rootfs using mkfs.ubifs with a different
>> compression than used
>> by the kernel?
> mkfs.ubifs -r /tmp/rootfs -m 1 -e 0xFF80 -c 1024 -o data.ubifs
> (I believe standard compression is used, lzo?)

To be honest, I'm not entirely sure what problem you are facing.

Can you please give these two approaches a try?
1. Create the filesystem with no compression and mount ubifs without compression,
to rule out compression related issues.
2. Try a (much) larger log size, you can specify it at mkfs.ubifs time.

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/