Nandsim, UBIFS and memory concerns

Nandsim, UBIFS and memory concerns

[Romain Izard]

On a regular but slow basis, I get report of devices based on UBIFS running
Linux 4.14 where the file system gets corrupted during an update. The update
process creates new files with temporary names to replace existing files,
and uses renames to replace these files atomically. What is observed is that
in some cases, the update log describes all steps for a complete update, and
yet some files contain the new version while others contain an older
version. Moreover, it seems that some files with temporary names that should
have been renamed are visible.

As the update process is also able to use tmpfs to create files, and will
use a large part of the available memory, I fear that this issue is related
with the behaviour of UBIFS in low memory conditions. I'm wondering about
UBIFS losing some parts of the log when a ENOMEM condition occurs during its
operations or when the OOM killer targets a process that is doing some UBIFS
processing.

To study this more easily, I started to create a QEMU image to be able to
easily test these conditions. While QEMU has a small MTD emulation support,
it will not work with x86 machines; but those run much faster than emulated
architectures, so I decided to use nandsim instead.

Starting from the qemu_x86_64_defconfig in Buildroot v2018.08, I have
modified the kernel configuration to add support for MTD, NAND flash,
Nandsim, UBI and UBIFS; the whole system was inserted as an initramfs
without modules in the kernel image to keep things simple; ttyS0 is also
selected as the default port for getty.

The command line used to start QEMU is:

qemu-system-x86_64 -nographic -m 64M -M q35 \
  -kernel output-x86/images/bzImage \
  -drive file=mtdfile.bin,if=virtio,format=raw \
  -net nic,model=virtio -net user \
  -append 'nandsim.id_bytes=0x20,0xaa,0,0x15
nandsim.cache_file=/dev/vda console=ttyS0'

The "mtdfile.bin" is a 256 MiB + 8 MiB file, with the same size as the NAND
flash chip described by the ID bytes. From the documentation and the code, I
expected that all accesses to the NAND would be simulated as read and writes
to this file through the /dev/vda device. But what I observe is that the
file is not touched when calling "ubiattach -m0", when observed from within
or from without QEMU.

Here, I am quite confused, as the use by nandsim of filp_open, kernel_read
and kernel_write look quite normal, and I do not see why the accesses are
only happening in RAM (as otherwise it would not possible to use UBI on
nandsim at all) but do not end up on the disk. When enabling the nandsim
debug traces, I can observe traces that match the cache_file mode.

As an additional effect of this, the memory for the system is very quickly
consumed by nandsim, and it makes it difficult to work with UBIFS to analyse
my original problem.

My questions are the following:
- Has UBIFS been tested in low memory conditions? Is there a suitable test
  suite to verify that everything works correctly?
- Is the nandsim file mode working as expected?  What can I do to be sure
  that the read and writes are committed to the file?

Best regards,
-- 
Romain Izard

Re: Nandsim, UBIFS and memory concerns

[Richard Weinberger]

On Thu, Oct 4, 2018 at 6:52 PM Romain Izard <romain.izard.pro@gmail.com> wrote:
> My questions are the following:
> - Has UBIFS been tested in low memory conditions? Is there a suitable test
>   suite to verify that everything works correctly?

The smallest device in my office with UBIFS support has 32MiB RAM.
But the kernel is not fresh, maybe recent kernels consume more memory wrt. VFS.
To answer your question, it has been designed to work with low memory.

> - Is the nandsim file mode working as expected?  What can I do to be sure
>   that the read and writes are committed to the file?

Nandsim works in memory. The file backend is a hack and what you might expect.

Please tell me more about the corruptions you're facing. Even in UBIFS
runs out of memory
it must not corrupt.
-- 
Thanks,
//richard

Re: Nandsim, UBIFS and memory concerns

[Richard Weinberger]

On Fri, Oct 5, 2018 at 12:02 AM Richard Weinberger
<richard.weinberger@gmail.com> wrote:
> Nandsim works in memory. The file backend is a hack and what you might expect.

Should be read, ... *not* what you might expect.

-- 
Thanks,
//richard

Re: Nandsim, UBIFS and memory concerns

[Steve deRosier]

Hi Romain,

On Thu, Oct 4, 2018 at 9:53 AM Romain Izard <romain.izard.pro@gmail.com> wrote:
>
> On a regular but slow basis, I get report of devices based on UBIFS running
> Linux 4.14 where the file system gets corrupted during an update. The update
> process creates new files with temporary names to replace existing files,
> and uses renames to replace these files atomically. What is observed is that
> in some cases, the update log describes all steps for a complete update, and
> yet some files contain the new version while others contain an older
> version. Moreover, it seems that some files with temporary names that should
> have been renamed are visible.
>
> As the update process is also able to use tmpfs to create files, and will
> use a large part of the available memory, I fear that this issue is related
> with the behaviour of UBIFS in low memory conditions. I'm wondering about
> UBIFS losing some parts of the log when a ENOMEM condition occurs during its
> operations or when the OOM killer targets a process that is doing some UBIFS
> processing.
>

I've seen these sort of symptoms that you describe in the wild. But
what I have seen has never had anything to do with UBIFS, but only
with problems with how updates (or other large filesystems operations)
are implemented. Specifically, the lack of a filesystem sync before a
reboot will have these exact effects. What you end up with is a
situation where the filesystem operations are done, yet the changes
haven't actually been flushed to "disk".  Doesn't mater if it's a HDD
or a UBIFS on flash, the effect is the same, though the time of
vulnerability might be different.

Especially since you mention the OOM killer and using tmpfs - I'd look
into if you're running out of RAM, and either causing an reboot oops
or at least killing the process before all file operations are
complete. Just because your log shows the operation was triggered at
the userspace level, doesn't mean the kernel has completed all
filesystem operations and written the physical device.

What you describe is not an UBIFS corruption, but a garden-variety
user-space file operations corruption issue.

As I said, I've encountered this before. The only thing you can do is
to examine your process and tailor it to be sure to complete it's
physical writes.  In our case, we had a few things to solve: * put
'sync' calls in our update scripts, * avoid the use of a problematic
utility, and * we tried using the `-osync` flag.  (-osync fixed the
problem at the cost of a performance hit. Later we decided not to go
that way and instead instructed our customers how to properly write
programs that wrote the filesystem).

- Steve

Re: Nandsim, UBIFS and memory concerns

[Romain Izard]

Hello Steve,

Le ven. 5 oct. 2018 à 00:43, Steve deRosier <derosier@gmail.com> a écrit :
>
> Hi Romain,
>
> On Thu, Oct 4, 2018 at 9:53 AM Romain Izard <romain.izard.pro@gmail.com> wrote:
> >
> > On a regular but slow basis, I get report of devices based on UBIFS running
> > Linux 4.14 where the file system gets corrupted during an update. The update
> > process creates new files with temporary names to replace existing files,
> > and uses renames to replace these files atomically. What is observed is that
> > in some cases, the update log describes all steps for a complete update, and
> > yet some files contain the new version while others contain an older
> > version. Moreover, it seems that some files with temporary names that should
> > have been renamed are visible.
> >
> > As the update process is also able to use tmpfs to create files, and will
> > use a large part of the available memory, I fear that this issue is related
> > with the behaviour of UBIFS in low memory conditions. I'm wondering about
> > UBIFS losing some parts of the log when a ENOMEM condition occurs during its
> > operations or when the OOM killer targets a process that is doing some UBIFS
> > processing.
> >
>
> I've seen these sort of symptoms that you describe in the wild. But
> what I have seen has never had anything to do with UBIFS, but only
> with problems with how updates (or other large filesystems operations)
> are implemented. Specifically, the lack of a filesystem sync before a
> reboot will have these exact effects. What you end up with is a
> situation where the filesystem operations are done, yet the changes
> haven't actually been flushed to "disk".  Doesn't mater if it's a HDD
> or a UBIFS on flash, the effect is the same, though the time of
> vulnerability might be different.
>
> Especially since you mention the OOM killer and using tmpfs - I'd look
> into if you're running out of RAM, and either causing an reboot oops
> or at least killing the process before all file operations are
> complete. Just because your log shows the operation was triggered at
> the userspace level, doesn't mean the kernel has completed all
> filesystem operations and written the physical device.
>
> What you describe is not an UBIFS corruption, but a garden-variety
> user-space file operations corruption issue.
>
> As I said, I've encountered this before. The only thing you can do is
> to examine your process and tailor it to be sure to complete it's
> physical writes.  In our case, we had a few things to solve: * put
> 'sync' calls in our update scripts, * avoid the use of a problematic
> utility, and * we tried using the `-osync` flag.  (-osync fixed the
> problem at the cost of a performance hit. Later we decided not to go
> that way and instead instructed our customers how to properly write
> programs that wrote the filesystem).

Thank you for sharing your experience on this topic. This will help
me to concentrate on checking my own code, rather than spending
time to analyse something that works.

Best regards,
-- 
Romain Izard

Re: Nandsim, UBIFS and memory concerns

[Richard Weinberger]

Romain,

On Fri, Oct 5, 2018 at 10:21 AM Romain Izard <romain.izard.pro@gmail.com> wrote:
> Thank you for sharing your experience on this topic. This will help
> me to concentrate on checking my own code, rather than spending
> time to analyse something that works.

So, what you are facing are not UBIFS corruptions in terms of
filesystem inconsistencies/errors?

-- 
Thanks,
//richard

Re: Nandsim, UBIFS and memory concerns

[Romain Izard]

Le ven. 5 oct. 2018 à 10:54, Richard Weinberger
<richard.weinberger@gmail.com> a écrit :
>
> Romain,
>
> On Fri, Oct 5, 2018 at 10:21 AM Romain Izard <romain.izard.pro@gmail.com> wrote:
> > Thank you for sharing your experience on this topic. This will help
> > me to concentrate on checking my own code, rather than spending
> > time to analyse something that works.
>
> So, what you are facing are not UBIFS corruptions in terms of
> filesystem inconsistencies/errors?
>
Yes, this looks like a good explanation. Thanks for taking some time
to look at my
issue.

Best regards,
-- 
Romain Izard

Re: Nandsim, UBIFS and memory concerns

[Richard Weinberger]

Am Freitag, 5. Oktober 2018, 10:55:51 CEST schrieb Romain Izard:
> Yes, this looks like a good explanation. Thanks for taking some time
> to look at my
> issue.

So the solution for your problem should be "make sure that userspace is sane and does
fsync/fdatasync/sync/umount.." :-)

Thanks,
//richard