Using FIO on systems with <=512MB RAM

Using FIO on systems with <=512MB RAM

[Philippe Schenker]

Hello

We use FIO to check the eMMC storage on our embedded boards. This works
great for our modules that have >= 1GB RAM but on all lower-memory ones
(<= 512MB) the OOM killer always kills FIO.

I can hook up a swap which helps but is just painfully slow.

I played with the settings mem, bs, iodepth and size but cannot achieve
a configuration that works on low-memory embedded systems.

Can anyone guide me how I can make this work?

Best Regards,
Philippe

Re: Using FIO on systems with <=512MB RAM

[Damien Le Moal]

On 2/21/22 23:40, Philippe Schenker wrote:
> Hello
> 
> We use FIO to check the eMMC storage on our embedded boards. This works
> great for our modules that have >= 1GB RAM but on all lower-memory ones
> (<= 512MB) the OOM killer always kills FIO.
> 
> I can hook up a swap which helps but is just painfully slow.

Well yes, your system will be trashing...

> 
> I played with the settings mem, bs, iodepth and size but cannot achieve
> a configuration that works on low-memory embedded systems.
> 
> Can anyone guide me how I can make this work?

See if playing with option "--alloc-size=" helps. You can also reduce
the maximum number of jobs with "--max-jobs=". Also try using threads
instead of (default) processes for jobs (--thread option). And of course
make sure that the total amount of IO buffers you need (num jobs * io
depth per job * bs) is not too large for your RAM size.

> 
> Best Regards,
> Philippe
> 


-- 
Damien Le Moal
Western Digital Research

Re: Using FIO on systems with <=512MB RAM

[Philippe Schenker]

On Tue, 2022-02-22 at 08:41 +0900, Damien Le Moal wrote:
> On 2/21/22 23:40, Philippe Schenker wrote:
> > Hello
> > 
> > We use FIO to check the eMMC storage on our embedded boards. This
> > works
> > great for our modules that have >= 1GB RAM but on all lower-memory
> > ones
> > (<= 512MB) the OOM killer always kills FIO.
> > 
> > I can hook up a swap which helps but is just painfully slow.
> 
> Well yes, your system will be trashing...
> 
> > 
> > I played with the settings mem, bs, iodepth and size but cannot
> > achieve
> > a configuration that works on low-memory embedded systems.
> > 
> > Can anyone guide me how I can make this work?
> 
> See if playing with option "--alloc-size=" helps. You can also reduce
> the maximum number of jobs with "--max-jobs=". Also try using threads
> instead of (default) processes for jobs (--thread option). And of
> course
> make sure that the total amount of IO buffers you need (num jobs * io
> depth per job * bs) is not too large for your RAM size.

Thank you very much for your help on this topic. I tried those
parameters back and forth, however it seems to always reserve ~400MB on
the biggest chunk (checked in /proc/<pid-of-fio>/smaps).

decreasing the max allowed shared memory in /proc/sys/kernel/shmmax to
~64M does solve our issue and fio works also on 256MB ram.

However this leads to the question if this is not a bug? I mean it seems
to reserve this memory despite it's not needing it?
What leads me to this assumption is that I can also set mem=malloc
instead of mem=shm and I get still the same error.

We use fio 3.17 is this maybe solved in newer versions?

Best Regards,
Philippe

> 
> > 
> > Best Regards,
> > Philippe
> > 
> 
> 

Re: Using FIO on systems with <=512MB RAM

[Philippe Schenker]

On Tue, 2022-02-22 at 15:00 +0100, Philippe Schenker wrote:
> On Tue, 2022-02-22 at 08:41 +0900, Damien Le Moal wrote:
> > On 2/21/22 23:40, Philippe Schenker wrote:
> > > Hello
> > > 
> > > We use FIO to check the eMMC storage on our embedded boards. This
> > > works
> > > great for our modules that have >= 1GB RAM but on all lower-memory
> > > ones
> > > (<= 512MB) the OOM killer always kills FIO.
> > > 
> > > I can hook up a swap which helps but is just painfully slow.
> > 
> > Well yes, your system will be trashing...
> > 
> > > 
> > > I played with the settings mem, bs, iodepth and size but cannot
> > > achieve
> > > a configuration that works on low-memory embedded systems.
> > > 
> > > Can anyone guide me how I can make this work?
> > 
> > See if playing with option "--alloc-size=" helps. You can also
> > reduce
> > the maximum number of jobs with "--max-jobs=". Also try using
> > threads
> > instead of (default) processes for jobs (--thread option). And of
> > course
> > make sure that the total amount of IO buffers you need (num jobs *
> > io
> > depth per job * bs) is not too large for your RAM size.
> 
> Thank you very much for your help on this topic. I tried those
> parameters back and forth, however it seems to always reserve ~400MB
> on
> the biggest chunk (checked in /proc/<pid-of-fio>/smaps).
> 
> decreasing the max allowed shared memory in /proc/sys/kernel/shmmax to
> ~64M does solve our issue and fio works also on 256MB ram.
> 
> However this leads to the question if this is not a bug? I mean it
> seems
> to reserve this memory despite it's not needing it?
> What leads me to this assumption is that I can also set mem=malloc
> instead of mem=shm and I get still the same error.
> 
> We use fio 3.17 is this maybe solved in newer versions?

I built fio 3.29 in the meantime and tried it with that and oh wonder I
have absolutely no issues, so it seems I hit an old bug.

Sorry for the noise and thanks again Damien for your kind help!

Philippe

> 
> Best Regards,
> Philippe
> 
> > 
> > > 
> > > Best Regards,
> > > Philippe
> > > 
> > 
> > 
> 

Re: Using FIO on systems with <=512MB RAM

[Volodymyr Hoida]

I'm developing and maintaining web based gui for fio (w3top). And I 
can't confirm your concern.

On x64 linux, 380Mb of RAM is enough for both w3top systemd service and 
fio executable.

On ARM32 the w3top+fio pair needs about 300M of ram.

What engines do you use? I'm about libaio, posixaio, io_uring, sync*** 
and mmap v2.21 ... v3.28 on headless linux

2/21/2022 4:40 PM, Philippe Schenker пишет:
> Hello
>
> We use FIO to check the eMMC storage on our embedded boards. This works
> great for our modules that have >= 1GB RAM but on all lower-memory ones
> (<= 512MB) the OOM killer always kills FIO.
>
> I can hook up a swap which helps but is just painfully slow.
>
> I played with the settings mem, bs, iodepth and size but cannot achieve
> a configuration that works on low-memory embedded systems.
>
> Can anyone guide me how I can make this work?
>
> Best Regards,
> Philippe
>

Re: Using FIO on systems with <=512MB RAM

[Philippe Schenker]

Hello Volodymyr

It turned out that the fio version we where using was either buggy or
the patches our buildsystem (Yocto/Openembedded) was applying did cause
that issue.
I updated to the latest fio version in our recipes and I have it running
on 256MB RAM systems now.

Best Regards,
Philippe

On Tue, 2022-03-01 at 00:41 +0200, Volodymyr Hoida wrote:
> I'm developing and maintaining web based gui for fio (w3top). And I 
> can't confirm your concern.
> 
> On x64 linux, 380Mb of RAM is enough for both w3top systemd service
> and 
> fio executable.
> 
> On ARM32 the w3top+fio pair needs about 300M of ram.
> 
> What engines do you use? I'm about libaio, posixaio, io_uring, sync***
> and mmap v2.21 ... v3.28 on headless linux
> 
> 2/21/2022 4:40 PM, Philippe Schenker пишет:
> > Hello
> > 
> > We use FIO to check the eMMC storage on our embedded boards. This
> > works
> > great for our modules that have >= 1GB RAM but on all lower-memory
> > ones
> > (<= 512MB) the OOM killer always kills FIO.
> > 
> > I can hook up a swap which helps but is just painfully slow.
> > 
> > I played with the settings mem, bs, iodepth and size but cannot
> > achieve
> > a configuration that works on low-memory embedded systems.
> > 
> > Can anyone guide me how I can make this work?
> > 
> > Best Regards,
> > Philippe
> >