liblzma: memory allocation failed

liblzma: memory allocation failed

[Peter Bergin]

Hi,

during the task do_package_write_rpm I get the error "liblzma: Memory 
allocation failed". It happens during packaging of binary RPM packages. 
The root cause seems to be the host environment that is used in our 
project. We run our builds on a big server with 32 cores and 256GB of 
physical RAM but each user has a limit of virtual memory usage to 32GB 
(ulimit -v). The packaging in rpm-native has been parallelized in the 
commit 
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3. 
What seems to happen is that rpm-native put up 32 parallel tasks with 
'#pragma omp', each task is using liblzma that also put up 32 tasks for 
the compression work. The memory calculations in liblzma is based on the 
amount of physical RAM but as the user is limited by 'ulimit -v' we get 
into a OOM situation in liblzma.

Here is the code snippet from rpm-native/build/pack.c where it happens:

     #pragma omp parallel
     #pragma omp single
     // re-declaring task variable is necessary, or older gcc versions will produce code that segfaults
     for (struct binaryPackageTaskData *task = tasks; task != NULL; task = task->next) {
         if (task != tasks)
         #pragma omp task
         {
             task->result = packageBinary(spec, task->pkg, cookie, cheating, &(task->filename), buildTime, buildHost);
             rpmlog(RPMLOG_NOTICE, _("Finished binary package job, result %d, filename %s\n"), task->result, task->filename);
         }
     }


Steps to reproduce is to set 'ulimit -v' in your shell to, for example, 
1/8 of the amount of physical RAM and then build for example 
glibc-locale. I have tested this with rocko. If the '#pragma omp' 
statements in code snippet above is removed the problem is solved. But 
that not good as the parallel processing speed up the process.

Is the host environment used here with restricted virtual memory 
supported by Yocto? If it is, someone that have any suggestion for a 
solution on this issue?

Best regards,
/Peter

Re: liblzma: memory allocation failed

[Manjukumar Harthikote Matha]

Hi Peter,

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org]
> On Behalf Of Peter Bergin
> Sent: Sunday, September 16, 2018 1:41 PM
> To: yocto@yoctoproject.org
> Subject: [yocto] liblzma: memory allocation failed
> 
> Hi,
> 
> during the task do_package_write_rpm I get the error "liblzma: Memory
> allocation failed". It happens during packaging of binary RPM packages.
> The root cause seems to be the host environment that is used in our
> project. We run our builds on a big server with 32 cores and 256GB of
> physical RAM but each user has a limit of virtual memory usage to 32GB
> (ulimit -v). The packaging in rpm-native has been parallelized in the
> commit
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-
> devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3.
> What seems to happen is that rpm-native put up 32 parallel tasks with
> '#pragma omp', each task is using liblzma that also put up 32 tasks for
> the compression work. The memory calculations in liblzma is based on the
> amount of physical RAM but as the user is limited by 'ulimit -v' we get
> into a OOM situation in liblzma.
> 
> Here is the code snippet from rpm-native/build/pack.c where it happens:
> 
>      #pragma omp parallel
>      #pragma omp single
>      // re-declaring task variable is necessary, or older gcc versions will produce code
> that segfaults
>      for (struct binaryPackageTaskData *task = tasks; task != NULL; task = task->next) {
>          if (task != tasks)
>          #pragma omp task
>          {
>              task->result = packageBinary(spec, task->pkg, cookie, cheating, &(task-
> >filename), buildTime, buildHost);
>              rpmlog(RPMLOG_NOTICE, _("Finished binary package job, result %d, filename
> %s\n"), task->result, task->filename);
>          }
>      }
> 
> 
> Steps to reproduce is to set 'ulimit -v' in your shell to, for example,
> 1/8 of the amount of physical RAM and then build for example
> glibc-locale. I have tested this with rocko. If the '#pragma omp'
> statements in code snippet above is removed the problem is solved. But
> that not good as the parallel processing speed up the process.
> 
> Is the host environment used here with restricted virtual memory
> supported by Yocto? If it is, someone that have any suggestion for a
> solution on this issue?
> 

We had seen this issue as well and concluded that some settings in the server was causing this.
See http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146705.html

Thanks,
Manju

Re: liblzma: memory allocation failed

[Randy MacLeod]

On 09/16/2018 04:40 PM, Peter Bergin wrote:
> Hi,
> 
> during the task do_package_write_rpm I get the error "liblzma: Memory 
> allocation failed". It happens during packaging of binary RPM packages. 
> The root cause seems to be the host environment that is used in our 
> project. We run our builds on a big server with 32 cores and 256GB of 
> physical RAM but each user has a limit of virtual memory usage to 32GB 
> (ulimit -v). The packaging in rpm-native has been parallelized in the 
> commit 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3. 
> What seems to happen is that rpm-native put up 32 parallel tasks with 
> '#pragma omp', each task is using liblzma that also put up 32 tasks for 

#pragma omp

Tha'ts OpenMP, right? I haven't played with that at all but
it looks like you can limit the number of threads using an
environment variable:
    OMP_NUM_THREADS num
https://www.openmp.org/wp-content/uploads/OpenMP3.0-SummarySpec.pdf

Doing that would be a little ugly but for now at least, there doesn't
seem to be that many packages using such a pragma.

Does that work for your case?

> the compression work. The memory calculations in liblzma is based on the 
> amount of physical RAM but as the user is limited by 'ulimit -v' we get 
> into a OOM situation in liblzma.
> 
> Here is the code snippet from rpm-native/build/pack.c where it happens:
> 
>      #pragma omp parallel
>      #pragma omp single
>      // re-declaring task variable is necessary, or older gcc versions 
> will produce code that segfaults
>      for (struct binaryPackageTaskData *task = tasks; task != NULL; task 
> = task->next) {
>          if (task != tasks)
>          #pragma omp task
>          {
>              task->result = packageBinary(spec, task->pkg, cookie, 
> cheating, &(task->filename), buildTime, buildHost);
>              rpmlog(RPMLOG_NOTICE, _("Finished binary package job, 
> result %d, filename %s\n"), task->result, task->filename);
>          }
>      }
> 
> 
> Steps to reproduce is to set 'ulimit -v' in your shell to, for example, 
> 1/8 of the amount of physical RAM and then build for example 
> glibc-locale. I have tested this with rocko. If the '#pragma omp' 
> statements in code snippet above is removed the problem is solved. But 
> that not good as the parallel processing speed up the process.
> 
> Is the host environment used here with restricted virtual memory 
> supported by Yocto? If it is, someone that have any suggestion for a 
> solution on this issue?


This is a little tricky.

 From bitbake's point of view, it's almost like you are building
on a 32 core, 32 GB box and runing out of RAM/swap.
Clearly we would not fix a build that OOMs in that case
(it does seem odd that 32 GB isn't enough ...)

Are you sure that there isn't something else going on?
I have a 24 core machine with 64 GB RAM that never comes
close to such a problem (so I haven't paid attention to RAM usage).


../Randy

> 
> Best regards,
> /Peter
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: liblzma: memory allocation failed

[Peter Bergin]

Hi Manju,

On 2018-09-17 00:25, Manjukumar Harthikote Matha wrote:
> Hi Peter,
>
>> -----Original Message-----
>> From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org]
>> On Behalf Of Peter Bergin
>> Sent: Sunday, September 16, 2018 1:41 PM
>> To: yocto@yoctoproject.org
>> Subject: [yocto] liblzma: memory allocation failed
>>
>> Hi,
>>
>> during the task do_package_write_rpm I get the error "liblzma: Memory
>> allocation failed". It happens during packaging of binary RPM packages.
>> The root cause seems to be the host environment that is used in our
>> project. We run our builds on a big server with 32 cores and 256GB of
>> physical RAM but each user has a limit of virtual memory usage to 32GB
>> (ulimit -v). The packaging in rpm-native has been parallelized in the
>> commit
>> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-
>> devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3.
>> What seems to happen is that rpm-native put up 32 parallel tasks with
>> '#pragma omp', each task is using liblzma that also put up 32 tasks for
>> the compression work. The memory calculations in liblzma is based on the
>> amount of physical RAM but as the user is limited by 'ulimit -v' we get
>> into a OOM situation in liblzma.
>>
>> Here is the code snippet from rpm-native/build/pack.c where it happens:
>>
>>       #pragma omp parallel
>>       #pragma omp single
>>       // re-declaring task variable is necessary, or older gcc versions will produce code
>> that segfaults
>>       for (struct binaryPackageTaskData *task = tasks; task != NULL; task = task->next) {
>>           if (task != tasks)
>>           #pragma omp task
>>           {
>>               task->result = packageBinary(spec, task->pkg, cookie, cheating, &(task-
>>> filename), buildTime, buildHost);
>>               rpmlog(RPMLOG_NOTICE, _("Finished binary package job, result %d, filename
>> %s\n"), task->result, task->filename);
>>           }
>>       }
>>
>>
>> Steps to reproduce is to set 'ulimit -v' in your shell to, for example,
>> 1/8 of the amount of physical RAM and then build for example
>> glibc-locale. I have tested this with rocko. If the '#pragma omp'
>> statements in code snippet above is removed the problem is solved. But
>> that not good as the parallel processing speed up the process.
>>
>> Is the host environment used here with restricted virtual memory
>> supported by Yocto? If it is, someone that have any suggestion for a
>> solution on this issue?
>>
> We had seen this issue as well and concluded that some settings in the server was causing this.
> See http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146705.html

yes , I saw your post also and the cases are related but not exactly the 
same. On our server the  "vm.overcommit_memory = 0". But I think in both 
cases the thing is that liblzma does memory calculations on physical RAM 
but on OS level we fiddle witht he virtual RAM settings. By changing the 
server settings I can avoid this but this is not an easy task as I'm not 
the sysadmin for the host.

Regards,
/Peter
> Thanks,
> Manju

Re: liblzma: memory allocation failed

[Peter Bergin]

Hi Randy,

On 2018-09-17 06:25, Randy MacLeod wrote:
> On 09/16/2018 04:40 PM, Peter Bergin wrote:
>> Hi,
>>
>> during the task do_package_write_rpm I get the error "liblzma: Memory 
>> allocation failed". It happens during packaging of binary RPM 
>> packages. The root cause seems to be the host environment that is 
>> used in our project. We run our builds on a big server with 32 cores 
>> and 256GB of physical RAM but each user has a limit of virtual memory 
>> usage to 32GB (ulimit -v). The packaging in rpm-native has been 
>> parallelized in the commit 
>> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3. 
>> What seems to happen is that rpm-native put up 32 parallel tasks with 
>> '#pragma omp', each task is using liblzma that also put up 32 tasks for 
>
> #pragma omp
>
> Tha'ts OpenMP, right? I haven't played with that at all but
> it looks like you can limit the number of threads using an
> environment variable:
>    OMP_NUM_THREADS num
> https://www.openmp.org/wp-content/uploads/OpenMP3.0-SummarySpec.pdf
>
> Doing that would be a little ugly but for now at least, there doesn't
> seem to be that many packages using such a pragma.
>
> Does that work for your case?
>
Yes, it's OpenMP. I tried '#pragma omp parallel num_thread(4)' and it 
worked as a workaround. On the failing server the build succeeded. The 
problem is to get this as a generic solution based on the host settings 
because the #pragma is a compiler directive. But for sure we can make a 
bbappend on this to get it working on our host.

>> the compression work. The memory calculations in liblzma is based on 
>> the amount of physical RAM but as the user is limited by 'ulimit -v' 
>> we get into a OOM situation in liblzma.
>>
>> Here is the code snippet from rpm-native/build/pack.c where it happens:
>>
>>      #pragma omp parallel
>>      #pragma omp single
>>      // re-declaring task variable is necessary, or older gcc 
>> versions will produce code that segfaults
>>      for (struct binaryPackageTaskData *task = tasks; task != NULL; 
>> task = task->next) {
>>          if (task != tasks)
>>          #pragma omp task
>>          {
>>              task->result = packageBinary(spec, task->pkg, cookie, 
>> cheating, &(task->filename), buildTime, buildHost);
>>              rpmlog(RPMLOG_NOTICE, _("Finished binary package job, 
>> result %d, filename %s\n"), task->result, task->filename);
>>          }
>>      }
>>
>>
>> Steps to reproduce is to set 'ulimit -v' in your shell to, for 
>> example, 1/8 of the amount of physical RAM and then build for example 
>> glibc-locale. I have tested this with rocko. If the '#pragma omp' 
>> statements in code snippet above is removed the problem is solved. 
>> But that not good as the parallel processing speed up the process.
>>
>> Is the host environment used here with restricted virtual memory 
>> supported by Yocto? If it is, someone that have any suggestion for a 
>> solution on this issue?
>
>
> This is a little tricky.
>
> From bitbake's point of view, it's almost like you are building
> on a 32 core, 32 GB box and runing out of RAM/swap.
> Clearly we would not fix a build that OOMs in that case
> (it does seem odd that 32 GB isn't enough ...)
>
> Are you sure that there isn't something else going on?
> I have a 24 core machine with 64 GB RAM that never comes
> close to such a problem (so I haven't paid attention to RAM usage).
>
I'm pretty sure I have narrowed down the root cause to the restriction 
of virtual memory and that liblzma base its memory calculations on 
physical RAM.

To prove this I added a printout in rpm-native/rpmio/rpmio.c and the 
function lzopen_internal.

         uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
	rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);


The value of memory_usage is the same regardless of which 'ulimit -v' 
value I set. On the host with 256GB of physical RAM and 32GB of virtual 
memory, memory_usage is ~5.1GB. On another host with 16GB of physical 
RAM I get memory_usage of ~660MB.

I guess you have not seen this kind of failure if you not have 
restricted virutal memory on your host. If you want to try to reproduce 
this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake 
glibc-locale -c package_write_rpm -f'.

Best regards,
/Peter

>
> ../Randy
>
>>
>> Best regards,
>> /Peter
>>
>>
>>
>>
>
>

Re: liblzma: memory allocation failed

[Burton, Ross]

On Mon, 17 Sep 2018 at 08:13, Peter Bergin <peter@berginkonsult.se> wrote:
> I'm pretty sure I have narrowed down the root cause to the restriction
> of virtual memory and that liblzma base its memory calculations on
> physical RAM.
>
> To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
> function lzopen_internal.
>
>          uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
>         rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
>
>
> The value of memory_usage is the same regardless of which 'ulimit -v'
> value I set. On the host with 256GB of physical RAM and 32GB of virtual
> memory, memory_usage is ~5.1GB. On another host with 16GB of physical
> RAM I get memory_usage of ~660MB.
>
> I guess you have not seen this kind of failure if you not have
> restricted virutal memory on your host. If you want to try to reproduce
> this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
> glibc-locale -c package_write_rpm -f'.

Wouldn't a solution be to change lzma to look at free memory, not
total physical memory?

Ross

Re: liblzma: memory allocation failed

[Andrea Adami]

On Mon, Sep 17, 2018 at 10:28 AM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Mon, 17 Sep 2018 at 08:13, Peter Bergin <peter@berginkonsult.se> wrote:
> > I'm pretty sure I have narrowed down the root cause to the restriction
> > of virtual memory and that liblzma base its memory calculations on
> > physical RAM.


Hello,

well, not only.
You can set the memory footprint for compression/decompression.

In OE for legacy kernels we use in our BSP:
# sane defaults for devices with only 32Mb RAM (see man xz)
XZ_COMPRESSION_LEVEL = "-2e"

Default is -3, the -2 uses right half the RAM for compressing,.
Pls check man xz.

Cheers
Andrea


> >
> > To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
> > function lzopen_internal.
> >
> >          uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
> >         rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
> >
> >
> > The value of memory_usage is the same regardless of which 'ulimit -v'
> > value I set. On the host with 256GB of physical RAM and 32GB of virtual
> > memory, memory_usage is ~5.1GB. On another host with 16GB of physical
> > RAM I get memory_usage of ~660MB.
> >
> > I guess you have not seen this kind of failure if you not have
> > restricted virutal memory on your host. If you want to try to reproduce
> > this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
> > glibc-locale -c package_write_rpm -f'.
>
> Wouldn't a solution be to change lzma to look at free memory, not
> total physical memory?
>
> Ross
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: liblzma: memory allocation failed

[Peter Bergin]

[-- Attachment #1: Type: text/plain, Size: 2476 bytes --]

On 2018-09-17 10:27, Burton, Ross wrote:
> On Mon, 17 Sep 2018 at 08:13, Peter Bergin<peter@berginkonsult.se>  wrote:
>> I'm pretty sure I have narrowed down the root cause to the restriction
>> of virtual memory and that liblzma base its memory calculations on
>> physical RAM.
>>
>> To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
>> function lzopen_internal.
>>
>>           uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
>>          rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
>>
>>
>> The value of memory_usage is the same regardless of which 'ulimit -v'
>> value I set. On the host with 256GB of physical RAM and 32GB of virtual
>> memory, memory_usage is ~5.1GB. On another host with 16GB of physical
>> RAM I get memory_usage of ~660MB.
>>
>> I guess you have not seen this kind of failure if you not have
>> restricted virutal memory on your host. If you want to try to reproduce
>> this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
>> glibc-locale -c package_write_rpm -f'.
> Wouldn't a solution be to change lzma to look at free memory, not
> total physical memory?
>
> Ross
I have been in contact with the maintainer of liblzma. There is 
currently no way to restrict the memory usage in liblzma during multi 
threaded compression. He recommended to adjust the number of threads 
used during compression. This kind of check is done for 32-bits but not 
when running 64-bits (in rpmio/rpmio.c lzopen_internal). To rewrite 
liblzma is another option but I took an easier way out.

I have come up with a patch 
(https://patchwork.openembedded.org/patch/155017/) that solves my 
problem and do a restriction of memory usage when the 'ulimit -v' is 
set. The calculation is based on the assumption that lzopen_internal is 
run in parallel with as many instances as cpu threads as '#pragma omp 
parallel' is used in build/pack.c.

When running test on my machine with 4 cores 16GB of physical RAM and 
'ulimit -v 2097152' (2GB). It works good and the log shows:

XZ: virtual memory restricted to 2147483648 and per CPU thread 536870912
XZ: Adjusted the number of threads from 4 to 3 to not exceed the memory 
usage limit of 2147483648 bytes

Didn't get a clear answer if this is something Yocto/OE should support 
but I hope my patch solves the issue and it will not affect the normal 
environments where 'ulimit -v' is not set.

/Peter

[-- Attachment #2: Type: text/html, Size: 3221 bytes --]

Re: liblzma: memory allocation failed

[Andrea Adami]

On Thu, Sep 20, 2018 at 1:03 PM Peter Bergin <peter@berginkonsult.se> wrote:
>
> On 2018-09-17 10:27, Burton, Ross wrote:
>
> On Mon, 17 Sep 2018 at 08:13, Peter Bergin <peter@berginkonsult.se> wrote:
>
> I'm pretty sure I have narrowed down the root cause to the restriction
> of virtual memory and that liblzma base its memory calculations on
> physical RAM.
>
> To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
> function lzopen_internal.
>
>          uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
>         rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
>
>
> The value of memory_usage is the same regardless of which 'ulimit -v'
> value I set. On the host with 256GB of physical RAM and 32GB of virtual
> memory, memory_usage is ~5.1GB. On another host with 16GB of physical
> RAM I get memory_usage of ~660MB.
>
> I guess you have not seen this kind of failure if you not have
> restricted virutal memory on your host. If you want to try to reproduce
> this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
> glibc-locale -c package_write_rpm -f'.
>
> Wouldn't a solution be to change lzma to look at free memory, not
> total physical memory?
>
> Ross
>
> I have been in contact with the maintainer of liblzma. There is currently no way to restrict the memory usage in liblzma during multi threaded compression. He recommended to adjust the number of threads used during compression. This kind of check is done for 32-bits but not when running 64-bits (in rpmio/rpmio.c lzopen_internal). To rewrite liblzma is another option but I took an easier way out.
>
> I have come up with a patch (https://patchwork.openembedded.org/patch/155017/) that solves my problem and do a restriction of memory usage when the 'ulimit -v' is set. The calculation is based on the assumption that lzopen_internal is run in parallel with as many instances as cpu threads as '#pragma omp parallel' is used in build/pack.c.
>
> When running test on my machine with 4 cores 16GB of physical RAM and 'ulimit -v 2097152' (2GB). It works good and the log shows:
>
> XZ: virtual memory restricted to 2147483648 and per CPU thread 536870912
> XZ: Adjusted the number of threads from 4 to 3 to not exceed the memory usage limit of 2147483648 bytes
>
> Didn't get a clear answer if this is something Yocto/OE should support but I hope my patch solves the issue and it will not affect the normal environments where 'ulimit -v' is not set.
>
> /Peter
> --

Peter,
first of all sorry for butting-in again. My bad I misunderstood the point,

Looking properly at it, I see the sources do restrict only #if __WORDSIZE == 32.

Again, there is the other way to reduce total memory footprint just
using lower preset in mt_options.
As far as I see default is  #define LZMA_PRESET_DEFAULT     UINT32_C(6)
so you could tr to modify   mt_options.preset to your needs.
I think more threads is normally  better.

Cheers
Andrea

> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: liblzma: memory allocation failed

[Andrea Adami]

On Fri, Sep 21, 2018 at 1:04 AM Andrea Adami <andrea.adami@gmail.com> wrote:
>
> On Thu, Sep 20, 2018 at 1:03 PM Peter Bergin <peter@berginkonsult.se> wrote:
> >
> > On 2018-09-17 10:27, Burton, Ross wrote:
> >
> > On Mon, 17 Sep 2018 at 08:13, Peter Bergin <peter@berginkonsult.se> wrote:
> >
> > I'm pretty sure I have narrowed down the root cause to the restriction
> > of virtual memory and that liblzma base its memory calculations on
> > physical RAM.
> >
> > To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
> > function lzopen_internal.
> >
> >          uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
> >         rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
> >
> >
> > The value of memory_usage is the same regardless of which 'ulimit -v'
> > value I set. On the host with 256GB of physical RAM and 32GB of virtual
> > memory, memory_usage is ~5.1GB. On another host with 16GB of physical
> > RAM I get memory_usage of ~660MB.
> >
> > I guess you have not seen this kind of failure if you not have
> > restricted virutal memory on your host. If you want to try to reproduce
> > this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
> > glibc-locale -c package_write_rpm -f'.
> >
> > Wouldn't a solution be to change lzma to look at free memory, not
> > total physical memory?
> >
> > Ross
> >
> > I have been in contact with the maintainer of liblzma. There is currently no way to restrict the memory usage in liblzma during multi threaded compression. He recommended to adjust the number of threads used during compression. This kind of check is done for 32-bits but not when running 64-bits (in rpmio/rpmio.c lzopen_internal). To rewrite liblzma is another option but I took an easier way out.
> >
> > I have come up with a patch (https://patchwork.openembedded.org/patch/155017/) that solves my problem and do a restriction of memory usage when the 'ulimit -v' is set. The calculation is based on the assumption that lzopen_internal is run in parallel with as many instances as cpu threads as '#pragma omp parallel' is used in build/pack.c.
> >
> > When running test on my machine with 4 cores 16GB of physical RAM and 'ulimit -v 2097152' (2GB). It works good and the log shows:
> >
> > XZ: virtual memory restricted to 2147483648 and per CPU thread 536870912
> > XZ: Adjusted the number of threads from 4 to 3 to not exceed the memory usage limit of 2147483648 bytes
> >
> > Didn't get a clear answer if this is something Yocto/OE should support but I hope my patch solves the issue and it will not affect the normal environments where 'ulimit -v' is not set.
> >
> > /Peter
> > --
>
> Peter,
> first of all sorry for butting-in again. My bad I misunderstood the point,
>
> Looking properly at it, I see the sources do restrict only #if __WORDSIZE == 32.
>
> Again, there is the other way to reduce total memory footprint just
> using lower preset in mt_options.
> As far as I see default is  #define LZMA_PRESET_DEFAULT     UINT32_C(6)
> so you could tr to modify   mt_options.preset to your needs.
> I think more threads is normally  better.
>
> Cheers
> Andrea
>
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto


Well,
call it luck, I now get similar problems with
-  opkg-utils: use multithreaded xz when building packages
   369c43cca4f0

I build in tmpfs (8GB) and have 8 cores.
Strange enough, it does happen with only one machine of our bsp (akita).

Build Configuration:
BB_VERSION           = "1.39.1"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-16.04"
TARGET_SYS           = "arm-oe-linux-gnueabi"
MACHINE              = "akita"
DISTRO               = "nodistro"
DISTRO_VERSION       = "nodistro.0"
TUNE_FEATURES        = "arm armv5 thumb dsp"
TARGET_FPU           = "soft"
meta                 = "master:8cc08e44cf3e969c124456d459c6c06a76cad018"
meta-handheld        = "master:61e16e253ea870b87592075efd87727e8b5c574d"
meta-initramfs
meta-oe              = "master:7ac8d82fec470a8c8e66cfc4216381e2364cb4b2"

...
Installing kernel-module-nls-cp1251-4.14.62-handheld (4.14) on root
Downloading file:/tmp/build/tmp-glibc/work/akita-oe-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/akita/kernel-module-nls-cp1251-4.14.62-handheld_4.14-r0_akita.ipk.
Installing kernel-mo
ERROR: core-image-base-1.0-r0 do_rootfs: Function failed: do_rootfs
ERROR: Logfile of failure stored in:
/tmp/build/tmp-glibc/work/akita-oe-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.28481
ERROR: Task (/oe/oe-core/meta/recipes-core/images/core-image-base.bb:do_rootfs)
failed with exit code '1'
...

and in dmesg you see opkg segfaulting and OOM kicking in.
It is the same issue...unnoticed building for other similar devices
(spitz). Maybe it did not happen using musl as TC_LIBC.

Reverting the patch fixes that.
Ross, worth opening a bug?

Cheers
Andrea

Re: liblzma: memory allocation failed

[Peter Bergin]

On 2018-09-24 00:06, Andrea Adami wrote:
> On Fri, Sep 21, 2018 at 1:04 AM Andrea Adami <andrea.adami@gmail.com> wrote:
>> On Thu, Sep 20, 2018 at 1:03 PM Peter Bergin <peter@berginkonsult.se> wrote:
>>> On 2018-09-17 10:27, Burton, Ross wrote:
>>>
>>> On Mon, 17 Sep 2018 at 08:13, Peter Bergin <peter@berginkonsult.se> wrote:
>>>
>>> I'm pretty sure I have narrowed down the root cause to the restriction
>>> of virtual memory and that liblzma base its memory calculations on
>>> physical RAM.
>>>
>>> To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
>>> function lzopen_internal.
>>>
>>>           uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
>>>          rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
>>>
>>>
>>> The value of memory_usage is the same regardless of which 'ulimit -v'
>>> value I set. On the host with 256GB of physical RAM and 32GB of virtual
>>> memory, memory_usage is ~5.1GB. On another host with 16GB of physical
>>> RAM I get memory_usage of ~660MB.
>>>
>>> I guess you have not seen this kind of failure if you not have
>>> restricted virutal memory on your host. If you want to try to reproduce
>>> this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
>>> glibc-locale -c package_write_rpm -f'.
>>>
>>> Wouldn't a solution be to change lzma to look at free memory, not
>>> total physical memory?
>>>
>>> Ross
>>>
>>> I have been in contact with the maintainer of liblzma. There is currently no way to restrict the memory usage in liblzma during multi threaded compression. He recommended to adjust the number of threads used during compression. This kind of check is done for 32-bits but not when running 64-bits (in rpmio/rpmio.c lzopen_internal). To rewrite liblzma is another option but I took an easier way out.
>>>
>>> I have come up with a patch (https://patchwork.openembedded.org/patch/155017/) that solves my problem and do a restriction of memory usage when the 'ulimit -v' is set. The calculation is based on the assumption that lzopen_internal is run in parallel with as many instances as cpu threads as '#pragma omp parallel' is used in build/pack.c.
>>>
>>> When running test on my machine with 4 cores 16GB of physical RAM and 'ulimit -v 2097152' (2GB). It works good and the log shows:
>>>
>>> XZ: virtual memory restricted to 2147483648 and per CPU thread 536870912
>>> XZ: Adjusted the number of threads from 4 to 3 to not exceed the memory usage limit of 2147483648 bytes
>>>
>>> Didn't get a clear answer if this is something Yocto/OE should support but I hope my patch solves the issue and it will not affect the normal environments where 'ulimit -v' is not set.
>>>
>>> /Peter
>>> --
>> Peter,
>> first of all sorry for butting-in again. My bad I misunderstood the point,
>>
>> Looking properly at it, I see the sources do restrict only #if __WORDSIZE == 32.
>>
>> Again, there is the other way to reduce total memory footprint just
>> using lower preset in mt_options.
>> As far as I see default is  #define LZMA_PRESET_DEFAULT     UINT32_C(6)
>> so you could tr to modify   mt_options.preset to your needs.
>> I think more threads is normally  better.
>>
>> Cheers
>> Andrea
>>
>>> _______________________________________________
>>> yocto mailing list
>>> yocto@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/yocto
>
> Well,
> call it luck, I now get similar problems with
> -  opkg-utils: use multithreaded xz when building packages
>     369c43cca4f0
>
> I build in tmpfs (8GB) and have 8 cores.
> Strange enough, it does happen with only one machine of our bsp (akita).
>
> Build Configuration:
> BB_VERSION           = "1.39.1"
> BUILD_SYS            = "x86_64-linux"
> NATIVELSBSTRING      = "ubuntu-16.04"
> TARGET_SYS           = "arm-oe-linux-gnueabi"
> MACHINE              = "akita"
> DISTRO               = "nodistro"
> DISTRO_VERSION       = "nodistro.0"
> TUNE_FEATURES        = "arm armv5 thumb dsp"
> TARGET_FPU           = "soft"
> meta                 = "master:8cc08e44cf3e969c124456d459c6c06a76cad018"
> meta-handheld        = "master:61e16e253ea870b87592075efd87727e8b5c574d"
> meta-initramfs
> meta-oe              = "master:7ac8d82fec470a8c8e66cfc4216381e2364cb4b2"
>
> ...
> Installing kernel-module-nls-cp1251-4.14.62-handheld (4.14) on root
> Downloading file:/tmp/build/tmp-glibc/work/akita-oe-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/akita/kernel-module-nls-cp1251-4.14.62-handheld_4.14-r0_akita.ipk.
> Installing kernel-mo
> ERROR: core-image-base-1.0-r0 do_rootfs: Function failed: do_rootfs
> ERROR: Logfile of failure stored in:
> /tmp/build/tmp-glibc/work/akita-oe-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.28481
> ERROR: Task (/oe/oe-core/meta/recipes-core/images/core-image-base.bb:do_rootfs)
> failed with exit code '1'
> ...
>
> and in dmesg you see opkg segfaulting and OOM kicking in.
> It is the same issue...unnoticed building for other similar devices
> (spitz). Maybe it did not happen using musl as TC_LIBC.
>
> Reverting the patch fixes that.
> Ross, worth opening a bug?
>
> Cheers
> Andrea
Andrea,

do you have any virtual memory restrictions (ulimit -v) or overcommit 
settings (vm.overcommit_memory)  on your build host?

Regarding my problem with rpm and liblzma when virtual memory is 
restricted I have supplied a patch to oe-core 
https://patchwork.openembedded.org/patch/155017/ with a suggested solution.

Regards,
/Peter

Re: liblzma: memory allocation failed

[Andrea Adami]

On Mon, Sep 24, 2018 at 9:08 AM Peter Bergin <peter@berginkonsult.se> wrote:
>
> On 2018-09-24 00:06, Andrea Adami wrote:
> > On Fri, Sep 21, 2018 at 1:04 AM Andrea Adami <andrea.adami@gmail.com> wrote:
> >> On Thu, Sep 20, 2018 at 1:03 PM Peter Bergin <peter@berginkonsult.se> wrote:
> >>> On 2018-09-17 10:27, Burton, Ross wrote:
> >>>
> >>> On Mon, 17 Sep 2018 at 08:13, Peter Bergin <peter@berginkonsult.se> wrote:
> >>>
> >>> I'm pretty sure I have narrowed down the root cause to the restriction
> >>> of virtual memory and that liblzma base its memory calculations on
> >>> physical RAM.
> >>>
> >>> To prove this I added a printout in rpm-native/rpmio/rpmio.c and the
> >>> function lzopen_internal.
> >>>
> >>>           uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
> >>>          rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);
> >>>
> >>>
> >>> The value of memory_usage is the same regardless of which 'ulimit -v'
> >>> value I set. On the host with 256GB of physical RAM and 32GB of virtual
> >>> memory, memory_usage is ~5.1GB. On another host with 16GB of physical
> >>> RAM I get memory_usage of ~660MB.
> >>>
> >>> I guess you have not seen this kind of failure if you not have
> >>> restricted virutal memory on your host. If you want to try to reproduce
> >>> this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake
> >>> glibc-locale -c package_write_rpm -f'.
> >>>
> >>> Wouldn't a solution be to change lzma to look at free memory, not
> >>> total physical memory?
> >>>
> >>> Ross
> >>>
> >>> I have been in contact with the maintainer of liblzma. There is currently no way to restrict the memory usage in liblzma during multi threaded compression. He recommended to adjust the number of threads used during compression. This kind of check is done for 32-bits but not when running 64-bits (in rpmio/rpmio.c lzopen_internal). To rewrite liblzma is another option but I took an easier way out.
> >>>
> >>> I have come up with a patch (https://patchwork.openembedded.org/patch/155017/) that solves my problem and do a restriction of memory usage when the 'ulimit -v' is set. The calculation is based on the assumption that lzopen_internal is run in parallel with as many instances as cpu threads as '#pragma omp parallel' is used in build/pack.c.
> >>>
> >>> When running test on my machine with 4 cores 16GB of physical RAM and 'ulimit -v 2097152' (2GB). It works good and the log shows:
> >>>
> >>> XZ: virtual memory restricted to 2147483648 and per CPU thread 536870912
> >>> XZ: Adjusted the number of threads from 4 to 3 to not exceed the memory usage limit of 2147483648 bytes
> >>>
> >>> Didn't get a clear answer if this is something Yocto/OE should support but I hope my patch solves the issue and it will not affect the normal environments where 'ulimit -v' is not set.
> >>>
> >>> /Peter
> >>> --
> >> Peter,
> >> first of all sorry for butting-in again. My bad I misunderstood the point,
> >>
> >> Looking properly at it, I see the sources do restrict only #if __WORDSIZE == 32.
> >>
> >> Again, there is the other way to reduce total memory footprint just
> >> using lower preset in mt_options.
> >> As far as I see default is  #define LZMA_PRESET_DEFAULT     UINT32_C(6)
> >> so you could tr to modify   mt_options.preset to your needs.
> >> I think more threads is normally  better.
> >>
> >> Cheers
> >> Andrea
> >>
> >>> _______________________________________________
> >>> yocto mailing list
> >>> yocto@yoctoproject.org
> >>> https://lists.yoctoproject.org/listinfo/yocto
> >
> > Well,
> > call it luck, I now get similar problems with
> > -  opkg-utils: use multithreaded xz when building packages
> >     369c43cca4f0
> >
> > I build in tmpfs (8GB) and have 8 cores.
> > Strange enough, it does happen with only one machine of our bsp (akita).
> >
> > Build Configuration:
> > BB_VERSION           = "1.39.1"
> > BUILD_SYS            = "x86_64-linux"
> > NATIVELSBSTRING      = "ubuntu-16.04"
> > TARGET_SYS           = "arm-oe-linux-gnueabi"
> > MACHINE              = "akita"
> > DISTRO               = "nodistro"
> > DISTRO_VERSION       = "nodistro.0"
> > TUNE_FEATURES        = "arm armv5 thumb dsp"
> > TARGET_FPU           = "soft"
> > meta                 = "master:8cc08e44cf3e969c124456d459c6c06a76cad018"
> > meta-handheld        = "master:61e16e253ea870b87592075efd87727e8b5c574d"
> > meta-initramfs
> > meta-oe              = "master:7ac8d82fec470a8c8e66cfc4216381e2364cb4b2"
> >
> > ...
> > Installing kernel-module-nls-cp1251-4.14.62-handheld (4.14) on root
> > Downloading file:/tmp/build/tmp-glibc/work/akita-oe-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/akita/kernel-module-nls-cp1251-4.14.62-handheld_4.14-r0_akita.ipk.
> > Installing kernel-mo
> > ERROR: core-image-base-1.0-r0 do_rootfs: Function failed: do_rootfs
> > ERROR: Logfile of failure stored in:
> > /tmp/build/tmp-glibc/work/akita-oe-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.28481
> > ERROR: Task (/oe/oe-core/meta/recipes-core/images/core-image-base.bb:do_rootfs)
> > failed with exit code '1'
> > ...
> >
> > and in dmesg you see opkg segfaulting and OOM kicking in.
> > It is the same issue...unnoticed building for other similar devices
> > (spitz). Maybe it did not happen using musl as TC_LIBC.
> >
> > Reverting the patch fixes that.
> > Ross, worth opening a bug?
> >
> > Cheers
> > Andrea
> Andrea,
>
> do you have any virtual memory restrictions (ulimit -v) or overcommit
> settings (vm.overcommit_memory)  on your build host?
>

Anything special, is just Ubuntu 16.04 on x86_64.

The point is, I do build since years with this setup:
- TMPDIR in tmpfs  (80% of 16GiB set in fstab)
- 8 cores
- rm_work

Lately I have been testing musl libc so probably I missed it back in march.
As for the opkg backend, the solution is very easy (if we want to keep
the parallelism):

- compressorargs="--threads 0"
+ compressorargs="--threads 0 -O 3"

The preset could be even 4 or 5, we need to test and accept a minimum.
I'll test on my setup.

> Regarding my problem with rpm and liblzma when virtual memory is
> restricted I have supplied a patch to oe-core
> https://patchwork.openembedded.org/patch/155017/ with a suggested solution.
>

Yes, I have seen it. IMHO the sources are buggy.

I still think we are giving too much memory (default is -O 6)
Cheers
Andrea

> Regards,
> /Peter