[Bug 1896298] [NEW] memory leak

[Bug 1896298] [NEW] memory leak

[Michael Slade]

Public bug reported:

qemu trunk as of today leaks memory FAST when freedos' edit is running.

To reproduce, download:

https://www.ibiblio.org/pub/micro/pc-
stuff/freedos/files/repositories/1.3/cdrom.iso

Then run:

$ qemu-system-i386 -cdrom cdrom.iso

select your language then select "return to DOS", then type

> edit

it will consume memory at ~10MB/s

This does NOT happen when adding -enable-kvm

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  memory leak

Status in QEMU:
  New

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

[Bug 1896298] Re: memory leak

[Michael Slade]

Note, this also occurs with freeDOS 1.2, at least.

Note 2, 4.2 stable does not exhibit the bug.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  memory leak

Status in QEMU:
  New

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

[Bug 1896298] Re: memory leak

[Thomas Huth]

Confirmed, this is still reproducible with the current v5.2-rc4...

** Changed in: qemu
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  memory leak

Status in QEMU:
  Confirmed

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

[Bug 1896298] Re: memory leak

[Thomas Huth]

** Tags added: tcg

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  memory leak

Status in QEMU:
  Confirmed

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

[Bug 1896298] Re: TCG memory leak with FreeDOS 'edit'

[Thomas Huth]

This is an automated cleanup. This bug report has been moved to QEMU's
new bug tracker on gitlab.com and thus gets marked as 'expired' now.
Please continue with the discussion here:

 https://gitlab.com/qemu-project/qemu/-/issues/283


** Summary changed:

- memory leak
+ TCG memory leak with FreeDOS 'edit'

** Changed in: qemu
       Status: Confirmed => Expired

** Bug watch added: gitlab.com/qemu-project/qemu/-/issues #283
   https://gitlab.com/qemu-project/qemu/-/issues/283

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  TCG memory leak with FreeDOS 'edit'

Status in QEMU:
  Expired

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

[Bug 1896298] Re: TCG memory leak with FreeDOS 'edit'

[Alex Bennée]

Just to repeat the work around discussed on the GitLab page: -accel tcg
,tb-size=32 will prevent the rapid increase of memory due to self
modifying code.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  TCG memory leak with FreeDOS 'edit'

Status in QEMU:
  Expired

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

[Bug 1896298] [RFC PATCH] accel/tcg: change default codegen buffer size for i386-softmmu

[Alex Bennée]

There are two justifications for making this change. The first is that
i386 emulation is typically for smaller machines where having a 1gb of
generated code is overkill for basic emulation. The second is the
propensity of self-modifying code (c.f. Doom/edit) utilised on i386
systems can trigger a rapid growth in invalidated and re-translated
buffers. This is seen in bug #283. Execution is still inefficient but
at least the host memory isn't so aggressively used up.

That said it's still really just a sticking plaster for user
convenience.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Thomas Huth <thuth@redhat.com>
Cc: 1896298@bugs.launchpad.net
---
 accel/tcg/translate-all.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 640ff6e3e7..f442165674 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -951,9 +951,13 @@ static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1,
  * Users running large scale system emulation may want to tweak their
  * runtime setup via the tb-size control on the command line.
  */
+#ifdef TARGET_I386
+#define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (32 * MiB)
+#else
 #define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (1 * GiB)
 #endif
 #endif
+#endif
 
 #define DEFAULT_CODE_GEN_BUFFER_SIZE \
   (DEFAULT_CODE_GEN_BUFFER_SIZE_1 < MAX_CODE_GEN_BUFFER_SIZE \
-- 
2.20.1

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  TCG memory leak with FreeDOS 'edit'

Status in QEMU:
  Expired

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

[RFC PATCH] accel/tcg: change default codegen buffer size for i386-softmmu

[Alex Bennée]

There are two justifications for making this change. The first is that
i386 emulation is typically for smaller machines where having a 1gb of
generated code is overkill for basic emulation. The second is the
propensity of self-modifying code (c.f. Doom/edit) utilised on i386
systems can trigger a rapid growth in invalidated and re-translated
buffers. This is seen in bug #283. Execution is still inefficient but
at least the host memory isn't so aggressively used up.

That said it's still really just a sticking plaster for user
convenience.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Thomas Huth <thuth@redhat.com>
Cc: 1896298@bugs.launchpad.net
---
 accel/tcg/translate-all.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 640ff6e3e7..f442165674 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -951,9 +951,13 @@ static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1,
  * Users running large scale system emulation may want to tweak their
  * runtime setup via the tb-size control on the command line.
  */
+#ifdef TARGET_I386
+#define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (32 * MiB)
+#else
 #define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (1 * GiB)
 #endif
 #endif
+#endif
 
 #define DEFAULT_CODE_GEN_BUFFER_SIZE \
   (DEFAULT_CODE_GEN_BUFFER_SIZE_1 < MAX_CODE_GEN_BUFFER_SIZE \
-- 
2.20.1

Re: [RFC PATCH] accel/tcg: change default codegen buffer size for i386-softmmu

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> There are two justifications for making this change. The first is that
> i386 emulation is typically for smaller machines where having a 1gb of
> generated code is overkill for basic emulation. The second is the
> propensity of self-modifying code (c.f. Doom/edit) utilised on i386
> systems can trigger a rapid growth in invalidated and re-translated
> buffers. This is seen in bug #283. Execution is still inefficient but
> at least the host memory isn't so aggressively used up.
>
> That said it's still really just a sticking plaster for user
> convenience.

ping?


-- 
Alex Bennée

[Bug 1896298] Re: [RFC PATCH] accel/tcg: change default codegen buffer size for i386-softmmu

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> There are two justifications for making this change. The first is that
> i386 emulation is typically for smaller machines where having a 1gb of
> generated code is overkill for basic emulation. The second is the
> propensity of self-modifying code (c.f. Doom/edit) utilised on i386
> systems can trigger a rapid growth in invalidated and re-translated
> buffers. This is seen in bug #283. Execution is still inefficient but
> at least the host memory isn't so aggressively used up.
>
> That said it's still really just a sticking plaster for user
> convenience.

ping?


-- 
Alex Bennée

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  TCG memory leak with FreeDOS 'edit'

Status in QEMU:
  Expired

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions

Re: [RFC PATCH] accel/tcg: change default codegen buffer size for i386-softmmu

[Richard Henderson]

On 5/25/21 9:45 AM, Alex Bennée wrote:
> There are two justifications for making this change. The first is that
> i386 emulation is typically for smaller machines where having a 1gb of
> generated code is overkill for basic emulation. The second is the
> propensity of self-modifying code (c.f. Doom/edit) utilised on i386
> systems can trigger a rapid growth in invalidated and re-translated
> buffers. This is seen in bug #283. Execution is still inefficient but
> at least the host memory isn't so aggressively used up.
> 
> That said it's still really just a sticking plaster for user
> convenience.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: 1896298@bugs.launchpad.net
> ---
>   accel/tcg/translate-all.c | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
> index 640ff6e3e7..f442165674 100644
> --- a/accel/tcg/translate-all.c
> +++ b/accel/tcg/translate-all.c
> @@ -951,9 +951,13 @@ static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1,
>    * Users running large scale system emulation may want to tweak their
>    * runtime setup via the tb-size control on the command line.
>    */
> +#ifdef TARGET_I386
> +#define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (32 * MiB)
> +#else
>   #define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (1 * GiB)
>   #endif
>   #endif
> +#endif
>   
>   #define DEFAULT_CODE_GEN_BUFFER_SIZE \
>     (DEFAULT_CODE_GEN_BUFFER_SIZE_1 < MAX_CODE_GEN_BUFFER_SIZE \
> 

I'm not thrilled, as it is ultra-hacky.

(1) I've got a re-org of this code out for review: 
https://patchew.org/QEMU/20210502231844.1977630-1-richard.henderson@linaro.org/

(2) I'm keen to reorg TCG such that it gets compiled once.  There's currently 
nothing standing in the way of that except work.  But this would introduce a 
use of a target-specific define for the first time into tcg/.  I guess I could 
leave the default sizing back in accel/tcg/ and pass in the default.

Other options?


r~

Re: [RFC PATCH] accel/tcg: change default codegen buffer size for i386-softmmu

[Alex Bennée]

Richard Henderson <richard.henderson@linaro.org> writes:

> On 5/25/21 9:45 AM, Alex Bennée wrote:
>> There are two justifications for making this change. The first is that
>> i386 emulation is typically for smaller machines where having a 1gb of
>> generated code is overkill for basic emulation. The second is the
>> propensity of self-modifying code (c.f. Doom/edit) utilised on i386
>> systems can trigger a rapid growth in invalidated and re-translated
>> buffers. This is seen in bug #283. Execution is still inefficient but
>> at least the host memory isn't so aggressively used up.
>> That said it's still really just a sticking plaster for user
>> convenience.
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Thomas Huth <thuth@redhat.com>
>> Cc: 1896298@bugs.launchpad.net
>> ---
>>   accel/tcg/translate-all.c | 4 ++++
>>   1 file changed, 4 insertions(+)
>> diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
>> index 640ff6e3e7..f442165674 100644
>> --- a/accel/tcg/translate-all.c
>> +++ b/accel/tcg/translate-all.c
>> @@ -951,9 +951,13 @@ static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1,
>>    * Users running large scale system emulation may want to tweak their
>>    * runtime setup via the tb-size control on the command line.
>>    */
>> +#ifdef TARGET_I386
>> +#define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (32 * MiB)
>> +#else
>>   #define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (1 * GiB)
>>   #endif
>>   #endif
>> +#endif
>>     #define DEFAULT_CODE_GEN_BUFFER_SIZE \
>>     (DEFAULT_CODE_GEN_BUFFER_SIZE_1 < MAX_CODE_GEN_BUFFER_SIZE \
>> 
>
> I'm not thrilled, as it is ultra-hacky.

I don't disagree.

> (1) I've got a re-org of this code out for review:
> https://patchew.org/QEMU/20210502231844.1977630-1-richard.henderson@linaro.org/

OK I'll have a look at that.

> (2) I'm keen to reorg TCG such that it gets compiled once.  There's
> currently nothing standing in the way of that except work.  But this
> would introduce a use of a target-specific define for the first time
> into tcg/.  I guess I could leave the default sizing back in
> accel/tcg/ and pass in the default.
>
> Other options?

Some random thoughts in no particular order:

 - a separately flushable translation region for code we detect as SMC heavy

 - a front-end interpreter for SMC code

 - smarter code generation that dynamically loads values from codemem
   (usually the SMC code is just tweaking an #imm value)

None of these seem particularly amenable to a clean non-complex
implementation though. A front-end interpreter would be useful for other
things though - it could even be incomplete and handle only common code
patterns falling back to full generation for anything it can't handle.

>
>
> r~


-- 
Alex Bennée

[Bug 1896298] Re: [RFC PATCH] accel/tcg: change default codegen buffer size for i386-softmmu

[Alex Bennée]

Richard Henderson <richard.henderson@linaro.org> writes:

> On 5/25/21 9:45 AM, Alex Bennée wrote:
>> There are two justifications for making this change. The first is that
>> i386 emulation is typically for smaller machines where having a 1gb of
>> generated code is overkill for basic emulation. The second is the
>> propensity of self-modifying code (c.f. Doom/edit) utilised on i386
>> systems can trigger a rapid growth in invalidated and re-translated
>> buffers. This is seen in bug #283. Execution is still inefficient but
>> at least the host memory isn't so aggressively used up.
>> That said it's still really just a sticking plaster for user
>> convenience.
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Thomas Huth <thuth@redhat.com>
>> Cc: 1896298@bugs.launchpad.net
>> ---
>>   accel/tcg/translate-all.c | 4 ++++
>>   1 file changed, 4 insertions(+)
>> diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
>> index 640ff6e3e7..f442165674 100644
>> --- a/accel/tcg/translate-all.c
>> +++ b/accel/tcg/translate-all.c
>> @@ -951,9 +951,13 @@ static void page_lock_pair(PageDesc **ret_p1, tb_page_addr_t phys1,
>>    * Users running large scale system emulation may want to tweak their
>>    * runtime setup via the tb-size control on the command line.
>>    */
>> +#ifdef TARGET_I386
>> +#define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (32 * MiB)
>> +#else
>>   #define DEFAULT_CODE_GEN_BUFFER_SIZE_1 (1 * GiB)
>>   #endif
>>   #endif
>> +#endif
>>     #define DEFAULT_CODE_GEN_BUFFER_SIZE \
>>     (DEFAULT_CODE_GEN_BUFFER_SIZE_1 < MAX_CODE_GEN_BUFFER_SIZE \
>> 
>
> I'm not thrilled, as it is ultra-hacky.

I don't disagree.

> (1) I've got a re-org of this code out for review:
> https://patchew.org/QEMU/20210502231844.1977630-1-richard.henderson@linaro.org/

OK I'll have a look at that.

> (2) I'm keen to reorg TCG such that it gets compiled once.  There's
> currently nothing standing in the way of that except work.  But this
> would introduce a use of a target-specific define for the first time
> into tcg/.  I guess I could leave the default sizing back in
> accel/tcg/ and pass in the default.
>
> Other options?

Some random thoughts in no particular order:

 - a separately flushable translation region for code we detect as SMC
heavy

 - a front-end interpreter for SMC code

 - smarter code generation that dynamically loads values from codemem
   (usually the SMC code is just tweaking an #imm value)

None of these seem particularly amenable to a clean non-complex
implementation though. A front-end interpreter would be useful for other
things though - it could even be incomplete and handle only common code
patterns falling back to full generation for anything it can't handle.

>
>
> r~


-- 
Alex Bennée

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1896298

Title:
  TCG memory leak with FreeDOS 'edit'

Status in QEMU:
  Expired

Bug description:
  qemu trunk as of today leaks memory FAST when freedos' edit is
  running.

  To reproduce, download:

  https://www.ibiblio.org/pub/micro/pc-
  stuff/freedos/files/repositories/1.3/cdrom.iso

  Then run:

  $ qemu-system-i386 -cdrom cdrom.iso

  select your language then select "return to DOS", then type

  > edit

  it will consume memory at ~10MB/s

  This does NOT happen when adding -enable-kvm

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1896298/+subscriptions