[Qemu-devel] [Migration][TCG] Page dirtying and migration in pure-QEMU VM mode

[Qemu-devel] [Migration][TCG] Page dirtying and migration in pure-QEMU VM mode

[Pavel Boldin]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]

Dear All,

As a result of fixing the bug [1] I discovered that QEMU in pure emulation
(TCG) sometimes misses page dirtying on the migration. This is happens at
least in the version 2.0.0 and should, according to the code, be the same
in the master as well.

The reason for that is that only pages missing from the TLB cache are
fetched using the `tlb_fill` that calls `x86_cpu_handle_mmu_fault` and
finally `stl_phys_notdirty` which marks the page as dirty in the
ram_list.dirty_memory[DIRTY_MEMORY_MIGRATION] bitmap.

However, if the page is in the TLB cache then no such code is run and page
is never marked as dirty, making memory dump created by `savevm` different
form the actual VM memory. Sometimes this leads to memory structures
corruption during the migration that results in the VM kernel Oops. This is
indeed happens very frequently with the data referenced by the APIC timer
IRQ handler, because these structures are almost always in the TLB-cache
especially on a non-busy VM. [2]

Sadly, just flushing the TLB on `ram_save_setup` will not be enough because
the appropriate TLB entry must be flushed whenever the appropriate page is
saved and marked clean. This however can not be done in a thread-safe
manner due to the race between the translated and migration code.


The question is: is there a portable and good way to adequately mark such
pages dirty?

One of the possibilities is to use `mprotect` on the VM memory and dirty
the pages on each write access. This is not a portable solution though.

Second one seems to be introducing a TCG variable that will disable the
code generation of the TLB-cache aware memory writes passing each such
access through the appropriate helper that will dirty out the pages. This
is something I have a draft hack-patch for in [3].


Are there any other possibilities I miss?


Pavel

[1] https://bugs.launchpad.net/mos/7.0.x/+bug/1371130
[2] https://bugs.launchpad.net/mos/7.0.x/+bug/1371130/comments/35
[3]
https://bugs.launchpad.net/mos/7.0.x/+bug/1371130/+attachment/4456256/+files/dont-use-tlb-migration.patch

[-- Attachment #2: Type: text/html, Size: 2621 bytes --]

Re: [Qemu-devel] [Migration][TCG] Page dirtying and migration in pure-QEMU VM mode

[Pavel Boldin]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

A little self-update here.

1. It is seems to be enough to just flush TLB entries with the iothread
lock held since the CPUs are stopped here.

2. The upstream version is not vulnerable to the bug due to the following
code call path: ram_save_setup -> memory_global_dirty_log_start ->
memory_region_transaction_commit -> tcg_commit -> CPU_FOREACH(cpu)
tlb_flush(cpu, 1);.
The fix was introduced by the commit [1]

3. QEMU versions 2.0.0 (including Ubuntu's one) and 2.3.1 are vulnerable to
the bug. I will report these as appropriate.

[1]
http://git.qemu.org/?p=qemu.git;a=commit;h=6f6a5ef3e429f92f987678ea8c396aab4dc6aa19

Pavel
​

[-- Attachment #2: Type: text/html, Size: 874 bytes --]