Re: Display update issue on M1 Macs

[BALATON Zoltan <balaton@eik.bme.hu>]

On Tue, 31 Jan 2023, BALATON Zoltan wrote:
> On Tue, 31 Jan 2023, Akihiko Odaki wrote:
[...]
To summarise previous discussion:

- There's a problem on Apple M1 Macs with sm501 and ati-vga 2d accel 
functions drawing from device model into the video memory of the emulated 
card which is not shown on screen when the display update callback is 
called from another thread. This works on x86_64 host so I suspect it may 
be related to missing memory synchronisation that ARM may need.

- This can be reproduced running AmigaOS4 on sam460ex or MorphOS (demo iso 
downliadable from their web site) on sam460ex, pegasos2 or mac99,via=pmu 
with -device ati-vga,romfile="" as described here: 
http://zero.eik.bme.hu/~balaton/qemu/amiga/

- I can't test it myself lacking hardware so I have to rely on reports 
from people who have this hardware so there may be some uncertainity in 
the info I get.

- We have confirmed it's not related to a known race condition as 
disabling dirty tracking and always doing full updates of whole screen 
did not fix it:

>>>>>>>> But there is an exception: memory_region_snapshot_and_clear_dirty() 
>>>>>>>> releases iothread lock, and that broke raspi3b display device:
>>>>>>>> https://lore.kernel.org/qemu-devel/CAFEAcA9odnPo2LPip295Uztri7JfoVnQbkJ=Wn+k8dQneB_ynQ@mail.gmail.com/T/
>>>>>>>> 
>>>>>>>> It is unexpected that gfx_update() callback releases iothread lock so 
>>>>>>>> it may break things in peculiar ways.
>>>>>>>> 
>>>>>>>> Peter, is there any change in the situation regarding the race 
>>>>>>>> introduced by memory_region_snapshot_and_clear_dirty()?
>>>>>>>> 
>>>>>>>> For now, to workaround the issue, I think you can create another 
>>>>>>>> mutex and make the entire sm501_2d_engine_write() and 
>>>>>>>> sm501_update_display() critical sections.
>>>>>>> 
>>>>>>> Interesting thread but not sure it's the same problem so this 
>>>>>>> workaround may not be enough to fix my issue. Here's a video posted by 
>>>>>>> one of the people who reported it showing the problem on M1 Mac:
>>>>>>> 
>>>>>>> https://www.youtube.com/watch?v=FDqoNbp6PQs
>>>>>>> 
>>>>>>> and here's how it looks like on other machines:
>>>>>>> 
>>>>>>> https://www.youtube.com/watch?v=ML7-F4HNFKQ
>>>>>>> 
>>>>>>> There are also videos showing it running on RPi 4 and G5 Mac without 
>>>>>>> this issue so it seems to only happen on Apple Silicon M1 Macs. What's 
>>>>>>> strange is that graphics elements are not just delayed which I think 
>>>>>>> should happen with missing thread synchronisation where the update 
>>>>>>> callback would miss some pixels rendered during it's running but 
>>>>>>> subsequent update callbacks would eventually draw those, woudn't they? 
>>>>>>> Also setting full_update to 1 in sm501_update_display() callback to 
>>>>>>> disable dirty tracking does not fix the problem. So it looks like as 
>>>>>>> if sm501_2d_operation() running on one CPU core only writes data to 
>>>>>>> the local cache of that core which sm501_update_display() running on 
>>>>>>> other core can't see, so maybe some cache synchronisation is needed in 
>>>>>>> memory_region_set_dirty() or if that's already there maybe I should 
>>>>>>> call it for all changes not only those in the visible display area? 
>>>>>>> I'm still not sure I understand the problem and don't know what could 
>>>>>>> be a fix for it so anything to test to identify the issue better might 
>>>>>>> also bring us closer to a solution.
>>>>>> 
>>>>>> If you set full_update to 1, you may also comment out 
>>>>>> memory_region_snapshot_and_clear_dirty() and 
>>>>>> memory_region_snapshot_get_dirty() to avoid the iothread mutex being 
>>>>>> unlocked. The iothread mutex should ensure cache coherency as well.
>>>>>> 
>>>>>> But as you say, it's weird that the rendered result is not just delayed 
>>>>>> but missed. That may imply other possibilities (e.g., the results are 
>>>>>> overwritten by someone else). If the problem persists after commenting 
>>>>>> out memory_region_snapshot_and_clear_dirty() and 
>>>>>> memory_region_snapshot_get_dirty(), I think you can assume the 
>>>>>> inter-thread coherency between sm501_2d_operation() and 
>>>>>> sm501_update_display() is not causing the problem.
>>>>> 
>>>>> I've asked people who reported and can reproduce it to test this but it 
>>>>> did not change anything so confirmed it's not that race condition but 
>>>>> looks more like some cache inconsistency maybe. Any other ideas?
>>>> 
>>>> I can come up with two important differences between x86 and Arm which 
>>>> can affect the execution of QEMU:
>>>> 1. Memory model. Arm uses a memory model more relaxed than x86 so it is 
>>>> more sensitive for synchronization failures among threads.
>>>> 2. Different instructions. TCG uses JIT so differences in instructions 
>>>> matter.
>>>> 
>>>> We should be able to exclude 1) as a potential cause of the problem. 
>>>> iothread mutex should take care of race condition and even cache 
>>>> coherency problem; mutex includes memory barrier functionality.
[...]
>>>> For difference 2), you may try to use TCI. You can find details of TCI in 
>>>> tcg/tci/README.
>>> 
>>> This was tested and also with TCI got the same results just much slower.
>>> 
>>>> The common sense tells, however, the memory model is usually the cause of 
>>>> the problem when you see behavioral differences between x86 and Arm, and 
>>>> TCG should work fine with both of x86 and Arm as they should have been 
>>>> tested well.
[...]
>> Fortunately macOS provides Rosetta 2 for x86 emulation on Apple M1, which 
>> makes it possible to compare x86 and Arm without concerning the difference 
>> of the microarchitecture.
>
> We've tried that before and even running x86 QEMU on M1 with Rosetta 2 it was 
> the same so it's probably not something about the code itself but how it's

As this was odd I've asked to re-test this and now I'm told at least QEMU 
5.1 x86_64 build from emaculation.com is working with Rosetta on M1 Mac so 
this suggests it may be a problem with memory sync but still don't know 
where and what to try. We're now try newer X86_64 builds to see if it 
broke somewhere along the way.

Anybody else with an M1 Mac wants to help testing? Can you reproduce the 
same with UTM with MorphOS and ati-vga? Here's what I've got showing the 
problem: https://www.youtube.com/watch?v=j5Ag5_Yq-Mk

Regards,
BALATON Zoltan