Re: [OE-core] [PATCH 1/2] qemu: add 34Kf-64tlb fictitious cpu type

["Victor Kamensky" <kamensky@cisco.com>]

Hi Paul,

Please forgive my horrible email agent that I have at work.
Please look for 'kamensky>' for responses inline.

________________________________________
From: Paul Barker <pbarker@konsulko.com>
Sent: Wednesday, October 7, 2020 1:46 PM
To: Victor Kamensky (kamensky)
Cc: openembedded-core; Richard Purdie; Khem Raj
Subject: Re: [OE-core] [PATCH 1/2] qemu: add 34Kf-64tlb fictitious cpu type

On Wed, 7 Oct 2020 at 21:39, Victor Kamensky via
lists.openembedded.org <kamensky=cisco.com@lists.openembedded.org>
wrote:
>
> In Yocto Project PR 13992 it was reported that qemumips
> in autobuilder runs almost twice slower then qemumips64 and
> some times hit time out.
>
> Upon investigations of qemu-system with perf, gdb, and
> SystemTap and comparing qemumips and qemumips64 machines
> behavior it was noticed that qemu soft mmu code behaves
> quite different and in case if qemumips tlbwr instruction
> called 16 times more oftern. It happens that in qemumips64
> case qemu runs with cpu type that contains 64 TLB, but in case
> of qemumips qemu runs with cpu type that contains only
> 16 TLBs.
>
> The idea of proposed qemu patch is to introduce fictitious
> 34Kf-64tlb cpu type that defined exactly as 34Kf but has
> 64 TLBs, instead of original 16 TLBs.
>
> Testing of core-image-full-cmdline:do_testimage with
> 34Kf-64tlb shows 40% or so test execution real time
> improvement.
>
> Note for future porters of the patch: easiest way to update
> the patch and be in sync with 34Kf definition is to copy
> 34Kf machine definition and apply the following changes to
> it (just change 15 to 63 of CP0C1_MMU bits value)
>
> [kamensky@coreos-lnx2 qemu]$ diff ~/34Kf.c ~/34Kf-64tlb.c
> 2c2
> <         .name = "34Kf",
> >         .name = "34Kf-64tlb",
> 6c6
> <         .CP0_Config1 = MIPS_CONFIG1 | (1 << CP0C1_FP) | (15 << CP0C1_MMU) |
> >         .CP0_Config1 = MIPS_CONFIG1 | (1 << CP0C1_FP) | (63 << CP0C1_MMU) |
>
> Fixes https://bugzilla.yoctoproject.org/show_bug.cgi?id=13992

Forgive my ignorance as to the range of MIPS processors available but
does any real MIPS CPU have 64 TLBs?

kamensky> I am not up to date wrt 32 bit only MIPS CPUs out there.
kamensky> I've experience with MIPS64 real CPUs that could
kamensky> operate both in 64 bit and 32 bit mode. In fact in our
kamensky> case even in time when we used 32bit user-land kernel
kamensky> itself was running 64 bit kernel (as OE MIPS multi-lib test
kamensky> case). In 32bit case h/w memory map quite unfriendly
kamensky> to linux kernel, kernel can see directly only 512Mb of
kamensky> unmapped phy memory KSEG0/KSEG1, and bigger amount would
kamensky> require CONFIG_HIGHMEM that drag additional mappings
kamensky> for kernel access of the memory.

kamensky> MIPS32 spec itself does allow 64 TLBs, for sure. 

If such a CPU model exists
shouldn't we be using this instead of inventing a new model?

kamensky> Note about other available MIPS 32bit existing CPU
kamensky> model with bigger number of TLBs below.

I'm a bit worried that targeting a unique, fictitious CPU model will
lead to us wasting time debugging obscure failures that other projects
have never seen and that would never occur on real hardware.

kamensky> I do share your concern up to some extent. But I think
kamensky> the risk is minimal.

kamensky> Please note in qemumips64 machine case we already
kamensky> use fictitious CPU type as "MIPS64R2-generic". AFAIK
kamensky> there is no such real CPU out there, it is generic thing
kamensky> inside of qemu. For sure, there is
kamensky> no such real combination as "MIPS64R2-generic" cpu type
kamensky> and machine "malta". So up to some extent a lot things
kamensky> under qemu are fictitious. 

kamensky> I've looked at other MIPS only CPUs that qemu supports.
kamensky> There is definition of "mips32r6-generic". Another fictitious
kamensky> CPU type but defined in qemu already. That one configured
kamensky> with 32 TLBs, but when I've tried it with qemumips it flat out
kamensky> did not boot. Which brings us to other practical issue: proper
kamensky> match between qemu cpu type and our qemumips kernel
kamensky> configuration, question about qemu CPU emulation correctness
kamensky> and whether it is bug free, etc, etc ..

kamensky> By modifying 34Kf cpu type that worked for us before, and doing
kamensky> minimal low risk changes in one limited small aspect of the
kamensky> system, that brings us performance gains, IMO is worth doing.

Thanks,
Victor

--
Paul Barker
Konsulko Group