Re: [PATCH v6 3/3] target/ppc: support single stepping with KVM HV

[Fabiano Rosas <farosas@linux.ibm.com>]

David Gibson <david@gibson.dropbear.id.au> writes:

(...)
>> > Hrm.... I don't actually see how changing env->msr helps you here.
>> > AFAICT if kvm_insert_breakpoint() resolves to kvm_arch_sw_breakpoint()
>> > it doesn't rely on the MSR value at all.  If it resolves to
>> > kvm_arch_hw_breakpoint(), then it looks like it just stashes
>> > information to be pushed into KVM when we re-enter the guest.  None of
>> > the information stashed appears to depend on the current MSR, and once
>> > we re-enter the MSR will already have been restored.
>> >
>>
>> This is the call chain:
>> 
>> kvm_arch_insert_sw_breakpoint -> cpu_memory_rw_debug ->
>> cpu_get_phys_page_attrs_debug -> ppc_cpu_get_phys_page_debug ->
>> ppc64_v3_get_phys_page_debug -> ppc_radix64_get_phys_page_debug:
>> 
>>     /* Handle Real Mode */
>>     if (msr_dr == 0) {
>>         /* In real mode top 4 effective addr bits (mostly) ignored */
>>         return eaddr & 0x0FFFFFFFFFFFFFFFULL;
>>     }
>
> Ah, right.  Basically the issue is that kvm_insert_breakpoint() takes
> an effective address, not a real address, but it might be happening in
> a different context than we're executing right now.
>
> Ok, that makes sense.  Though.. aren't you always inserting the
> breakpoint into an interrupt vector?  So wouldn't it always be MMU
> off?  Under what circumstances would this get called with mmu_on =
> true?
>

Well, the MSR state at the moment of the breakpoint is that of the
currently executing instruction. So this gets called with mmu_on = true
very often because we're often debugging code than runs with IR|DR=1.

However, we could be at a point when IR|DR=1, but the next traced
instruction will execute with IR|DR=0. This happens at the rfid at the
end of __enter_rtas, for instance.

So ppc_radix64_get_phys_page_debug will check the MSR, see that we are
(now) not in real mode and proceed with the page table walk, which could
fail.

In the particular case of the __enter_rtas rfid, we have PIDR=1 [1] so
if we don't exit ppc_radix64_get_phys_page_debug at the msr_dr == 0
check, it will fail to translate the address.

1 - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eeb715c3e995fbdda0cc05e61216c6c5609bce66

>> Actually, I think there is a bug after ppc_cpu_get_phys_page_debug
>> somewhere. There are some cases where GDB wants to read/write to some
>> memory, but it gets denied access. Presumably because of one such
>> discrepancy as the one above. I need to spend more time looking at this
>> to define the problem properly, though.
>
> Hm, ok.
>
>> >> +    /*
>> >> +     * MSR_SE = 1 will cause a Trace Interrupt in the guest after the
>> >> +     * next instruction executes. If this is a rfid, use SRR1 instead
>> >> +     * of MSR.
>> >> +     */
>> >> +    if (rfid) {
>> >> +        if ((env->spr[SPR_SRR1] >> MSR_SE) & 1) {
>> >> +            /*
>> >> +             * The guest is doing a single step itself. Make sure we
>> >> +             * restore it later.
>> >> +             */
>> >> +            env->sstep_kind = SSTEP_GUEST;
>> >> +        }
>> >> +
>> >> +        env->spr[SPR_SRR1] |= (1ULL << MSR_SE);
>> >> +        mmu_on = srr1_ir & srr1_dr;
>> >
>> > s/&/&&/
>> >
>> 
>> Ack.
>> 
>> >> +    } else {
>> >> +        env->msr |= (1ULL << MSR_SE);
>> >> +        mmu_on = msr_ir & msr_dr;
>> >
>> > s/&/&&/
>> >
>> 
>> Ack.
>> 
>> > Also, what happens if the guest is using MSR[DR] != MSR[IR]?  It's
>> > rare, but it is occasionally useful.
>> 
>> I understand from the ISA that for the purposes of AIL, both bits need
>> to be set. So mmu_on = 0 is correct here.
>
> I'm not sure what you mean by "for the purposes of AIL".
>

The reason I'm tracking the translation state here is to be able to tell
what will be the value of AIL, since an Alternate Interrupt Location is
not used when translation is disabled. In the ISA, under "Alternate
Interrupt Location" the only mention of MSR_IR != MSR_DR is:

"Other interrupts that occur when MSR IR=0 or MSR DR=0, are taken as
if LPCR AIL=0."

and my interpretation of that text is that AIL value is 0 when IR DR are
either 0b00, 0b01 or 0b10.

So "for the purposes of AIL", I'm considering either IR=0 or DR=0 as
meaning MMU off.

>> 
>> >> +    }
>> >> +
>> >> +    kvm_insert_singlestep_breakpoint(cs, mmu_on);
>> >> +}
>> >> +
>> >> +void kvm_singlestep_ail_change(CPUState *cs)
>> >> +{
>> >> +    PowerPCCPU *cpu = POWERPC_CPU(cs);
>> >> +    CPUPPCState *env = &cpu->env;
>> >> +
>> >> +    if (kvm_arch_can_singlestep(cs)) {
>> >> +        return;
>> >> +    }
>> >> +
>> >> +    /*
>> >> +     * The instruction being stepped altered the interrupt vectors
>> >> +     * location (AIL). Re-insert the single step breakpoint at the new
>> >> +     * location
>> >> +     */
>> >> +    kvm_remove_breakpoint(cs, env->trace_handler_addr, 4, GDB_BREAKPOINT_SW);
>> >> +    kvm_insert_singlestep_breakpoint(cs, (msr_ir & msr_dr));
>> >
>> > s/&/&&/
>> >
>> 
>> Ack.
>> 
>> >> +}
>> >> +
>> >>  void kvm_arch_update_guest_debug(CPUState *cs, struct kvm_guest_debug *dbg)
>> >>  {
>> >>      int n;
>> >> @@ -1585,6 +1781,98 @@ void kvm_arch_update_guest_debug(CPUState *cs, struct kvm_guest_debug *dbg)
>> >>      }
>> >>  }
>> >>  
>> >> +/* Revert any side-effects caused during single step */
>> >> +static void restore_singlestep_env(CPUState *cs)
>> >> +{
>> >> +    PowerPCCPU *cpu = POWERPC_CPU(cs);
>> >> +    CPUPPCState *env = &cpu->env;
>> >> +    uint32_t insn = env->sstep_insn;
>> >> +    int reg;
>> >> +    int spr;
>> >> +
>> >> +    env->spr[SPR_SRR0] = env->sstep_srr0;
>> >> +    env->spr[SPR_SRR1] = env->sstep_srr1;
>> >> +
>> >> +    if (ppc_gdb_get_op(insn) != OP_MOV) {
>> >> +        return;
>> >> +    }
>> >> +
>> >> +    reg = ppc_gdb_get_rt(insn);
>> >> +
>> >> +    switch (ppc_gdb_get_xop(insn)) {
>> >> +    case XOP_MTSPR:
>> >> +        /*
>> >> +         * mtspr: the guest altered the SRR, so do not use the
>> >> +         *        pre-step value.
>> >> +         */
>> >> +        spr = ppc_gdb_get_spr(insn);
>> >> +        if (spr == SPR_SRR0 || spr == SPR_SRR1) {
>> >> +            env->spr[spr] = env->gpr[reg];
>> >> +        }
>> >> +        break;
>> >> +    case XOP_MFMSR:
>> >> +        /*
>> >> +         * mfmsr: clear MSR_SE bit to avoid the guest knowing
>> >> +         *         that it is being single-stepped.
>> >> +         */
>> >> +        env->gpr[reg] &= ~(1ULL << MSR_SE);
>> >
>> > Don't you need to check for the case where the guest also thinks it is
>> > single stepping here?
>> >
>> 
>> Hm. I had this in some previous version but removed it for some
>> reason. I'll review it.
>> 
>> 
>> Thanks
>>