VMI: singlestep event not received

VMI: singlestep event not received

[Mathieu Tarral]

Hi,

I'm having an issue with Xen's VMI subsystem.

My goal is to build a small debugger that can break at an application's entrypoint
on Windows XP, when a new process is being created.

To accomplish this, I first set a software breakpoint on KiThreadStartup (ntoskrnl.exe),
then on RtlUserThreadStart (ntdll.dll).

The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.

To my surprise, it didn't work as expected, since my application just hanged, waiting for the next singlestep event:

--Waiting for xen events...(0 ms)
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
Enabling single step
INFO:WindowsDebugContext:[105] at: 0x806d32d6
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
--Setting MTF flag on vcpu 0
Enabling single step
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(0 ms)
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
Enabling single step
INFO:WindowsDebugContext:[106] at: 0x806d32dc
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
--Setting MTF flag on vcpu 0
Enabling single step
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)

The reason why i'm disabling end enabling the singlestep successively is because i already
have a libvmi singlestep event registered, with the MTF flag disabled.
I only use it for breakpoint recoil situations.
It's a limitation of the libvmi API where you cannot modified a registered event to enable singlestep at will.
Therefore my solution:
https://github.com/Wenzel/pyvmidbg/blob/37ee12fb7bee612b0dbee9356e21aa959e750746/vmidbg/breakpoint.py#L79

The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
at 0x806d32dc:

[0x7c90e514]> s 0x806d32dc
[0x806d32dc]> pd 10
            0x806d32dc      890d8000feff   mov dword [0xfffe0080], ecx
            0x806d32e2      a18000feff     mov eax, dword [0xfffe0080] ; [0xfffe0080:4]=-1
            0x806d32e7      c3             ret
            0x806d32e8      a18000feff     mov eax, dword [0xfffe0080] ; [0xfffe0080:4]=-1
            0x806d32ed      c1e804         shr eax, 4
            0x806d32f0      0fb68088e06d.  movzx eax, byte [eax - 0x7f921f78]
            0x806d32f7      c3             ret
            0x806d32f8      b91f000000     mov ecx, 0x1f               ; 31
            0x806d32fd      e876ffffff     call 0x806d3278
            0x806d3302      c3             ret
[0x806d32dc]>


I'm running on Xen packaged by Fedora 29: 4.11.1, in a nested virt environment on top of KVM.

I discussed this bug on IRC with andyhpp, who convinced me to move the discussion on the mailing list.
Apparently the singlestepping in Xen was in a poor quality state because of multiple layers of refactoring.

If you have any ideas, please share.

Thanks !

Mathieu Tarral.

Sent with ProtonMail Secure Email.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

Hi,

I'm having an issue with Xen's VMI subsystem.

My goal is to build a small debugger that can break at an application's entrypoint
on Windows XP, when a new process is being created.

To accomplish this, I first set a software breakpoint on KiThreadStartup (ntoskrnl.exe),
then on RtlUserThreadStart (ntdll.dll).

The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.

To my surprise, it didn't work as expected, since my application just hanged, waiting for the next singlestep event:

--Waiting for xen events...(0 ms)
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
Enabling single step
INFO:WindowsDebugContext:[105] at: 0x806d32d6
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
--Setting MTF flag on vcpu 0
Enabling single step
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(0 ms)
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
Enabling single step
INFO:WindowsDebugContext:[106] at: 0x806d32dc
Disabling single step on vcpu: 0
--Removing MTF flag from vcpu 0
--Shutting down single step on domain 1
--Removing MTF flag from vcpu 0
--Starting single step on domain 1
--Setting MTF flag on vcpu 0
Enabling single step
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)
--Waiting for xen events...(1000 ms)

The reason why i'm disabling end enabling the singlestep successively is because i already
have a libvmi singlestep event registered, with the MTF flag disabled.
I only use it for breakpoint recoil situations.
It's a limitation of the libvmi API where you cannot modified a registered event to enable singlestep at will.
Therefore my solution:
https://github.com/Wenzel/pyvmidbg/blob/37ee12fb7bee612b0dbee9356e21aa959e750746/vmidbg/breakpoint.py#L79

The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
at 0x806d32dc:

[0x7c90e514]> s 0x806d32dc
[0x806d32dc]> pd 10
            0x806d32dc      890d8000feff   mov dword [0xfffe0080], ecx
            0x806d32e2      a18000feff     mov eax, dword [0xfffe0080] ; [0xfffe0080:4]=-1
            0x806d32e7      c3             ret
            0x806d32e8      a18000feff     mov eax, dword [0xfffe0080] ; [0xfffe0080:4]=-1
            0x806d32ed      c1e804         shr eax, 4
            0x806d32f0      0fb68088e06d.  movzx eax, byte [eax - 0x7f921f78]
            0x806d32f7      c3             ret
            0x806d32f8      b91f000000     mov ecx, 0x1f               ; 31
            0x806d32fd      e876ffffff     call 0x806d3278
            0x806d3302      c3             ret
[0x806d32dc]>


I'm running on Xen packaged by Fedora 29: 4.11.1, in a nested virt environment on top of KVM.

I discussed this bug on IRC with andyhpp, who convinced me to move the discussion on the mailing list.
Apparently the singlestepping in Xen was in a poor quality state because of multiple layers of refactoring.

If you have any ideas, please share.

Thanks !

Mathieu Tarral.

Sent with ProtonMail Secure Email.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Razvan Cojocaru]

On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.

FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault 
parameter to cause the OS to bring a paged out page back into main memory.


HTH,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Razvan Cojocaru]

On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.

FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault 
parameter to cause the OS to bring a paged out page back into main memory.


HTH,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Andrew Cooper]

On 21/04/2019 23:26, Mathieu Tarral wrote:

Answering out of order.

> I discussed this bug on IRC with andyhpp, who convinced me to move the discussion on the mailing list.
> Apparently the singlestepping in Xen was in a poor quality state because of multiple layers of refactoring.

What I said was that x86 is a complete mess.  Various aspects of debug
behaviour aren't even documented, and have had to be reversed engineered
after finding security vulnerabilities with how software handles them
(caused by a lack of documentation).

> I'm running on Xen packaged by Fedora 29: 4.11.1, in a nested virt environment on top of KVM.

I didn't realise you were nested on top of KVM.  That definitely isn't
helping things, and there is an equal chance that you've hit a nested
virt bug in KVM.

Please try and reproduce it without KVM, to rule out some of the complexity.

> The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
> at 0x806d32dc:
>
> [0x7c90e514]> s 0x806d32dc
> [0x806d32dc]> pd 10
>             0x806d32dc      890d8000feff   mov dword [0xfffe0080], ecx

This is a read of the APIC TPR, which is very commonly an operation
accelerated by hardware (because without it, virtualising windows XP is
exceedingly slow).

What is your CPU, and how exactly are you trying to singlestep.  Is it
with MTF, or using the trap flag inside the guest?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Andrew Cooper]

On 21/04/2019 23:26, Mathieu Tarral wrote:

Answering out of order.

> I discussed this bug on IRC with andyhpp, who convinced me to move the discussion on the mailing list.
> Apparently the singlestepping in Xen was in a poor quality state because of multiple layers of refactoring.

What I said was that x86 is a complete mess.  Various aspects of debug
behaviour aren't even documented, and have had to be reversed engineered
after finding security vulnerabilities with how software handles them
(caused by a lack of documentation).

> I'm running on Xen packaged by Fedora 29: 4.11.1, in a nested virt environment on top of KVM.

I didn't realise you were nested on top of KVM.  That definitely isn't
helping things, and there is an equal chance that you've hit a nested
virt bug in KVM.

Please try and reproduce it without KVM, to rule out some of the complexity.

> The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
> at 0x806d32dc:
>
> [0x7c90e514]> s 0x806d32dc
> [0x806d32dc]> pd 10
>             0x806d32dc      890d8000feff   mov dword [0xfffe0080], ecx

This is a read of the APIC TPR, which is very commonly an operation
accelerated by hardware (because without it, virtualising windows XP is
exceedingly slow).

What is your CPU, and how exactly are you trying to singlestep.  Is it
with MTF, or using the trap flag inside the guest?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Mathieu Tarral]

On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:

> On 4/22/19 1:26 AM, Mathieu Tarral wrote:
>
> > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
>
> FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> parameter to cause the OS to bring a paged out page back into main memory.

The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.

I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.

Maybe we should open an issue to track the progress about it.

Thanks anyway for the suggestion !


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:

> On 4/22/19 1:26 AM, Mathieu Tarral wrote:
>
> > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
>
> FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> parameter to cause the OS to bring a paged out page back into main memory.

The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.

I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.

Maybe we should open an issue to track the progress about it.

Thanks anyway for the suggestion !


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Mathieu Tarral]

On Monday 22 April 2019 12:28, Andrew Cooper <andrew.cooper3@citrix.com> wrote:

> On 21/04/2019 23:26, Mathieu Tarral wrote:
>
> Answering out of order.
>
> > I discussed this bug on IRC with andyhpp, who convinced me to move the discussion on the mailing list.
> > Apparently the singlestepping in Xen was in a poor quality state because of multiple layers of refactoring.
>
> What I said was that x86 is a complete mess.  Various aspects of debug
> behaviour aren't even documented, and have had to be reversed engineered
> after finding security vulnerabilities with how software handles them
> (caused by a lack of documentation).

My bad, i understood that i was related to Xen's code, not just x86.

> > I'm running on Xen packaged by Fedora 29: 4.11.1, in a nested virt environment on top of KVM.
>
> I didn't realise you were nested on top of KVM.  That definitely isn't
> helping things, and there is an equal chance that you've hit a nested
> virt bug in KVM.
>
> Please try and reproduce it without KVM, to rule out some of the complexity.

I would like to, but Grub cannot boot on Xen's kernel on my machine, i need to run it in a VM.

I'm hitting a bug in Fedora where module2.mod and multiboot2.mod are absent:
https://bugzilla.redhat.com/show_bug.cgi?id=1486002

Also, it's not the first time i'm having this bug,
I was already working on Xen 6 months before:
https://github.com/libvmi/libvmi/issues/636


> > The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
> > at 0x806d32dc:
> > [0x7c90e514]> s 0x806d32dc
> > [0x806d32dc]> pd 10
> > 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
>
> This is a read of the APIC TPR, which is very commonly an operation
> accelerated by hardware (because without it, virtualising windows XP is
> exceedingly slow).
>
> What is your CPU, and how exactly are you trying to singlestep.  Is it
> with MTF, or using the trap flag inside the guest?

My CPU is an Intel i7-8750H.
I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.

Thanks.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

On Monday 22 April 2019 12:28, Andrew Cooper <andrew.cooper3@citrix.com> wrote:

> On 21/04/2019 23:26, Mathieu Tarral wrote:
>
> Answering out of order.
>
> > I discussed this bug on IRC with andyhpp, who convinced me to move the discussion on the mailing list.
> > Apparently the singlestepping in Xen was in a poor quality state because of multiple layers of refactoring.
>
> What I said was that x86 is a complete mess.  Various aspects of debug
> behaviour aren't even documented, and have had to be reversed engineered
> after finding security vulnerabilities with how software handles them
> (caused by a lack of documentation).

My bad, i understood that i was related to Xen's code, not just x86.

> > I'm running on Xen packaged by Fedora 29: 4.11.1, in a nested virt environment on top of KVM.
>
> I didn't realise you were nested on top of KVM.  That definitely isn't
> helping things, and there is an equal chance that you've hit a nested
> virt bug in KVM.
>
> Please try and reproduce it without KVM, to rule out some of the complexity.

I would like to, but Grub cannot boot on Xen's kernel on my machine, i need to run it in a VM.

I'm hitting a bug in Fedora where module2.mod and multiboot2.mod are absent:
https://bugzilla.redhat.com/show_bug.cgi?id=1486002

Also, it's not the first time i'm having this bug,
I was already working on Xen 6 months before:
https://github.com/libvmi/libvmi/issues/636


> > The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
> > at 0x806d32dc:
> > [0x7c90e514]> s 0x806d32dc
> > [0x806d32dc]> pd 10
> > 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
>
> This is a read of the APIC TPR, which is very commonly an operation
> accelerated by hardware (because without it, virtualising windows XP is
> exceedingly slow).
>
> What is your CPU, and how exactly are you trying to singlestep.  Is it
> with MTF, or using the trap flag inside the guest?

My CPU is an Intel i7-8750H.
I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.

Thanks.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Mathieu Tarral]

> Also, it's not the first time i'm having this bug,
> I was already working on Xen 6 months before:
> https://github.com/libvmi/libvmi/issues/636

I forgot to add that I was running Xen bare-metal at that time,
so yes, the bug is reproducible without KVM and nested virtualization.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

> Also, it's not the first time i'm having this bug,
> I was already working on Xen 6 months before:
> https://github.com/libvmi/libvmi/issues/636

I forgot to add that I was running Xen bare-metal at that time,
so yes, the bug is reproducible without KVM and nested virtualization.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Andrew Cooper]

On 23/04/2019 22:59, Mathieu Tarral wrote:
>>> The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
>>> at 0x806d32dc:
>>> [0x7c90e514]> s 0x806d32dc
>>> [0x806d32dc]> pd 10
>>> 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
>> This is a read of the APIC TPR, which is very commonly an operation
>> accelerated by hardware (because without it, virtualising windows XP is
>> exceedingly slow).
>>
>> What is your CPU, and how exactly are you trying to singlestep.  Is it
>> with MTF, or using the trap flag inside the guest?
> My CPU is an Intel i7-8750H.
> I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.

Ok - that is a CoffeeLake.

I've had a quick play with a simple example (XTF test and xen-access)
and MTF does appear to work fine with both TPR shadowing and full APIC
virtualisation.  I've tried on Haswell, KabyLake and CoffeeLake.

Can you describe the exact VMI configuration you've got enabled -
specifically which other intercepts are active?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Andrew Cooper]

On 23/04/2019 22:59, Mathieu Tarral wrote:
>>> The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
>>> at 0x806d32dc:
>>> [0x7c90e514]> s 0x806d32dc
>>> [0x806d32dc]> pd 10
>>> 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
>> This is a read of the APIC TPR, which is very commonly an operation
>> accelerated by hardware (because without it, virtualising windows XP is
>> exceedingly slow).
>>
>> What is your CPU, and how exactly are you trying to singlestep.  Is it
>> with MTF, or using the trap flag inside the guest?
> My CPU is an Intel i7-8750H.
> I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.

Ok - that is a CoffeeLake.

I've had a quick play with a simple example (XTF test and xen-access)
and MTF does appear to work fine with both TPR shadowing and full APIC
virtualisation.  I've tried on Haswell, KabyLake and CoffeeLake.

Can you describe the exact VMI configuration you've got enabled -
specifically which other intercepts are active?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Nuernberger, Stefan]

On Sun, 2019-04-21 at 22:26 +0000, Mathieu Tarral wrote:
> Hi,
> 
> I'm having an issue with Xen's VMI subsystem.
> 
> My goal is to build a small debugger that can break at an
> application's entrypoint
> on Windows XP, when a new process is being created.
> 

There was an announcement a while back on this list of a more modern
xen debugger for VMI purposes. Maybe you are more lucky with that one?

https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/ja
nuary/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/

https://github.com/nccgroup/xendbg

(I have not yet tried it, though.)

- Stefan



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Nuernberger, Stefan]

On Sun, 2019-04-21 at 22:26 +0000, Mathieu Tarral wrote:
> Hi,
> 
> I'm having an issue with Xen's VMI subsystem.
> 
> My goal is to build a small debugger that can break at an
> application's entrypoint
> on Windows XP, when a new process is being created.
> 

There was an announcement a while back on this list of a more modern
xen debugger for VMI purposes. Maybe you are more lucky with that one?

https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/ja
nuary/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/

https://github.com/nccgroup/xendbg

(I have not yet tried it, though.)

- Stefan



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Tamas K Lengyel]

On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
<mathieu.tarral@protonmail.com> wrote:
>
> On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:
>
> > On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> >
> > > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> >
> > FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> > parameter to cause the OS to bring a paged out page back into main memory.
>
> The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
>
> I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
>
> Maybe we should open an issue to track the progress about it.
>
> Thanks anyway for the suggestion !

Not having a LibVMI wrapper should not prevent you from using a libxc
function directly. There are many corner-cases regarding when it is
safe to inject a pagefault into the guest to bring back pages and that
logic is not trivial enough to be implemented in LibVMI. It's way too
specific to the usecase to decide when it's safe to do that. So in
this case, if you know you are in ring3 of your target process you can
just call the libxc function directly.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Tamas K Lengyel]

On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
<mathieu.tarral@protonmail.com> wrote:
>
> On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:
>
> > On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> >
> > > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> >
> > FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> > parameter to cause the OS to bring a paged out page back into main memory.
>
> The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
>
> I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
>
> Maybe we should open an issue to track the progress about it.
>
> Thanks anyway for the suggestion !

Not having a LibVMI wrapper should not prevent you from using a libxc
function directly. There are many corner-cases regarding when it is
safe to inject a pagefault into the guest to bring back pages and that
logic is not trivial enough to be implemented in LibVMI. It's way too
specific to the usecase to decide when it's safe to do that. So in
this case, if you know you are in ring3 of your target process you can
just call the libxc function directly.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Razvan Cojocaru]

On 4/24/19 6:25 PM, Tamas K Lengyel wrote:
> On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
> <mathieu.tarral@protonmail.com> wrote:
>>
>> On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:
>>
>>> On 4/22/19 1:26 AM, Mathieu Tarral wrote:
>>>
>>>> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
>>>
>>> FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
>>> parameter to cause the OS to bring a paged out page back into main memory.
>>
>> The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
>>
>> I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
>>
>> Maybe we should open an issue to track the progress about it.
>>
>> Thanks anyway for the suggestion !
> 
> Not having a LibVMI wrapper should not prevent you from using a libxc
> function directly. There are many corner-cases regarding when it is
> safe to inject a pagefault into the guest to bring back pages and that
> logic is not trivial enough to be implemented in LibVMI. It's way too
> specific to the usecase to decide when it's safe to do that. So in
> this case, if you know you are in ring3 of your target process you can
> just call the libxc function directly.

That's very very true, however I think we should assume that libraries 
such as libvmi and libbdvmi are used by client code smart enough to 
figure out, using said libraries, enough things about what's appropriate 
(or not) to do, and  let them do it rather than second-guess them. 
Hence, in libbdvmi I've wrapped that, and added a comment basically 
saying what you're saying above:

https://github.com/bitdefender/libbdvmi/blob/master/src/xendriver.cpp#L542

If we end up having the client code call backend-specific code directly 
we're not, after all, really abstracting it with our wrapper libraries.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Razvan Cojocaru]

On 4/24/19 6:25 PM, Tamas K Lengyel wrote:
> On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
> <mathieu.tarral@protonmail.com> wrote:
>>
>> On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:
>>
>>> On 4/22/19 1:26 AM, Mathieu Tarral wrote:
>>>
>>>> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
>>>
>>> FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
>>> parameter to cause the OS to bring a paged out page back into main memory.
>>
>> The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
>>
>> I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
>>
>> Maybe we should open an issue to track the progress about it.
>>
>> Thanks anyway for the suggestion !
> 
> Not having a LibVMI wrapper should not prevent you from using a libxc
> function directly. There are many corner-cases regarding when it is
> safe to inject a pagefault into the guest to bring back pages and that
> logic is not trivial enough to be implemented in LibVMI. It's way too
> specific to the usecase to decide when it's safe to do that. So in
> this case, if you know you are in ring3 of your target process you can
> just call the libxc function directly.

That's very very true, however I think we should assume that libraries 
such as libvmi and libbdvmi are used by client code smart enough to 
figure out, using said libraries, enough things about what's appropriate 
(or not) to do, and  let them do it rather than second-guess them. 
Hence, in libbdvmi I've wrapped that, and added a comment basically 
saying what you're saying above:

https://github.com/bitdefender/libbdvmi/blob/master/src/xendriver.cpp#L542

If we end up having the client code call backend-specific code directly 
we're not, after all, really abstracting it with our wrapper libraries.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Tamas K Lengyel]

On Sun, Apr 21, 2019 at 4:27 PM Mathieu Tarral
<mathieu.tarral@protonmail.com> wrote:
>
> Hi,
>
> I'm having an issue with Xen's VMI subsystem.
>
> My goal is to build a small debugger that can break at an application's entrypoint
> on Windows XP, when a new process is being created.
>
> To accomplish this, I first set a software breakpoint on KiThreadStartup (ntoskrnl.exe),
> then on RtlUserThreadStart (ntdll.dll).
>
> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
>
> To my surprise, it didn't work as expected, since my application just hanged, waiting for the next singlestep event:
>
> --Waiting for xen events...(0 ms)
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> Enabling single step
> INFO:WindowsDebugContext:[105] at: 0x806d32d6
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> --Setting MTF flag on vcpu 0
> Enabling single step
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(0 ms)
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> Enabling single step
> INFO:WindowsDebugContext:[106] at: 0x806d32dc
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> --Setting MTF flag on vcpu 0
> Enabling single step
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
>
> The reason why i'm disabling end enabling the singlestep successively is because i already
> have a libvmi singlestep event registered, with the MTF flag disabled.
> I only use it for breakpoint recoil situations.
> It's a limitation of the libvmi API where you cannot modified a registered event to enable singlestep at will.

I'm not entirely sure what you mean here (and perhaps that's a
discussion to be moved to the LibVMI issues page). If you already have
an event registered for singlestepping why would you want to disable
it just to re-enable it? If it's because you just have the handler
registered without MTF actually active, that's specifically made for
the situation where turn on/off MTF using the VM_EVENT_RESPONSE_FLAG.

Also, using MTF to reach parts of the code that are several hundred
instructions down the pipe will most likely not work due to the
extreme overhead it adds. You are more likely to get an interrupt and
land somewhere in the kernel long before you reach your target. At
least that has been my experience. You likely want to investigate
other options, such as what Razvan recommended.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Tamas K Lengyel]

On Sun, Apr 21, 2019 at 4:27 PM Mathieu Tarral
<mathieu.tarral@protonmail.com> wrote:
>
> Hi,
>
> I'm having an issue with Xen's VMI subsystem.
>
> My goal is to build a small debugger that can break at an application's entrypoint
> on Windows XP, when a new process is being created.
>
> To accomplish this, I first set a software breakpoint on KiThreadStartup (ntoskrnl.exe),
> then on RtlUserThreadStart (ntdll.dll).
>
> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
>
> To my surprise, it didn't work as expected, since my application just hanged, waiting for the next singlestep event:
>
> --Waiting for xen events...(0 ms)
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> Enabling single step
> INFO:WindowsDebugContext:[105] at: 0x806d32d6
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> --Setting MTF flag on vcpu 0
> Enabling single step
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(0 ms)
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> Enabling single step
> INFO:WindowsDebugContext:[106] at: 0x806d32dc
> Disabling single step on vcpu: 0
> --Removing MTF flag from vcpu 0
> --Shutting down single step on domain 1
> --Removing MTF flag from vcpu 0
> --Starting single step on domain 1
> --Setting MTF flag on vcpu 0
> Enabling single step
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
> --Waiting for xen events...(1000 ms)
>
> The reason why i'm disabling end enabling the singlestep successively is because i already
> have a libvmi singlestep event registered, with the MTF flag disabled.
> I only use it for breakpoint recoil situations.
> It's a limitation of the libvmi API where you cannot modified a registered event to enable singlestep at will.

I'm not entirely sure what you mean here (and perhaps that's a
discussion to be moved to the LibVMI issues page). If you already have
an event registered for singlestepping why would you want to disable
it just to re-enable it? If it's because you just have the handler
registered without MTF actually active, that's specifically made for
the situation where turn on/off MTF using the VM_EVENT_RESPONSE_FLAG.

Also, using MTF to reach parts of the code that are several hundred
instructions down the pipe will most likely not work due to the
extreme overhead it adds. You are more likely to get an interrupt and
land somewhere in the kernel long before you reach your target. At
least that has been my experience. You likely want to investigate
other options, such as what Razvan recommended.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Tamas K Lengyel]

On Wed, Apr 24, 2019 at 9:34 AM Razvan Cojocaru
<rcojocaru@bitdefender.com> wrote:
>
>
>
> On 4/24/19 6:25 PM, Tamas K Lengyel wrote:
> > On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
> > <mathieu.tarral@protonmail.com> wrote:
> >>
> >> On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:
> >>
> >>> On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> >>>
> >>>> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> >>>
> >>> FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> >>> parameter to cause the OS to bring a paged out page back into main memory.
> >>
> >> The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
> >>
> >> I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
> >>
> >> Maybe we should open an issue to track the progress about it.
> >>
> >> Thanks anyway for the suggestion !
> >
> > Not having a LibVMI wrapper should not prevent you from using a libxc
> > function directly. There are many corner-cases regarding when it is
> > safe to inject a pagefault into the guest to bring back pages and that
> > logic is not trivial enough to be implemented in LibVMI. It's way too
> > specific to the usecase to decide when it's safe to do that. So in
> > this case, if you know you are in ring3 of your target process you can
> > just call the libxc function directly.
>
> That's very very true, however I think we should assume that libraries
> such as libvmi and libbdvmi are used by client code smart enough to
> figure out, using said libraries, enough things about what's appropriate
> (or not) to do, and  let them do it rather than second-guess them.
> Hence, in libbdvmi I've wrapped that, and added a comment basically
> saying what you're saying above:
>
> https://github.com/bitdefender/libbdvmi/blob/master/src/xendriver.cpp#L542
>
> If we end up having the client code call backend-specific code directly
> we're not, after all, really abstracting it with our wrapper libraries.

If someone adds that wrapper to LibVMI I wouldn't veto it - but it's
also a function that I'm 100% expecting will generate a lot of
confusion and whining about it not working in all cases. So I chose
not to add it myself because of that. ¯\_(ツ)_/¯

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Tamas K Lengyel]

On Wed, Apr 24, 2019 at 9:34 AM Razvan Cojocaru
<rcojocaru@bitdefender.com> wrote:
>
>
>
> On 4/24/19 6:25 PM, Tamas K Lengyel wrote:
> > On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
> > <mathieu.tarral@protonmail.com> wrote:
> >>
> >> On Monday 22 April 2019 11:22, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote:
> >>
> >>> On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> >>>
> >>>> The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> >>>
> >>> FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> >>> parameter to cause the OS to bring a paged out page back into main memory.
> >>
> >> The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
> >>
> >> I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
> >>
> >> Maybe we should open an issue to track the progress about it.
> >>
> >> Thanks anyway for the suggestion !
> >
> > Not having a LibVMI wrapper should not prevent you from using a libxc
> > function directly. There are many corner-cases regarding when it is
> > safe to inject a pagefault into the guest to bring back pages and that
> > logic is not trivial enough to be implemented in LibVMI. It's way too
> > specific to the usecase to decide when it's safe to do that. So in
> > this case, if you know you are in ring3 of your target process you can
> > just call the libxc function directly.
>
> That's very very true, however I think we should assume that libraries
> such as libvmi and libbdvmi are used by client code smart enough to
> figure out, using said libraries, enough things about what's appropriate
> (or not) to do, and  let them do it rather than second-guess them.
> Hence, in libbdvmi I've wrapped that, and added a comment basically
> saying what you're saying above:
>
> https://github.com/bitdefender/libbdvmi/blob/master/src/xendriver.cpp#L542
>
> If we end up having the client code call backend-specific code directly
> we're not, after all, really abstracting it with our wrapper libraries.

If someone adds that wrapper to LibVMI I wouldn't veto it - but it's
also a function that I'm 100% expecting will generate a lot of
confusion and whining about it not working in all cases. So I chose
not to add it myself because of that. ¯\_(ツ)_/¯

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Mathieu Tarral]

On Wednesday 24 April 2019 16:27, Nuernberger, Stefan <snu@amazon.de> wrote:

> On Sun, 2019-04-21 at 22:26 +0000, Mathieu Tarral wrote:
>
> > Hi,
> > I'm having an issue with Xen's VMI subsystem.
> > My goal is to build a small debugger that can break at an
> > application's entrypoint
> > on Windows XP, when a new process is being created.
>
> There was an announcement a while back on this list of a more modern
> xen debugger for VMI purposes. Maybe you are more lucky with that one?
>
> https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/ja
> nuary/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/
>
> https://github.com/nccgroup/xendbg
>
> (I have not yet tried it, though.)

Yes, I heard of it via Tamas.
I actually contacted the developer and tried to see how we could work together on the same project.

He likes the idea of being hypervisor-agnostic, and being built upon an abstraction layer.

The reason he chooses not to use LibVMI was because of the lack of support for Xen PV guests, IIRC.

Ping @Tamas, maybe you have an opinion here.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

On Wednesday 24 April 2019 16:27, Nuernberger, Stefan <snu@amazon.de> wrote:

> On Sun, 2019-04-21 at 22:26 +0000, Mathieu Tarral wrote:
>
> > Hi,
> > I'm having an issue with Xen's VMI subsystem.
> > My goal is to build a small debugger that can break at an
> > application's entrypoint
> > on Windows XP, when a new process is being created.
>
> There was an announcement a while back on this list of a more modern
> xen debugger for VMI purposes. Maybe you are more lucky with that one?
>
> https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/ja
> nuary/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/
>
> https://github.com/nccgroup/xendbg
>
> (I have not yet tried it, though.)

Yes, I heard of it via Tamas.
I actually contacted the developer and tried to see how we could work together on the same project.

He likes the idea of being hypervisor-agnostic, and being built upon an abstraction layer.

The reason he chooses not to use LibVMI was because of the lack of support for Xen PV guests, IIRC.

Ping @Tamas, maybe you have an opinion here.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Mathieu Tarral]

On Wednesday 24 April 2019 17:38, Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:

> On Wed, Apr 24, 2019 at 9:34 AM Razvan Cojocaru
> rcojocaru@bitdefender.com wrote:
>
> > On 4/24/19 6:25 PM, Tamas K Lengyel wrote:
> >
> > > On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
> > > mathieu.tarral@protonmail.com wrote:
> > >
> > > > On Monday 22 April 2019 11:22, Razvan Cojocaru rcojocaru@bitdefender.com wrote:
> > > >
> > > > > On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> > > > >
> > > > > > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> > > > >
> > > > > FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> > > > > parameter to cause the OS to bring a paged out page back into main memory.
> > > >
> > > > The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
> > > > I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
> > > > Maybe we should open an issue to track the progress about it.
> > > > Thanks anyway for the suggestion !
> > >
> > > Not having a LibVMI wrapper should not prevent you from using a libxc
> > > function directly. There are many corner-cases regarding when it is
> > > safe to inject a pagefault into the guest to bring back pages and that
> > > logic is not trivial enough to be implemented in LibVMI. It's way too
> > > specific to the usecase to decide when it's safe to do that. So in
> > > this case, if you know you are in ring3 of your target process you can
> > > just call the libxc function directly.
> >
> > That's very very true, however I think we should assume that libraries
> > such as libvmi and libbdvmi are used by client code smart enough to
> > figure out, using said libraries, enough things about what's appropriate
> > (or not) to do, and let them do it rather than second-guess them.
> > Hence, in libbdvmi I've wrapped that, and added a comment basically
> > saying what you're saying above:
> > https://github.com/bitdefender/libbdvmi/blob/master/src/xendriver.cpp#L542
> > If we end up having the client code call backend-specific code directly
> > we're not, after all, really abstracting it with our wrapper libraries.
>
> If someone adds that wrapper to LibVMI I wouldn't veto it - but it's
> also a function that I'm 100% expecting will generate a lot of
> confusion and whining about it not working in all cases. So I chose
> not to add it myself because of that. ¯\(ツ)/¯

Tamas, i'm using Python so i can't directly call libxc.

Also, and as Razvan said, I would like to stick with a coherent and clearly defined abstractions layers, first because it will allow me to be hypervisor-agnostic in the future, if new LibVMI drivers are developed, and then just for the sack of API consistency.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

On Wednesday 24 April 2019 17:38, Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:

> On Wed, Apr 24, 2019 at 9:34 AM Razvan Cojocaru
> rcojocaru@bitdefender.com wrote:
>
> > On 4/24/19 6:25 PM, Tamas K Lengyel wrote:
> >
> > > On Mon, Apr 22, 2019 at 6:00 PM Mathieu Tarral
> > > mathieu.tarral@protonmail.com wrote:
> > >
> > > > On Monday 22 April 2019 11:22, Razvan Cojocaru rcojocaru@bitdefender.com wrote:
> > > >
> > > > > On 4/22/19 1:26 AM, Mathieu Tarral wrote:
> > > > >
> > > > > > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> > > > >
> > > > > FWIW, you can just use xc_hvm_inject_trap() with a TRAP_page_fault
> > > > > parameter to cause the OS to bring a paged out page back into main memory.
> > > >
> > > > The problem is that i'm fully based on LibVMI, which doesn't provide an API for page injection yet.
> > > > I think Tamas was waiting for a full support of pagefault injection on Xen before opening an API.
> > > > Maybe we should open an issue to track the progress about it.
> > > > Thanks anyway for the suggestion !
> > >
> > > Not having a LibVMI wrapper should not prevent you from using a libxc
> > > function directly. There are many corner-cases regarding when it is
> > > safe to inject a pagefault into the guest to bring back pages and that
> > > logic is not trivial enough to be implemented in LibVMI. It's way too
> > > specific to the usecase to decide when it's safe to do that. So in
> > > this case, if you know you are in ring3 of your target process you can
> > > just call the libxc function directly.
> >
> > That's very very true, however I think we should assume that libraries
> > such as libvmi and libbdvmi are used by client code smart enough to
> > figure out, using said libraries, enough things about what's appropriate
> > (or not) to do, and let them do it rather than second-guess them.
> > Hence, in libbdvmi I've wrapped that, and added a comment basically
> > saying what you're saying above:
> > https://github.com/bitdefender/libbdvmi/blob/master/src/xendriver.cpp#L542
> > If we end up having the client code call backend-specific code directly
> > we're not, after all, really abstracting it with our wrapper libraries.
>
> If someone adds that wrapper to LibVMI I wouldn't veto it - but it's
> also a function that I'm 100% expecting will generate a lot of
> confusion and whining about it not working in all cases. So I chose
> not to add it myself because of that. ¯\(ツ)/¯

Tamas, i'm using Python so i can't directly call libxc.

Also, and as Razvan said, I would like to stick with a coherent and clearly defined abstractions layers, first because it will allow me to be hypervisor-agnostic in the future, if new LibVMI drivers are developed, and then just for the sack of API consistency.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Andrew Cooper]

On 24/04/2019 20:11, Mathieu Tarral wrote:
> On Wednesday 24 April 2019 16:27, Nuernberger, Stefan <snu@amazon.de> wrote:
>
>> On Sun, 2019-04-21 at 22:26 +0000, Mathieu Tarral wrote:
>>
>>> Hi,
>>> I'm having an issue with Xen's VMI subsystem.
>>> My goal is to build a small debugger that can break at an
>>> application's entrypoint
>>> on Windows XP, when a new process is being created.
>> There was an announcement a while back on this list of a more modern
>> xen debugger for VMI purposes. Maybe you are more lucky with that one?
>>
>> https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/ja
>> nuary/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/
>>
>> https://github.com/nccgroup/xendbg
>>
>> (I have not yet tried it, though.)
> Yes, I heard of it via Tamas.
> I actually contacted the developer and tried to see how we could work together on the same project.
>
> He likes the idea of being hypervisor-agnostic, and being built upon an abstraction layer.
>
> The reason he chooses not to use LibVMI was because of the lack of support for Xen PV guests, IIRC.


The lack of support for PV guests is (amongst other things) a side
effect of the vm_event ring registration API being HVM-specific.

There is ongoing work from Bitdefender to move to a completely different
mechanism which would be compatible with PV guests, after which basic
support for breakpoints/etc ought to be easy (although PV guests will
never be as fully featured as HVM guests, seeing as some of the VMI
options are specific to using hardware non-root mode support).

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Andrew Cooper]

On 24/04/2019 20:11, Mathieu Tarral wrote:
> On Wednesday 24 April 2019 16:27, Nuernberger, Stefan <snu@amazon.de> wrote:
>
>> On Sun, 2019-04-21 at 22:26 +0000, Mathieu Tarral wrote:
>>
>>> Hi,
>>> I'm having an issue with Xen's VMI subsystem.
>>> My goal is to build a small debugger that can break at an
>>> application's entrypoint
>>> on Windows XP, when a new process is being created.
>> There was an announcement a while back on this list of a more modern
>> xen debugger for VMI purposes. Maybe you are more lucky with that one?
>>
>> https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/ja
>> nuary/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/
>>
>> https://github.com/nccgroup/xendbg
>>
>> (I have not yet tried it, though.)
> Yes, I heard of it via Tamas.
> I actually contacted the developer and tried to see how we could work together on the same project.
>
> He likes the idea of being hypervisor-agnostic, and being built upon an abstraction layer.
>
> The reason he chooses not to use LibVMI was because of the lack of support for Xen PV guests, IIRC.


The lack of support for PV guests is (amongst other things) a side
effect of the vm_event ring registration API being HVM-specific.

There is ongoing work from Bitdefender to move to a completely different
mechanism which would be compatible with PV guests, after which basic
support for breakpoints/etc ought to be easy (although PV guests will
never be as fully featured as HVM guests, seeing as some of the VMI
options are specific to using hardware non-root mode support).

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Mathieu Tarral]

On Wednesday 24 April 2019 17:35, Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:

> On Sun, Apr 21, 2019 at 4:27 PM Mathieu Tarral
> mathieu.tarral@protonmail.com wrote:
>
> > Hi,
> > I'm having an issue with Xen's VMI subsystem.
> > My goal is to build a small debugger that can break at an application's entrypoint
> > on Windows XP, when a new process is being created.
> > To accomplish this, I first set a software breakpoint on KiThreadStartup (ntoskrnl.exe),
> > then on RtlUserThreadStart (ntdll.dll).
> > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> > To my surprise, it didn't work as expected, since my application just hanged, waiting for the next singlestep event:
> > --Waiting for xen events...(0 ms)
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > Enabling single step
> > INFO:WindowsDebugContext:[105] at: 0x806d32d6
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > --Setting MTF flag on vcpu 0
> > Enabling single step
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(0 ms)
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > Enabling single step
> > INFO:WindowsDebugContext:[106] at: 0x806d32dc
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > --Setting MTF flag on vcpu 0
> > Enabling single step
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > The reason why i'm disabling end enabling the singlestep successively is because i already
> > have a libvmi singlestep event registered, with the MTF flag disabled.
> > I only use it for breakpoint recoil situations.
> > It's a limitation of the libvmi API where you cannot modified a registered event to enable singlestep at will.
>
> I'm not entirely sure what you mean here (and perhaps that's a
> discussion to be moved to the LibVMI issues page). If you already have
> an event registered for singlestepping why would you want to disable
> it just to re-enable it? If it's because you just have the handler
> registered without MTF actually active, that's specifically made for
> the situation where turn on/off MTF using the VM_EVENT_RESPONSE_FLAG.

The reason is because i needed an on-demand activation of MTF, without being in an event callback.

Let's say my GDB client asks me to do one singlestep.
I will need to register a new singlestep event, enabled, listen
one event and then return.

The problem is that I already have my singlestep event for breakpoint recoiling purposes which is registered.

That's the reason I'm unregistering this recoil singlestep event and reregistering it afterwards, because I can't "reuse" it:
https://github.com/Wenzel/pyvmidbg/blob/master/vmidbg/breakpoint.py#L79

I actually discussed it with you here:
https://github.com/libvmi/libvmi/issues/665

> Also, using MTF to reach parts of the code that are several hundred
> instructions down the pipe will most likely not work due to the
> extreme overhead it adds. You are more likely to get an interrupt and
> land somewhere in the kernel long before you reach your target. At
> least that has been my experience. You likely want to investigate
> other options, such as what Razvan recommended.

I agree, it was a backup solution from the beginning.
I would like to use hardware breakpoint instead, so I won't have to deal with pagefault injection.

As I reported to you, I implemented this solution already:
https://github.com/Wenzel/pyvmidbg/blob/entrypoint.hwbreakpoint/vmidbg/windowsdebugcontext.py#L210

But as soon as I restart my process, DR7 gets rewritten at some point by Windows, maybe in the NtContinue syscall where it sets a new Thread CONTEXT.

Anyway, i would need to intercept a MOV-TO-DRx to prevent Windows from disabling my hardware breakpoints :)

As I understood, this was not exposed by the Xen VMI API yet:
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/vm_event.h;h=b2bafc0d77f9758e42b0d53c05a7e6bb86c86686;hb=HEAD#l154

Mathieu


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

On Wednesday 24 April 2019 17:35, Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:

> On Sun, Apr 21, 2019 at 4:27 PM Mathieu Tarral
> mathieu.tarral@protonmail.com wrote:
>
> > Hi,
> > I'm having an issue with Xen's VMI subsystem.
> > My goal is to build a small debugger that can break at an application's entrypoint
> > on Windows XP, when a new process is being created.
> > To accomplish this, I first set a software breakpoint on KiThreadStartup (ntoskrnl.exe),
> > then on RtlUserThreadStart (ntdll.dll).
> > The problem is that RtlUserThreadStart is paged-out, so i'm trying to reach it via singlestepping as a backup solution.
> > To my surprise, it didn't work as expected, since my application just hanged, waiting for the next singlestep event:
> > --Waiting for xen events...(0 ms)
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > Enabling single step
> > INFO:WindowsDebugContext:[105] at: 0x806d32d6
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > --Setting MTF flag on vcpu 0
> > Enabling single step
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(0 ms)
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > Enabling single step
> > INFO:WindowsDebugContext:[106] at: 0x806d32dc
> > Disabling single step on vcpu: 0
> > --Removing MTF flag from vcpu 0
> > --Shutting down single step on domain 1
> > --Removing MTF flag from vcpu 0
> > --Starting single step on domain 1
> > --Setting MTF flag on vcpu 0
> > Enabling single step
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > --Waiting for xen events...(1000 ms)
> > The reason why i'm disabling end enabling the singlestep successively is because i already
> > have a libvmi singlestep event registered, with the MTF flag disabled.
> > I only use it for breakpoint recoil situations.
> > It's a limitation of the libvmi API where you cannot modified a registered event to enable singlestep at will.
>
> I'm not entirely sure what you mean here (and perhaps that's a
> discussion to be moved to the LibVMI issues page). If you already have
> an event registered for singlestepping why would you want to disable
> it just to re-enable it? If it's because you just have the handler
> registered without MTF actually active, that's specifically made for
> the situation where turn on/off MTF using the VM_EVENT_RESPONSE_FLAG.

The reason is because i needed an on-demand activation of MTF, without being in an event callback.

Let's say my GDB client asks me to do one singlestep.
I will need to register a new singlestep event, enabled, listen
one event and then return.

The problem is that I already have my singlestep event for breakpoint recoiling purposes which is registered.

That's the reason I'm unregistering this recoil singlestep event and reregistering it afterwards, because I can't "reuse" it:
https://github.com/Wenzel/pyvmidbg/blob/master/vmidbg/breakpoint.py#L79

I actually discussed it with you here:
https://github.com/libvmi/libvmi/issues/665

> Also, using MTF to reach parts of the code that are several hundred
> instructions down the pipe will most likely not work due to the
> extreme overhead it adds. You are more likely to get an interrupt and
> land somewhere in the kernel long before you reach your target. At
> least that has been my experience. You likely want to investigate
> other options, such as what Razvan recommended.

I agree, it was a backup solution from the beginning.
I would like to use hardware breakpoint instead, so I won't have to deal with pagefault injection.

As I reported to you, I implemented this solution already:
https://github.com/Wenzel/pyvmidbg/blob/entrypoint.hwbreakpoint/vmidbg/windowsdebugcontext.py#L210

But as soon as I restart my process, DR7 gets rewritten at some point by Windows, maybe in the NtContinue syscall where it sets a new Thread CONTEXT.

Anyway, i would need to intercept a MOV-TO-DRx to prevent Windows from disabling my hardware breakpoints :)

As I understood, this was not exposed by the Xen VMI API yet:
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/vm_event.h;h=b2bafc0d77f9758e42b0d53c05a7e6bb86c86686;hb=HEAD#l154

Mathieu


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Tamas K Lengyel]

> Anyway, i would need to intercept a MOV-TO-DRx to prevent Windows from disabling my hardware breakpoints :)
>
> As I understood, this was not exposed by the Xen VMI API yet:
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/vm_event.h;h=b2bafc0d77f9758e42b0d53c05a7e6bb86c86686;hb=HEAD#l154

Correct, but nothing stops you from adding the glue that enables that
;) The hardware is fully capable of trapping on that event, it's just
a matter of wiring it all up. Should be largely a copy-paste job if
you follow the example of other vm_event enable/disable functions.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Tamas K Lengyel]

> Anyway, i would need to intercept a MOV-TO-DRx to prevent Windows from disabling my hardware breakpoints :)
>
> As I understood, this was not exposed by the Xen VMI API yet:
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/vm_event.h;h=b2bafc0d77f9758e42b0d53c05a7e6bb86c86686;hb=HEAD#l154

Correct, but nothing stops you from adding the glue that enables that
;) The hardware is fully capable of trapping on that event, it's just
a matter of wiring it all up. Should be largely a copy-paste job if
you follow the example of other vm_event enable/disable functions.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Tamas K Lengyel]

> > I'm not entirely sure what you mean here (and perhaps that's a
> > discussion to be moved to the LibVMI issues page). If you already have
> > an event registered for singlestepping why would you want to disable
> > it just to re-enable it? If it's because you just have the handler
> > registered without MTF actually active, that's specifically made for
> > the situation where turn on/off MTF using the VM_EVENT_RESPONSE_FLAG.
>
> The reason is because i needed an on-demand activation of MTF, without being in an event callback.
>
> Let's say my GDB client asks me to do one singlestep.
> I will need to register a new singlestep event, enabled, listen
> one event and then return.
>
> The problem is that I already have my singlestep event for breakpoint recoiling purposes which is registered.
>
> That's the reason I'm unregistering this recoil singlestep event and reregistering it afterwards, because I can't "reuse" it:
> https://github.com/Wenzel/pyvmidbg/blob/master/vmidbg/breakpoint.py#L79

Yea, that seems suboptimal. Provided there is already a
vmi_stop_single_step_vcpu function in LibVMI which just removes the
MTF from a given vCPU, I would recommend replacing that function with
a generic one that you can use to both enable/disable MTF on a
per-vCPU base.

>
> I actually discussed it with you here:
> https://github.com/libvmi/libvmi/issues/665

I see I actually made a very similar recommendation in that discussion
then what I just described above ;)

Cheers,
Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Tamas K Lengyel]

> > I'm not entirely sure what you mean here (and perhaps that's a
> > discussion to be moved to the LibVMI issues page). If you already have
> > an event registered for singlestepping why would you want to disable
> > it just to re-enable it? If it's because you just have the handler
> > registered without MTF actually active, that's specifically made for
> > the situation where turn on/off MTF using the VM_EVENT_RESPONSE_FLAG.
>
> The reason is because i needed an on-demand activation of MTF, without being in an event callback.
>
> Let's say my GDB client asks me to do one singlestep.
> I will need to register a new singlestep event, enabled, listen
> one event and then return.
>
> The problem is that I already have my singlestep event for breakpoint recoiling purposes which is registered.
>
> That's the reason I'm unregistering this recoil singlestep event and reregistering it afterwards, because I can't "reuse" it:
> https://github.com/Wenzel/pyvmidbg/blob/master/vmidbg/breakpoint.py#L79

Yea, that seems suboptimal. Provided there is already a
vmi_stop_single_step_vcpu function in LibVMI which just removes the
MTF from a given vCPU, I would recommend replacing that function with
a generic one that you can use to both enable/disable MTF on a
per-vCPU base.

>
> I actually discussed it with you here:
> https://github.com/libvmi/libvmi/issues/665

I see I actually made a very similar recommendation in that discussion
then what I just described above ;)

Cheers,
Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Andrew Cooper]

On 25/04/2019 00:43, Mathieu Tarral wrote:
> On Wednesday 24 April 2019 14:00, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
>> On 23/04/2019 22:59, Mathieu Tarral wrote:
>>
>>>>> The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
>>>>> at 0x806d32dc:
>>>>> [0x7c90e514]> s 0x806d32dc
>>>>> [0x806d32dc]> pd 10
>>>>> 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
>>>>> This is a read of the APIC TPR, which is very commonly an operation
>>>>> accelerated by hardware (because without it, virtualising windows XP is
>>>>> exceedingly slow).
>>>> What is your CPU, and how exactly are you trying to singlestep.  Is it
>>>> with MTF, or using the trap flag inside the guest?
>>>> My CPU is an Intel i7-8750H.
>>>> I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.
>> Ok - that is a CoffeeLake.
>>
>> I've had a quick play with a simple example (XTF test and xen-access)
>> and MTF does appear to work fine with both TPR shadowing and full APIC
>> virtualisation.  I've tried on Haswell, KabyLake and CoffeeLake.
> Thank you for testing.
>
>> Can you describe the exact VMI configuration you've got enabled -
>> specifically which other intercepts are active?
> My BreakpointManager class is dealing with VMI events.
> It registers 3 events:
> - int3
> - debug event (for hardware breakpoints)
> - singlestep event (for breakpoint recoil on int3)
>
> https://github.com/Wenzel/pyvmidbg/blob/1a2e97c238a96c98c19cdbbdefa6047be246e504/vmidbg/breakpoint.py#L20

What about control register intercepts, or memory permission restrictions?

I expect things are going wrong when the TPR access is emulated by Xen,
rather than completed directly by hardware, but I can't explain how
you'd be getting into that situation without some other VMI settings
being active at the same time.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Andrew Cooper]

On 25/04/2019 00:43, Mathieu Tarral wrote:
> On Wednesday 24 April 2019 14:00, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
>> On 23/04/2019 22:59, Mathieu Tarral wrote:
>>
>>>>> The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
>>>>> at 0x806d32dc:
>>>>> [0x7c90e514]> s 0x806d32dc
>>>>> [0x806d32dc]> pd 10
>>>>> 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
>>>>> This is a read of the APIC TPR, which is very commonly an operation
>>>>> accelerated by hardware (because without it, virtualising windows XP is
>>>>> exceedingly slow).
>>>> What is your CPU, and how exactly are you trying to singlestep.  Is it
>>>> with MTF, or using the trap flag inside the guest?
>>>> My CPU is an Intel i7-8750H.
>>>> I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.
>> Ok - that is a CoffeeLake.
>>
>> I've had a quick play with a simple example (XTF test and xen-access)
>> and MTF does appear to work fine with both TPR shadowing and full APIC
>> virtualisation.  I've tried on Haswell, KabyLake and CoffeeLake.
> Thank you for testing.
>
>> Can you describe the exact VMI configuration you've got enabled -
>> specifically which other intercepts are active?
> My BreakpointManager class is dealing with VMI events.
> It registers 3 events:
> - int3
> - debug event (for hardware breakpoints)
> - singlestep event (for breakpoint recoil on int3)
>
> https://github.com/Wenzel/pyvmidbg/blob/1a2e97c238a96c98c19cdbbdefa6047be246e504/vmidbg/breakpoint.py#L20

What about control register intercepts, or memory permission restrictions?

I expect things are going wrong when the TPR access is emulated by Xen,
rather than completed directly by hardware, but I can't explain how
you'd be getting into that situation without some other VMI settings
being active at the same time.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: VMI: singlestep event not received

[Mathieu Tarral]

Le jeudi, avril 25, 2019 9:34 PM, Andrew Cooper <andrew.cooper3@citrix.com> a écrit :

> On 25/04/2019 00:43, Mathieu Tarral wrote:
>
> > On Wednesday 24 April 2019 14:00, Andrew Cooper andrew.cooper3@citrix.com wrote:
> >
> > > On 23/04/2019 22:59, Mathieu Tarral wrote:
> > >
> > > > > > The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
> > > > > > at 0x806d32dc:
> > > > > > [0x7c90e514]> s 0x806d32dc
> > > > > > [0x806d32dc]> pd 10
> > > > > > 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
> > > > > > This is a read of the APIC TPR, which is very commonly an operation
> > > > > > accelerated by hardware (because without it, virtualising windows XP is
> > > > > > exceedingly slow).
> > > > > > What is your CPU, and how exactly are you trying to singlestep.  Is it
> > > > > > with MTF, or using the trap flag inside the guest?
> > > > > > My CPU is an Intel i7-8750H.
> > > > > > I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.
> > > > > > Ok - that is a CoffeeLake.
> > >
> > > I've had a quick play with a simple example (XTF test and xen-access)
> > > and MTF does appear to work fine with both TPR shadowing and full APIC
> > > virtualisation.  I've tried on Haswell, KabyLake and CoffeeLake.
> > > Thank you for testing.
> >
> > > Can you describe the exact VMI configuration you've got enabled -
> > > specifically which other intercepts are active?
> > > My BreakpointManager class is dealing with VMI events.
> > > It registers 3 events:
> >
> > -   int3
> > -   debug event (for hardware breakpoints)
> > -   singlestep event (for breakpoint recoil on int3)
> >
> > https://github.com/Wenzel/pyvmidbg/blob/1a2e97c238a96c98c19cdbbdefa6047be246e504/vmidbg/breakpoint.py#L20
>
> What about control register intercepts, or memory permission restrictions?

I'm not using them, not CR3 intercepts, or mem_events registered.

> I expect things are going wrong when the TPR access is emulated by Xen,
> rather than completed directly by hardware, but I can't explain how
> you'd be getting into that situation without some other VMI settings
> being active at the same time.
I don't really know Xen, so I' can't explain it myself.
How can I debug this ?
Do you want me to add a printk somewhere ?

Mathieu



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] VMI: singlestep event not received

[Mathieu Tarral]

Le jeudi, avril 25, 2019 9:34 PM, Andrew Cooper <andrew.cooper3@citrix.com> a écrit :

> On 25/04/2019 00:43, Mathieu Tarral wrote:
>
> > On Wednesday 24 April 2019 14:00, Andrew Cooper andrew.cooper3@citrix.com wrote:
> >
> > > On 23/04/2019 22:59, Mathieu Tarral wrote:
> > >
> > > > > > The funny thing is that it's always at the same instruction that it fails, the 106th singlestep,
> > > > > > at 0x806d32dc:
> > > > > > [0x7c90e514]> s 0x806d32dc
> > > > > > [0x806d32dc]> pd 10
> > > > > > 0x806d32dc 890d8000feff mov dword [0xfffe0080], ecx
> > > > > > This is a read of the APIC TPR, which is very commonly an operation
> > > > > > accelerated by hardware (because without it, virtualising windows XP is
> > > > > > exceedingly slow).
> > > > > > What is your CPU, and how exactly are you trying to singlestep.  Is it
> > > > > > with MTF, or using the trap flag inside the guest?
> > > > > > My CPU is an Intel i7-8750H.
> > > > > > I'm using the libvmi API, which is enabling the MTF flag to singlestep, if i judge by the debug logs.
> > > > > > Ok - that is a CoffeeLake.
> > >
> > > I've had a quick play with a simple example (XTF test and xen-access)
> > > and MTF does appear to work fine with both TPR shadowing and full APIC
> > > virtualisation.  I've tried on Haswell, KabyLake and CoffeeLake.
> > > Thank you for testing.
> >
> > > Can you describe the exact VMI configuration you've got enabled -
> > > specifically which other intercepts are active?
> > > My BreakpointManager class is dealing with VMI events.
> > > It registers 3 events:
> >
> > -   int3
> > -   debug event (for hardware breakpoints)
> > -   singlestep event (for breakpoint recoil on int3)
> >
> > https://github.com/Wenzel/pyvmidbg/blob/1a2e97c238a96c98c19cdbbdefa6047be246e504/vmidbg/breakpoint.py#L20
>
> What about control register intercepts, or memory permission restrictions?

I'm not using them, not CR3 intercepts, or mem_events registered.

> I expect things are going wrong when the TPR access is emulated by Xen,
> rather than completed directly by hardware, but I can't explain how
> you'd be getting into that situation without some other VMI settings
> being active at the same time.
I don't really know Xen, so I' can't explain it myself.
How can I debug this ?
Do you want me to add a printk somewhere ?

Mathieu



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel