[Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Alexey Kardashevskiy]

sPAPR PHB emulates IO ports on PCI via a special memory region which
routes all reads/writes further via cpu_in*/cpu_out* which are eventually
processed by MemoryRegionOps implemented by devices.

As devices normally take care of endianness themselves by setting
correct MemoryRegionOps::endianness for their memory regions, it is
better to set endianness in spapr_io_ops to NATIVE.

Tested on sPAPR KVM with e1000, rtl8139, virtio-net.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---

I would really appreciate if someone told me what exactly changed
in QEMU and broke spapr-pci. It used to work... Thanks!

---
 hw/ppc/spapr_pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index ca588aa..dfe4d04 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -472,7 +472,7 @@ static void spapr_io_write(void *opaque, hwaddr addr,
 }
 
 static const MemoryRegionOps spapr_io_ops = {
-    .endianness = DEVICE_LITTLE_ENDIAN,
+    .endianness = DEVICE_NATIVE_ENDIAN,
     .read = spapr_io_read,
     .write = spapr_io_write
 };
-- 
1.8.3.2

[Qemu-devel] BUG report Re: [PATCH] spapr-pci: change endianness for io ports space

[Alexey Kardashevskiy]

Hi!

Got a small lesson from Ben how to post stuff to the list and here are more
details :)

Every PCI device which uses IO ports (at least vga, e1000, virtio-pci,
rtl8139) is broken in the master branch of QEMU for powerpc-kvm. The
problem is exactly with endianness. For example, setup_vq() from
drivers/virtio/virtio_pci.c (guest kernel) fails on:

iowrite16(msix_vec, vp_dev->ioaddr + VIRTIO_MSI_QUEUE_VECTOR);

as QEMU receives msix_vec==0x100 instead of 0x1 in virtio_ioport_write().

The patch which broke it is b40acf99bef69fa8ab0f9092ff162fde945eec12
"ioport: Switch dispatching to memory core layer".

The patch below fixes this but I am not really sure it is correct. Is it?
Thanks.


On 07/12/2013 05:37 PM, Alexey Kardashevskiy wrote:
> sPAPR PHB emulates IO ports on PCI via a special memory region which
> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
> processed by MemoryRegionOps implemented by devices.
> 
> As devices normally take care of endianness themselves by setting
> correct MemoryRegionOps::endianness for their memory regions, it is
> better to set endianness in spapr_io_ops to NATIVE.
> 
> Tested on sPAPR KVM with e1000, rtl8139, virtio-net.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
> 
> I would really appreciate if someone told me what exactly changed
> in QEMU and broke spapr-pci. It used to work... Thanks!
> 
> ---
>  hw/ppc/spapr_pci.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index ca588aa..dfe4d04 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -472,7 +472,7 @@ static void spapr_io_write(void *opaque, hwaddr addr,
>  }
>  
>  static const MemoryRegionOps spapr_io_ops = {
> -    .endianness = DEVICE_LITTLE_ENDIAN,
> +    .endianness = DEVICE_NATIVE_ENDIAN,
>      .read = spapr_io_read,
>      .write = spapr_io_write
>  };
> 


-- 
Alexey

Re: [Qemu-devel] BUG report Re: [PATCH] spapr-pci: change endianness for io ports space

[Alexander Graf]

Am 12.07.2013 um 10:59 schrieb Alexey Kardashevskiy <aik@ozlabs.ru>:

> Hi!
> 
> Got a small lesson from Ben how to post stuff to the list and here are more
> details :)
> 
> Every PCI device which uses IO ports (at least vga, e1000, virtio-pci,
> rtl8139) is broken in the master branch of QEMU for powerpc-kvm. The
> problem is exactly with endianness. For example, setup_vq() from
> drivers/virtio/virtio_pci.c (guest kernel) fails on:
> 
> iowrite16(msix_vec, vp_dev->ioaddr + VIRTIO_MSI_QUEUE_VECTOR);
> 
> as QEMU receives msix_vec==0x100 instead of 0x1 in virtio_ioport_write().
> 
> The patch which broke it is b40acf99bef69fa8ab0f9092ff162fde945eec12
> "ioport: Switch dispatching to memory core layer".
> 
> The patch below fixes this but I am not really sure it is correct. Is it?

It is not. Please read my mail on the topic.

Alex

> Thanks.
> 
> 
> On 07/12/2013 05:37 PM, Alexey Kardashevskiy wrote:
>> sPAPR PHB emulates IO ports on PCI via a special memory region which
>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
>> processed by MemoryRegionOps implemented by devices.
>> 
>> As devices normally take care of endianness themselves by setting
>> correct MemoryRegionOps::endianness for their memory regions, it is
>> better to set endianness in spapr_io_ops to NATIVE.
>> 
>> Tested on sPAPR KVM with e1000, rtl8139, virtio-net.
>> 
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>> 
>> I would really appreciate if someone told me what exactly changed
>> in QEMU and broke spapr-pci. It used to work... Thanks!
>> 
>> ---
>> hw/ppc/spapr_pci.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>> index ca588aa..dfe4d04 100644
>> --- a/hw/ppc/spapr_pci.c
>> +++ b/hw/ppc/spapr_pci.c
>> @@ -472,7 +472,7 @@ static void spapr_io_write(void *opaque, hwaddr addr,
>> }
>> 
>> static const MemoryRegionOps spapr_io_ops = {
>> -    .endianness = DEVICE_LITTLE_ENDIAN,
>> +    .endianness = DEVICE_NATIVE_ENDIAN,
>>     .read = spapr_io_read,
>>     .write = spapr_io_write
>> };
> 
> 
> -- 
> Alexey

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote:
> sPAPR PHB emulates IO ports on PCI via a special memory region which
> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
> processed by MemoryRegionOps implemented by devices.

Hrm.  That double dispatch was a workaround for bugs in the plain
memory region dispatching which meant we couldn't directly map regions
in memory space to IO areas.

It would be worth checking if that workaround is still necessary.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Alexey Kardashevskiy]

On 07/13/2013 06:03 PM, David Gibson wrote:
> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote:
>> sPAPR PHB emulates IO ports on PCI via a special memory region which
>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
>> processed by MemoryRegionOps implemented by devices.

> Hrm.  That double dispatch was a workaround for bugs in the plain
> memory region dispatching which meant we couldn't directly map regions
> in memory space to IO areas.
> 
> It would be worth checking if that workaround is still necessary.

Hm. Good point, thanks! It seems memory_region_init_io is not necessary any
more. Will make a patch for it.


-- 
Alexey

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Anthony Liguori]

Alexey Kardashevskiy <aik@ozlabs.ru> writes:

> On 07/13/2013 06:03 PM, David Gibson wrote:
>> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote:
>>> sPAPR PHB emulates IO ports on PCI via a special memory region which
>>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
>>> processed by MemoryRegionOps implemented by devices.
>
>> Hrm.  That double dispatch was a workaround for bugs in the plain
>> memory region dispatching which meant we couldn't directly map regions
>> in memory space to IO areas.
>> 
>> It would be worth checking if that workaround is still necessary.
>
> Hm. Good point, thanks! It seems memory_region_init_io is not necessary any
> more. Will make a patch for it.

You should try the latest qemu.git commit.  There shouldn't be a problem
anymore.

Regards,

Anthony Liguori

>
>
> -- 
> Alexey

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Alexey Kardashevskiy]

On 07/16/2013 01:02 AM, Anthony Liguori wrote:
> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
> 
>> On 07/13/2013 06:03 PM, David Gibson wrote:
>>> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote:
>>>> sPAPR PHB emulates IO ports on PCI via a special memory region which
>>>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
>>>> processed by MemoryRegionOps implemented by devices.
>>
>>> Hrm.  That double dispatch was a workaround for bugs in the plain
>>> memory region dispatching which meant we couldn't directly map regions
>>> in memory space to IO areas.
>>>
>>> It would be worth checking if that workaround is still necessary.
>>
>> Hm. Good point, thanks! It seems memory_region_init_io is not necessary any
>> more. Will make a patch for it.
> 
> You should try the latest qemu.git commit.  There shouldn't be a problem
> anymore.


Does this mean sPAPR still needs an additional IO memory region? It looks
redundand and everything (almost) works without it...


-- 
Alexey

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Anthony Liguori]

Alexey Kardashevskiy <aik@ozlabs.ru> writes:

> On 07/16/2013 01:02 AM, Anthony Liguori wrote:
>> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>> 
>>> On 07/13/2013 06:03 PM, David Gibson wrote:
>>>> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote:
>>>>> sPAPR PHB emulates IO ports on PCI via a special memory region which
>>>>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
>>>>> processed by MemoryRegionOps implemented by devices.
>>>
>>>> Hrm.  That double dispatch was a workaround for bugs in the plain
>>>> memory region dispatching which meant we couldn't directly map regions
>>>> in memory space to IO areas.
>>>>
>>>> It would be worth checking if that workaround is still necessary.
>>>
>>> Hm. Good point, thanks! It seems memory_region_init_io is not necessary any
>>> more. Will make a patch for it.
>> 
>> You should try the latest qemu.git commit.  There shouldn't be a problem
>> anymore.
>
>
> Does this mean sPAPR still needs an additional IO memory region? It looks
> redundand and everything (almost) works without it...

There's more brokenness...

Some ISA devices mark themselves as "little endian" whereas others mark
themselves as "native endian".

"little endian" really means "do byte lane swapping during dispatch" if
host endian != target endian.

So on sPAPR, what you're getting is the redundant IO memory region
causing a byte lane swap which is then negated by the ISA devices that
mark themselves as little endian (such as VGA).

The right solution is to drop the additional IO region on sPAPR and
remove the ISA devices marking themselves as "little endian".

But that requires careful testing and fixing the other platforms that
also are relying on the doube byte lane swapping.

Regards,

Anthony Liguori


>
>
> -- 
> Alexey

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Alexey Kardashevskiy]

On 07/16/2013 10:28 AM, Anthony Liguori wrote:
> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
> 
>> On 07/16/2013 01:02 AM, Anthony Liguori wrote:
>>> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>>>
>>>> On 07/13/2013 06:03 PM, David Gibson wrote:
>>>>> On Fri, Jul 12, 2013 at 05:37:19PM +1000, Alexey Kardashevskiy wrote:
>>>>>> sPAPR PHB emulates IO ports on PCI via a special memory region which
>>>>>> routes all reads/writes further via cpu_in*/cpu_out* which are eventually
>>>>>> processed by MemoryRegionOps implemented by devices.
>>>>
>>>>> Hrm.  That double dispatch was a workaround for bugs in the plain
>>>>> memory region dispatching which meant we couldn't directly map regions
>>>>> in memory space to IO areas.
>>>>>
>>>>> It would be worth checking if that workaround is still necessary.
>>>>
>>>> Hm. Good point, thanks! It seems memory_region_init_io is not necessary any
>>>> more. Will make a patch for it.
>>>
>>> You should try the latest qemu.git commit.  There shouldn't be a problem
>>> anymore.
>>
>>
>> Does this mean sPAPR still needs an additional IO memory region? It looks
>> redundand and everything (almost) works without it...
> 
> There's more brokenness...
> 
> Some ISA devices mark themselves as "little endian" whereas others mark
> themselves as "native endian".
> 
> "little endian" really means "do byte lane swapping during dispatch" if
> host endian != target endian.
> 
> So on sPAPR, what you're getting is the redundant IO memory region
> causing a byte lane swap which is then negated by the ISA devices that
> mark themselves as little endian (such as VGA).
> 
> The right solution is to drop the additional IO region on sPAPR and
> remove the ISA devices marking themselves as "little endian".


No, this is not endianness, this is something different caused by a
difference in IO port registration (subpage? section? memoryregion? I am
going to draw a graph and try realize what is what here :) ).

Even very first IO port access does not reach VGA, fails somehow in
address_space_translate_internal() but devices other than VGA (well, at
least network devices) work perfectly.


> But that requires careful testing and fixing the other platforms that
> also are relying on the doube byte lane swapping.


-- 
Alexey

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Anthony Liguori]

On Mon, Jul 15, 2013 at 7:39 PM, Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> On 07/16/2013 10:28 AM, Anthony Liguori wrote:
>> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>>
>> The right solution is to drop the additional IO region on sPAPR and
>> remove the ISA devices marking themselves as "little endian".
>
>
> No, this is not endianness, this is something different caused by a
> difference in IO port registration (subpage? section? memoryregion? I am
> going to draw a graph and try realize what is what here :) ).
>
> Even very first IO port access does not reach VGA, fails somehow in
> address_space_translate_internal() but devices other than VGA (well, at
> least network devices) work perfectly.

Can you be more specific about what's failing?  VGA works fine for me with TCG.

Regards,

Anthony Liguori

>
>
>> But that requires careful testing and fixing the other platforms that
>> also are relying on the doube byte lane swapping.
>
>
> --
> Alexey

Re: [Qemu-devel] [PATCH] spapr-pci: change endianness for io ports space

[Alexey Kardashevskiy]

On 07/16/2013 12:05 PM, Anthony Liguori wrote:
> On Mon, Jul 15, 2013 at 7:39 PM, Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
>> On 07/16/2013 10:28 AM, Anthony Liguori wrote:
>>> Alexey Kardashevskiy <aik@ozlabs.ru> writes:
>>>
>>> The right solution is to drop the additional IO region on sPAPR and
>>> remove the ISA devices marking themselves as "little endian".
>>
>>
>> No, this is not endianness, this is something different caused by a
>> difference in IO port registration (subpage? section? memoryregion? I am
>> going to draw a graph and try realize what is what here :) ).
>>
>> Even very first IO port access does not reach VGA, fails somehow in
>> address_space_translate_internal() but devices other than VGA (well, at
>> least network devices) work perfectly.
> 
> Can you be more specific about what's failing?  VGA works fine for me with TCG.

With the patch we are replying to? Wow. Does not work for me (full
emulation or kvm) - does not show even a single symbol.


-- 
Alexey