[PATCH v2 00/15] xen/arm: Add GICv3 support

[PATCH v2 00/15] xen/arm: Add GICv3 support

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Made changes to existing gic and vgic drivers
to make it generic and added support for GIC v3
hw version

Tested with ARM64 simulator with single core

Major Changes in v2:
 - Added per domain io handler
 - Use device api to initialize gic driver
 - Avoid use of void pointer to access common data
 - Moved vgic address information data from domain.h
 - Coding style

Vijaya Kumar K (15):
  xen/arm: register mmio handler at runtime
  xen/arm: move vgic rank data to gic header file
  arm/xen: move gic save and restore registers to gic driver
  xen/arm: move gic definitions to seperate file
  xen/arm: segregate GIC low level functionality
  xen/arm: move gic lock out of gic data structure
  xen/arm: split gic driver into generic and gic-v2 driver
  xen/arm: use device api to detect GIC version
  xen/arm: segregate VGIC low level functionality
  xen/arm: split vgic driver into generic and vgic-v2 driver
  xen/arm: make GIC context data version specific
  xen/arm: move GIC data to driver from domain structure
  xen/arm: Add support for GIC v3
  xen/arm: Add vgic support for GIC v3
  xen/arm: update GIC dt node with GIC v3 information

 xen/arch/arm/Makefile             |    4 +
 xen/arch/arm/domain.c             |    8 +
 xen/arch/arm/domain_build.c       |   41 +-
 xen/arch/arm/gic-v2.c             |  559 ++++++++++++++++++++++
 xen/arch/arm/gic-v3.c             |  927 +++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic.c                |  463 ++++--------------
 xen/arch/arm/io.c                 |   34 +-
 xen/arch/arm/io.h                 |    8 +-
 xen/arch/arm/smp.c                |    1 +
 xen/arch/arm/vgic-v2.c            |  639 +++++++++++++++++++++++++
 xen/arch/arm/vgic-v3.c            |  927 +++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic.c               |  577 ++---------------------
 xen/arch/arm/vuart.c              |    6 +-
 xen/include/asm-arm/device.h      |    4 +-
 xen/include/asm-arm/domain.h      |   22 +-
 xen/include/asm-arm/gic.h         |  219 +++++----
 xen/include/asm-arm/gic_v2_defs.h |  133 ++++++
 xen/include/asm-arm/gic_v3_defs.h |  187 ++++++++
 18 files changed, 3691 insertions(+), 1068 deletions(-)
 create mode 100644 xen/arch/arm/gic-v2.c
 create mode 100644 xen/arch/arm/gic-v3.c
 create mode 100644 xen/arch/arm/vgic-v2.c
 create mode 100644 xen/arch/arm/vgic-v3.c
 create mode 100644 xen/include/asm-arm/gic_v2_defs.h
 create mode 100644 xen/include/asm-arm/gic_v3_defs.h

-- 
1.7.9.5

[PATCH v2 01/15] xen/arm: register mmio handler at runtime

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

register mmio handlers at runtime and make
mmio handlers domain specific.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/domain.c        |    4 ++++
 xen/arch/arm/io.c            |   34 +++++++++++++++++++++++-----------
 xen/arch/arm/io.h            |    8 ++++++--
 xen/arch/arm/vgic.c          |    4 +++-
 xen/arch/arm/vuart.c         |    6 +++++-
 xen/include/asm-arm/domain.h |    1 +
 6 files changed, 42 insertions(+), 15 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index f26b77b..e2f0cc4 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -34,6 +34,7 @@
 #include <asm/platform.h>
 #include "vtimer.h"
 #include "vuart.h"
+#include "io.h"
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
@@ -521,6 +522,9 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags)
     share_xen_page_with_guest(
         virt_to_page(d->shared_info), d, XENSHARE_writable);
 
+    if ( (d->arch.io_handlers = xmalloc(struct io_handler)) == NULL )
+        goto fail;
+
     if ( (rc = p2m_alloc_table(d)) != 0 )
         goto fail;
 
diff --git a/xen/arch/arm/io.c b/xen/arch/arm/io.c
index a6db00b..d6231d0 100644
--- a/xen/arch/arm/io.c
+++ b/xen/arch/arm/io.c
@@ -17,31 +17,43 @@
  */
 
 #include <xen/config.h>
+#include <xen/init.h>
+#include <xen/kernel.h>
 #include <xen/lib.h>
+#include <xen/spinlock.h>
 #include <asm/current.h>
+#include <xen/sched.h>
 
 #include "io.h"
 
-static const struct mmio_handler *const mmio_handlers[] =
-{
-    &vgic_distr_mmio_handler,
-    &vuart_mmio_handler,
-};
-#define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
+static DEFINE_SPINLOCK(handler_lock);
 
 int handle_mmio(mmio_info_t *info)
 {
     struct vcpu *v = current;
     int i;
+    struct io_handler *mmio_handle = v->domain->arch.io_handlers;
 
-    for ( i = 0; i < MMIO_HANDLER_NR; i++ )
-        if ( mmio_handlers[i]->check_handler(v, info->gpa) )
+    for ( i = 0; i < mmio_handle->num_entries; i++ )
+    {
+        if ( mmio_handle->mmio_handlers[i]->check_handler(v, info->gpa) )
             return info->dabt.write ?
-                mmio_handlers[i]->write_handler(v, info) :
-                mmio_handlers[i]->read_handler(v, info);
-
+                mmio_handle->mmio_handlers[i]->write_handler(v, info) :
+                mmio_handle->mmio_handlers[i]->read_handler(v, info);
+    }
     return 0;
 }
+
+void register_mmio_handler(struct domain *d, struct mmio_handler * handle)
+{
+    unsigned long flags;
+    struct io_handler *handler = d->arch.io_handlers;
+    BUG_ON(handler->num_entries >= MAX_IO_HANDLER);
+
+    spin_lock_irqsave(&handler_lock, flags);
+    handler->mmio_handlers[handler->num_entries++] = handle;
+    spin_unlock_irqrestore(&handler_lock, flags);
+}
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
index 8d252c0..5fc1660 100644
--- a/xen/arch/arm/io.h
+++ b/xen/arch/arm/io.h
@@ -40,10 +40,14 @@ struct mmio_handler {
     mmio_write_t write_handler;
 };
 
-extern const struct mmio_handler vgic_distr_mmio_handler;
-extern const struct mmio_handler vuart_mmio_handler;
+#define MAX_IO_HANDLER  16
+struct io_handler {
+    int num_entries;
+    struct mmio_handler *mmio_handlers[MAX_IO_HANDLER];
+};
 
 extern int handle_mmio(mmio_info_t *info);
+void register_mmio_handler(struct domain *d, struct mmio_handler * handle);
 
 #endif
 
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 8616534..77b561e 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -35,6 +35,7 @@
 /* Number of ranks of interrupt registers for a domain */
 #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
 
+static struct mmio_handler vgic_distr_mmio_handler;
 /*
  * Rank containing GICD_<FOO><n> for GICD_<FOO> with
  * <b>-bits-per-interrupt
@@ -107,6 +108,7 @@ int domain_vgic_init(struct domain *d)
     }
     for (i=0; i<DOMAIN_NR_RANKS(d); i++)
         spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
+    register_mmio_handler(d, &vgic_distr_mmio_handler);
     return 0;
 }
 
@@ -673,7 +675,7 @@ static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
     return (addr >= (d->arch.vgic.dbase)) && (addr < (d->arch.vgic.dbase + PAGE_SIZE));
 }
 
-const struct mmio_handler vgic_distr_mmio_handler = {
+static struct mmio_handler vgic_distr_mmio_handler = {
     .check_handler = vgic_distr_mmio_check,
     .read_handler  = vgic_distr_mmio_read,
     .write_handler = vgic_distr_mmio_write,
diff --git a/xen/arch/arm/vuart.c b/xen/arch/arm/vuart.c
index b9d3ced..96df6b3 100644
--- a/xen/arch/arm/vuart.c
+++ b/xen/arch/arm/vuart.c
@@ -44,6 +44,8 @@
 
 #define domain_has_vuart(d) ((d)->arch.vuart.info != NULL)
 
+static struct mmio_handler vuart_mmio_handler;
+
 int domain_vuart_init(struct domain *d)
 {
     ASSERT( !d->domain_id );
@@ -59,6 +61,8 @@ int domain_vuart_init(struct domain *d)
     if ( !d->arch.vuart.buf )
         return -ENOMEM;
 
+    register_mmio_handler(d, &vuart_mmio_handler);
+
     return 0;
 }
 
@@ -133,7 +137,7 @@ static int vuart_mmio_write(struct vcpu *v, mmio_info_t *info)
     return 1;
 }
 
-const struct mmio_handler vuart_mmio_handler = {
+static struct mmio_handler vuart_mmio_handler = {
     .check_handler = vuart_mmio_check,
     .read_handler  = vuart_mmio_read,
     .write_handler = vuart_mmio_write,
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 50b9b54..23dac85 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -116,6 +116,7 @@ struct arch_domain
     struct hvm_domain hvm_domain;
     xen_pfn_t *grant_table_gpfn;
 
+    struct io_handler *io_handlers;
     /* Continuable domain_relinquish_resources(). */
     enum {
         RELMEM_not_started,
-- 
1.7.9.5

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

Hello vijay,

Thank your for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:

[..]

> -#define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
> +static DEFINE_SPINLOCK(handler_lock);

Why a global lock rather than a domain lock?

[..]

> +
> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle)
> +{
> +    unsigned long flags;
> +    struct io_handler *handler = d->arch.io_handlers;
> +    BUG_ON(handler->num_entries >= MAX_IO_HANDLER);
> +
> +    spin_lock_irqsave(&handler_lock, flags);

You don't need to disable the IRQ here.

> +    handler->mmio_handlers[handler->num_entries++] = handle;
> +    spin_unlock_irqrestore(&handler_lock, flags);
> +}
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
> index 8d252c0..5fc1660 100644
> --- a/xen/arch/arm/io.h
> +++ b/xen/arch/arm/io.h

As you are modifying this header. Can you move it in include/asm-arm?

> @@ -40,10 +40,14 @@ struct mmio_handler {
>      mmio_write_t write_handler;
>  };
>  
> -extern const struct mmio_handler vgic_distr_mmio_handler;
> -extern const struct mmio_handler vuart_mmio_handler;
> +#define MAX_IO_HANDLER  16
> +struct io_handler {
> +    int num_entries;
> +    struct mmio_handler *mmio_handlers[MAX_IO_HANDLER];
> +};
>  
>  extern int handle_mmio(mmio_info_t *info);
> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle);

As I said on the previous version, it would be nice to remove check
callback in the mmio_handler and replace it by addr/size. It's better if
we might want to change the place in the memory following the guest.

So the result function would be:

register_mmio_handler(struct domain *d, read_t read, write_t write,
addr, size);

>  
>  #endif
>  
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 8616534..77b561e 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -35,6 +35,7 @@
>  /* Number of ranks of interrupt registers for a domain */
>  #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
>  
> +static struct mmio_handler vgic_distr_mmio_handler;
>  /*
>   * Rank containing GICD_<FOO><n> for GICD_<FOO> with
>   * <b>-bits-per-interrupt
> @@ -107,6 +108,7 @@ int domain_vgic_init(struct domain *d)
>      }
>      for (i=0; i<DOMAIN_NR_RANKS(d); i++)
>          spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
> +    register_mmio_handler(d, &vgic_distr_mmio_handler);
>      return 0;
>  }
>  
> @@ -673,7 +675,7 @@ static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
>      return (addr >= (d->arch.vgic.dbase)) && (addr < (d->arch.vgic.dbase + PAGE_SIZE));
>  }
>  
> -const struct mmio_handler vgic_distr_mmio_handler = {
> +static struct mmio_handler vgic_distr_mmio_handler = {

Why did you remove the const?

[..]

> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 50b9b54..23dac85 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -116,6 +116,7 @@ struct arch_domain
>      struct hvm_domain hvm_domain;
>      xen_pfn_t *grant_table_gpfn;
>  
> +    struct io_handler *io_handlers;

Why do you need a pointer here? I think can can directly use
struct io_handler iohandlers.

Regards,

-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Vijay Kilari]

On Fri, Apr 4, 2014 at 5:48 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hello vijay,
>
> Thank your for the patch.
>
> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>
> [..]
>
>> -#define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
>> +static DEFINE_SPINLOCK(handler_lock);
>
> Why a global lock rather than a domain lock?
>
OK can move into domain io_handler structure

> [..]
>
>> +
>> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle)
>> +{
>> +    unsigned long flags;
>> +    struct io_handler *handler = d->arch.io_handlers;
>> +    BUG_ON(handler->num_entries >= MAX_IO_HANDLER);
>> +
>> +    spin_lock_irqsave(&handler_lock, flags);
>
> You don't need to disable the IRQ here.
  My first version is without IRQ disable. I remember Andrii suggested it
>
>> +    handler->mmio_handlers[handler->num_entries++] = handle;
>> +    spin_unlock_irqrestore(&handler_lock, flags);
>> +}
>>  /*
>>   * Local variables:
>>   * mode: C
>> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
>> index 8d252c0..5fc1660 100644
>> --- a/xen/arch/arm/io.h
>> +++ b/xen/arch/arm/io.h
>
> As you are modifying this header. Can you move it in include/asm-arm?
OK.

>
>> @@ -40,10 +40,14 @@ struct mmio_handler {
>>      mmio_write_t write_handler;
>>  };
>>
>> -extern const struct mmio_handler vgic_distr_mmio_handler;
>> -extern const struct mmio_handler vuart_mmio_handler;
>> +#define MAX_IO_HANDLER  16
>> +struct io_handler {
>> +    int num_entries;
>> +    struct mmio_handler *mmio_handlers[MAX_IO_HANDLER];
>> +};
>>
>>  extern int handle_mmio(mmio_info_t *info);
>> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle);
>
> As I said on the previous version, it would be nice to remove check
> callback in the mmio_handler and replace it by addr/size. It's better if
> we might want to change the place in the memory following the guest.
>
> So the result function would be:
>
> register_mmio_handler(struct domain *d, read_t read, write_t write,
> addr, size);
>
OK.
>>
>>  #endif
>>
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 8616534..77b561e 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -35,6 +35,7 @@
>>  /* Number of ranks of interrupt registers for a domain */
>>  #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
>>
>> +static struct mmio_handler vgic_distr_mmio_handler;
>>  /*
>>   * Rank containing GICD_<FOO><n> for GICD_<FOO> with
>>   * <b>-bits-per-interrupt
>> @@ -107,6 +108,7 @@ int domain_vgic_init(struct domain *d)
>>      }
>>      for (i=0; i<DOMAIN_NR_RANKS(d); i++)
>>          spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
>> +    register_mmio_handler(d, &vgic_distr_mmio_handler);
>>      return 0;
>>  }
>>
>> @@ -673,7 +675,7 @@ static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
>>      return (addr >= (d->arch.vgic.dbase)) && (addr < (d->arch.vgic.dbase + PAGE_SIZE));
>>  }
>>
>> -const struct mmio_handler vgic_distr_mmio_handler = {
>> +static struct mmio_handler vgic_distr_mmio_handler = {
>
> Why did you remove the const?

I think, compiler is allowing forward declaration with const

>
> [..]
>
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 50b9b54..23dac85 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -116,6 +116,7 @@ struct arch_domain
>>      struct hvm_domain hvm_domain;
>>      xen_pfn_t *grant_table_gpfn;
>>
>> +    struct io_handler *io_handlers;
>
> Why do you need a pointer here? I think can can directly use
> struct io_handler iohandlers.
>
  I just adds to increase the size of this arch_domain struct.
so allocated memory at runtime.

> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Ian Campbell]

On Fri, 2014-04-04 at 18:00 +0530, Vijay Kilari wrote:
> >> +    handler->mmio_handlers[handler->num_entries++] = handle;
> >> +    spin_unlock_irqrestore(&handler_lock, flags);
> >> +}
> >>  /*
> >>   * Local variables:
> >>   * mode: C
> >> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
> >> index 8d252c0..5fc1660 100644
> >> --- a/xen/arch/arm/io.h
> >> +++ b/xen/arch/arm/io.h
> >
> > As you are modifying this header. Can you move it in
> include/asm-arm?
> OK.

Please don't mix movement and modifications. Unless you are adding uses
from outside xen/arch/arm requiring ugly relative includes there is no
need to move this IMHO.

There is already asm/io.h which relates to readl/writel and friends. If
this were to move then it would need to be to asm/mmio.h or vmmio.h
something.

Ian.

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

On 04/04/2014 01:42 PM, Ian Campbell wrote:
> On Fri, 2014-04-04 at 18:00 +0530, Vijay Kilari wrote:
>>>> +    handler->mmio_handlers[handler->num_entries++] = handle;
>>>> +    spin_unlock_irqrestore(&handler_lock, flags);
>>>> +}
>>>>  /*
>>>>   * Local variables:
>>>>   * mode: C
>>>> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
>>>> index 8d252c0..5fc1660 100644
>>>> --- a/xen/arch/arm/io.h
>>>> +++ b/xen/arch/arm/io.h
>>>
>>> As you are modifying this header. Can you move it in
>> include/asm-arm?
>> OK.
> 
> Please don't mix movement and modifications. Unless you are adding uses
> from outside xen/arch/arm requiring ugly relative includes there is no
> need to move this IMHO.

My point was to clean up and having a single place for every headers.

> There is already asm/io.h which relates to readl/writel and friends. If
> this were to move then it would need to be to asm/mmio.h or vmmio.h
> something.

-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Ian Campbell]

On Fri, 2014-04-04 at 13:54 +0100, Julien Grall wrote:
> On 04/04/2014 01:42 PM, Ian Campbell wrote:
> > On Fri, 2014-04-04 at 18:00 +0530, Vijay Kilari wrote:
> >>>> +    handler->mmio_handlers[handler->num_entries++] = handle;
> >>>> +    spin_unlock_irqrestore(&handler_lock, flags);
> >>>> +}
> >>>>  /*
> >>>>   * Local variables:
> >>>>   * mode: C
> >>>> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
> >>>> index 8d252c0..5fc1660 100644
> >>>> --- a/xen/arch/arm/io.h
> >>>> +++ b/xen/arch/arm/io.h
> >>>
> >>> As you are modifying this header. Can you move it in
> >> include/asm-arm?
> >> OK.
> > 
> > Please don't mix movement and modifications. Unless you are adding uses
> > from outside xen/arch/arm requiring ugly relative includes there is no
> > need to move this IMHO.
> 
> My point was to clean up and having a single place for every headers.

Why? What your goal with this "clean up"?

If a header is only used by .c files in the same directory then it is
absolutely fine for the header to live in that directory and be included
with
	#include "foo.h"

Ian.

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

On 04/04/2014 01:59 PM, Ian Campbell wrote:
> On Fri, 2014-04-04 at 13:54 +0100, Julien Grall wrote:
>> On 04/04/2014 01:42 PM, Ian Campbell wrote:
>>> On Fri, 2014-04-04 at 18:00 +0530, Vijay Kilari wrote:
>>>>>> +    handler->mmio_handlers[handler->num_entries++] = handle;
>>>>>> +    spin_unlock_irqrestore(&handler_lock, flags);
>>>>>> +}
>>>>>>  /*
>>>>>>   * Local variables:
>>>>>>   * mode: C
>>>>>> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
>>>>>> index 8d252c0..5fc1660 100644
>>>>>> --- a/xen/arch/arm/io.h
>>>>>> +++ b/xen/arch/arm/io.h
>>>>>
>>>>> As you are modifying this header. Can you move it in
>>>> include/asm-arm?
>>>> OK.
>>>
>>> Please don't mix movement and modifications. Unless you are adding uses
>>> from outside xen/arch/arm requiring ugly relative includes there is no
>>> need to move this IMHO.
>>
>> My point was to clean up and having a single place for every headers.
> 
> Why? What your goal with this "clean up"?

His changes will basically rewrite 80% of the header. So it was a good
time to move it.

> If a header is only used by .c files in the same directory then it is
> absolutely fine for the header to live in that directory and be included
> with
> 	#include "foo.h"

We have many headers in include/asm-arm which are only used on arch/arm.

Anyway it was a suggestion... nothing else.

Regards,


-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

On 04/04/2014 01:30 PM, Vijay Kilari wrote:
> On Fri, Apr 4, 2014 at 5:48 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> Hello vijay,
>>
>> Thank your for the patch.
>>
>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>
>> [..]
>>
>>> -#define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
>>> +static DEFINE_SPINLOCK(handler_lock);
>>
>> Why a global lock rather than a domain lock?
>>
> OK can move into domain io_handler structure

>> [..]
>>
>>> +
>>> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle)
>>> +{
>>> +    unsigned long flags;
>>> +    struct io_handler *handler = d->arch.io_handlers;
>>> +    BUG_ON(handler->num_entries >= MAX_IO_HANDLER);
>>> +
>>> +    spin_lock_irqsave(&handler_lock, flags);
>>
>> You don't need to disable the IRQ here.
>   My first version is without IRQ disable. I remember Andrii suggested it

(Adding Andrii in the mail).

Unless you are planning to use it in interrupt context, IHMO irqsave is
not needed.

>>
>>> +    handler->mmio_handlers[handler->num_entries++] = handle;
>>> +    spin_unlock_irqrestore(&handler_lock, flags);
>>> +}
>>>  /*
>>>   * Local variables:
>>>   * mode: C
>>> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
>>> index 8d252c0..5fc1660 100644
>>> --- a/xen/arch/arm/io.h
>>> +++ b/xen/arch/arm/io.h
>>
>> As you are modifying this header. Can you move it in include/asm-arm?
> OK.
> 
>>
>>> @@ -40,10 +40,14 @@ struct mmio_handler {
>>>      mmio_write_t write_handler;
>>>  };
>>>
>>> -extern const struct mmio_handler vgic_distr_mmio_handler;
>>> -extern const struct mmio_handler vuart_mmio_handler;
>>> +#define MAX_IO_HANDLER  16
>>> +struct io_handler {
>>> +    int num_entries;
>>> +    struct mmio_handler *mmio_handlers[MAX_IO_HANDLER];
>>> +};
>>>
>>>  extern int handle_mmio(mmio_info_t *info);
>>> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle);
>>
>> As I said on the previous version, it would be nice to remove check
>> callback in the mmio_handler and replace it by addr/size. It's better if
>> we might want to change the place in the memory following the guest.
>>
>> So the result function would be:
>>
>> register_mmio_handler(struct domain *d, read_t read, write_t write,
>> addr, size);
>>
> OK.
>>>
>>>  #endif
>>>
>>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>>> index 8616534..77b561e 100644
>>> --- a/xen/arch/arm/vgic.c
>>> +++ b/xen/arch/arm/vgic.c
>>> @@ -35,6 +35,7 @@
>>>  /* Number of ranks of interrupt registers for a domain */
>>>  #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
>>>
>>> +static struct mmio_handler vgic_distr_mmio_handler;
>>>  /*
>>>   * Rank containing GICD_<FOO><n> for GICD_<FOO> with
>>>   * <b>-bits-per-interrupt
>>> @@ -107,6 +108,7 @@ int domain_vgic_init(struct domain *d)
>>>      }
>>>      for (i=0; i<DOMAIN_NR_RANKS(d); i++)
>>>          spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
>>> +    register_mmio_handler(d, &vgic_distr_mmio_handler);
>>>      return 0;
>>>  }
>>>
>>> @@ -673,7 +675,7 @@ static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
>>>      return (addr >= (d->arch.vgic.dbase)) && (addr < (d->arch.vgic.dbase + PAGE_SIZE));
>>>  }
>>>
>>> -const struct mmio_handler vgic_distr_mmio_handler = {
>>> +static struct mmio_handler vgic_distr_mmio_handler = {
>>
>> Why did you remove the const?
> 
> I think, compiler is allowing forward declaration with const

You meant doesn't allow? If so, please move the whole structure. I'd
prefer code movement rather than removing a const because of compiler issue.

> 
>>
>> [..]
>>
>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>>> index 50b9b54..23dac85 100644
>>> --- a/xen/include/asm-arm/domain.h
>>> +++ b/xen/include/asm-arm/domain.h
>>> @@ -116,6 +116,7 @@ struct arch_domain
>>>      struct hvm_domain hvm_domain;
>>>      xen_pfn_t *grant_table_gpfn;
>>>
>>> +    struct io_handler *io_handlers;
>>
>> Why do you need a pointer here? I think can can directly use
>> struct io_handler iohandlers.
>>
>   I just adds to increase the size of this arch_domain struct.
> so allocated memory at runtime.

Do you hit the page size?

Regards,

-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Vijay Kilari]

>>>
>>>>
>>>> -const struct mmio_handler vgic_distr_mmio_handler = {
>>>> +static struct mmio_handler vgic_distr_mmio_handler = {
>>>
>>> Why did you remove the const?
>>
>> I think, compiler is allowing forward declaration with const
>
> You meant doesn't allow? If so, please move the whole structure. I'd
> prefer code movement rather than removing a const because of compiler issue.
>
Yes, compiler does not allow. I will move the code
>>
>>>
>>> [..]
>>>
>>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>>>> index 50b9b54..23dac85 100644
>>>> --- a/xen/include/asm-arm/domain.h
>>>> +++ b/xen/include/asm-arm/domain.h
>>>> @@ -116,6 +116,7 @@ struct arch_domain
>>>>      struct hvm_domain hvm_domain;
>>>>      xen_pfn_t *grant_table_gpfn;
>>>>
>>>> +    struct io_handler *io_handlers;
>>>
>>> Why do you need a pointer here? I think can can directly use
>>> struct io_handler iohandlers.
>>>
>>   I just adds to increase the size of this arch_domain struct.
>> so allocated memory at runtime.
>
> Do you hit the page size?
Yes, I have faced page size exhausted issue which I have reported earlier

>
> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

On 04/08/2014 05:47 AM, Vijay Kilari wrote:
>>>>
>>>>>
>>>>> -const struct mmio_handler vgic_distr_mmio_handler = {
>>>>> +static struct mmio_handler vgic_distr_mmio_handler = {
>>>>
>>>> Why did you remove the const?
>>>
>>> I think, compiler is allowing forward declaration with const
>>
>> You meant doesn't allow? If so, please move the whole structure. I'd
>> prefer code movement rather than removing a const because of compiler issue.
>>
> Yes, compiler does not allow. I will move the code

This is from the specification... if you declare something const you
have to assign the value directly.

>>>
>>>>
>>>> [..]
>>>>
>>>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>>>>> index 50b9b54..23dac85 100644
>>>>> --- a/xen/include/asm-arm/domain.h
>>>>> +++ b/xen/include/asm-arm/domain.h
>>>>> @@ -116,6 +116,7 @@ struct arch_domain
>>>>>      struct hvm_domain hvm_domain;
>>>>>      xen_pfn_t *grant_table_gpfn;
>>>>>
>>>>> +    struct io_handler *io_handlers;
>>>>
>>>> Why do you need a pointer here? I think can can directly use
>>>> struct io_handler iohandlers.
>>>>
>>>   I just adds to increase the size of this arch_domain struct.
>>> so allocated memory at runtime.
>>
>> Do you hit the page size?
> Yes, I have faced page size exhausted issue which I have reported earlier

Did you still have the issue when you moved out GICv stuff outside?

Regards,

-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Vijay Kilari]

On Tue, Apr 8, 2014 at 3:47 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 04/08/2014 05:47 AM, Vijay Kilari wrote:
>>>>>
>>>>>>
>>>>>> -const struct mmio_handler vgic_distr_mmio_handler = {
>>>>>> +static struct mmio_handler vgic_distr_mmio_handler = {
>>>>>
>>>>> Why did you remove the const?
>>>>
>>>> I think, compiler is allowing forward declaration with const
>>>
>>> You meant doesn't allow? If so, please move the whole structure. I'd
>>> prefer code movement rather than removing a const because of compiler issue.
>>>
>> Yes, compiler does not allow. I will move the code
>
> This is from the specification... if you declare something const you
> have to assign the value directly.
>
Also we cannot keep it as const for the reason that
register_mmio_handler fails to
assign this const pointer to handler list.

Anyway we will get rid of this structure completely after implementing
check handler where read & write handler are passed as parameter to
register_mmio_handler()
call.

>>>>
>>>>>
>>>>> [..]
>>>>>
>>>>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>>>>>> index 50b9b54..23dac85 100644
>>>>>> --- a/xen/include/asm-arm/domain.h
>>>>>> +++ b/xen/include/asm-arm/domain.h
>>>>>> @@ -116,6 +116,7 @@ struct arch_domain
>>>>>>      struct hvm_domain hvm_domain;
>>>>>>      xen_pfn_t *grant_table_gpfn;
>>>>>>
>>>>>> +    struct io_handler *io_handlers;
>>>>>
>>>>> Why do you need a pointer here? I think can can directly use
>>>>> struct io_handler iohandlers.
>>>>>
>>>>   I just adds to increase the size of this arch_domain struct.
>>>> so allocated memory at runtime.
>>>
>>> Do you hit the page size?
>> Yes, I have faced page size exhausted issue which I have reported earlier
>
> Did you still have the issue when you moved out GICv stuff outside?
Now I am not getting this issue. But I feel we are on the edge. So it
is always better
to allocate runtime.

>
> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

On 04/08/2014 11:34 AM, Vijay Kilari wrote:
>>>>>
>>>>>>
>>>>>> [..]
>>>>>>
>>>>>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>>>>>>> index 50b9b54..23dac85 100644
>>>>>>> --- a/xen/include/asm-arm/domain.h
>>>>>>> +++ b/xen/include/asm-arm/domain.h
>>>>>>> @@ -116,6 +116,7 @@ struct arch_domain
>>>>>>>      struct hvm_domain hvm_domain;
>>>>>>>      xen_pfn_t *grant_table_gpfn;
>>>>>>>
>>>>>>> +    struct io_handler *io_handlers;
>>>>>>
>>>>>> Why do you need a pointer here? I think can can directly use
>>>>>> struct io_handler iohandlers.
>>>>>>
>>>>>   I just adds to increase the size of this arch_domain struct.
>>>>> so allocated memory at runtime.
>>>>
>>>> Do you hit the page size?
>>> Yes, I have faced page size exhausted issue which I have reported earlier
>>
>> Did you still have the issue when you moved out GICv stuff outside?
> Now I am not getting this issue. But I feel we are on the edge. So it
> is always better
> to allocate runtime.

Hmmmm... we are not on the edge with the current Xen:
   - arm32: sizeof(struct domain) = 1024
   - arm64: sizeof(struct domain) = 1408

Unless if you have added lots of fields in the structure, there is no
reason to use a pointer here.

Regards,

-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Vijay Kilari]

On Tue, Apr 8, 2014 at 4:21 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 04/08/2014 11:34 AM, Vijay Kilari wrote:
>>>>>>
>>>>>>>
>>>>>>> [..]
>>>>>>>
>>>>>>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>>>>>>>> index 50b9b54..23dac85 100644
>>>>>>>> --- a/xen/include/asm-arm/domain.h
>>>>>>>> +++ b/xen/include/asm-arm/domain.h
>>>>>>>> @@ -116,6 +116,7 @@ struct arch_domain
>>>>>>>>      struct hvm_domain hvm_domain;
>>>>>>>>      xen_pfn_t *grant_table_gpfn;
>>>>>>>>
>>>>>>>> +    struct io_handler *io_handlers;
>>>>>>>
>>>>>>> Why do you need a pointer here? I think can can directly use
>>>>>>> struct io_handler iohandlers.
>>>>>>>
>>>>>>   I just adds to increase the size of this arch_domain struct.
>>>>>> so allocated memory at runtime.
>>>>>
>>>>> Do you hit the page size?
>>>> Yes, I have faced page size exhausted issue which I have reported earlier
>>>
>>> Did you still have the issue when you moved out GICv stuff outside?
>> Now I am not getting this issue. But I feel we are on the edge. So it
>> is always better
>> to allocate runtime.
>
> Hmmmm... we are not on the edge with the current Xen:
>    - arm32: sizeof(struct domain) = 1024
>    - arm64: sizeof(struct domain) = 1408
>
> Unless if you have added lots of fields in the structure, there is no
> reason to use a pointer here.
>
OK, I can remove this pointer.  When I make static allocation as below

struct io_handler iohandlers;

Now all the files that includes domain.h now expects io.h defined
in arch/arm to include this file for io_handler definition.
So I plan to move io.h to include directory under different name as
pio.h file or device_io.h?

> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Ian Campbell]

On Tue, 2014-04-08 at 17:11 +0530, Vijay Kilari wrote:
> On Tue, Apr 8, 2014 at 4:21 PM, Julien Grall <julien.grall@linaro.org> wrote:
> > On 04/08/2014 11:34 AM, Vijay Kilari wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> [..]
> >>>>>>>
> >>>>>>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> >>>>>>>> index 50b9b54..23dac85 100644
> >>>>>>>> --- a/xen/include/asm-arm/domain.h
> >>>>>>>> +++ b/xen/include/asm-arm/domain.h
> >>>>>>>> @@ -116,6 +116,7 @@ struct arch_domain
> >>>>>>>>      struct hvm_domain hvm_domain;
> >>>>>>>>      xen_pfn_t *grant_table_gpfn;
> >>>>>>>>
> >>>>>>>> +    struct io_handler *io_handlers;
> >>>>>>>
> >>>>>>> Why do you need a pointer here? I think can can directly use
> >>>>>>> struct io_handler iohandlers.
> >>>>>>>
> >>>>>>   I just adds to increase the size of this arch_domain struct.
> >>>>>> so allocated memory at runtime.
> >>>>>
> >>>>> Do you hit the page size?
> >>>> Yes, I have faced page size exhausted issue which I have reported earlier
> >>>
> >>> Did you still have the issue when you moved out GICv stuff outside?
> >> Now I am not getting this issue. But I feel we are on the edge. So it
> >> is always better
> >> to allocate runtime.
> >
> > Hmmmm... we are not on the edge with the current Xen:
> >    - arm32: sizeof(struct domain) = 1024
> >    - arm64: sizeof(struct domain) = 1408
> >
> > Unless if you have added lots of fields in the structure, there is no
> > reason to use a pointer here.
> >
> OK, I can remove this pointer.  When I make static allocation as below
> 
> struct io_handler iohandlers;
> 
> Now all the files that includes domain.h now expects io.h defined
> in arch/arm to include this file for io_handler definition.
> So I plan to move io.h to include directory under different name as
> pio.h file or device_io.h?

mmio.h I think fits the bill.

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Vijay Kilari]

 Hi Julien,

On Fri, Apr 4, 2014 at 6:00 PM, Vijay Kilari <vijay.kilari@gmail.com> wrote:
> On Fri, Apr 4, 2014 at 5:48 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> Hello vijay,
>>
>> Thank your for the patch.
>>
>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>
>> [..]
>>
>>> -#define MMIO_HANDLER_NR ARRAY_SIZE(mmio_handlers)
>>> +static DEFINE_SPINLOCK(handler_lock);
>>
>> Why a global lock rather than a domain lock?
>>
> OK can move into domain io_handler structure
>
>> [..]
>>
>>> +
>>> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle)
>>> +{
>>> +    unsigned long flags;
>>> +    struct io_handler *handler = d->arch.io_handlers;
>>> +    BUG_ON(handler->num_entries >= MAX_IO_HANDLER);
>>> +
>>> +    spin_lock_irqsave(&handler_lock, flags);
>>
>> You don't need to disable the IRQ here.
>   My first version is without IRQ disable. I remember Andrii suggested it
>>
>>> +    handler->mmio_handlers[handler->num_entries++] = handle;
>>> +    spin_unlock_irqrestore(&handler_lock, flags);
>>> +}
>>>  /*
>>>   * Local variables:
>>>   * mode: C
>>> diff --git a/xen/arch/arm/io.h b/xen/arch/arm/io.h
>>> index 8d252c0..5fc1660 100644
>>> --- a/xen/arch/arm/io.h
>>> +++ b/xen/arch/arm/io.h
>>
>> As you are modifying this header. Can you move it in include/asm-arm?
> OK.
>
>>
>>> @@ -40,10 +40,14 @@ struct mmio_handler {
>>>      mmio_write_t write_handler;
>>>  };
>>>
>>> -extern const struct mmio_handler vgic_distr_mmio_handler;
>>> -extern const struct mmio_handler vuart_mmio_handler;
>>> +#define MAX_IO_HANDLER  16
>>> +struct io_handler {
>>> +    int num_entries;
>>> +    struct mmio_handler *mmio_handlers[MAX_IO_HANDLER];
>>> +};
>>>
>>>  extern int handle_mmio(mmio_info_t *info);
>>> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle);
>>
>> As I said on the previous version, it would be nice to remove check
>> callback in the mmio_handler and replace it by addr/size. It's better if
>> we might want to change the place in the memory following the guest.
>>
>> So the result function would be:
>>
>> register_mmio_handler(struct domain *d, read_t read, write_t write,
>> addr, size);
>>
> OK.
   Though you suggested, I kept this check handler because it gives flexibility
for driver to make additional checks if required apart from just checking for
address range.

>>>
>>>  #endif

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

On 04/04/2014 04:24 PM, Vijay Kilari wrote:

>>> As I said on the previous version, it would be nice to remove check
>>> callback in the mmio_handler and replace it by addr/size. It's better if
>>> we might want to change the place in the memory following the guest.
>>>
>>> So the result function would be:
>>>
>>> register_mmio_handler(struct domain *d, read_t read, write_t write,
>>> addr, size);
>>>
>> OK.
>    Though you suggested, I kept this check handler because it gives flexibility
> for driver to make additional checks if required apart from just checking for
> address range.

What kind of checks? If the handler is registered to a domain that means
we want to use it, right? It would seem strange to trap once a while.

Regards,

-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Vijay Kilari]

On Fri, Apr 4, 2014 at 8:57 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 04/04/2014 04:24 PM, Vijay Kilari wrote:
>
>>>> As I said on the previous version, it would be nice to remove check
>>>> callback in the mmio_handler and replace it by addr/size. It's better if
>>>> we might want to change the place in the memory following the guest.
>>>>
>>>> So the result function would be:
>>>>
>>>> register_mmio_handler(struct domain *d, read_t read, write_t write,
>>>> addr, size);
>>>>
>>> OK.
>>    Though you suggested, I kept this check handler because it gives flexibility
>> for driver to make additional checks if required apart from just checking for
>> address range.
>
> What kind of checks? If the handler is registered to a domain that means
> we want to use it, right? It would seem strange to trap once a while.

   IMO, with check handler it will give more flexibility for driver to
various kinds of checks.
Ex: Driver wants to check for multiple addr ranges etc.,
>
> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Julien Grall]

On 04/08/2014 07:35 AM, Vijay Kilari wrote:
> On Fri, Apr 4, 2014 at 8:57 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> On 04/04/2014 04:24 PM, Vijay Kilari wrote:
>>
>>>>> As I said on the previous version, it would be nice to remove check
>>>>> callback in the mmio_handler and replace it by addr/size. It's better if
>>>>> we might want to change the place in the memory following the guest.
>>>>>
>>>>> So the result function would be:
>>>>>
>>>>> register_mmio_handler(struct domain *d, read_t read, write_t write,
>>>>> addr, size);
>>>>>
>>>> OK.
>>>    Though you suggested, I kept this check handler because it gives flexibility
>>> for driver to make additional checks if required apart from just checking for
>>> address range.
>>
>> What kind of checks? If the handler is registered to a domain that means
>> we want to use it, right? It would seem strange to trap once a while.
> 
>    IMO, with check handler it will give more flexibility for driver to
> various kinds of checks.
> Ex: Driver wants to check for multiple addr ranges etc.,

>From driver POV it's easier to have one range by set of callbacks rather
than multiple range.
Because you will end up to check in 2 place the address (once in check
and once in read/write callback).

You don't use this feature in your code and I don't think it will be
use, so I would prefer to get a rid of check and let handle_mmio do the
job for us.

Regards,

-- 
Julien Grall

Re: [PATCH v2 01/15] xen/arm: register mmio handler at runtime

[Ian Campbell]

On Fri, 2014-04-04 at 18:00 +0530, Vijay Kilari wrote:

> >> +void register_mmio_handler(struct domain *d, struct mmio_handler * handle);
> >
> > As I said on the previous version, it would be nice to remove check
> > callback in the mmio_handler and replace it by addr/size. It's better if
> > we might want to change the place in the memory following the guest.
> >
> > So the result function would be:
> >
> > register_mmio_handler(struct domain *d, read_t read, write_t write,
> > addr, size);
> >
> OK.

Regardless of addr+size vs check() I don't think it is necessary to
explode struct mmio_handler into separate arguments here, passing a
pointer to the static struct in each driver seems much nicer.

Ian.

[PATCH v2 02/15] xen/arm: move vgic rank data to gic header file

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

vgic_irq_rank structure contains gic specific
data elements. Move this out of domain.h and
allocate memory dynamically in vgic driver.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/domain.c        |    1 +
 xen/arch/arm/smp.c           |    1 +
 xen/arch/arm/vgic.c          |   17 ++++++++++++++---
 xen/include/asm-arm/domain.h |   13 +++----------
 xen/include/asm-arm/gic.h    |   10 ++++++++++
 5 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index e2f0cc4..3d7e685 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -495,6 +495,7 @@ int vcpu_initialise(struct vcpu *v)
 void vcpu_destroy(struct vcpu *v)
 {
     vcpu_timer_destroy(v);
+    vcpu_vgic_free(v);
     free_xenheap_pages(v->arch.stack, STACK_ORDER);
 }
 
diff --git a/xen/arch/arm/smp.c b/xen/arch/arm/smp.c
index 30203b8..3cfd6ac 100644
--- a/xen/arch/arm/smp.c
+++ b/xen/arch/arm/smp.c
@@ -3,6 +3,7 @@
 #include <asm/smp.h>
 #include <asm/cpregs.h>
 #include <asm/page.h>
+#include <xen/irq.h>
 #include <asm/gic.h>
 #include <asm/flushtlb.h>
 
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 77b561e..d86bede 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -67,7 +67,7 @@ static struct vgic_irq_rank *vgic_irq_rank(struct vcpu *v, int b, int n)
     int rank = REG_RANK_NR(b, n);
 
     if ( rank == 0 )
-        return &v->arch.vgic.private_irqs;
+        return v->arch.vgic.private_irqs;
     else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
         return &v->domain->arch.vgic.shared_irqs[rank - 1];
     else
@@ -121,9 +121,14 @@ void domain_vgic_free(struct domain *d)
 int vcpu_vgic_init(struct vcpu *v)
 {
     int i;
+
+    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
+    if ( v->arch.vgic.private_irqs == NULL )
+      return -ENOMEM;
+
     memset(&v->arch.vgic.private_irqs, 0, sizeof(v->arch.vgic.private_irqs));
 
-    spin_lock_init(&v->arch.vgic.private_irqs.lock);
+    spin_lock_init(&v->arch.vgic.private_irqs->lock);
 
     memset(&v->arch.vgic.pending_irqs, 0, sizeof(v->arch.vgic.pending_irqs));
     for (i = 0; i < 32; i++)
@@ -134,7 +139,7 @@ int vcpu_vgic_init(struct vcpu *v)
 
     /* For SGI and PPI the target is always this CPU */
     for ( i = 0 ; i < 8 ; i++ )
-        v->arch.vgic.private_irqs.itargets[i] =
+        v->arch.vgic.private_irqs->itargets[i] =
               (1<<(v->vcpu_id+0))
             | (1<<(v->vcpu_id+8))
             | (1<<(v->vcpu_id+16))
@@ -146,6 +151,12 @@ int vcpu_vgic_init(struct vcpu *v)
     return 0;
 }
 
+int vcpu_vgic_free(struct vcpu *v)
+{
+   xfree(v->arch.vgic.private_irqs);
+   return 0;
+}
+
 #define vgic_lock(v)   spin_lock_irq(&(v)->domain->arch.vgic.lock)
 #define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
 
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 23dac85..5b23d07 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -9,15 +9,8 @@
 #include <asm/vfp.h>
 #include <public/hvm/params.h>
 #include <xen/serial.h>
-
-/* Represents state corresponding to a block of 32 interrupts */
-struct vgic_irq_rank {
-    spinlock_t lock; /* Covers access to all other members of this struct */
-    uint32_t ienable, iactive, ipend, pendsgi;
-    uint32_t icfg[2];
-    uint32_t ipriority[8];
-    uint32_t itargets[8];
-};
+#include <xen/irq.h>
+#include <asm/gic.h>
 
 struct pending_irq
 {
@@ -270,7 +263,7 @@ struct arch_vcpu
          * struct arch_domain.
          */
         struct pending_irq pending_irqs[32];
-        struct vgic_irq_rank private_irqs;
+        struct vgic_irq_rank *private_irqs;
 
         /* This list is ordered by IRQ priority and it is used to keep
          * track of the IRQs that the VGIC injected into the guest.
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 5d8f7f1..dd7e891 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -160,10 +160,20 @@
 #define DT_MATCH_GIC    DT_MATCH_COMPATIBLE("arm,cortex-a15-gic"), \
                         DT_MATCH_COMPATIBLE("arm,cortex-a7-gic")
 
+/* Represents state corresponding to a block of 32 interrupts */
+struct vgic_irq_rank {
+    spinlock_t lock; /* Covers access to all other members of this struct */
+    uint32_t ienable, iactive, ipend, pendsgi;
+    uint32_t icfg[2];
+    uint32_t ipriority[8];
+    uint32_t itargets[8];
+};
+
 extern int domain_vgic_init(struct domain *d);
 extern void domain_vgic_free(struct domain *d);
 
 extern int vcpu_vgic_init(struct vcpu *v);
+extern int vcpu_vgic_free(struct vcpu *v);
 
 extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
-- 
1.7.9.5

Re: [PATCH v2 02/15] xen/arm: move vgic rank data to gic header file

[Julien Grall]

Hello Vijay,

Thank you for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> vgic_irq_rank structure contains gic specific
> data elements. Move this out of domain.h and
> allocate memory dynamically in vgic driver.

I guess it's because you hit the maximum size of vgic structure, right?
If so can you specify it in the commit message?

[..]

> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 77b561e..d86bede 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -67,7 +67,7 @@ static struct vgic_irq_rank *vgic_irq_rank(struct vcpu *v, int b, int n)
>      int rank = REG_RANK_NR(b, n);
>  
>      if ( rank == 0 )
> -        return &v->arch.vgic.private_irqs;
> +        return v->arch.vgic.private_irqs;
>      else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
>          return &v->domain->arch.vgic.shared_irqs[rank - 1];
>      else
> @@ -121,9 +121,14 @@ void domain_vgic_free(struct domain *d)
>  int vcpu_vgic_init(struct vcpu *v)
>  {
>      int i;
> +
> +    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
> +    if ( v->arch.vgic.private_irqs == NULL )
> +      return -ENOMEM;
> +
>      memset(&v->arch.vgic.private_irqs, 0, sizeof(v->arch.vgic.private_irqs));

You forgot to remove & in front of v->arch.vgic.private_irqs.

Regards,

-- 
Julien Grall

Re: [PATCH v2 02/15] xen/arm: move vgic rank data to gic header file

[Vijay Kilari]

On Fri, Apr 4, 2014 at 6:46 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hello Vijay,
>
> Thank you for the patch.
>
> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>
>> vgic_irq_rank structure contains gic specific
>> data elements. Move this out of domain.h and
>> allocate memory dynamically in vgic driver.
>
> I guess it's because you hit the maximum size of vgic structure, right?
> If so can you specify it in the commit message?
>
   That was one reason. Also it has GIC hw version specific variables.
So I moved it. I will take care in next revision

> [..]
>
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 77b561e..d86bede 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -67,7 +67,7 @@ static struct vgic_irq_rank *vgic_irq_rank(struct vcpu *v, int b, int n)
>>      int rank = REG_RANK_NR(b, n);
>>
>>      if ( rank == 0 )
>> -        return &v->arch.vgic.private_irqs;
>> +        return v->arch.vgic.private_irqs;
>>      else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
>>          return &v->domain->arch.vgic.shared_irqs[rank - 1];
>>      else
>> @@ -121,9 +121,14 @@ void domain_vgic_free(struct domain *d)
>>  int vcpu_vgic_init(struct vcpu *v)
>>  {
>>      int i;
>> +
>> +    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
>> +    if ( v->arch.vgic.private_irqs == NULL )
>> +      return -ENOMEM;
>> +
>>      memset(&v->arch.vgic.private_irqs, 0, sizeof(v->arch.vgic.private_irqs));
>
> You forgot to remove & in front of v->arch.vgic.private_irqs.
OK

> Regards,
>
> --
> Julien Grall

[PATCH v2 03/15] arm/xen: move gic save and restore registers to gic driver

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

gic saved registers are moved to gic driver.
This required structure is allocated at runtime
and is saved & restored.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/domain.c        |    3 +++
 xen/arch/arm/gic.c           |   26 +++++++++++++++++---------
 xen/include/asm-arm/domain.h |    3 +--
 xen/include/asm-arm/gic.h    |    8 ++++++++
 4 files changed, 29 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 3d7e685..701298a 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -467,6 +467,9 @@ int vcpu_initialise(struct vcpu *v)
     v->arch.saved_context.sp = (register_t)v->arch.cpu_info;
     v->arch.saved_context.pc = (register_t)continue_new_vcpu;
 
+    if ( (rc = vcpu_gic_init(v)) != 0 )
+        return rc;
+
     /* Idle VCPUs don't need the rest of this setup */
     if ( is_idle_vcpu(v) )
         return rc;
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 86c52ac..a59118f 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -106,11 +106,11 @@ void gic_save_state(struct vcpu *v)
      * this call and it only accesses struct vcpu fields that cannot be
      * accessed simultaneously by another pCPU.
      */
-    for ( i=0; i<nr_lrs; i++)
-        v->arch.gic_lr[i] = GICH[GICH_LR + i];
+    for ( i = 0; i < nr_lrs; i++ )
+        v->arch.gic_state->gic_lr[i] = GICH[GICH_LR + i];
     v->arch.lr_mask = this_cpu(lr_mask);
-    v->arch.gic_apr = GICH[GICH_APR];
-    v->arch.gic_vmcr = GICH[GICH_VMCR];
+    v->arch.gic_state->gic_apr = GICH[GICH_APR];
+    v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];
     /* Disable until next VCPU scheduled */
     GICH[GICH_HCR] = 0;
     isb();
@@ -125,10 +125,10 @@ void gic_restore_state(struct vcpu *v)
         return;
 
     this_cpu(lr_mask) = v->arch.lr_mask;
-    for ( i=0; i<nr_lrs; i++)
-        GICH[GICH_LR + i] = v->arch.gic_lr[i];
-    GICH[GICH_APR] = v->arch.gic_apr;
-    GICH[GICH_VMCR] = v->arch.gic_vmcr;
+    for ( i = 0; i < nr_lrs; i++ )
+        GICH[GICH_LR + i] = v->arch.gic_state->gic_lr[i];
+    GICH[GICH_APR] = v->arch.gic_state->gic_apr;
+    GICH[GICH_VMCR] = v->arch.gic_state->gic_vmcr;
     GICH[GICH_HCR] = GICH_HCR_EN;
     isb();
 
@@ -975,6 +975,14 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
     } while (1);
 }
 
+int vcpu_gic_init(struct vcpu *v)
+{
+     v->arch.gic_state = xzalloc(struct gic_state_data);
+     if(!v->arch.gic_state)
+        return -ENOMEM;
+     return 0;
+}
+
 int gicv_setup(struct domain *d)
 {
     int ret;
@@ -1044,7 +1052,7 @@ void gic_dump_info(struct vcpu *v)
             printk("   HW_LR[%d]=%x\n", i, GICH[GICH_LR + i]);
     } else {
         for ( i = 0; i < nr_lrs; i++ )
-            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_lr[i]);
+            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->gic_lr[i]);
     }
 
     list_for_each_entry ( p, &v->arch.vgic.inflight_irqs, inflight )
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 5b23d07..c51b3cb 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -252,8 +252,7 @@ struct arch_vcpu
     uint32_t csselr;
     register_t vmpidr;
 
-    uint32_t gic_hcr, gic_vmcr, gic_apr;
-    uint32_t gic_lr[64];
+    struct gic_state_data *gic_state;
     uint64_t event_mask;
     uint64_t lr_mask;
 
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index dd7e891..8a28c4a 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -169,12 +169,20 @@ struct vgic_irq_rank {
     uint32_t itargets[8];
 };
 
+struct gic_state_data {
+    uint32_t gic_hcr, gic_vmcr;
+    uint32_t gic_apr;
+    uint32_t gic_lr[64];
+};
+
 extern int domain_vgic_init(struct domain *d);
 extern void domain_vgic_free(struct domain *d);
 
 extern int vcpu_vgic_init(struct vcpu *v);
 extern int vcpu_vgic_free(struct vcpu *v);
 
+extern int vcpu_gic_init(struct vcpu *v);
+
 extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
 extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
-- 
1.7.9.5

Re: [PATCH v2 03/15] arm/xen: move gic save and restore registers to gic driver

[Julien Grall]

Hello Vijay,

Thank your for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> gic saved registers are moved to gic driver.
> This required structure is allocated at runtime
> and is saved & restored.
> 
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> ---
>  xen/arch/arm/domain.c        |    3 +++
>  xen/arch/arm/gic.c           |   26 +++++++++++++++++---------
>  xen/include/asm-arm/domain.h |    3 +--
>  xen/include/asm-arm/gic.h    |    8 ++++++++
>  4 files changed, 29 insertions(+), 11 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 3d7e685..701298a 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -467,6 +467,9 @@ int vcpu_initialise(struct vcpu *v)
>      v->arch.saved_context.sp = (register_t)v->arch.cpu_info;
>      v->arch.saved_context.pc = (register_t)continue_new_vcpu;
>  
> +    if ( (rc = vcpu_gic_init(v)) != 0 )
> +        return rc;
> +

You don't need to allocate it for idle VCPU. Can you move it after the
if ( is_idle_vcpu(v)) ?

>      /* Idle VCPUs don't need the rest of this setup */
>      if ( is_idle_vcpu(v) )
>          return rc;

Somewhere here.

> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 86c52ac..a59118f 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -106,11 +106,11 @@ void gic_save_state(struct vcpu *v)
>       * this call and it only accesses struct vcpu fields that cannot be
>       * accessed simultaneously by another pCPU.
>       */


I'm surprised that we save theses registers for idle VCPU but not
restored...

With my request, you will have to add an if ( is_idle_vcpu(v) ) return
in gic_save_state.

> -    for ( i=0; i<nr_lrs; i++)
> -        v->arch.gic_lr[i] = GICH[GICH_LR + i];
> +    for ( i = 0; i < nr_lrs; i++ )
> +        v->arch.gic_state->gic_lr[i] = GICH[GICH_LR + i];
>      v->arch.lr_mask = this_cpu(lr_mask);
> -    v->arch.gic_apr = GICH[GICH_APR];
> -    v->arch.gic_vmcr = GICH[GICH_VMCR];
> +    v->arch.gic_state->gic_apr = GICH[GICH_APR];
> +    v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];
>      /* Disable until next VCPU scheduled */
>      GICH[GICH_HCR] = 0;
>      isb();
> @@ -125,10 +125,10 @@ void gic_restore_state(struct vcpu *v)
>          return;
>  
>      this_cpu(lr_mask) = v->arch.lr_mask;
> -    for ( i=0; i<nr_lrs; i++)
> -        GICH[GICH_LR + i] = v->arch.gic_lr[i];
> -    GICH[GICH_APR] = v->arch.gic_apr;
> -    GICH[GICH_VMCR] = v->arch.gic_vmcr;
> +    for ( i = 0; i < nr_lrs; i++ )
> +        GICH[GICH_LR + i] = v->arch.gic_state->gic_lr[i];
> +    GICH[GICH_APR] = v->arch.gic_state->gic_apr;
> +    GICH[GICH_VMCR] = v->arch.gic_state->gic_vmcr;
>      GICH[GICH_HCR] = GICH_HCR_EN;
>      isb();
>  
> @@ -975,6 +975,14 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
>      } while (1);
>  }
>  
> +int vcpu_gic_init(struct vcpu *v)
> +{
> +     v->arch.gic_state = xzalloc(struct gic_state_data);
> +     if(!v->arch.gic_state)

if ( !v->arch.gic_state )

Regards,

-- 
Julien Grall

Re: [PATCH v2 03/15] arm/xen: move gic save and restore registers to gic driver

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:

> gic saved registers are moved to gic driver.
> This required structure is allocated at runtime
> and is saved & restored.

I don't follow why this change required switch to allocating them
dynamically. If it is a v2 vs v3 thing then I think a union in struct
arch_vcpu would be fine, unless the v3 stuff is relatively enormous.

> -    v->arch.gic_apr = GICH[GICH_APR];
> -    v->arch.gic_vmcr = GICH[GICH_VMCR];
> +    v->arch.gic_state->gic_apr = GICH[GICH_APR];
> +    v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];

I think the gic on the field name is now redundant, at I reckon you can
drop _state too.

IOW v->arch.gic->apr, v->arch.gic->vmcr, etc.

(or v->arch.gic.v2.apr / v->arch.gic_v2.apr based on the first comment)

Ian.

Re: [PATCH v2 03/15] arm/xen: move gic save and restore registers to gic driver

[Vijay Kilari]

On Wed, Apr 9, 2014 at 10:21 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>
>> gic saved registers are moved to gic driver.
>> This required structure is allocated at runtime
>> and is saved & restored.
>
> I don't follow why this change required switch to allocating them
> dynamically. If it is a v2 vs v3 thing then I think a union in struct
> arch_vcpu would be fine, unless the v3 stuff is relatively enormous.
>
v2:
struct gic_state_data {
    uint32_t gic_hcr, gic_vmcr;
    uint32_t gic_apr;
    uint32_t gic_lr[64];
};

v3:
 struct gic_state_data {
     uint32_t gic_hcr, gic_vmcr;
     uint32_t gic_apr0[4];
      uint32_t gic_apr1[4];
      uint64_t gic_lr[16];
  };

Except hcr & vmcr registers, v2 & v3 are different.
Does static allocation helps in caching the domain struct and
is this the reason for not choosing dynamic allocation?
If so, I would suggest to move this structure to gic.h without
dynamic allocation?

>> -    v->arch.gic_apr = GICH[GICH_APR];
>> -    v->arch.gic_vmcr = GICH[GICH_VMCR];
>> +    v->arch.gic_state->gic_apr = GICH[GICH_APR];
>> +    v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];
>
> I think the gic on the field name is now redundant, at I reckon you can
> drop _state too.
>
> IOW v->arch.gic->apr, v->arch.gic->vmcr, etc.
>
> (or v->arch.gic.v2.apr / v->arch.gic_v2.apr based on the first comment)
>
> Ian.
>

Re: [PATCH v2 03/15] arm/xen: move gic save and restore registers to gic driver

[Ian Campbell]

On Thu, 2014-04-10 at 10:20 +0530, Vijay Kilari wrote:
> On Wed, Apr 9, 2014 at 10:21 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> >
> >> gic saved registers are moved to gic driver.
> >> This required structure is allocated at runtime
> >> and is saved & restored.
> >
> > I don't follow why this change required switch to allocating them
> > dynamically. If it is a v2 vs v3 thing then I think a union in struct
> > arch_vcpu would be fine, unless the v3 stuff is relatively enormous.
> >
> v2:
> struct gic_state_data {
>     uint32_t gic_hcr, gic_vmcr;
>     uint32_t gic_apr;
>     uint32_t gic_lr[64];
> };
> 
> v3:
>  struct gic_state_data {
>      uint32_t gic_hcr, gic_vmcr;
>      uint32_t gic_apr0[4];
>       uint32_t gic_apr1[4];
>       uint64_t gic_lr[16];
>   };
> 
> Except hcr & vmcr registers, v2 & v3 are different.

looks like it is 268 bytes for v2 and 168 for v3 (surprised it gets
smaller! Fewer LRs I suppose)

> Does static allocation helps in caching the domain struct and
> is this the reason for not choosing dynamic allocation?

Right plus it avoids a pointer indirection. Maybe that's not so
important, but it also reduces code complexity a bit and we can defer
that complexity until the size of the vcpu struct gets too big.

> If so, I would suggest to move this structure to gic.h without
> dynamic allocation?

Shouldn't they be in gic_v2_defs.h and gic_v3_defs.h respectively, adn
with a distinct name? Then in struct arch_vcpu:
	union {
		struct gic_v2_state v2;
		struct gic_v3_state v3;
	} gic;

[PATCH v2 04/15] xen/arm: move gic definitions to seperate file

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Move gic v2 register definitions to separate file
so that gic.h will hold only common definitions
and helps to define gic v3 definitions.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/gic.c                |    1 +
 xen/arch/arm/vgic.c               |    1 +
 xen/include/asm-arm/gic.h         |  114 ------------------------------
 xen/include/asm-arm/gic_v2_defs.h |  139 +++++++++++++++++++++++++++++++++++++
 4 files changed, 141 insertions(+), 114 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index a59118f..64699e4 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -32,6 +32,7 @@
 #include <asm/domain.h>
 #include <asm/platform.h>
 
+#include <asm/gic_v2_defs.h>
 #include <asm/gic.h>
 
 /* Access to the GIC Distributor registers through the fixmap */
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index d86bede..a98da82 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -28,6 +28,7 @@
 #include <asm/current.h>
 
 #include "io.h"
+#include <asm/gic_v2_defs.h>
 #include <asm/gic.h>
 
 #define REG(n) (n/4)
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 8a28c4a..5f49eb1 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -18,120 +18,6 @@
 #ifndef __ASM_ARM_GIC_H__
 #define __ASM_ARM_GIC_H__
 
-#define GICD_CTLR       (0x000/4)
-#define GICD_TYPER      (0x004/4)
-#define GICD_IIDR       (0x008/4)
-#define GICD_IGROUPR    (0x080/4)
-#define GICD_IGROUPRN   (0x0FC/4)
-#define GICD_ISENABLER  (0x100/4)
-#define GICD_ISENABLERN (0x17C/4)
-#define GICD_ICENABLER  (0x180/4)
-#define GICD_ICENABLERN (0x1fC/4)
-#define GICD_ISPENDR    (0x200/4)
-#define GICD_ISPENDRN   (0x27C/4)
-#define GICD_ICPENDR    (0x280/4)
-#define GICD_ICPENDRN   (0x2FC/4)
-#define GICD_ISACTIVER  (0x300/4)
-#define GICD_ISACTIVERN (0x37C/4)
-#define GICD_ICACTIVER  (0x380/4)
-#define GICD_ICACTIVERN (0x3FC/4)
-#define GICD_IPRIORITYR (0x400/4)
-#define GICD_IPRIORITYRN (0x7F8/4)
-#define GICD_ITARGETSR  (0x800/4)
-#define GICD_ITARGETSRN (0xBF8/4)
-#define GICD_ICFGR      (0xC00/4)
-#define GICD_ICFGRN     (0xCFC/4)
-#define GICD_NSACR      (0xE00/4)
-#define GICD_NSACRN     (0xEFC/4)
-#define GICD_SGIR       (0xF00/4)
-#define GICD_CPENDSGIR  (0xF10/4)
-#define GICD_CPENDSGIRN (0xF1C/4)
-#define GICD_SPENDSGIR  (0xF20/4)
-#define GICD_SPENDSGIRN (0xF2C/4)
-#define GICD_ICPIDR2    (0xFE8/4)
-
-#define GICD_SGI_TARGET_LIST_SHIFT   (24)
-#define GICD_SGI_TARGET_LIST_MASK    (0x3UL << GICD_SGI_TARGET_LIST_SHIFT)
-#define GICD_SGI_TARGET_LIST         (0UL<<GICD_SGI_TARGET_LIST_SHIFT)
-#define GICD_SGI_TARGET_OTHERS       (1UL<<GICD_SGI_TARGET_LIST_SHIFT)
-#define GICD_SGI_TARGET_SELF         (2UL<<GICD_SGI_TARGET_LIST_SHIFT)
-#define GICD_SGI_TARGET_SHIFT        (16)
-#define GICD_SGI_TARGET_MASK         (0xFFUL<<GICD_SGI_TARGET_SHIFT)
-#define GICD_SGI_GROUP1              (1UL<<15)
-#define GICD_SGI_INTID_MASK          (0xFUL)
-
-#define GICC_CTLR       (0x0000/4)
-#define GICC_PMR        (0x0004/4)
-#define GICC_BPR        (0x0008/4)
-#define GICC_IAR        (0x000C/4)
-#define GICC_EOIR       (0x0010/4)
-#define GICC_RPR        (0x0014/4)
-#define GICC_HPPIR      (0x0018/4)
-#define GICC_APR        (0x00D0/4)
-#define GICC_NSAPR      (0x00E0/4)
-#define GICC_DIR        (0x1000/4)
-
-#define GICH_HCR        (0x00/4)
-#define GICH_VTR        (0x04/4)
-#define GICH_VMCR       (0x08/4)
-#define GICH_MISR       (0x10/4)
-#define GICH_EISR0      (0x20/4)
-#define GICH_EISR1      (0x24/4)
-#define GICH_ELSR0      (0x30/4)
-#define GICH_ELSR1      (0x34/4)
-#define GICH_APR        (0xF0/4)
-#define GICH_LR         (0x100/4)
-
-/* Register bits */
-#define GICD_CTL_ENABLE 0x1
-
-#define GICD_TYPE_LINES 0x01f
-#define GICD_TYPE_CPUS  0x0e0
-#define GICD_TYPE_SEC   0x400
-
-#define GICC_CTL_ENABLE 0x1
-#define GICC_CTL_EOI    (0x1 << 9)
-
-#define GICC_IA_IRQ       0x03ff
-#define GICC_IA_CPU_MASK  0x1c00
-#define GICC_IA_CPU_SHIFT 10
-
-#define GICH_HCR_EN       (1 << 0)
-#define GICH_HCR_UIE      (1 << 1)
-#define GICH_HCR_LRENPIE  (1 << 2)
-#define GICH_HCR_NPIE     (1 << 3)
-#define GICH_HCR_VGRP0EIE (1 << 4)
-#define GICH_HCR_VGRP0DIE (1 << 5)
-#define GICH_HCR_VGRP1EIE (1 << 6)
-#define GICH_HCR_VGRP1DIE (1 << 7)
-
-#define GICH_MISR_EOI     (1 << 0)
-#define GICH_MISR_U       (1 << 1)
-#define GICH_MISR_LRENP   (1 << 2)
-#define GICH_MISR_NP      (1 << 3)
-#define GICH_MISR_VGRP0E  (1 << 4)
-#define GICH_MISR_VGRP0D  (1 << 5)
-#define GICH_MISR_VGRP1E  (1 << 6)
-#define GICH_MISR_VGRP1D  (1 << 7)
-
-#define GICH_LR_VIRTUAL_MASK    0x3ff
-#define GICH_LR_VIRTUAL_SHIFT   0
-#define GICH_LR_PHYSICAL_MASK   0x3ff
-#define GICH_LR_PHYSICAL_SHIFT  10
-#define GICH_LR_STATE_MASK      0x3
-#define GICH_LR_STATE_SHIFT     28
-#define GICH_LR_PRIORITY_SHIFT  23
-#define GICH_LR_MAINTENANCE_IRQ (1<<19)
-#define GICH_LR_PENDING         (1<<28)
-#define GICH_LR_ACTIVE          (1<<29)
-#define GICH_LR_GRP1            (1<<30)
-#define GICH_LR_HW              (1<<31)
-#define GICH_LR_CPUID_SHIFT     9
-#define GICH_VTR_NRLRGS         0x3f
-
-#define GICH_VMCR_PRIORITY_MASK   0x1f
-#define GICH_VMCR_PRIORITY_SHIFT  27
-
 /*
  * The minimum GICC_BPR is required to be in the range 0-3. We set
  * GICC_BPR to 0 but we must expect that it might be 3. This means we
diff --git a/xen/include/asm-arm/gic_v2_defs.h b/xen/include/asm-arm/gic_v2_defs.h
new file mode 100644
index 0000000..f9ff885
--- /dev/null
+++ b/xen/include/asm-arm/gic_v2_defs.h
@@ -0,0 +1,139 @@
+/*
+ * ARM Generic Interrupt Controller support v2
+ *
+ * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
+ * Copyright (c) 2014 Cavium Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#define GICD_CTLR       (0x000/4)
+#define GICD_TYPER      (0x004/4)
+#define GICD_IIDR       (0x008/4)
+#define GICD_IGROUPR    (0x080/4)
+#define GICD_IGROUPRN   (0x0FC/4)
+#define GICD_ISENABLER  (0x100/4)
+#define GICD_ISENABLERN (0x17C/4)
+#define GICD_ICENABLER  (0x180/4)
+#define GICD_ICENABLERN (0x1fC/4)
+#define GICD_ISPENDR    (0x200/4)
+#define GICD_ISPENDRN   (0x27C/4)
+#define GICD_ICPENDR    (0x280/4)
+#define GICD_ICPENDRN   (0x2FC/4)
+#define GICD_ISACTIVER  (0x300/4)
+#define GICD_ISACTIVERN (0x37C/4)
+#define GICD_ICACTIVER  (0x380/4)
+#define GICD_ICACTIVERN (0x3FC/4)
+#define GICD_IPRIORITYR (0x400/4)
+#define GICD_IPRIORITYRN (0x7F8/4)
+#define GICD_ITARGETSR  (0x800/4)
+#define GICD_ITARGETSRN (0xBF8/4)
+#define GICD_ICFGR      (0xC00/4)
+#define GICD_ICFGRN     (0xCFC/4)
+#define GICD_NSACR      (0xE00/4)
+#define GICD_NSACRN     (0xEFC/4)
+#define GICD_SGIR       (0xF00/4)
+#define GICD_CPENDSGIR  (0xF10/4)
+#define GICD_CPENDSGIRN (0xF1C/4)
+#define GICD_SPENDSGIR  (0xF20/4)
+#define GICD_SPENDSGIRN (0xF2C/4)
+#define GICD_ICPIDR2    (0xFE8/4)
+
+#define GICD_SGI_TARGET_LIST_SHIFT   (24)
+#define GICD_SGI_TARGET_LIST_MASK    (0x3UL << GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_LIST         (0UL<<GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_OTHERS       (1UL<<GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_SELF         (2UL<<GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_SHIFT        (16)
+#define GICD_SGI_TARGET_MASK         (0xFFUL<<GICD_SGI_TARGET_SHIFT)
+#define GICD_SGI_GROUP1              (1UL<<15)
+#define GICD_SGI_INTID_MASK          (0xFUL)
+
+#define GICC_CTLR       (0x0000/4)
+#define GICC_PMR        (0x0004/4)
+#define GICC_BPR        (0x0008/4)
+#define GICC_IAR        (0x000C/4)
+#define GICC_EOIR       (0x0010/4)
+#define GICC_RPR        (0x0014/4)
+#define GICC_HPPIR      (0x0018/4)
+#define GICC_APR        (0x00D0/4)
+#define GICC_NSAPR      (0x00E0/4)
+#define GICC_DIR        (0x1000/4)
+
+#define GICH_HCR        (0x00/4)
+#define GICH_VTR        (0x04/4)
+#define GICH_VMCR       (0x08/4)
+#define GICH_MISR       (0x10/4)
+#define GICH_EISR0      (0x20/4)
+#define GICH_EISR1      (0x24/4)
+#define GICH_ELSR0      (0x30/4)
+#define GICH_ELSR1      (0x34/4)
+#define GICH_APR        (0xF0/4)
+#define GICH_LR         (0x100/4)
+
+/* Register bits */
+#define GICD_CTL_ENABLE 0x1
+
+#define GICD_TYPE_LINES 0x01f
+#define GICD_TYPE_CPUS  0x0e0
+#define GICD_TYPE_SEC   0x400
+
+#define GICC_CTL_ENABLE 0x1
+#define GICC_CTL_EOI    (0x1 << 9)
+
+#define GICC_IA_IRQ       0x03ff
+#define GICC_IA_CPU_MASK  0x1c00
+#define GICC_IA_CPU_SHIFT 10
+
+#define GICH_HCR_EN       (1 << 0)
+#define GICH_HCR_UIE      (1 << 1)
+#define GICH_HCR_LRENPIE  (1 << 2)
+#define GICH_HCR_NPIE     (1 << 3)
+#define GICH_HCR_VGRP0EIE (1 << 4)
+#define GICH_HCR_VGRP0DIE (1 << 5)
+#define GICH_HCR_VGRP1EIE (1 << 6)
+#define GICH_HCR_VGRP1DIE (1 << 7)
+
+#define GICH_MISR_EOI     (1 << 0)
+#define GICH_MISR_U       (1 << 1)
+#define GICH_MISR_LRENP   (1 << 2)
+#define GICH_MISR_NP      (1 << 3)
+#define GICH_MISR_VGRP0E  (1 << 4)
+#define GICH_MISR_VGRP0D  (1 << 5)
+#define GICH_MISR_VGRP1E  (1 << 6)
+#define GICH_MISR_VGRP1D  (1 << 7)
+
+#define GICH_LR_VIRTUAL_MASK    0x3ff
+#define GICH_LR_VIRTUAL_SHIFT   0
+#define GICH_LR_PHYSICAL_MASK   0x3ff
+#define GICH_LR_PHYSICAL_SHIFT  10
+#define GICH_LR_STATE_MASK      0x3
+#define GICH_LR_STATE_SHIFT     28
+#define GICH_LR_PRIORITY_SHIFT  23
+#define GICH_LR_MAINTENANCE_IRQ (1<<19)
+#define GICH_LR_PENDING         (1<<28)
+#define GICH_LR_ACTIVE          (1<<29)
+#define GICH_LR_GRP1            (1<<30)
+#define GICH_LR_HW              (1<<31)
+#define GICH_LR_CPUID_SHIFT     9
+#define GICH_VTR_NRLRGS         0x3f
+
+#define GICH_VMCR_PRIORITY_MASK   0x1f
+#define GICH_VMCR_PRIORITY_SHIFT  27
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
1.7.9.5

Re: [PATCH v2 04/15] xen/arm: move gic definitions to seperate file

[Julien Grall]

Hi Vijaya,

Thank you for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> diff --git a/xen/include/asm-arm/gic_v2_defs.h b/xen/include/asm-arm/gic_v2_defs.h
> new file mode 100644
> index 0000000..f9ff885
> --- /dev/null
> +++ b/xen/include/asm-arm/gic_v2_defs.h
> @@ -0,0 +1,139 @@
> +/*
> + * ARM Generic Interrupt Controller support v2
> + *
> + * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
> + * Copyright (c) 2014 Cavium Inc.

I'm not very familiar with copyright in general... but as this is code
movement, should not you keep the copyright of Tim?

Regards,

-- 
Julien Grall

Re: [PATCH v2 04/15] xen/arm: move gic definitions to seperate file

[Vijay Kilari]

On Fri, Apr 4, 2014 at 6:57 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Vijaya,
>
> Thank you for the patch.
>
> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>> diff --git a/xen/include/asm-arm/gic_v2_defs.h b/xen/include/asm-arm/gic_v2_defs.h
>> new file mode 100644
>> index 0000000..f9ff885
>> --- /dev/null
>> +++ b/xen/include/asm-arm/gic_v2_defs.h
>> @@ -0,0 +1,139 @@
>> +/*
>> + * ARM Generic Interrupt Controller support v2
>> + *
>> + * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
>> + * Copyright (c) 2014 Cavium Inc.
>
> I'm not very familiar with copyright in general... but as this is code
> movement, should not you keep the copyright of Tim?
>
I am also not aware it. I kept as it in V1.  You have asked me to change it
so I did. Anyways, I will put copyright of Tim
> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 04/15] xen/arm: move gic definitions to seperate file

[Julien Grall]

On 04/04/2014 04:29 PM, Vijay Kilari wrote:
> On Fri, Apr 4, 2014 at 6:57 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> Hi Vijaya,
>>
>> Thank you for the patch.
>>
>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>> diff --git a/xen/include/asm-arm/gic_v2_defs.h b/xen/include/asm-arm/gic_v2_defs.h
>>> new file mode 100644
>>> index 0000000..f9ff885
>>> --- /dev/null
>>> +++ b/xen/include/asm-arm/gic_v2_defs.h
>>> @@ -0,0 +1,139 @@
>>> +/*
>>> + * ARM Generic Interrupt Controller support v2
>>> + *
>>> + * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
>>> + * Copyright (c) 2014 Cavium Inc.
>>
>> I'm not very familiar with copyright in general... but as this is code
>> movement, should not you keep the copyright of Tim?
>>
> I am also not aware it. I kept as it in V1.  You have asked me to change it
> so I did. Anyways, I will put copyright of Tim

Hrm, I read my mail and I didn't ask to modify the copyright...  by
guard I meant include guard:

#ifndef __ASM_ARM_GIC_V2_DEFS_H__
#define __ASM_ARM_GIC_V2_DEFS_H__

your content

#endif /* __ASM_ARM_GIC_V2_DEFS_H__

Without this the header is not protected against multiple inclusion.

Regards,

-- 
Julien Grall

Re: [PATCH v2 04/15] xen/arm: move gic definitions to seperate file

[Ian Campbell]

On Fri, 2014-04-04 at 20:59 +0530, Vijay Kilari wrote:
> On Fri, Apr 4, 2014 at 6:57 PM, Julien Grall <julien.grall@linaro.org> wrote:
> > Hi Vijaya,
> >
> > Thank you for the patch.
> >
> > On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> >> diff --git a/xen/include/asm-arm/gic_v2_defs.h b/xen/include/asm-arm/gic_v2_defs.h
> >> new file mode 100644
> >> index 0000000..f9ff885
> >> --- /dev/null
> >> +++ b/xen/include/asm-arm/gic_v2_defs.h
> >> @@ -0,0 +1,139 @@
> >> +/*
> >> + * ARM Generic Interrupt Controller support v2
> >> + *
> >> + * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
> >> + * Copyright (c) 2014 Cavium Inc.
> >
> > I'm not very familiar with copyright in general... but as this is code
> > movement, should not you keep the copyright of Tim?
> >
> I am also not aware it. I kept as it in V1.  You have asked me to change it
> so I did. Anyways, I will put copyright of Tim

Yes, you need to keep the copyrights in place for sure.

Ian.

[PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

GIC low level functionality is segregated into
separate functions and are called using registered
callback wherever required.

This helps to separate generic and hardware functionality
later

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/gic.c                |  362 ++++++++++++++++++++++++++++---------
 xen/include/asm-arm/gic.h         |   50 +++++
 xen/include/asm-arm/gic_v2_defs.h |   16 +-
 3 files changed, 328 insertions(+), 100 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 64699e4..9f03135 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -57,8 +57,21 @@ static irq_desc_t irq_desc[NR_IRQS];
 static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
 static DEFINE_PER_CPU(uint64_t, lr_mask);
 
+static struct gic_hw_operations *gic_hw_ops;
+static struct gic_hw_operations gic_ops;
+
+void register_gic_ops(struct gic_hw_operations *ops)
+{
+    gic_hw_ops = ops;
+}
+
+void update_cpu_lr_mask(void)
+{
+    this_cpu(lr_mask) = 0ULL;
+}
+
 static uint8_t nr_lrs;
-#define lr_all_full() (this_cpu(lr_mask) == ((1 << nr_lrs) - 1))
+#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->nr_lrs()) - 1))
 
 /* The GIC mapping of CPU interfaces does not necessarily match the
  * logical CPU numbering. Let's use mapping as returned by the GIC
@@ -89,48 +102,124 @@ static unsigned int gic_cpu_mask(const cpumask_t *cpumask)
 
 unsigned int gic_number_lines(void)
 {
+    return gic_hw_ops->nr_lines();
+}
+
+static unsigned int gic_nr_lines(void)
+{
     return gic.lines;
 }
 
-irq_desc_t *__irq_to_desc(int irq)
+static unsigned int gic_nr_lrs(void)
 {
-    if (irq < NR_LOCAL_IRQS) return &this_cpu(local_irq_desc)[irq];
-    return &irq_desc[irq-NR_LOCAL_IRQS];
+    return nr_lrs;
 }
 
-void gic_save_state(struct vcpu *v)
+static int gic_state_init(struct vcpu *v)
+{
+     v->arch.gic_state = xzalloc(struct gic_state_data);
+     if(!v->arch.gic_state)
+        return -ENOMEM;
+     return 0;
+}
+
+static void save_state(struct vcpu *v)
 {
     int i;
-    ASSERT(!local_irq_is_enabled());
 
     /* No need for spinlocks here because interrupts are disabled around
      * this call and it only accesses struct vcpu fields that cannot be
      * accessed simultaneously by another pCPU.
      */
-    for ( i = 0; i < nr_lrs; i++ )
+    for ( i = 0; i < nr_lrs; i++)
         v->arch.gic_state->gic_lr[i] = GICH[GICH_LR + i];
-    v->arch.lr_mask = this_cpu(lr_mask);
     v->arch.gic_state->gic_apr = GICH[GICH_APR];
     v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];
     /* Disable until next VCPU scheduled */
     GICH[GICH_HCR] = 0;
+}
+
+static void restore_state(struct vcpu *v)
+{
+    int i;
+
+    for ( i = 0; i < nr_lrs; i++ )
+        GICH[GICH_LR + i] = v->arch.gic_state->gic_lr[i];
+    GICH[GICH_APR] = v->arch.gic_state->gic_apr;
+    GICH[GICH_VMCR] = v->arch.gic_state->gic_vmcr;
+    GICH[GICH_HCR] = GICH_HCR_EN;
+}
+
+static void gic_dump_state(struct vcpu *v)
+{
+    int i;
+    if ( v == current )
+    {
+        for ( i = 0; i < nr_lrs; i++ )
+            printk("   HW_LR[%d]=%x\n", i, GICH[GICH_LR + i]);
+    } else {
+        for ( i = 0; i < nr_lrs; i++ )
+            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->gic_lr[i]);
+    }
+}
+
+static void gic_enable_irq(int irq)
+{
+    /* Enable routing */
+    GICD[GICD_ISENABLER + irq / 32] = (1u << (irq % 32));
+}
+
+static void gic_disable_irq(int irq)
+{
+    /* Disable routing */
+    GICD[GICD_ICENABLER + irq / 32] = (1u << (irq % 32));
+}
+
+static void gic_eoi_irq(int irq)
+{
+    /* Lower the priority */
+    GICC[GICC_EOIR] = irq;
+}
+
+static void gic_dir_irq(int irq)
+{
+    /* Deactivate */
+    GICC[GICC_DIR] = irq;
+}
+
+static unsigned int gic_ack_irq(void)
+{
+    return (GICC[GICC_IAR] & GICC_IA_IRQ);
+}
+
+irq_desc_t *__irq_to_desc(int irq)
+{
+    if (irq < NR_LOCAL_IRQS) return &this_cpu(local_irq_desc)[irq];
+    return &irq_desc[irq-NR_LOCAL_IRQS];
+}
+
+void gic_save_state(struct vcpu *v)
+{
+    ASSERT(!local_irq_is_enabled());
+
+    /* No need for spinlocks here because interrupts are disabled around
+     * this call and it only accesses struct vcpu fields that cannot be
+     * accessed simultaneously by another pCPU.
+     */
+    v->arch.lr_mask = this_cpu(lr_mask);
+    gic_hw_ops->save_state(v);
     isb();
 }
 
 void gic_restore_state(struct vcpu *v)
 {
-    int i;
     ASSERT(!local_irq_is_enabled());
 
     if ( is_idle_vcpu(v) )
         return;
 
     this_cpu(lr_mask) = v->arch.lr_mask;
-    for ( i = 0; i < nr_lrs; i++ )
-        GICH[GICH_LR + i] = v->arch.gic_state->gic_lr[i];
-    GICH[GICH_APR] = v->arch.gic_state->gic_apr;
-    GICH[GICH_VMCR] = v->arch.gic_state->gic_vmcr;
-    GICH[GICH_HCR] = GICH_HCR_EN;
+    gic_hw_ops->restore_state(v);
     isb();
 
     gic_restore_pending_irqs(v);
@@ -146,7 +235,7 @@ static void gic_irq_enable(struct irq_desc *desc)
     desc->status &= ~IRQ_DISABLED;
     dsb(sy);
     /* Enable routing */
-    GICD[GICD_ISENABLER + irq / 32] = (1u << (irq % 32));
+    gic_hw_ops->enable_irq(irq);
     spin_unlock(&gic.lock);
     spin_unlock_irqrestore(&desc->lock, flags);
 }
@@ -159,7 +248,7 @@ static void gic_irq_disable(struct irq_desc *desc)
     spin_lock_irqsave(&desc->lock, flags);
     spin_lock(&gic.lock);
     /* Disable routing */
-    GICD[GICD_ICENABLER + irq / 32] = (1u << (irq % 32));
+    gic_hw_ops->disable_irq(irq);
     desc->status |= IRQ_DISABLED;
     spin_unlock(&gic.lock);
     spin_unlock_irqrestore(&desc->lock, flags);
@@ -185,16 +274,16 @@ static void gic_host_irq_end(struct irq_desc *desc)
 {
     int irq = desc->irq;
     /* Lower the priority */
-    GICC[GICC_EOIR] = irq;
+    gic_hw_ops->eoi_irq(irq);
     /* Deactivate */
-    GICC[GICC_DIR] = irq;
+    gic_hw_ops->deactivate_irq(irq);
 }
 
 static void gic_guest_irq_end(struct irq_desc *desc)
 {
     int irq = desc->irq;
     /* Lower the priority of the IRQ */
-    GICC[GICC_EOIR] = irq;
+    gic_hw_ops->eoi_irq(irq);
     /* Deactivation happens in maintenance interrupt / via GICV */
 }
 
@@ -230,7 +319,7 @@ static hw_irq_controller gic_guest_irq_type = {
  * - needs to be called with a valid cpu_mask, ie each cpu in the mask has
  * already called gic_cpu_init
  */
-static void gic_set_irq_properties(unsigned int irq, bool_t level,
+static void gic_set_irq_property(unsigned int irq, bool_t level,
                                    const cpumask_t *cpu_mask,
                                    unsigned int priority)
 {
@@ -257,6 +346,13 @@ static void gic_set_irq_properties(unsigned int irq, bool_t level,
 
 }
 
+static void gic_set_irq_properties(unsigned int irq, bool_t level,
+                                   const cpumask_t *cpu_mask,
+                                   unsigned int priority)
+{
+   return gic_hw_ops->set_irq_property(irq, level, cpu_mask, priority);
+}
+
 /* Program the GIC to route an interrupt */
 static int gic_route_irq(unsigned int irq, bool_t level,
                          const cpumask_t *cpu_mask, unsigned int priority)
@@ -377,7 +473,7 @@ static void __cpuinit gic_hyp_init(void)
     nr_lrs  = (vtr & GICH_VTR_NRLRGS) + 1;
 
     GICH[GICH_MISR] = GICH_MISR_EOI;
-    this_cpu(lr_mask) = 0ULL;
+    update_cpu_lr_mask();
 }
 
 static void __cpuinit gic_hyp_disable(void)
@@ -478,10 +574,22 @@ void __init gic_init(void)
     gic_cpu_init();
     gic_hyp_init();
 
+    register_gic_ops(&gic_ops);
     spin_unlock(&gic.lock);
 }
 
-void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
+static void gic_secondary_cpu_init(void)
+{
+    gic_cpu_init();
+    gic_hyp_init();
+}
+
+static struct dt_irq * gic_maintenance_irq(void)
+{
+    return &gic.maintenance;
+}
+
+static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
 {
     unsigned int mask = 0;
     cpumask_t online_mask;
@@ -498,30 +606,26 @@ void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
         | sgi;
 }
 
-void send_SGI_one(unsigned int cpu, enum gic_sgi sgi)
+void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
 {
-    ASSERT(cpu < NR_GIC_CPU_IF);  /* Targets bitmap only supports 8 CPUs */
-    send_SGI_mask(cpumask_of(cpu), sgi);
+   gic_hw_ops->send_sgi(cpumask, sgi);
 }
 
-void send_SGI_self(enum gic_sgi sgi)
+void send_SGI_one(unsigned int cpu, enum gic_sgi sgi)
 {
-    ASSERT(sgi < 16); /* There are only 16 SGIs */
-
-    dsb(sy);
-
-    GICD[GICD_SGIR] = GICD_SGI_TARGET_SELF
-        | sgi;
+    ASSERT(cpu < NR_GIC_CPU_IF);  /* Targets bitmap only supports 8 CPUs */
+    send_SGI_mask(cpumask_of(cpu), sgi);
 }
 
 void send_SGI_allbutself(enum gic_sgi sgi)
 {
-   ASSERT(sgi < 16); /* There are only 16 SGIs */
+    cpumask_t all_others_mask;
+    ASSERT(sgi < 16); /* There are only 16 SGIs */
 
-   dsb(sy);
+    cpumask_andnot(&all_others_mask, &cpu_possible_map, cpumask_of(smp_processor_id()));
 
-   GICD[GICD_SGIR] = GICD_SGI_TARGET_OTHERS
-       | sgi;
+    dsb(sy);
+    send_SGI_mask(&all_others_mask, sgi);
 }
 
 void smp_send_state_dump(unsigned int cpu)
@@ -533,26 +637,30 @@ void smp_send_state_dump(unsigned int cpu)
 void __cpuinit gic_init_secondary_cpu(void)
 {
     spin_lock(&gic.lock);
-    gic_cpu_init();
-    gic_hyp_init();
+    gic_hw_ops->secondary_init();
     spin_unlock(&gic.lock);
 }
 
 /* Shut down the per-CPU GIC interface */
+static void gic_disable_interface(void)
+{
+    gic_cpu_disable();
+    gic_hyp_disable();
+}
+
 void gic_disable_cpu(void)
 {
     ASSERT(!local_irq_is_enabled());
 
     spin_lock(&gic.lock);
-    gic_cpu_disable();
-    gic_hyp_disable();
+    gic_hw_ops->disable_interface();
     spin_unlock(&gic.lock);
 }
 
 void gic_route_ppis(void)
 {
     /* GIC maintenance */
-    gic_route_dt_irq(&gic.maintenance, cpumask_of(smp_processor_id()),
+    gic_route_dt_irq(gic_hw_ops->get_maintenance_irq(), cpumask_of(smp_processor_id()),
                      GIC_PRI_IRQ);
     /* Route timer interrupt */
     route_timer_interrupt();
@@ -627,23 +735,31 @@ int __init setup_dt_irq(const struct dt_irq *irq, struct irqaction *new)
     return rc;
 }
 
-static inline void gic_set_lr(int lr, struct pending_irq *p,
-        unsigned int state)
+static void gic_update_lr(int lr, struct pending_irq *p, unsigned int state)
 {
-    uint32_t lr_reg;
+    int maintenance_int = GICH_LR_MAINTENANCE_IRQ;
 
-    ASSERT(!local_irq_is_enabled());
     BUG_ON(lr >= nr_lrs);
     BUG_ON(lr < 0);
     BUG_ON(state & ~(GICH_LR_STATE_MASK<<GICH_LR_STATE_SHIFT));
 
-    lr_reg = state | ((p->priority >> 3) << GICH_LR_PRIORITY_SHIFT) |
+    GICH[GICH_LR + lr] = ((state & 0x3) << GICH_LR_STATE_SHIFT) |
+        maintenance_int |
+        ((p->priority >> 3) << GICH_LR_PRIORITY_SHIFT) |
         ((p->irq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT);
-    if ( p->desc != NULL )
-        lr_reg |= GICH_LR_HW | (p->desc->irq << GICH_LR_PHYSICAL_SHIFT);
+}
 
-    GICH[GICH_LR + lr] = lr_reg;
+static void gic_clear_lr(int lr)
+{
+    GICH[GICH_LR + lr] = 0;
+}
+
+static inline void gic_set_lr(int lr, struct pending_irq *p,
+        unsigned int state)
+{
+    ASSERT(!local_irq_is_enabled());
 
+    gic_hw_ops->update_lr(lr, p, state);
     set_bit(GIC_IRQ_GUEST_VISIBLE, &p->status);
     clear_bit(GIC_IRQ_GUEST_PENDING, &p->status);
     p->lr = lr;
@@ -685,6 +801,7 @@ void gic_raise_guest_irq(struct vcpu *v, unsigned int virtual_irq,
 {
     int i;
     struct pending_irq *n = irq_to_pending(v, virtual_irq);
+    unsigned int nr_lrs = gic_hw_ops->nr_lrs();
 
     ASSERT(spin_is_locked(&v->arch.vgic.lock));
 
@@ -711,29 +828,32 @@ void gic_raise_guest_irq(struct vcpu *v, unsigned int virtual_irq,
 static void gic_clear_one_lr(struct vcpu *v, int i)
 {
     struct pending_irq *p;
-    uint32_t lr;
     int irq;
+    struct gic_lr lr_val;
 
     ASSERT(!local_irq_is_enabled());
     ASSERT(spin_is_locked(&v->arch.vgic.lock));
 
     ASSERT(!local_irq_is_enabled());
 
-    lr = GICH[GICH_LR + i];
-    irq = (lr >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
+    gic_hw_ops->read_lr(i, &lr_val);
+    irq = lr_val.virq;
     p = irq_to_pending(v, irq);
-    if ( lr & GICH_LR_ACTIVE )
+    if ( lr_val.state & GICH_LR_ACTIVE )
     {
         set_bit(GIC_IRQ_GUEST_ACTIVE, &p->status);
         /* HW interrupts cannot be ACTIVE and PENDING */
         if ( p->desc == NULL &&
              test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) &&
              test_and_clear_bit(GIC_IRQ_GUEST_PENDING, &p->status) )
-            GICH[GICH_LR + i] = lr | GICH_LR_PENDING;
-    } else if ( lr & GICH_LR_PENDING ) {
+        {
+                 lr_val.state |= GICH_LR_PENDING;
+                 gic_hw_ops->write_lr(i, &lr_val);
+        }
+    } else if ( lr_val.state & GICH_LR_PENDING ) {
         clear_bit(GIC_IRQ_GUEST_PENDING, &p->status);
     } else {
-        GICH[GICH_LR + i] = 0;
+        gic_hw_ops->clear_lr(i);
         clear_bit(i, &this_cpu(lr_mask));
 
         if ( p->desc != NULL )
@@ -754,6 +874,7 @@ void gic_clear_lrs(struct vcpu *v)
 {
     int i = 0;
     unsigned long flags;
+    unsigned int nr_lrs = gic_hw_ops->nr_lrs();
 
     spin_lock_irqsave(&v->arch.vgic.lock, flags);
 
@@ -768,10 +889,12 @@ void gic_clear_lrs(struct vcpu *v)
 
 static void gic_restore_pending_irqs(struct vcpu *v)
 {
-    int i = 0, lrs = nr_lrs;
+    int i = 0, lrs;
     struct pending_irq *p, *t, *p_r;
     unsigned long flags;
+    unsigned int nr_lrs = gic_hw_ops->nr_lrs();
 
+    lrs = nr_lrs;
     if ( list_empty(&v->arch.vgic.lr_pending) )
         return;
 
@@ -828,13 +951,15 @@ void gic_clear_pending_irqs(struct vcpu *v)
 
 int gic_events_need_delivery(void)
 {
-    int mask_priority, lrs = nr_lrs;
+    int mask_priority, lrs;
     int max_priority = 0xff, active_priority = 0xff;
     struct vcpu *v = current;
     struct pending_irq *p;
     unsigned long flags;
+    unsigned int nr_lrs = gic_hw_ops->nr_lrs();
+    lrs = nr_lrs;
 
-    mask_priority = (GICH[GICH_VMCR] >> GICH_VMCR_PRIORITY_SHIFT) & GICH_VMCR_PRIORITY_MASK;
+    mask_priority = gic_hw_ops->read_vmcr_priority();
 
     spin_lock_irqsave(&v->arch.vgic.lock, flags);
 
@@ -871,9 +996,9 @@ void gic_inject(void)
     gic_restore_pending_irqs(current);
 
     if ( !list_empty(&current->arch.vgic.lr_pending) && lr_all_full() )
-        GICH[GICH_HCR] |= GICH_HCR_UIE;
+        gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 1);
     else
-        GICH[GICH_HCR] &= ~GICH_HCR_UIE;
+        gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 0);
 }
 
 int gic_route_irq_to_guest(struct domain *d, const struct dt_irq *irq,
@@ -921,10 +1046,10 @@ out:
     return retval;
 }
 
-static void do_sgi(struct cpu_user_regs *regs, int othercpu, enum gic_sgi sgi)
+static void do_sgi(struct cpu_user_regs *regs, enum gic_sgi sgi)
 {
     /* Lower the priority */
-    GICC[GICC_EOIR] = sgi;
+    gic_hw_ops->eoi_irq(sgi);
 
     switch (sgi)
     {
@@ -943,19 +1068,16 @@ static void do_sgi(struct cpu_user_regs *regs, int othercpu, enum gic_sgi sgi)
     }
 
     /* Deactivate */
-    GICC[GICC_DIR] = sgi;
+    gic_hw_ops->deactivate_irq(sgi);
 }
 
 /* Accept an interrupt from the GIC and dispatch its handler */
 void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
 {
-    uint32_t intack;
     unsigned int irq;
 
-
     do  {
-        intack = GICC[GICC_IAR];
-        irq = intack & GICC_IA_IRQ;
+        irq = gic_hw_ops->ack_irq();
 
         if ( likely(irq >= 16 && irq < 1021) )
         {
@@ -965,8 +1087,7 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
         }
         else if (unlikely(irq < 16))
         {
-            unsigned int cpu = (intack & GICC_IA_CPU_MASK) >> GICC_IA_CPU_SHIFT;
-            do_sgi(regs, cpu, irq);
+            do_sgi(regs, irq);
         }
         else
         {
@@ -976,15 +1097,7 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
     } while (1);
 }
 
-int vcpu_gic_init(struct vcpu *v)
-{
-     v->arch.gic_state = xzalloc(struct gic_state_data);
-     if(!v->arch.gic_state)
-        return -ENOMEM;
-     return 0;
-}
-
-int gicv_setup(struct domain *d)
+static int gicv_init(struct domain *d)
 {
     int ret;
 
@@ -1031,6 +1144,85 @@ int gicv_setup(struct domain *d)
 
 }
 
+int vcpu_gic_init(struct vcpu *v)
+{
+     return gic_hw_ops->state_init(v);
+}
+
+int gicv_setup(struct domain *d)
+{
+    int ret;
+
+    ret = gic_hw_ops->gicv_setup(d);
+    return ret;
+
+}
+
+static void gic_read_lr(int lr, struct gic_lr *lr_reg)
+{
+    uint32_t lrv;
+
+    lrv = GICH[GICH_LR + lr];
+    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
+    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
+    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
+    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
+    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
+    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
+}
+
+static void gic_write_lr(int lr, struct gic_lr *lr_reg)
+{
+    uint32_t lrv = 0;
+    lrv = ( ((lr_reg->pirq & GICH_LR_PHYSICAL_MASK) << GICH_LR_PHYSICAL_SHIFT)  |
+            ((lr_reg->virq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT)   |
+            ((uint32_t)(lr_reg->priority & GICH_LR_PRIORITY_MASK) << GICH_LR_PRIORITY_SHIFT) |
+            ((uint32_t)(lr_reg->state & GICH_LR_STATE_MASK) << GICH_LR_STATE_SHIFT) |
+            ((uint32_t)(lr_reg->hw_status & GICH_LR_HW_MASK)  << GICH_LR_HW_SHIFT)  |
+            ((uint32_t)(lr_reg->grp & GICH_LR_GRP_MASK) << GICH_LR_GRP_SHIFT) );
+
+    GICH[GICH_LR + lr] = lrv;
+}
+
+static void gic_hcr_status(uint32_t flag, uint8_t status)
+{
+    if ( status )
+      GICH[GICH_HCR] |= flag;
+    else
+      GICH[GICH_HCR] &= ~flag;
+}
+
+static unsigned int gic_read_vmcr_priority(void)
+{
+   return (GICH[GICH_VMCR] >> GICH_VMCR_PRIORITY_SHIFT) & GICH_VMCR_PRIORITY_MASK;
+}
+
+static struct gic_hw_operations gic_ops = {
+    .nr_lines            = gic_nr_lines,
+    .nr_lrs              = gic_nr_lrs,
+    .secondary_init      = gic_secondary_cpu_init,
+    .get_maintenance_irq = gic_maintenance_irq,
+    .state_init          = gic_state_init,
+    .save_state          = save_state,
+    .restore_state       = restore_state,
+    .dump_state          = gic_dump_state,
+    .gicv_setup          = gicv_init,
+    .enable_irq          = gic_enable_irq,
+    .disable_irq         = gic_disable_irq,
+    .eoi_irq             = gic_eoi_irq,
+    .deactivate_irq      = gic_dir_irq,
+    .ack_irq             = gic_ack_irq,
+    .set_irq_property    = gic_set_irq_property,
+    .send_sgi            = gic_send_sgi,
+    .disable_interface   = gic_disable_interface,
+    .update_lr           = gic_update_lr,
+    .update_hcr_status   = gic_hcr_status,
+    .clear_lr            = gic_clear_lr,
+    .read_lr             = gic_read_lr,
+    .write_lr            = gic_write_lr,
+    .read_vmcr_priority  = gic_read_vmcr_priority,
+};
+
 static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
 {
     /* 
@@ -1043,18 +1235,10 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
 
 void gic_dump_info(struct vcpu *v)
 {
-    int i;
     struct pending_irq *p;
 
     printk("GICH_LRs (vcpu %d) mask=%"PRIx64"\n", v->vcpu_id, v->arch.lr_mask);
-    if ( v == current )
-    {
-        for ( i = 0; i < nr_lrs; i++ )
-            printk("   HW_LR[%d]=%x\n", i, GICH[GICH_LR + i]);
-    } else {
-        for ( i = 0; i < nr_lrs; i++ )
-            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->gic_lr[i]);
-    }
+    gic_hw_ops->dump_state(v);
 
     list_for_each_entry ( p, &v->arch.vgic.inflight_irqs, inflight )
     {
@@ -1070,7 +1254,7 @@ void gic_dump_info(struct vcpu *v)
 
 void __cpuinit init_maintenance_interrupt(void)
 {
-    request_dt_irq(&gic.maintenance, maintenance_interrupt,
+    request_dt_irq(gic_hw_ops->get_maintenance_irq(), maintenance_interrupt,
                    "irq-maintenance", NULL);
 }
 
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 5f49eb1..27d2792 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -39,6 +39,17 @@
 #define GIC_PRI_IPI        0x90 /* IPIs must preempt normal interrupts */
 #define GIC_PRI_HIGHEST    0x80 /* Higher priorities belong to Secure-World */
 
+#define GICH_LR_PENDING    1
+#define GICH_LR_ACTIVE     2
+
+#define GICH_HCR_EN       (1 << 0)
+#define GICH_HCR_UIE      (1 << 1)
+#define GICH_HCR_LRENPIE  (1 << 2)
+#define GICH_HCR_NPIE     (1 << 3)
+#define GICH_HCR_VGRP0EIE (1 << 4)
+#define GICH_HCR_VGRP0DIE (1 << 5)
+#define GICH_HCR_VGRP1EIE (1 << 6)
+#define GICH_HCR_VGRP1DIE (1 << 7)
 
 #ifndef __ASSEMBLY__
 #include <xen/device_tree.h>
@@ -61,6 +72,15 @@ struct gic_state_data {
     uint32_t gic_lr[64];
 };
 
+struct gic_lr {
+   uint32_t pirq;
+   uint32_t virq;
+   uint8_t priority;
+   uint8_t state;
+   uint8_t hw_status;
+   uint8_t grp;
+};
+
 extern int domain_vgic_init(struct domain *d);
 extern void domain_vgic_free(struct domain *d);
 
@@ -129,6 +149,36 @@ int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
                   unsigned int *out_hwirq, unsigned int *out_type);
 void gic_clear_lrs(struct vcpu *v);
 
+struct gic_hw_operations {
+    struct dt_irq * (*get_maintenance_irq)(void);
+    unsigned int (*nr_lines)(void);
+    unsigned int (*nr_lrs)(void);
+    int (*state_init)(struct vcpu *);
+    void (*save_state)(struct vcpu *);
+    void (*restore_state)(struct vcpu *);
+    void (*dump_state)(struct vcpu *);
+    int (*gicv_setup)(struct domain *);
+    void (*enable_irq)(int);
+    void (*disable_irq)(int);
+    void (*eoi_irq)(int);
+    void (*deactivate_irq)(int);
+    unsigned int (*ack_irq)(void);
+    void (*set_irq_property)(unsigned int irq, bool_t level,
+                            const cpumask_t *cpu_mask,
+                            unsigned int priority);
+    void (*send_sgi)(const cpumask_t *, enum gic_sgi);
+    void (*disable_interface)(void);
+    void (*update_lr)(int lr, struct pending_irq *, unsigned int state);
+    void (*update_hcr_status)(uint32_t flag, uint8_t set);
+    void (*clear_lr)(int lr);
+    void (*read_lr)(int lr, struct gic_lr *);
+    void (*write_lr)(int lr, struct gic_lr *);
+    unsigned int (*read_vmcr_priority)(void);
+    void (*secondary_init)(void);
+};
+
+void register_gic_ops(struct gic_hw_operations *ops);
+extern void update_cpu_lr_mask(void);
 #endif /* __ASSEMBLY__ */
 #endif
 
diff --git a/xen/include/asm-arm/gic_v2_defs.h b/xen/include/asm-arm/gic_v2_defs.h
index f9ff885..e1bb09c 100644
--- a/xen/include/asm-arm/gic_v2_defs.h
+++ b/xen/include/asm-arm/gic_v2_defs.h
@@ -93,15 +93,6 @@
 #define GICC_IA_CPU_MASK  0x1c00
 #define GICC_IA_CPU_SHIFT 10
 
-#define GICH_HCR_EN       (1 << 0)
-#define GICH_HCR_UIE      (1 << 1)
-#define GICH_HCR_LRENPIE  (1 << 2)
-#define GICH_HCR_NPIE     (1 << 3)
-#define GICH_HCR_VGRP0EIE (1 << 4)
-#define GICH_HCR_VGRP0DIE (1 << 5)
-#define GICH_HCR_VGRP1EIE (1 << 6)
-#define GICH_HCR_VGRP1DIE (1 << 7)
-
 #define GICH_MISR_EOI     (1 << 0)
 #define GICH_MISR_U       (1 << 1)
 #define GICH_MISR_LRENP   (1 << 2)
@@ -118,9 +109,12 @@
 #define GICH_LR_STATE_MASK      0x3
 #define GICH_LR_STATE_SHIFT     28
 #define GICH_LR_PRIORITY_SHIFT  23
+#define GICH_LR_PRIORITY_MASK   0x1f
+#define GICH_LR_HW_SHIFT        31
+#define GICH_LR_HW_MASK         0x1
+#define GICH_LR_GRP_SHIFT       30
+#define GICH_LR_GRP_MASK        0x1
 #define GICH_LR_MAINTENANCE_IRQ (1<<19)
-#define GICH_LR_PENDING         (1<<28)
-#define GICH_LR_ACTIVE          (1<<29)
 #define GICH_LR_GRP1            (1<<30)
 #define GICH_LR_HW              (1<<31)
 #define GICH_LR_CPUID_SHIFT     9
-- 
1.7.9.5

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Julien Grall]

Hi Vijaya,

Thank you for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> GIC low level functionality is segregated into
> separate functions and are called using registered
> callback wherever required.
> 
> This helps to separate generic and hardware functionality
> later

Honestly, your patch on V1 was far better to read. As you are nearly
modify every functions, you should directly move it to a new file (e.g
merge with patch #9).

> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> ---
>  xen/arch/arm/gic.c                |  362 ++++++++++++++++++++++++++++---------
>  xen/include/asm-arm/gic.h         |   50 +++++
>  xen/include/asm-arm/gic_v2_defs.h |   16 +-
>  3 files changed, 328 insertions(+), 100 deletions(-)
> 
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 64699e4..9f03135 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -57,8 +57,21 @@ static irq_desc_t irq_desc[NR_IRQS];
>  static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
>  static DEFINE_PER_CPU(uint64_t, lr_mask);
>  
> +static struct gic_hw_operations *gic_hw_ops;
> +static struct gic_hw_operations gic_ops;
> +
> +void register_gic_ops(struct gic_hw_operations *ops)

the callback should not change during runtime. Should the prototype
should be:

void register_gic_ops(const struct gic_hw_operations *ops);

> +{
> +    gic_hw_ops = ops;
> +}
> +
> +void update_cpu_lr_mask(void)
> +{
> +    this_cpu(lr_mask) = 0ULL;
> +}
> +
>  static uint8_t nr_lrs;
> -#define lr_all_full() (this_cpu(lr_mask) == ((1 << nr_lrs) - 1))
> +#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->nr_lrs()) - 1))
>  
>  /* The GIC mapping of CPU interfaces does not necessarily match the
>   * logical CPU numbering. Let's use mapping as returned by the GIC
> @@ -89,48 +102,124 @@ static unsigned int gic_cpu_mask(const cpumask_t *cpumask)
>  
>  unsigned int gic_number_lines(void)
>  {
> +    return gic_hw_ops->nr_lines();
> +}
> +
> +static unsigned int gic_nr_lines(void)
> +{
>      return gic.lines;
>  }

If you had directly move the code to another files you would avoid this
kind of non sense name...

>  
> -irq_desc_t *__irq_to_desc(int irq)
> +static unsigned int gic_nr_lrs(void)
>  {
> -    if (irq < NR_LOCAL_IRQS) return &this_cpu(local_irq_desc)[irq];
> -    return &irq_desc[irq-NR_LOCAL_IRQS];

Why did you had to modify __irq_to_desc?

> +    return nr_lrs;
>  }
>  
> -void gic_save_state(struct vcpu *v)
> +static int gic_state_init(struct vcpu *v)
> +{
> +     v->arch.gic_state = xzalloc(struct gic_state_data);
> +     if(!v->arch.gic_state)

if ( !v->arch.gic_state )

> +        return -ENOMEM;
> +     return 0;
> +}
> +
> +static void save_state(struct vcpu *v)
>  {
>      int i;
> -    ASSERT(!local_irq_is_enabled());
>  
>      /* No need for spinlocks here because interrupts are disabled around
>       * this call and it only accesses struct vcpu fields that cannot be
>       * accessed simultaneously by another pCPU.
>       */
> -    for ( i = 0; i < nr_lrs; i++ )
> +    for ( i = 0; i < nr_lrs; i++)

Spurious change

[..]

> @@ -129,6 +149,36 @@ int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
>                    unsigned int *out_hwirq, unsigned int *out_type);
>  void gic_clear_lrs(struct vcpu *v);
>  
> +struct gic_hw_operations {

I made some remarks on v1, that you completely forgot to address on this
structure.

Can you document every callbacks (see asm-arm/platform.h for an
example)? Why do you need them...

> +    struct dt_irq * (*get_maintenance_irq)(void);

With my interrupts patch series dt_irq will be removed soon.

> +    unsigned int (*nr_lines)(void);
> +    unsigned int (*nr_lrs)(void);

I don't think we need a callbacks for both of them. 2 unsigned int are fine.

[..]

> +    void (*enable_irq)(int);
> +    void (*disable_irq)(int);
> +    void (*eoi_irq)(int);
> +    void (*deactivate_irq)(int);
> +    unsigned int (*ack_irq)(void);

I would prefer to introduce a new hw_irq_controller here rather than
adding another abstraction.

> +    void (*set_irq_property)(unsigned int irq, bool_t level,
> +                            const cpumask_t *cpu_mask,
> +                            unsigned int priority);

I would rename the function into set_irq_properties.

[..]

> diff --git a/xen/include/asm-arm/gic_v2_defs.h b/xen/include/asm-arm/gic_v2_defs.h
> index f9ff885..e1bb09c 100644
> --- a/xen/include/asm-arm/gic_v2_defs.h
> +++ b/xen/include/asm-arm/gic_v2_defs.h
> @@ -93,15 +93,6 @@
>  #define GICC_IA_CPU_MASK  0x1c00
>  #define GICC_IA_CPU_SHIFT 10
>  
> -#define GICH_HCR_EN       (1 << 0)
> -#define GICH_HCR_UIE      (1 << 1)
> -#define GICH_HCR_LRENPIE  (1 << 2)
> -#define GICH_HCR_NPIE     (1 << 3)
> -#define GICH_HCR_VGRP0EIE (1 << 4)
> -#define GICH_HCR_VGRP0DIE (1 << 5)
> -#define GICH_HCR_VGRP1EIE (1 << 6)
> -#define GICH_HCR_VGRP1DIE (1 << 7)
> -
>  #define GICH_MISR_EOI     (1 << 0)
>  #define GICH_MISR_U       (1 << 1)
>  #define GICH_MISR_LRENP   (1 << 2)
> @@ -118,9 +109,12 @@
>  #define GICH_LR_STATE_MASK      0x3
>  #define GICH_LR_STATE_SHIFT     28
>  #define GICH_LR_PRIORITY_SHIFT  23
> +#define GICH_LR_PRIORITY_MASK   0x1f
> +#define GICH_LR_HW_SHIFT        31
> +#define GICH_LR_HW_MASK         0x1
> +#define GICH_LR_GRP_SHIFT       30
> +#define GICH_LR_GRP_MASK        0x1
>  #define GICH_LR_MAINTENANCE_IRQ (1<<19)
> -#define GICH_LR_PENDING         (1<<28)
> -#define GICH_LR_ACTIVE          (1<<29)
>  #define GICH_LR_GRP1            (1<<30)
>  #define GICH_LR_HW              (1<<31)
>  #define GICH_LR_CPUID_SHIFT     9

Why did you move all theses define on gic_v2_defs.h, and move back to
gic.h again?

Regards,


-- 
Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Vijay Kilari]

>> +    unsigned int (*nr_lines)(void);
>> +    unsigned int (*nr_lrs)(void);
>
> I don't think we need a callbacks for both of them. 2 unsigned int are fine.
   nr_lines returns number of gic lines supported.
   nr_lrs returns number of LR registers supported.

>
> [..]
>
>> +    void (*enable_irq)(int);
>> +    void (*disable_irq)(int);
>> +    void (*eoi_irq)(int);
>> +    void (*deactivate_irq)(int);
>> +    unsigned int (*ack_irq)(void);
>
> I would prefer to introduce a new hw_irq_controller here rather than
> adding another abstraction.
>
  hw_irq_controller can be used only for enable_irq, disable_irq  & eoi_irq.
other callbacks in hw_irq_controller cannot be used.

Ex: ack_irq in our case returns the irq where as 'ack' callback in
hw_irq_controller
does not return a value. So not suitable here.

>
> --
> Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Julien Grall]

On 09/04/14 08:43, Vijay Kilari wrote:
>>> +    unsigned int (*nr_lines)(void);
>>> +    unsigned int (*nr_lrs)(void);
>>
>> I don't think we need a callbacks for both of them. 2 unsigned int are fine.
>     nr_lines returns number of gic lines supported.
>     nr_lrs returns number of LR registers supported.

I know it ... my point is to replace the both callbacks by:
    unsigned nr_lines;
    unsigned nr_lrs;

We don't have to use functions as it won't change after boot setup.

>
>>
>> [..]
>>
>>> +    void (*enable_irq)(int);
>>> +    void (*disable_irq)(int);
>>> +    void (*eoi_irq)(int);
>>> +    void (*deactivate_irq)(int);
>>> +    unsigned int (*ack_irq)(void);
>>
>> I would prefer to introduce a new hw_irq_controller here rather than
>> adding another abstraction.
>>
>    hw_irq_controller can be used only for enable_irq, disable_irq  & eoi_irq.
> other callbacks in hw_irq_controller cannot be used.
>
> Ex: ack_irq in our case returns the irq where as 'ack' callback in
> hw_irq_controller
> does not return a value. So not suitable here.

Hmmm... rigth for deactivate_irq and ack_irq. For the others you can 
definitely use the hw_irq_controller.

Regards,

-- 
Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Ian Campbell]

On Wed, 2014-04-09 at 09:36 +0100, Julien Grall wrote:
> >> [..]
> >>
> >>> +    void (*enable_irq)(int);
> >>> +    void (*disable_irq)(int);
> >>> +    void (*eoi_irq)(int);
> >>> +    void (*deactivate_irq)(int);
> >>> +    unsigned int (*ack_irq)(void);
> >>
> >> I would prefer to introduce a new hw_irq_controller here rather than
> >> adding another abstraction.
> >>
> >    hw_irq_controller can be used only for enable_irq, disable_irq  & eoi_irq.
> > other callbacks in hw_irq_controller cannot be used.
> >
> > Ex: ack_irq in our case returns the irq where as 'ack' callback in
> > hw_irq_controller
> > does not return a value. So not suitable here.
> 
> Hmmm... rigth for deactivate_irq and ack_irq. For the others you can 
> definitely use the hw_irq_controller.

Perhaps
	struct gic_irq_controller {
		struct hw_irq_controller hw_irq;
		unsigned int (*ack_irq)(void)
	}

And use container_of to go fom a hw_Irq to a gic_irq_controller when
needed?

Ian.

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Ian Campbell]

On Fri, 2014-04-04 at 14:55 +0100, Julien Grall wrote:
> Hi Vijaya,
> 
> Thank you for the patch.
> 
> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> > From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> > 
> > GIC low level functionality is segregated into
> > separate functions and are called using registered
> > callback wherever required.
> > 
> > This helps to separate generic and hardware functionality
> > later
> 
> Honestly, your patch on V1 was far better to read. As you are nearly
> modify every functions, you should directly move it to a new file (e.g
> merge with patch #9).

This is a tricky judgement call, if there is to be mass code motion it
should be done strictly separately from any functional changes. If that
can be done all at once in a way which is reviewable then fine, but if
not then I would much rather err on the side of refactoring and then
moving as two steps even if the interim version looks a bit odd.

v1 of this patch certainly did mix the motion with functional changes
and so separating things out was the correct thing to do. Perhaps the
functional changes are now done elsewhere and it would be possible to
revert to a single patch which moved blocks of code out into callbacks,
but I wouldn't require it.

Ian.

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Julien Grall]

On 04/09/2014 06:00 PM, Ian Campbell wrote:
> On Fri, 2014-04-04 at 14:55 +0100, Julien Grall wrote:
>> Hi Vijaya,
>>
>> Thank you for the patch.
>>
>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>>
>>> GIC low level functionality is segregated into
>>> separate functions and are called using registered
>>> callback wherever required.
>>>
>>> This helps to separate generic and hardware functionality
>>> later
>>
>> Honestly, your patch on V1 was far better to read. As you are nearly
>> modify every functions, you should directly move it to a new file (e.g
>> merge with patch #9).
> 
> This is a tricky judgement call, if there is to be mass code motion it
> should be done strictly separately from any functional changes. If that
> can be done all at once in a way which is reviewable then fine, but if
> not then I would much rather err on the side of refactoring and then
> moving as two steps even if the interim version looks a bit odd.
> 
> v1 of this patch certainly did mix the motion with functional changes
> and so separating things out was the correct thing to do. Perhaps the
> functional changes are now done elsewhere and it would be possible to
> revert to a single patch which moved blocks of code out into callbacks,
> but I wouldn't require it.

IHMO, I think everything that is more than splitting the function in 2
(e.g, prototype change, merging functions...) should not be part of this
patch.

Keeping one patch (i'm not taking into account my previous comment) for
code movement and function splitting will avoid strange renaming that is
very confusing in this patch.

If we keep two patch, perhaps the gicv2 callbacks should be prefix by
gicv2_...

Regards,

-- 
Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Vijay Kilari]

Hi Julien & Ian,

On Wed, Apr 9, 2014 at 10:37 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 04/09/2014 06:00 PM, Ian Campbell wrote:
>> On Fri, 2014-04-04 at 14:55 +0100, Julien Grall wrote:
>>> Hi Vijaya,
>>>
>>> Thank you for the patch.
>>>
>>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>>>
>>>> GIC low level functionality is segregated into
>>>> separate functions and are called using registered
>>>> callback wherever required.
>>>>
>>>> This helps to separate generic and hardware functionality
>>>> later
>>>
>>> Honestly, your patch on V1 was far better to read. As you are nearly
>>> modify every functions, you should directly move it to a new file (e.g
>>> merge with patch #9).
>>
>> This is a tricky judgement call, if there is to be mass code motion it
>> should be done strictly separately from any functional changes. If that
>> can be done all at once in a way which is reviewable then fine, but if
>> not then I would much rather err on the side of refactoring and then
>> moving as two steps even if the interim version looks a bit odd.
>>
>> v1 of this patch certainly did mix the motion with functional changes
>> and so separating things out was the correct thing to do. Perhaps the
>> functional changes are now done elsewhere and it would be possible to
>> revert to a single patch which moved blocks of code out into callbacks,
>> but I wouldn't require it.
>
> IHMO, I think everything that is more than splitting the function in 2
> (e.g, prototype change, merging functions...) should not be part of this
> patch.
>
> Keeping one patch (i'm not taking into account my previous comment) for
> code movement and function splitting will avoid strange renaming that is
> very confusing in this patch.
>
> If we keep two patch, perhaps the gicv2 callbacks should be prefix by
> gicv2_...
>

Having 2 patches one for with code changes and
other with pure code movement is fine. but first patch is always confusing
to make this changes.

I prefer to have single patch with changes and code movement as in V1 and this
makes job simple and I think we should bear with this.

> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Ian Campbell]

On Wed, 2014-04-09 at 18:07 +0100, Julien Grall wrote:
> If we keep two patch, perhaps the gicv2 callbacks should be prefix by
> gicv2_...

I think this should be done regardless of everything else. Although it
doesn't matter much once the functions are in gic-v2.c it will make it
obvious in stack traces etc which driver is in use.

Ian.

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Julien Grall]

Hello Vijaya,

On 04/04/14 12:56, vijay.kilari@gmail.com wrote:
> +static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
>   {
>       unsigned int mask = 0;
>       cpumask_t online_mask;
> @@ -498,30 +606,26 @@ void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
>           | sgi;
>   }
>
> -void send_SGI_one(unsigned int cpu, enum gic_sgi sgi)
> +void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
>   {
> -    ASSERT(cpu < NR_GIC_CPU_IF);  /* Targets bitmap only supports 8 CPUs */
> -    send_SGI_mask(cpumask_of(cpu), sgi);
> +   gic_hw_ops->send_sgi(cpumask, sgi);
>   }
>
> -void send_SGI_self(enum gic_sgi sgi)
> +void send_SGI_one(unsigned int cpu, enum gic_sgi sgi)
>   {
> -    ASSERT(sgi < 16); /* There are only 16 SGIs */
> -
> -    dsb(sy);
> -
> -    GICD[GICD_SGIR] = GICD_SGI_TARGET_SELF
> -        | sgi;
> +    ASSERT(cpu < NR_GIC_CPU_IF);  /* Targets bitmap only supports 8 CPUs */
> +    send_SGI_mask(cpumask_of(cpu), sgi);
>   }
>
>   void send_SGI_allbutself(enum gic_sgi sgi)
>   {
> -   ASSERT(sgi < 16); /* There are only 16 SGIs */
> +    cpumask_t all_others_mask;
> +    ASSERT(sgi < 16); /* There are only 16 SGIs */
>
> -   dsb(sy);
> +    cpumask_andnot(&all_others_mask, &cpu_possible_map, cpumask_of(smp_processor_id()));
>
> -   GICD[GICD_SGIR] = GICD_SGI_TARGET_OTHERS
> -       | sgi;
> +    dsb(sy);
> +    send_SGI_mask(&all_others_mask, sgi);
>   }

As I said in V1, this change breaks SMP boot with GICv2...

GICD_SGI_TARGERT_OTHERS will send an SGI to every CPUs even if the CPU 
is not yet online (i.e. not registered by Xen). It's used during 
secondary boot (cpu_up_send_sgi).

Your solution won't work because send_SGI_mask only deal with online CPU.

All the changes of send_sgi is more than splitting functions... this 
should be moved on another patch and you should justify the modifications.

[..]

>   int gic_route_irq_to_guest(struct domain *d, const struct dt_irq *irq,
> @@ -921,10 +1046,10 @@ out:
>       return retval;
>   }
>
> -static void do_sgi(struct cpu_user_regs *regs, int othercpu, enum gic_sgi sgi)
> +static void do_sgi(struct cpu_user_regs *regs, enum gic_sgi sgi)

Why did you drop the othercpu here?

Again, please justify every change on the prototype of every functions. 
If it's not trivial, split in small patches...

-- 
Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Vijay Kilari]

On Wed, Apr 9, 2014 at 2:20 PM, Julien Grall <julien.grall@linaro.org> wrote:
>
> Hello Vijaya,
>
>
> On 04/04/14 12:56, vijay.kilari@gmail.com wrote:
>>
>> +static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
>>   {
>>       unsigned int mask = 0;
>>       cpumask_t online_mask;
>> @@ -498,30 +606,26 @@ void send_SGI_mask(const cpumask_t *cpumask, enum
>> gic_sgi sgi)
>>           | sgi;
>>   }
>>
>> -void send_SGI_one(unsigned int cpu, enum gic_sgi sgi)
>> +void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
>>   {
>> -    ASSERT(cpu < NR_GIC_CPU_IF);  /* Targets bitmap only supports 8 CPUs
>> */
>> -    send_SGI_mask(cpumask_of(cpu), sgi);
>> +   gic_hw_ops->send_sgi(cpumask, sgi);
>>   }
>>
>> -void send_SGI_self(enum gic_sgi sgi)
>> +void send_SGI_one(unsigned int cpu, enum gic_sgi sgi)
>>   {
>> -    ASSERT(sgi < 16); /* There are only 16 SGIs */
>> -
>> -    dsb(sy);
>> -
>> -    GICD[GICD_SGIR] = GICD_SGI_TARGET_SELF
>> -        | sgi;
>> +    ASSERT(cpu < NR_GIC_CPU_IF);  /* Targets bitmap only supports 8 CPUs
>> */
>> +    send_SGI_mask(cpumask_of(cpu), sgi);
>>   }
>>
>>   void send_SGI_allbutself(enum gic_sgi sgi)
>>   {
>> -   ASSERT(sgi < 16); /* There are only 16 SGIs */
>> +    cpumask_t all_others_mask;
>> +    ASSERT(sgi < 16); /* There are only 16 SGIs */
>>
>> -   dsb(sy);
>> +    cpumask_andnot(&all_others_mask, &cpu_possible_map,
>> cpumask_of(smp_processor_id()));
>>
>> -   GICD[GICD_SGIR] = GICD_SGI_TARGET_OTHERS
>> -       | sgi;
>> +    dsb(sy);
>> +    send_SGI_mask(&all_others_mask, sgi);
>>   }
>
>
> As I said in V1, this change breaks SMP boot with GICv2...
>
> GICD_SGI_TARGERT_OTHERS will send an SGI to every CPUs even if the CPU is
> not yet online (i.e. not registered by Xen). It's used during secondary boot
> (cpu_up_send_sgi).


cpumask_andnot(&all_others_mask,
&cpu_possible_map,cpumask_of(smp_processor_id()));

In my understanding, with the above statement, I am using
cpu_possible_map (all possible cpu's) which should
contains all the cpu possible cpu masks. so this is fine.

The issue could be in gic_send_sgi() call which is always "and" with
cpu_online_map

static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
{
    unsigned int mask = 0;
    cpumask_t online_mask;

    ASSERT(sgi < 16); /* There are only 16 SGIs */

    cpumask_and(&online_mask, cpumask, &cpu_online_map);
    mask = gic_cpu_mask(&online_mask);

    dsb(sy);

    GICD[GICD_SGIR] = GICD_SGI_TARGET_LIST
        | (mask<<GICD_SGI_TARGET_SHIFT)
        | sgi;
}

I think gic_send_sgi should be passed with absolute mask value
and get rid of and'ing with cpu_online_map

> Your solution won't work because send_SGI_mask only deal with online CPU.
>
> All the changes of send_sgi is more than splitting functions... this should
> be moved on another patch and you should justify the modifications.
>

I will check if I could make this fix in a separate patch.

> [..]
>
>
>>   int gic_route_irq_to_guest(struct domain *d, const struct dt_irq *irq,
>> @@ -921,10 +1046,10 @@ out:
>>       return retval;
>>   }
>>
>> -static void do_sgi(struct cpu_user_regs *regs, int othercpu, enum gic_sgi
>> sgi)
>> +static void do_sgi(struct cpu_user_regs *regs, enum gic_sgi sgi)
>
>
> Why did you drop the othercpu here?
>
othercpu is not used at all. and also othercpu is computed with
IAR fields #defines which is not required in this generic code.

> Again, please justify every change on the prototype of every functions. If
> it's not trivial, split in small patches...
>
> --
> Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Julien Grall]

On 04/09/2014 12:34 PM, Vijay Kilari wrote:
>> GICD_SGI_TARGERT_OTHERS will send an SGI to every CPUs even if the CPU is
>> not yet online (i.e. not registered by Xen). It's used during secondary boot
>> (cpu_up_send_sgi).
> 
> 
> cpumask_andnot(&all_others_mask,
> &cpu_possible_map,cpumask_of(smp_processor_id()));
> 
> In my understanding, with the above statement, I am using
> cpu_possible_map (all possible cpu's) which should
> contains all the cpu possible cpu masks. so this is fine.
> 
> The issue could be in gic_send_sgi() call which is always "and" with
> cpu_online_map
> 
> static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
> {
>     unsigned int mask = 0;
>     cpumask_t online_mask;
> 
>     ASSERT(sgi < 16); /* There are only 16 SGIs */
> 
>     cpumask_and(&online_mask, cpumask, &cpu_online_map);
>     mask = gic_cpu_mask(&online_mask);
> 
>     dsb(sy);
> 
>     GICD[GICD_SGIR] = GICD_SGI_TARGET_LIST
>         | (mask<<GICD_SGI_TARGET_SHIFT)
>         | sgi;
> }
> 
> I think gic_send_sgi should be passed with absolute mask value
> and get rid of and'ing with cpu_online_map

It won't work... gic_cpu_mask will translate the set of CPU ID into a
set of GIC CPU ID.

The mapping is initialized by gic_cpu_init. This function is only called
when the CPU is booting. So the mask is invalid for offline CPU...

>>> -static void do_sgi(struct cpu_user_regs *regs, int othercpu, enum gic_sgi
>>> sgi)
>>> +static void do_sgi(struct cpu_user_regs *regs, enum gic_sgi sgi)
>>
>>
>> Why did you drop the othercpu here?
>>
> othercpu is not used at all. and also othercpu is computed with
> IAR fields #defines which is not required in this generic code.

It should not be part of this patch. Please send a separate patch for
this change.

Regards,

-- 
Julien Grall

Re: [PATCH v2 05/15] xen/arm: segregate GIC low level functionality

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> GIC low level functionality is segregated into
> separate functions and are called using registered
> callback wherever required.
> 
> This helps to separate generic and hardware functionality
> later
> 
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> ---
>  xen/arch/arm/gic.c                |  362 ++++++++++++++++++++++++++++---------
>  xen/include/asm-arm/gic.h         |   50 +++++
>  xen/include/asm-arm/gic_v2_defs.h |   16 +-
>  3 files changed, 328 insertions(+), 100 deletions(-)
> 
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 64699e4..9f03135 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -57,8 +57,21 @@ static irq_desc_t irq_desc[NR_IRQS];
>  static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
>  static DEFINE_PER_CPU(uint64_t, lr_mask);
>  
> +static struct gic_hw_operations *gic_hw_ops;
> +static struct gic_hw_operations gic_ops;

Why two? Is the second one actually gic_v2_ops?

> +void register_gic_ops(struct gic_hw_operations *ops)
> +{
> +    gic_hw_ops = ops;
> +}
> +
> +void update_cpu_lr_mask(void)
> +{
> +    this_cpu(lr_mask) = 0ULL;
> +}

This looks more like init_cpu_lr_mask.

> +
>  static uint8_t nr_lrs;

Shouldn't everywhere be using gic_hw_ops->nr_lrs() rendering this
unused?

> -#define lr_all_full() (this_cpu(lr_mask) == ((1 << nr_lrs) - 1))
> +#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->nr_lrs()) - 1))

I think nr_lrs can just be an integer field of gic_hw_ops, rather than
using a function to return it, or perhaps a global in gic. set by the
lower level driver somehow. Unless nr_lrs is somehow for gicv3?

> +static void restore_state(struct vcpu *v)

I suspect that all of these callbacks shoujld actually be
gic_v2_restore_state or some such?

> @@ -230,7 +319,7 @@ static hw_irq_controller gic_guest_irq_type = {
>   * - needs to be called with a valid cpu_mask, ie each cpu in the mask has
>   * already called gic_cpu_init
>   */
> -static void gic_set_irq_properties(unsigned int irq, bool_t level,
> +static void gic_set_irq_property(unsigned int irq, bool_t level,
>                                     const cpumask_t *cpu_mask,
>                                     unsigned int priority)

Why did this need to become singular?

> +static struct dt_irq * gic_maintenance_irq(void)
> +{
> +    return &gic.maintenance;
> +}

This isn't using gic_hw_ops, so what is it for?


Ian.

[PATCH v2 06/15] xen/arm: move gic lock out of gic data structure

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

spinlock is used across generic and GIC low level
functions. Move this lock out of gic data.
This helps to separate generic and low level functions
later.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/gic.c |   34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 9f03135..77243cb 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -50,9 +50,9 @@ static struct {
     unsigned int lines;  /* Number of interrupts (SPIs + PPIs + SGIs) */
     struct dt_irq maintenance; /* IRQ maintenance */
     unsigned int cpus;
-    spinlock_t lock;
 } gic;
 
+static spinlock_t gic_lock;
 static irq_desc_t irq_desc[NR_IRQS];
 static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
 static DEFINE_PER_CPU(uint64_t, lr_mask);
@@ -231,12 +231,12 @@ static void gic_irq_enable(struct irq_desc *desc)
     unsigned long flags;
 
     spin_lock_irqsave(&desc->lock, flags);
-    spin_lock(&gic.lock);
+    spin_lock(&gic_lock);
     desc->status &= ~IRQ_DISABLED;
     dsb(sy);
     /* Enable routing */
     gic_hw_ops->enable_irq(irq);
-    spin_unlock(&gic.lock);
+    spin_unlock(&gic_lock);
     spin_unlock_irqrestore(&desc->lock, flags);
 }
 
@@ -246,11 +246,11 @@ static void gic_irq_disable(struct irq_desc *desc)
     unsigned long flags;
 
     spin_lock_irqsave(&desc->lock, flags);
-    spin_lock(&gic.lock);
+    spin_lock(&gic_lock);
     /* Disable routing */
     gic_hw_ops->disable_irq(irq);
     desc->status |= IRQ_DISABLED;
-    spin_unlock(&gic.lock);
+    spin_unlock(&gic_lock);
     spin_unlock_irqrestore(&desc->lock, flags);
 }
 
@@ -315,7 +315,7 @@ static hw_irq_controller gic_guest_irq_type = {
 };
 
 /*
- * - needs to be called with gic.lock held
+ * - needs to be called with gic_lock held
  * - needs to be called with a valid cpu_mask, ie each cpu in the mask has
  * already called gic_cpu_init
  */
@@ -373,9 +373,9 @@ static int gic_route_irq(unsigned int irq, bool_t level,
 
     desc->handler = &gic_host_irq_type;
 
-    spin_lock(&gic.lock);
+    spin_lock(&gic_lock);
     gic_set_irq_properties(irq, level, cpu_mask, priority);
-    spin_unlock(&gic.lock);
+    spin_unlock(&gic_lock);
 
     spin_unlock_irqrestore(&desc->lock, flags);
     return 0;
@@ -567,15 +567,15 @@ void __init gic_init(void)
     set_fixmap(FIXMAP_GICH, gic.hbase >> PAGE_SHIFT, DEV_SHARED);
 
     /* Global settings: interrupt distributor */
-    spin_lock_init(&gic.lock);
-    spin_lock(&gic.lock);
+    spin_lock_init(&gic_lock);
+    spin_lock(&gic_lock);
 
     gic_dist_init();
     gic_cpu_init();
     gic_hyp_init();
 
     register_gic_ops(&gic_ops);
-    spin_unlock(&gic.lock);
+    spin_unlock(&gic_lock);
 }
 
 static void gic_secondary_cpu_init(void)
@@ -636,9 +636,9 @@ void smp_send_state_dump(unsigned int cpu)
 /* Set up the per-CPU parts of the GIC for a secondary CPU */
 void __cpuinit gic_init_secondary_cpu(void)
 {
-    spin_lock(&gic.lock);
+    spin_lock(&gic_lock);
     gic_hw_ops->secondary_init();
-    spin_unlock(&gic.lock);
+    spin_unlock(&gic_lock);
 }
 
 /* Shut down the per-CPU GIC interface */
@@ -652,9 +652,9 @@ void gic_disable_cpu(void)
 {
     ASSERT(!local_irq_is_enabled());
 
-    spin_lock(&gic.lock);
+    spin_lock(&gic_lock);
     gic_hw_ops->disable_interface();
-    spin_unlock(&gic.lock);
+    spin_unlock(&gic_lock);
 }
 
 void gic_route_ppis(void)
@@ -1020,7 +1020,7 @@ int gic_route_irq_to_guest(struct domain *d, const struct dt_irq *irq,
     action->free_on_release = 1;
 
     spin_lock_irqsave(&desc->lock, flags);
-    spin_lock(&gic.lock);
+    spin_lock(&gic_lock);
 
     desc->handler = &gic_guest_irq_type;
     desc->status |= IRQ_GUEST;
@@ -1041,7 +1041,7 @@ int gic_route_irq_to_guest(struct domain *d, const struct dt_irq *irq,
     p->desc = desc;
 
 out:
-    spin_unlock(&gic.lock);
+    spin_unlock(&gic_lock);
     spin_unlock_irqrestore(&desc->lock, flags);
     return retval;
 }
-- 
1.7.9.5

Re: [PATCH v2 06/15] xen/arm: move gic lock out of gic data structure

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> spinlock is used across generic and GIC low level
> functions. Move this lock out of gic data.
> This helps to separate generic and low level functions
> later.

Is there a subsequent patch which exposes this lock to  gic-{v2,v3}.c? I
grepped but couldn't find it. AFAIK the pattern is always:
	common: take lock
	common: call per gic callback
        	per gic callback: do stuff assuming lock held
	common: release lock

IOW the lock is only used from the common code?

Ian.

Re: [PATCH v2 06/15] xen/arm: move gic lock out of gic data structure

[Vijay Kilari]

On Thu, Apr 10, 2014 at 2:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>
>> spinlock is used across generic and GIC low level
>> functions. Move this lock out of gic data.
>> This helps to separate generic and low level functions
>> later.
>
> Is there a subsequent patch which exposes this lock to  gic-{v2,v3}.c? I
> grepped but couldn't find it. AFAIK the pattern is always:
>         common: take lock
>         common: call per gic callback
>                 per gic callback: do stuff assuming lock held
>         common: release lock
>
> IOW the lock is only used from the common code?
>
Yes, lock is only used in common code and your pattern is correct.
I made it separate patch as this is not trivial change

> Ian.
>

Re: [PATCH v2 06/15] xen/arm: move gic lock out of gic data structure

[Ian Campbell]

On Thu, 2014-04-10 at 14:54 +0530, Vijay Kilari wrote:
> On Thu, Apr 10, 2014 at 2:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> >> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> >>
> >> spinlock is used across generic and GIC low level
> >> functions. Move this lock out of gic data.
> >> This helps to separate generic and low level functions
> >> later.
> >
> > Is there a subsequent patch which exposes this lock to  gic-{v2,v3}.c? I
> > grepped but couldn't find it. AFAIK the pattern is always:
> >         common: take lock
> >         common: call per gic callback
> >                 per gic callback: do stuff assuming lock held
> >         common: release lock
> >
> > IOW the lock is only used from the common code?
> >
> Yes, lock is only used in common code and your pattern is correct.

Why is gic.lock not alright?

> I made it separate patch as this is not trivial change
> 
> > Ian.
> >

Re: [PATCH v2 06/15] xen/arm: move gic lock out of gic data structure

[Vijay Kilari]

On Thu, Apr 10, 2014 at 3:32 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-04-10 at 14:54 +0530, Vijay Kilari wrote:
>> On Thu, Apr 10, 2014 at 2:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>> >> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>> >>
>> >> spinlock is used across generic and GIC low level
>> >> functions. Move this lock out of gic data.
>> >> This helps to separate generic and low level functions
>> >> later.
>> >
>> > Is there a subsequent patch which exposes this lock to  gic-{v2,v3}.c? I
>> > grepped but couldn't find it. AFAIK the pattern is always:
>> >         common: take lock
>> >         common: call per gic callback
>> >                 per gic callback: do stuff assuming lock held
>> >         common: release lock
>> >
>> > IOW the lock is only used from the common code?
>> >
>> Yes, lock is only used in common code and your pattern is correct.
>
> Why is gic.lock not alright?
>
gic.lock also protects some common code as well in the existing v2 driver.
So it is retained in common code and locking mechanism is not changed.

>> I made it separate patch as this is not trivial change
>>
>> > Ian.
>> >
>
>

Re: [PATCH v2 06/15] xen/arm: move gic lock out of gic data structure

[Ian Campbell]

On Thu, 2014-04-10 at 15:42 +0530, Vijay Kilari wrote:
> On Thu, Apr 10, 2014 at 3:32 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Thu, 2014-04-10 at 14:54 +0530, Vijay Kilari wrote:
> >> On Thu, Apr 10, 2014 at 2:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> >> >> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> >> >>
> >> >> spinlock is used across generic and GIC low level
> >> >> functions. Move this lock out of gic data.
> >> >> This helps to separate generic and low level functions
> >> >> later.
> >> >
> >> > Is there a subsequent patch which exposes this lock to  gic-{v2,v3}.c? I
> >> > grepped but couldn't find it. AFAIK the pattern is always:
> >> >         common: take lock
> >> >         common: call per gic callback
> >> >                 per gic callback: do stuff assuming lock held
> >> >         common: release lock
> >> >
> >> > IOW the lock is only used from the common code?
> >> >
> >> Yes, lock is only used in common code and your pattern is correct.
> >
> > Why is gic.lock not alright?
> >
> gic.lock also protects some common code as well in the existing v2 driver.
> So it is retained in common code and locking mechanism is not changed.

Ah. I was missing that the struct gic was going to become v2 specific
and move to gic-v2.c, but this field needs to stay behind in the common
code.

Can you make that explicit in your commit log please.

Ian,

[PATCH v2 07/15] xen/arm: split gic driver into generic and gic-v2 driver

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Existing GIC driver has both generic code and hw specific
code. Segregate GIC low level driver into gic-v2.c and
keep generic code in existing gic.c file

GIC v2 driver registers required functions
to generic GIC driver. This helps to plug in next version
of GIC drivers like GIC v3.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/Makefile     |    2 +-
 xen/arch/arm/gic-v2.c     |  538 +++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic.c        |  492 +----------------------------------------
 xen/include/asm-arm/gic.h |    1 +
 4 files changed, 542 insertions(+), 491 deletions(-)

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 63e0460..969ee52 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -10,7 +10,7 @@ obj-y += vpsci.o
 obj-y += domctl.o
 obj-y += sysctl.o
 obj-y += domain_build.o
-obj-y += gic.o
+obj-y += gic.o gic-v2.o
 obj-y += io.o
 obj-y += irq.o
 obj-y += kernel.o
diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
new file mode 100644
index 0000000..e39099f
--- /dev/null
+++ b/xen/arch/arm/gic-v2.c
@@ -0,0 +1,538 @@
+/*
+ * xen/arch/arm/gic-v2.c
+ *
+ * ARM Generic Interrupt Controller support v2
+ *
+ * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
+ * Copyright (c) 2014 Cavium Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/config.h>
+#include <xen/lib.h>
+#include <xen/init.h>
+#include <xen/mm.h>
+#include <xen/irq.h>
+#include <xen/sched.h>
+#include <xen/errno.h>
+#include <xen/serial.h>
+#include <xen/softirq.h>
+#include <xen/list.h>
+#include <xen/device_tree.h>
+#include <asm/p2m.h>
+#include <asm/domain.h>
+#include <asm/platform.h>
+
+#include <asm/gic_v2_defs.h>
+#include <asm/gic.h>
+
+/* Access to the GIC Distributor registers through the fixmap */
+#define GICD ((volatile uint32_t *) FIXMAP_ADDR(FIXMAP_GICD))
+#define GICC ((volatile uint32_t *) FIXMAP_ADDR(FIXMAP_GICC1))
+#define GICH ((volatile uint32_t *) FIXMAP_ADDR(FIXMAP_GICH))
+
+/* Global state */
+static struct {
+    paddr_t dbase;       /* Address of distributor registers */
+    paddr_t cbase;       /* Address of CPU interface registers */
+    paddr_t hbase;       /* Address of virtual interface registers */
+    paddr_t vbase;       /* Address of virtual cpu interface registers */
+    unsigned int lines;  /* Number of interrupts (SPIs + PPIs + SGIs) */
+    struct dt_irq maintenance; /* IRQ maintenance */
+    unsigned int cpus;
+} gic;
+
+static struct gic_hw_operations gic_ops;
+
+static uint8_t nr_lrs;
+
+/* The GIC mapping of CPU interfaces does not necessarily match the
+ * logical CPU numbering. Let's use mapping as returned by the GIC
+ * itself
+ */
+static DEFINE_PER_CPU(u8, gic_cpu_id);
+
+/* Maximum cpu interface per GIC */
+#define NR_GIC_CPU_IF 8
+
+static unsigned int gic_cpu_mask(const cpumask_t *cpumask)
+{
+    unsigned int cpu;
+    unsigned int mask = 0;
+    cpumask_t possible_mask;
+
+    cpumask_and(&possible_mask, cpumask, &cpu_possible_map);
+    for_each_cpu(cpu, &possible_mask)
+    {
+        ASSERT(cpu < NR_GIC_CPU_IF);
+        mask |= per_cpu(gic_cpu_id, cpu);
+    }
+
+    return mask;
+}
+
+static unsigned int gic_nr_lines(void)
+{
+    return gic.lines;
+}
+
+static unsigned int gic_nr_lrs(void)
+{
+    return nr_lrs;
+}
+
+static int gic_state_init(struct vcpu *v)
+{
+     v->arch.gic_state = xzalloc(struct gic_state_data);
+     if ( !v->arch.gic_state )
+        return -ENOMEM;
+     return 0;
+}
+
+static void save_state(struct vcpu *v)
+{
+    int i;
+
+    /* No need for spinlocks here because interrupts are disabled around
+     * this call and it only accesses struct vcpu fields that cannot be
+     * accessed simultaneously by another pCPU.
+     */
+    for ( i = 0; i < nr_lrs; i++ )
+        v->arch.gic_state->gic_lr[i] = GICH[GICH_LR + i];
+    v->arch.gic_state->gic_apr = GICH[GICH_APR];
+    v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];
+    /* Disable until next VCPU scheduled */
+    GICH[GICH_HCR] = 0;
+}
+
+static void restore_state(struct vcpu *v)
+{
+    int i;
+
+    for ( i = 0; i < nr_lrs; i++ )
+        GICH[GICH_LR + i] = v->arch.gic_state->gic_lr[i];
+    GICH[GICH_APR] = v->arch.gic_state->gic_apr;
+    GICH[GICH_VMCR] = v->arch.gic_state->gic_vmcr;
+    GICH[GICH_HCR] = GICH_HCR_EN;
+}
+
+static void gic_dump_state(struct vcpu *v)
+{
+    int i;
+    if ( v == current )
+    {
+        for ( i = 0; i < nr_lrs; i++ )
+            printk("   HW_LR[%d]=%x\n", i, GICH[GICH_LR + i]);
+    } else {
+        for ( i = 0; i < nr_lrs; i++ )
+            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->gic_lr[i]);
+    }
+}
+
+static void gic_enable_irq(int irq)
+{
+    /* Enable routing */
+    GICD[GICD_ISENABLER + irq / 32] = (1u << (irq % 32));
+}
+
+static void gic_disable_irq(int irq)
+{
+    /* Disable routing */
+    GICD[GICD_ICENABLER + irq / 32] = (1u << (irq % 32));
+}
+
+static void gic_eoi_irq(int irq)
+{
+    /* Lower the priority */
+    GICC[GICC_EOIR] = irq;
+}
+
+static void gic_dir_irq(int irq)
+{
+    /* Deactivate */
+    GICC[GICC_DIR] = irq;
+}
+
+static unsigned int gic_ack_irq(void)
+{
+    return (GICC[GICC_IAR] & GICC_IA_IRQ);
+}
+
+/*
+ * - needs to be called with gic_lock held
+ * - needs to be called with a valid cpu_mask, ie each cpu in the mask has
+ * already called gic_cpu_init
+ */
+static void gic_set_irq_property(unsigned int irq, bool_t level,
+                                   const cpumask_t *cpu_mask,
+                                   unsigned int priority)
+{
+    volatile unsigned char *bytereg;
+    uint32_t cfg, edgebit;
+    unsigned int mask = gic_cpu_mask(cpu_mask);
+
+    /* Set edge / level */
+    cfg = GICD[GICD_ICFGR + irq / 16];
+    edgebit = 2u << (2 * (irq % 16));
+    if ( level )
+        cfg &= ~edgebit;
+    else
+        cfg |= edgebit;
+    GICD[GICD_ICFGR + irq / 16] = cfg;
+
+    /* Set target CPU mask (RAZ/WI on uniprocessor) */
+    bytereg = (unsigned char *) (GICD + GICD_ITARGETSR);
+    bytereg[irq] = mask;
+
+    /* Set priority */
+    bytereg = (unsigned char *) (GICD + GICD_IPRIORITYR);
+    bytereg[irq] = priority;
+
+}
+
+static void __init gic_dist_init(void)
+{
+    uint32_t type;
+    uint32_t cpumask;
+    int i;
+
+    cpumask = GICD[GICD_ITARGETSR] & 0xff;
+    cpumask |= cpumask << 8;
+    cpumask |= cpumask << 16;
+
+    /* Disable the distributor */
+    GICD[GICD_CTLR] = 0;
+
+    type = GICD[GICD_TYPER];
+    gic.lines = 32 * ((type & GICD_TYPE_LINES) + 1);
+    gic.cpus = 1 + ((type & GICD_TYPE_CPUS) >> 5);
+    printk("GIC: %d lines, %d cpu%s%s (IID %8.8x).\n",
+           gic.lines, gic.cpus, (gic.cpus == 1) ? "" : "s",
+           (type & GICD_TYPE_SEC) ? ", secure" : "",
+           GICD[GICD_IIDR]);
+
+    /* Default all global IRQs to level, active low */
+    for ( i = 32; i < gic.lines; i += 16 )
+        GICD[GICD_ICFGR + i / 16] = 0x0;
+
+    /* Route all global IRQs to this CPU */
+    for ( i = 32; i < gic.lines; i += 4 )
+        GICD[GICD_ITARGETSR + i / 4] = cpumask;
+
+    /* Default priority for global interrupts */
+    for ( i = 32; i < gic.lines; i += 4 )
+        GICD[GICD_IPRIORITYR + i / 4] =
+            GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ;
+
+    /* Disable all global interrupts */
+    for ( i = 32; i < gic.lines; i += 32 )
+        GICD[GICD_ICENABLER + i / 32] = (uint32_t)~0ul;
+
+    /* Turn on the distributor */
+    GICD[GICD_CTLR] = GICD_CTL_ENABLE;
+}
+
+static void __cpuinit gic_cpu_init(void)
+{
+    int i;
+
+    this_cpu(gic_cpu_id) = GICD[GICD_ITARGETSR] & 0xff;
+
+    /* The first 32 interrupts (PPI and SGI) are banked per-cpu, so
+     * even though they are controlled with GICD registers, they must
+     * be set up here with the other per-cpu state. */
+    GICD[GICD_ICENABLER] = 0xffff0000; /* Disable all PPI */
+    GICD[GICD_ISENABLER] = 0x0000ffff; /* Enable all SGI */
+    /* Set SGI priorities */
+    for ( i = 0; i < 16; i += 4 )
+        GICD[GICD_IPRIORITYR + i / 4] =
+            GIC_PRI_IPI<<24 | GIC_PRI_IPI<<16 | GIC_PRI_IPI<<8 | GIC_PRI_IPI;
+    /* Set PPI priorities */
+    for ( i = 16; i < 32; i += 4 )
+        GICD[GICD_IPRIORITYR + i / 4] =
+            GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ;
+
+    /* Local settings: interface controller */
+    GICC[GICC_PMR] = 0xff;                /* Don't mask by priority */
+    GICC[GICC_BPR] = 0;                   /* Finest granularity of priority */
+    GICC[GICC_CTLR] = GICC_CTL_ENABLE|GICC_CTL_EOI;    /* Turn on delivery */
+}
+
+static void gic_cpu_disable(void)
+{
+    GICC[GICC_CTLR] = 0;
+}
+
+static void __cpuinit gic_hyp_init(void)
+{
+    uint32_t vtr;
+
+    vtr = GICH[GICH_VTR];
+    nr_lrs  = (vtr & GICH_VTR_NRLRGS) + 1;
+
+    GICH[GICH_MISR] = GICH_MISR_EOI;
+    update_cpu_lr_mask();
+}
+
+static void __cpuinit gic_hyp_disable(void)
+{
+    GICH[GICH_HCR] = 0;
+}
+
+/* Set up the GIC */
+void __init gicv2_init(void)
+{
+    static const struct dt_device_match gic_ids[] __initconst =
+    {
+        DT_MATCH_GIC,
+        { /* sentinel */ },
+    };
+    struct dt_device_node *node;
+    int res;
+
+    node = dt_find_interrupt_controller(gic_ids);
+    if ( !node )
+        panic("Unable to find compatible GIC in the device tree");
+
+    dt_device_set_used_by(node, DOMID_XEN);
+
+    res = dt_device_get_address(node, 0, &gic.dbase, NULL);
+    if ( res || !gic.dbase || (gic.dbase & ~PAGE_MASK) )
+        panic("GIC: Cannot find a valid address for the distributor");
+
+    res = dt_device_get_address(node, 1, &gic.cbase, NULL);
+    if ( res || !gic.cbase || (gic.cbase & ~PAGE_MASK) )
+        panic("GIC: Cannot find a valid address for the CPU");
+
+    res = dt_device_get_address(node, 2, &gic.hbase, NULL);
+    if ( res || !gic.hbase || (gic.hbase & ~PAGE_MASK) )
+        panic("GIC: Cannot find a valid address for the hypervisor");
+
+    res = dt_device_get_address(node, 3, &gic.vbase, NULL);
+    if ( res || !gic.vbase || (gic.vbase & ~PAGE_MASK) )
+        panic("GIC: Cannot find a valid address for the virtual CPU");
+
+    res = dt_device_get_irq(node, 0, &gic.maintenance);
+    if ( res )
+        panic("GIC: Cannot find the maintenance IRQ");
+
+    /* Set the GIC as the primary interrupt controller */
+    dt_interrupt_controller = node;
+
+    /* TODO: Add check on distributor, cpu size */
+
+    printk("GIC initialization:\n"
+              "        gic_dist_addr=%"PRIpaddr"\n"
+              "        gic_cpu_addr=%"PRIpaddr"\n"
+              "        gic_hyp_addr=%"PRIpaddr"\n"
+              "        gic_vcpu_addr=%"PRIpaddr"\n"
+              "        gic_maintenance_irq=%u\n",
+              gic.dbase, gic.cbase, gic.hbase, gic.vbase,
+              gic.maintenance.irq);
+
+    if ( (gic.dbase & ~PAGE_MASK) || (gic.cbase & ~PAGE_MASK) ||
+         (gic.hbase & ~PAGE_MASK) || (gic.vbase & ~PAGE_MASK) )
+        panic("GIC interfaces not page aligned");
+
+    set_fixmap(FIXMAP_GICD, gic.dbase >> PAGE_SHIFT, DEV_SHARED);
+    BUILD_BUG_ON(FIXMAP_ADDR(FIXMAP_GICC1) !=
+                 FIXMAP_ADDR(FIXMAP_GICC2)-PAGE_SIZE);
+    set_fixmap(FIXMAP_GICC1, gic.cbase >> PAGE_SHIFT, DEV_SHARED);
+    if ( platform_has_quirk(PLATFORM_QUIRK_GIC_64K_STRIDE) )
+        set_fixmap(FIXMAP_GICC2, (gic.cbase >> PAGE_SHIFT) + 0x10, DEV_SHARED);
+    else
+        set_fixmap(FIXMAP_GICC2, (gic.cbase >> PAGE_SHIFT) + 0x1, DEV_SHARED);
+    set_fixmap(FIXMAP_GICH, gic.hbase >> PAGE_SHIFT, DEV_SHARED);
+
+    /* Global settings: interrupt distributor */
+
+    gic_dist_init();
+    gic_cpu_init();
+    gic_hyp_init();
+
+    register_gic_ops(&gic_ops);
+}
+
+static void gic_secondary_cpu_init(void)
+{
+    gic_cpu_init();
+    gic_hyp_init();
+}
+
+static struct dt_irq * gic_maintenance_irq(void)
+{
+    return &gic.maintenance;
+}
+
+static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
+{
+    unsigned int mask = 0;
+    cpumask_t online_mask;
+
+    ASSERT(sgi < 16); /* There are only 16 SGIs */
+
+    cpumask_and(&online_mask, cpumask, &cpu_online_map);
+    mask = gic_cpu_mask(&online_mask);
+
+    dsb(sy);
+
+    GICD[GICD_SGIR] = GICD_SGI_TARGET_LIST
+        | (mask<<GICD_SGI_TARGET_SHIFT)
+        | sgi;
+}
+
+/* Shut down the per-CPU GIC interface */
+static void gic_disable_interface(void)
+{
+    gic_cpu_disable();
+    gic_hyp_disable();
+}
+
+static void gic_update_lr(int lr, struct pending_irq *p, unsigned int state)
+{
+    int maintenance_int = GICH_LR_MAINTENANCE_IRQ;
+
+    BUG_ON(lr >= nr_lrs);
+    BUG_ON(lr < 0);
+    BUG_ON(state & ~(GICH_LR_STATE_MASK<<GICH_LR_STATE_SHIFT));
+
+    GICH[GICH_LR + lr] = ((state & 0x3) << GICH_LR_STATE_SHIFT) |
+        maintenance_int |
+        ((p->priority >> 3) << GICH_LR_PRIORITY_SHIFT) |
+        ((p->irq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT);
+}
+
+static void gic_clear_lr(int lr)
+{
+    GICH[GICH_LR + lr] = 0;
+}
+
+static int gicv_init(struct domain *d)
+{
+    int ret;
+
+    /*
+     * Domain 0 gets the hardware address.
+     * Guests get the virtual platform layout.
+     */
+    if ( d->domain_id == 0 )
+    {
+        d->arch.vgic.dbase = gic.dbase;
+        d->arch.vgic.cbase = gic.cbase;
+    }
+    else
+    {
+        d->arch.vgic.dbase = GUEST_GICD_BASE;
+        d->arch.vgic.cbase = GUEST_GICC_BASE;
+    }
+
+    d->arch.vgic.nr_lines = 0;
+
+    /*
+     * Map the gic virtual cpu interface in the gic cpu interface
+     * region of the guest.
+     *
+     * The second page is always mapped at +4K irrespective of the
+     * GIC_64K_STRIDE quirk. The DTB passed to the guest reflects this.
+     */
+    ret = map_mmio_regions(d, d->arch.vgic.cbase,
+                           d->arch.vgic.cbase + PAGE_SIZE - 1,
+                           gic.vbase);
+    if ( ret )
+        return ret;
+
+    if ( !platform_has_quirk(PLATFORM_QUIRK_GIC_64K_STRIDE) )
+        ret = map_mmio_regions(d, d->arch.vgic.cbase + PAGE_SIZE,
+                               d->arch.vgic.cbase + (2 * PAGE_SIZE) - 1,
+                               gic.vbase + PAGE_SIZE);
+    else
+        ret = map_mmio_regions(d, d->arch.vgic.cbase + PAGE_SIZE,
+                               d->arch.vgic.cbase + (2 * PAGE_SIZE) - 1,
+                               gic.vbase + 16*PAGE_SIZE);
+
+    return ret;
+
+}
+
+static void gic_read_lr(int lr, struct gic_lr *lr_reg)
+{
+    uint32_t lrv;
+
+    lrv = GICH[GICH_LR + lr];
+    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
+    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
+    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
+    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
+    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
+    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
+}
+
+static void gic_write_lr(int lr, struct gic_lr *lr_reg)
+{
+    uint32_t lrv = 0;
+    lrv = ( ((lr_reg->pirq & GICH_LR_PHYSICAL_MASK) << GICH_LR_PHYSICAL_SHIFT)  |
+            ((lr_reg->virq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT)   |
+            ((uint32_t)(lr_reg->priority & GICH_LR_PRIORITY_MASK) << GICH_LR_PRIORITY_SHIFT) |
+            ((uint32_t)(lr_reg->state & GICH_LR_STATE_MASK) << GICH_LR_STATE_SHIFT) |
+            ((uint32_t)(lr_reg->hw_status & GICH_LR_HW_MASK)  << GICH_LR_HW_SHIFT)  |
+            ((uint32_t)(lr_reg->grp & GICH_LR_GRP_MASK) << GICH_LR_GRP_SHIFT) );
+
+    GICH[GICH_LR + lr] = lrv;
+}
+
+static void gic_hcr_status(uint32_t flag, uint8_t status)
+{
+    if ( status )
+      GICH[GICH_HCR] |= flag;
+    else
+      GICH[GICH_HCR] &= ~flag;
+}
+
+static unsigned int gic_read_vmcr_priority(void)
+{
+   return (GICH[GICH_VMCR] >> GICH_VMCR_PRIORITY_SHIFT) & GICH_VMCR_PRIORITY_MASK;
+}
+
+static struct gic_hw_operations gic_ops = {
+    .nr_lines            = gic_nr_lines,
+    .nr_lrs              = gic_nr_lrs,
+    .secondary_init      = gic_secondary_cpu_init,
+    .get_maintenance_irq = gic_maintenance_irq,
+    .state_init          = gic_state_init,
+    .save_state          = save_state,
+    .restore_state       = restore_state,
+    .dump_state          = gic_dump_state,
+    .gicv_setup          = gicv_init,
+    .enable_irq          = gic_enable_irq,
+    .disable_irq         = gic_disable_irq,
+    .eoi_irq             = gic_eoi_irq,
+    .deactivate_irq      = gic_dir_irq,
+    .ack_irq             = gic_ack_irq,
+    .set_irq_property    = gic_set_irq_property,
+    .send_sgi            = gic_send_sgi,
+    .disable_interface   = gic_disable_interface,
+    .update_lr           = gic_update_lr,
+    .update_hcr_status   = gic_hcr_status,
+    .clear_lr            = gic_clear_lr,
+    .read_lr             = gic_read_lr,
+    .write_lr            = gic_write_lr,
+    .read_vmcr_priority  = gic_read_vmcr_priority,
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 77243cb..eb2f2d4 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -32,33 +32,16 @@
 #include <asm/domain.h>
 #include <asm/platform.h>
 
-#include <asm/gic_v2_defs.h>
 #include <asm/gic.h>
 
-/* Access to the GIC Distributor registers through the fixmap */
-#define GICD ((volatile uint32_t *) FIXMAP_ADDR(FIXMAP_GICD))
-#define GICC ((volatile uint32_t *) FIXMAP_ADDR(FIXMAP_GICC1))
-#define GICH ((volatile uint32_t *) FIXMAP_ADDR(FIXMAP_GICH))
 static void gic_restore_pending_irqs(struct vcpu *v);
 
-/* Global state */
-static struct {
-    paddr_t dbase;       /* Address of distributor registers */
-    paddr_t cbase;       /* Address of CPU interface registers */
-    paddr_t hbase;       /* Address of virtual interface registers */
-    paddr_t vbase;       /* Address of virtual cpu interface registers */
-    unsigned int lines;  /* Number of interrupts (SPIs + PPIs + SGIs) */
-    struct dt_irq maintenance; /* IRQ maintenance */
-    unsigned int cpus;
-} gic;
-
 static spinlock_t gic_lock;
 static irq_desc_t irq_desc[NR_IRQS];
 static DEFINE_PER_CPU(irq_desc_t[NR_LOCAL_IRQS], local_irq_desc);
 static DEFINE_PER_CPU(uint64_t, lr_mask);
 
 static struct gic_hw_operations *gic_hw_ops;
-static struct gic_hw_operations gic_ops;
 
 void register_gic_ops(struct gic_hw_operations *ops)
 {
@@ -70,128 +53,15 @@ void update_cpu_lr_mask(void)
     this_cpu(lr_mask) = 0ULL;
 }
 
-static uint8_t nr_lrs;
 #define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->nr_lrs()) - 1))
 
-/* The GIC mapping of CPU interfaces does not necessarily match the
- * logical CPU numbering. Let's use mapping as returned by the GIC
- * itself
- */
-static DEFINE_PER_CPU(u8, gic_cpu_id);
-
-/* Maximum cpu interface per GIC */
-#define NR_GIC_CPU_IF 8
-
 static void gic_clear_one_lr(struct vcpu *v, int i);
 
-static unsigned int gic_cpu_mask(const cpumask_t *cpumask)
-{
-    unsigned int cpu;
-    unsigned int mask = 0;
-    cpumask_t possible_mask;
-
-    cpumask_and(&possible_mask, cpumask, &cpu_possible_map);
-    for_each_cpu(cpu, &possible_mask)
-    {
-        ASSERT(cpu < NR_GIC_CPU_IF);
-        mask |= per_cpu(gic_cpu_id, cpu);
-    }
-
-    return mask;
-}
-
 unsigned int gic_number_lines(void)
 {
     return gic_hw_ops->nr_lines();
 }
 
-static unsigned int gic_nr_lines(void)
-{
-    return gic.lines;
-}
-
-static unsigned int gic_nr_lrs(void)
-{
-    return nr_lrs;
-}
-
-static int gic_state_init(struct vcpu *v)
-{
-     v->arch.gic_state = xzalloc(struct gic_state_data);
-     if(!v->arch.gic_state)
-        return -ENOMEM;
-     return 0;
-}
-
-static void save_state(struct vcpu *v)
-{
-    int i;
-
-    /* No need for spinlocks here because interrupts are disabled around
-     * this call and it only accesses struct vcpu fields that cannot be
-     * accessed simultaneously by another pCPU.
-     */
-    for ( i = 0; i < nr_lrs; i++)
-        v->arch.gic_state->gic_lr[i] = GICH[GICH_LR + i];
-    v->arch.gic_state->gic_apr = GICH[GICH_APR];
-    v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];
-    /* Disable until next VCPU scheduled */
-    GICH[GICH_HCR] = 0;
-}
-
-static void restore_state(struct vcpu *v)
-{
-    int i;
-
-    for ( i = 0; i < nr_lrs; i++ )
-        GICH[GICH_LR + i] = v->arch.gic_state->gic_lr[i];
-    GICH[GICH_APR] = v->arch.gic_state->gic_apr;
-    GICH[GICH_VMCR] = v->arch.gic_state->gic_vmcr;
-    GICH[GICH_HCR] = GICH_HCR_EN;
-}
-
-static void gic_dump_state(struct vcpu *v)
-{
-    int i;
-    if ( v == current )
-    {
-        for ( i = 0; i < nr_lrs; i++ )
-            printk("   HW_LR[%d]=%x\n", i, GICH[GICH_LR + i]);
-    } else {
-        for ( i = 0; i < nr_lrs; i++ )
-            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->gic_lr[i]);
-    }
-}
-
-static void gic_enable_irq(int irq)
-{
-    /* Enable routing */
-    GICD[GICD_ISENABLER + irq / 32] = (1u << (irq % 32));
-}
-
-static void gic_disable_irq(int irq)
-{
-    /* Disable routing */
-    GICD[GICD_ICENABLER + irq / 32] = (1u << (irq % 32));
-}
-
-static void gic_eoi_irq(int irq)
-{
-    /* Lower the priority */
-    GICC[GICC_EOIR] = irq;
-}
-
-static void gic_dir_irq(int irq)
-{
-    /* Deactivate */
-    GICC[GICC_DIR] = irq;
-}
-
-static unsigned int gic_ack_irq(void)
-{
-    return (GICC[GICC_IAR] & GICC_IA_IRQ);
-}
-
 irq_desc_t *__irq_to_desc(int irq)
 {
     if (irq < NR_LOCAL_IRQS) return &this_cpu(local_irq_desc)[irq];
@@ -314,38 +184,6 @@ static hw_irq_controller gic_guest_irq_type = {
     .set_affinity = gic_irq_set_affinity,
 };
 
-/*
- * - needs to be called with gic_lock held
- * - needs to be called with a valid cpu_mask, ie each cpu in the mask has
- * already called gic_cpu_init
- */
-static void gic_set_irq_property(unsigned int irq, bool_t level,
-                                   const cpumask_t *cpu_mask,
-                                   unsigned int priority)
-{
-    volatile unsigned char *bytereg;
-    uint32_t cfg, edgebit;
-    unsigned int mask = gic_cpu_mask(cpu_mask);
-
-    /* Set edge / level */
-    cfg = GICD[GICD_ICFGR + irq / 16];
-    edgebit = 2u << (2 * (irq % 16));
-    if ( level )
-        cfg &= ~edgebit;
-    else
-        cfg |= edgebit;
-    GICD[GICD_ICFGR + irq / 16] = cfg;
-
-    /* Set target CPU mask (RAZ/WI on uniprocessor) */
-    bytereg = (unsigned char *) (GICD + GICD_ITARGETSR);
-    bytereg[irq] = mask;
-
-    /* Set priority */
-    bytereg = (unsigned char *) (GICD + GICD_IPRIORITYR);
-    bytereg[irq] = priority;
-
-}
-
 static void gic_set_irq_properties(unsigned int irq, bool_t level,
                                    const cpumask_t *cpu_mask,
                                    unsigned int priority)
@@ -361,7 +199,7 @@ static int gic_route_irq(unsigned int irq, bool_t level,
     unsigned long flags;
 
     ASSERT(priority <= 0xff);     /* Only 8 bits of priority */
-    ASSERT(irq < gic.lines);      /* Can't route interrupts that don't exist */
+    ASSERT(irq < gic_number_lines());
 
     if ( desc->action != NULL )
         return -EBUSY;
@@ -392,95 +230,6 @@ void gic_route_dt_irq(const struct dt_irq *irq, const cpumask_t *cpu_mask,
     gic_route_irq(irq->irq, level, cpu_mask, priority);
 }
 
-static void __init gic_dist_init(void)
-{
-    uint32_t type;
-    uint32_t cpumask;
-    int i;
-
-    cpumask = GICD[GICD_ITARGETSR] & 0xff;
-    cpumask |= cpumask << 8;
-    cpumask |= cpumask << 16;
-
-    /* Disable the distributor */
-    GICD[GICD_CTLR] = 0;
-
-    type = GICD[GICD_TYPER];
-    gic.lines = 32 * ((type & GICD_TYPE_LINES) + 1);
-    gic.cpus = 1 + ((type & GICD_TYPE_CPUS) >> 5);
-    printk("GIC: %d lines, %d cpu%s%s (IID %8.8x).\n",
-           gic.lines, gic.cpus, (gic.cpus == 1) ? "" : "s",
-           (type & GICD_TYPE_SEC) ? ", secure" : "",
-           GICD[GICD_IIDR]);
-
-    /* Default all global IRQs to level, active low */
-    for ( i = 32; i < gic.lines; i += 16 )
-        GICD[GICD_ICFGR + i / 16] = 0x0;
-
-    /* Route all global IRQs to this CPU */
-    for ( i = 32; i < gic.lines; i += 4 )
-        GICD[GICD_ITARGETSR + i / 4] = cpumask;
-
-    /* Default priority for global interrupts */
-    for ( i = 32; i < gic.lines; i += 4 )
-        GICD[GICD_IPRIORITYR + i / 4] =
-            GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ;
-
-    /* Disable all global interrupts */
-    for ( i = 32; i < gic.lines; i += 32 )
-        GICD[GICD_ICENABLER + i / 32] = (uint32_t)~0ul;
-
-    /* Turn on the distributor */
-    GICD[GICD_CTLR] = GICD_CTL_ENABLE;
-}
-
-static void __cpuinit gic_cpu_init(void)
-{
-    int i;
-
-    this_cpu(gic_cpu_id) = GICD[GICD_ITARGETSR] & 0xff;
-
-    /* The first 32 interrupts (PPI and SGI) are banked per-cpu, so
-     * even though they are controlled with GICD registers, they must
-     * be set up here with the other per-cpu state. */
-    GICD[GICD_ICENABLER] = 0xffff0000; /* Disable all PPI */
-    GICD[GICD_ISENABLER] = 0x0000ffff; /* Enable all SGI */
-    /* Set SGI priorities */
-    for (i = 0; i < 16; i += 4)
-        GICD[GICD_IPRIORITYR + i / 4] =
-            GIC_PRI_IPI<<24 | GIC_PRI_IPI<<16 | GIC_PRI_IPI<<8 | GIC_PRI_IPI;
-    /* Set PPI priorities */
-    for (i = 16; i < 32; i += 4)
-        GICD[GICD_IPRIORITYR + i / 4] =
-            GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ;
-
-    /* Local settings: interface controller */
-    GICC[GICC_PMR] = 0xff;                /* Don't mask by priority */
-    GICC[GICC_BPR] = 0;                   /* Finest granularity of priority */
-    GICC[GICC_CTLR] = GICC_CTL_ENABLE|GICC_CTL_EOI;    /* Turn on delivery */
-}
-
-static void gic_cpu_disable(void)
-{
-    GICC[GICC_CTLR] = 0;
-}
-
-static void __cpuinit gic_hyp_init(void)
-{
-    uint32_t vtr;
-
-    vtr = GICH[GICH_VTR];
-    nr_lrs  = (vtr & GICH_VTR_NRLRGS) + 1;
-
-    GICH[GICH_MISR] = GICH_MISR_EOI;
-    update_cpu_lr_mask();
-}
-
-static void __cpuinit gic_hyp_disable(void)
-{
-    GICH[GICH_HCR] = 0;
-}
-
 int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
                   unsigned int *out_hwirq,
                   unsigned int *out_type)
@@ -504,106 +253,8 @@ int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
 /* Set up the GIC */
 void __init gic_init(void)
 {
-    static const struct dt_device_match gic_ids[] __initconst =
-    {
-        DT_MATCH_GIC,
-        { /* sentinel */ },
-    };
-    struct dt_device_node *node;
-    int res;
-
-    node = dt_find_interrupt_controller(gic_ids);
-    if ( !node )
-        panic("Unable to find compatible GIC in the device tree");
-
-    dt_device_set_used_by(node, DOMID_XEN);
-
-    res = dt_device_get_address(node, 0, &gic.dbase, NULL);
-    if ( res || !gic.dbase || (gic.dbase & ~PAGE_MASK) )
-        panic("GIC: Cannot find a valid address for the distributor");
-
-    res = dt_device_get_address(node, 1, &gic.cbase, NULL);
-    if ( res || !gic.cbase || (gic.cbase & ~PAGE_MASK) )
-        panic("GIC: Cannot find a valid address for the CPU");
-
-    res = dt_device_get_address(node, 2, &gic.hbase, NULL);
-    if ( res || !gic.hbase || (gic.hbase & ~PAGE_MASK) )
-        panic("GIC: Cannot find a valid address for the hypervisor");
-
-    res = dt_device_get_address(node, 3, &gic.vbase, NULL);
-    if ( res || !gic.vbase || (gic.vbase & ~PAGE_MASK) )
-        panic("GIC: Cannot find a valid address for the virtual CPU");
-
-    res = dt_device_get_irq(node, 0, &gic.maintenance);
-    if ( res )
-        panic("GIC: Cannot find the maintenance IRQ");
-
-    /* Set the GIC as the primary interrupt controller */
-    dt_interrupt_controller = node;
-
-    /* TODO: Add check on distributor, cpu size */
-
-    printk("GIC initialization:\n"
-              "        gic_dist_addr=%"PRIpaddr"\n"
-              "        gic_cpu_addr=%"PRIpaddr"\n"
-              "        gic_hyp_addr=%"PRIpaddr"\n"
-              "        gic_vcpu_addr=%"PRIpaddr"\n"
-              "        gic_maintenance_irq=%u\n",
-              gic.dbase, gic.cbase, gic.hbase, gic.vbase,
-              gic.maintenance.irq);
-
-    if ( (gic.dbase & ~PAGE_MASK) || (gic.cbase & ~PAGE_MASK) ||
-         (gic.hbase & ~PAGE_MASK) || (gic.vbase & ~PAGE_MASK) )
-        panic("GIC interfaces not page aligned");
-
-    set_fixmap(FIXMAP_GICD, gic.dbase >> PAGE_SHIFT, DEV_SHARED);
-    BUILD_BUG_ON(FIXMAP_ADDR(FIXMAP_GICC1) !=
-                 FIXMAP_ADDR(FIXMAP_GICC2)-PAGE_SIZE);
-    set_fixmap(FIXMAP_GICC1, gic.cbase >> PAGE_SHIFT, DEV_SHARED);
-    if ( platform_has_quirk(PLATFORM_QUIRK_GIC_64K_STRIDE) )
-        set_fixmap(FIXMAP_GICC2, (gic.cbase >> PAGE_SHIFT) + 0x10, DEV_SHARED);
-    else
-        set_fixmap(FIXMAP_GICC2, (gic.cbase >> PAGE_SHIFT) + 0x1, DEV_SHARED);
-    set_fixmap(FIXMAP_GICH, gic.hbase >> PAGE_SHIFT, DEV_SHARED);
-
-    /* Global settings: interrupt distributor */
+    gicv2_init();
     spin_lock_init(&gic_lock);
-    spin_lock(&gic_lock);
-
-    gic_dist_init();
-    gic_cpu_init();
-    gic_hyp_init();
-
-    register_gic_ops(&gic_ops);
-    spin_unlock(&gic_lock);
-}
-
-static void gic_secondary_cpu_init(void)
-{
-    gic_cpu_init();
-    gic_hyp_init();
-}
-
-static struct dt_irq * gic_maintenance_irq(void)
-{
-    return &gic.maintenance;
-}
-
-static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
-{
-    unsigned int mask = 0;
-    cpumask_t online_mask;
-
-    ASSERT(sgi < 16); /* There are only 16 SGIs */
-
-    cpumask_and(&online_mask, cpumask, &cpu_online_map);
-    mask = gic_cpu_mask(&online_mask);
-
-    dsb(sy);
-
-    GICD[GICD_SGIR] = GICD_SGI_TARGET_LIST
-        | (mask<<GICD_SGI_TARGET_SHIFT)
-        | sgi;
 }
 
 void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
@@ -613,7 +264,6 @@ void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
 
 void send_SGI_one(unsigned int cpu, enum gic_sgi sgi)
 {
-    ASSERT(cpu < NR_GIC_CPU_IF);  /* Targets bitmap only supports 8 CPUs */
     send_SGI_mask(cpumask_of(cpu), sgi);
 }
 
@@ -641,13 +291,6 @@ void __cpuinit gic_init_secondary_cpu(void)
     spin_unlock(&gic_lock);
 }
 
-/* Shut down the per-CPU GIC interface */
-static void gic_disable_interface(void)
-{
-    gic_cpu_disable();
-    gic_hyp_disable();
-}
-
 void gic_disable_cpu(void)
 {
     ASSERT(!local_irq_is_enabled());
@@ -735,25 +378,6 @@ int __init setup_dt_irq(const struct dt_irq *irq, struct irqaction *new)
     return rc;
 }
 
-static void gic_update_lr(int lr, struct pending_irq *p, unsigned int state)
-{
-    int maintenance_int = GICH_LR_MAINTENANCE_IRQ;
-
-    BUG_ON(lr >= nr_lrs);
-    BUG_ON(lr < 0);
-    BUG_ON(state & ~(GICH_LR_STATE_MASK<<GICH_LR_STATE_SHIFT));
-
-    GICH[GICH_LR + lr] = ((state & 0x3) << GICH_LR_STATE_SHIFT) |
-        maintenance_int |
-        ((p->priority >> 3) << GICH_LR_PRIORITY_SHIFT) |
-        ((p->irq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT);
-}
-
-static void gic_clear_lr(int lr)
-{
-    GICH[GICH_LR + lr] = 0;
-}
-
 static inline void gic_set_lr(int lr, struct pending_irq *p,
         unsigned int state)
 {
@@ -1097,53 +721,6 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
     } while (1);
 }
 
-static int gicv_init(struct domain *d)
-{
-    int ret;
-
-    /*
-     * Domain 0 gets the hardware address.
-     * Guests get the virtual platform layout.
-     */
-    if ( d->domain_id == 0 )
-    {
-        d->arch.vgic.dbase = gic.dbase;
-        d->arch.vgic.cbase = gic.cbase;
-    }
-    else
-    {
-        d->arch.vgic.dbase = GUEST_GICD_BASE;
-        d->arch.vgic.cbase = GUEST_GICC_BASE;
-    }
-
-    d->arch.vgic.nr_lines = 0;
-
-    /*
-     * Map the gic virtual cpu interface in the gic cpu interface
-     * region of the guest.
-     *
-     * The second page is always mapped at +4K irrespective of the
-     * GIC_64K_STRIDE quirk. The DTB passed to the guest reflects this.
-     */
-    ret = map_mmio_regions(d, d->arch.vgic.cbase,
-                           d->arch.vgic.cbase + PAGE_SIZE - 1,
-                           gic.vbase);
-    if (ret)
-        return ret;
-
-    if ( !platform_has_quirk(PLATFORM_QUIRK_GIC_64K_STRIDE) )
-        ret = map_mmio_regions(d, d->arch.vgic.cbase + PAGE_SIZE,
-                               d->arch.vgic.cbase + (2 * PAGE_SIZE) - 1,
-                               gic.vbase + PAGE_SIZE);
-    else
-        ret = map_mmio_regions(d, d->arch.vgic.cbase + PAGE_SIZE,
-                               d->arch.vgic.cbase + (2 * PAGE_SIZE) - 1,
-                               gic.vbase + 16*PAGE_SIZE);
-
-    return ret;
-
-}
-
 int vcpu_gic_init(struct vcpu *v)
 {
      return gic_hw_ops->state_init(v);
@@ -1158,71 +735,6 @@ int gicv_setup(struct domain *d)
 
 }
 
-static void gic_read_lr(int lr, struct gic_lr *lr_reg)
-{
-    uint32_t lrv;
-
-    lrv = GICH[GICH_LR + lr];
-    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
-    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
-    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
-    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
-    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
-    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
-}
-
-static void gic_write_lr(int lr, struct gic_lr *lr_reg)
-{
-    uint32_t lrv = 0;
-    lrv = ( ((lr_reg->pirq & GICH_LR_PHYSICAL_MASK) << GICH_LR_PHYSICAL_SHIFT)  |
-            ((lr_reg->virq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT)   |
-            ((uint32_t)(lr_reg->priority & GICH_LR_PRIORITY_MASK) << GICH_LR_PRIORITY_SHIFT) |
-            ((uint32_t)(lr_reg->state & GICH_LR_STATE_MASK) << GICH_LR_STATE_SHIFT) |
-            ((uint32_t)(lr_reg->hw_status & GICH_LR_HW_MASK)  << GICH_LR_HW_SHIFT)  |
-            ((uint32_t)(lr_reg->grp & GICH_LR_GRP_MASK) << GICH_LR_GRP_SHIFT) );
-
-    GICH[GICH_LR + lr] = lrv;
-}
-
-static void gic_hcr_status(uint32_t flag, uint8_t status)
-{
-    if ( status )
-      GICH[GICH_HCR] |= flag;
-    else
-      GICH[GICH_HCR] &= ~flag;
-}
-
-static unsigned int gic_read_vmcr_priority(void)
-{
-   return (GICH[GICH_VMCR] >> GICH_VMCR_PRIORITY_SHIFT) & GICH_VMCR_PRIORITY_MASK;
-}
-
-static struct gic_hw_operations gic_ops = {
-    .nr_lines            = gic_nr_lines,
-    .nr_lrs              = gic_nr_lrs,
-    .secondary_init      = gic_secondary_cpu_init,
-    .get_maintenance_irq = gic_maintenance_irq,
-    .state_init          = gic_state_init,
-    .save_state          = save_state,
-    .restore_state       = restore_state,
-    .dump_state          = gic_dump_state,
-    .gicv_setup          = gicv_init,
-    .enable_irq          = gic_enable_irq,
-    .disable_irq         = gic_disable_irq,
-    .eoi_irq             = gic_eoi_irq,
-    .deactivate_irq      = gic_dir_irq,
-    .ack_irq             = gic_ack_irq,
-    .set_irq_property    = gic_set_irq_property,
-    .send_sgi            = gic_send_sgi,
-    .disable_interface   = gic_disable_interface,
-    .update_lr           = gic_update_lr,
-    .update_hcr_status   = gic_hcr_status,
-    .clear_lr            = gic_clear_lr,
-    .read_lr             = gic_read_lr,
-    .write_lr            = gic_write_lr,
-    .read_vmcr_priority  = gic_read_vmcr_priority,
-};
-
 static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
 {
     /* 
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 27d2792..cc6d8c4 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -116,6 +116,7 @@ extern int gic_route_irq_to_guest(struct domain *d,
 extern void gic_interrupt(struct cpu_user_regs *regs, int is_fiq);
 /* Bring up the interrupt controller, and report # cpus attached */
 extern void gic_init(void);
+extern void gicv2_init(void);
 /* Bring up a secondary CPU's per-CPU GIC interface */
 extern void gic_init_secondary_cpu(void);
 /* Take down a CPU's per-CPU GIC interface */
-- 
1.7.9.5

Re: [PATCH v2 07/15] xen/arm: split gic driver into generic and gic-v2 driver

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> Existing GIC driver has both generic code and hw specific
> code. Segregate GIC low level driver into gic-v2.c and
> keep generic code in existing gic.c file

This is substantially just code motion, correct?

I notice that gicv2_init does not hold the gic lock while calling +
gic_{dist,cpu,hyp}_init(), is that intentional?

> +static struct gic_hw_operations gic_ops;

If you order this file:
	callbacks
	gic_hw_ops
	gicv2_init

Then you can avoid this forward declaration and more importantly you can
probably mark the struct as const.

Ian.

Re: [PATCH v2 07/15] xen/arm: split gic driver into generic and gic-v2 driver

[Vijay Kilari]

On Thu, Apr 10, 2014 at 2:28 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>
>> Existing GIC driver has both generic code and hw specific
>> code. Segregate GIC low level driver into gic-v2.c and
>> keep generic code in existing gic.c file
>
> This is substantially just code motion, correct?
>
  Yes, but now I think to avoid confusion I will make single
patch for spliting v2 driver & code movement as in V1

> I notice that gicv2_init does not hold the gic lock while calling +
> gic_{dist,cpu,hyp}_init(), is that intentional?
>
>> +static struct gic_hw_operations gic_ops;
>
> If you order this file:
>         callbacks
>         gic_hw_ops
>         gicv2_init
>
Yes, this is called only during init by boot cpu and is not called by
any other cpu
So, I have not taken the lock.

> Then you can avoid this forward declaration and more importantly you can
> probably mark the struct as const.
>
> Ian.
>

[PATCH v2 08/15] xen/arm: use device api to detect GIC version

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Run through the device tree to detect
compatible GIC version and initialize GIC driver.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/gic-v2.c        |   26 +++++++++++++++-----------
 xen/arch/arm/gic.c           |   17 ++++++++++++++++-
 xen/include/asm-arm/device.h |    3 ++-
 xen/include/asm-arm/gic.h    |    1 -
 4 files changed, 33 insertions(+), 14 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index e39099f..06ed12b 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -31,6 +31,7 @@
 #include <asm/p2m.h>
 #include <asm/domain.h>
 #include <asm/platform.h>
+#include <asm/device.h>
 
 #include <asm/gic_v2_defs.h>
 #include <asm/gic.h>
@@ -289,20 +290,10 @@ static void __cpuinit gic_hyp_disable(void)
 }
 
 /* Set up the GIC */
-void __init gicv2_init(void)
+int __init gicv2_init(struct dt_device_node *node, const void *data)
 {
-    static const struct dt_device_match gic_ids[] __initconst =
-    {
-        DT_MATCH_GIC,
-        { /* sentinel */ },
-    };
-    struct dt_device_node *node;
     int res;
 
-    node = dt_find_interrupt_controller(gic_ids);
-    if ( !node )
-        panic("Unable to find compatible GIC in the device tree");
-
     dt_device_set_used_by(node, DOMID_XEN);
 
     res = dt_device_get_address(node, 0, &gic.dbase, NULL);
@@ -360,6 +351,7 @@ void __init gicv2_init(void)
     gic_hyp_init();
 
     register_gic_ops(&gic_ops);
+    return 0;
 }
 
 static void gic_secondary_cpu_init(void)
@@ -528,6 +520,18 @@ static struct gic_hw_operations gic_ops = {
     .read_vmcr_priority  = gic_read_vmcr_priority,
 };
 
+static const char * const gicv2_dt_compat[] __initconst =
+{
+    "arm,cortex-a15-gic",
+    "arm,cortex-a9-gic",
+    NULL
+};
+
+DT_DEVICE_START(gicv2, "GIC", DEVICE_GIC)
+        .compatible = gicv2_dt_compat,
+        .init = gicv2_init,
+DT_DEVICE_END
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index eb2f2d4..87a7ad0 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -31,6 +31,7 @@
 #include <asm/p2m.h>
 #include <asm/domain.h>
 #include <asm/platform.h>
+#include <asm/device.h>
 
 #include <asm/gic.h>
 
@@ -253,8 +254,22 @@ int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
 /* Set up the GIC */
 void __init gic_init(void)
 {
-    gicv2_init();
+    int rc;
+    struct dt_device_node *node;
+    unsigned int num_gics = 0;
+
     spin_lock_init(&gic_lock);
+    spin_lock(&gic_lock);
+
+    dt_for_each_device_node(dt_host, node)
+    {
+        rc = device_init(node, DEVICE_GIC, NULL);
+        if ( !rc )
+            num_gics++;
+    }
+    spin_unlock(&gic_lock);
+    if ( !num_gics )
+        panic("Unable to find compatible GIC in the device tree");
 }
 
 void send_SGI_mask(const cpumask_t *cpumask, enum gic_sgi sgi)
diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
index 9e47ca6..61412e6 100644
--- a/xen/include/asm-arm/device.h
+++ b/xen/include/asm-arm/device.h
@@ -6,7 +6,8 @@
 
 enum device_type
 {
-    DEVICE_SERIAL
+    DEVICE_SERIAL,
+    DEVICE_GIC
 };
 
 struct device_desc {
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index cc6d8c4..27d2792 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -116,7 +116,6 @@ extern int gic_route_irq_to_guest(struct domain *d,
 extern void gic_interrupt(struct cpu_user_regs *regs, int is_fiq);
 /* Bring up the interrupt controller, and report # cpus attached */
 extern void gic_init(void);
-extern void gicv2_init(void);
 /* Bring up a secondary CPU's per-CPU GIC interface */
 extern void gic_init_secondary_cpu(void);
 /* Take down a CPU's per-CPU GIC interface */
-- 
1.7.9.5

Re: [PATCH v2 08/15] xen/arm: use device api to detect GIC version

[Julien Grall]

Hi Vijaya,

Thank you for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> @@ -253,8 +254,22 @@ int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
>  /* Set up the GIC */
>  void __init gic_init(void)
>  {
> -    gicv2_init();
> +    int rc;
> +    struct dt_device_node *node;
> +    unsigned int num_gics = 0;
> +
>      spin_lock_init(&gic_lock);
> +    spin_lock(&gic_lock);

Why do you take the lock here?

> +
> +    dt_for_each_device_node(dt_host, node)
> +    {
> +        rc = device_init(node, DEVICE_GIC, NULL);
> +        if ( !rc )
> +            num_gics++;

Xen only supports a single GIC for now. You have to initialized the
correct one (see dt_find_interrupt_controller).

Regards,

-- 
Julien Grall

Re: [PATCH v2 08/15] xen/arm: use device api to detect GIC version

[Vijay Kilari]

Hi Julien,

>
>> +
>> +    dt_for_each_device_node(dt_host, node)
>> +    {
>> +        rc = device_init(node, DEVICE_GIC, NULL);
>> +        if ( !rc )
>> +            num_gics++;
>
> Xen only supports a single GIC for now. You have to initialized the
> correct one (see dt_find_interrupt_controller).

dt_find_interrupt_controller expects compatibility string as below to fetch
the gic node which can be passed to device_init().

static const struct dt_device_match gic_ids[] __initconst =
    {
        DT_MATCH_GIC,
        { /* sentinel */ },
    };

In such case, generic driver has to pass compatibility strings of both
GIC v3 & GIC v2 to find the node in dt. Is this ok?

Other way is to break when num_gics = 1 when it finds the first gic node.

>
> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 08/15] xen/arm: use device api to detect GIC version

[Julien Grall]

On 04/09/2014 03:28 PM, Vijay Kilari wrote:
>>> +
>>> +    dt_for_each_device_node(dt_host, node)
>>> +    {
>>> +        rc = device_init(node, DEVICE_GIC, NULL);
>>> +        if ( !rc )
>>> +            num_gics++;
>>
>> Xen only supports a single GIC for now. You have to initialized the
>> correct one (see dt_find_interrupt_controller).
> 
> dt_find_interrupt_controller expects compatibility string as below to fetch
> the gic node which can be passed to device_init().
> 
> static const struct dt_device_match gic_ids[] __initconst =
>     {
>         DT_MATCH_GIC,
>         { /* sentinel */ },
>     };
> 
> In such case, generic driver has to pass compatibility strings of both
> GIC v3 & GIC v2 to find the node in dt. Is this ok?
> 
> Other way is to break when num_gics = 1 when it finds the first gic node.

Why can't you modify dt_find_interrupt_controller? Or even better, take
the code from it and use it directly in your function...

-- 
Julien Grall

Re: [PATCH v2 08/15] xen/arm: use device api to detect GIC version

[Ian Campbell]

On Wed, 2014-04-09 at 15:32 +0100, Julien Grall wrote:
> On 04/09/2014 03:28 PM, Vijay Kilari wrote:
> >>> +
> >>> +    dt_for_each_device_node(dt_host, node)
> >>> +    {
> >>> +        rc = device_init(node, DEVICE_GIC, NULL);
> >>> +        if ( !rc )
> >>> +            num_gics++;
> >>
> >> Xen only supports a single GIC for now. You have to initialized the
> >> correct one (see dt_find_interrupt_controller).
> > 
> > dt_find_interrupt_controller expects compatibility string as below to fetch
> > the gic node which can be passed to device_init().
> > 
> > static const struct dt_device_match gic_ids[] __initconst =
> >     {
> >         DT_MATCH_GIC,
> >         { /* sentinel */ },
> >     };
> > 
> > In such case, generic driver has to pass compatibility strings of both
> > GIC v3 & GIC v2 to find the node in dt. Is this ok?
> > 
> > Other way is to break when num_gics = 1 when it finds the first gic node.
> 
> Why can't you modify dt_find_interrupt_controller? Or even better, take
> the code from it and use it directly in your function...

The patch here looks like a pretty natural use of the DT device API to
me, I think all that is needed is for gicv2_init to call
dt_get_parent(np) and skip any non-root gic with a warning message and
return error.

Something should also check for the "interrupt-controller" property,
either the inner or outer init function.

Ian.

[PATCH v2 09/15] xen/arm: segregate VGIC low level functionality

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

VGIC low level functionality is segregated into
separate functions and are called using registered
callback wherever required.

This helps to separate generic and hardware functionality
later

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/vgic.c          |  120 ++++++++++++++++++++++++++++++++----------
 xen/include/asm-arm/device.h |    3 +-
 xen/include/asm-arm/gic.h    |    7 +++
 3 files changed, 101 insertions(+), 29 deletions(-)

diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index a98da82..6c0189e 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -26,6 +26,7 @@
 #include <xen/sched.h>
 
 #include <asm/current.h>
+#include <asm/device.h>
 
 #include "io.h"
 #include <asm/gic_v2_defs.h>
@@ -36,6 +37,8 @@
 /* Number of ranks of interrupt registers for a domain */
 #define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
 
+static struct vgic_ops *vgic_ops;
+
 static struct mmio_handler vgic_distr_mmio_handler;
 /*
  * Rank containing GICD_<FOO><n> for GICD_<FOO> with
@@ -53,6 +56,11 @@ static inline int REG_RANK_NR(int b, uint32_t n)
     }
 }
 
+void register_vgic_ops(struct vgic_ops *ops)
+{
+   vgic_ops = ops;
+}
+
 /*
  * Offset of GICD_<FOO><n> with its rank, for GICD_<FOO> with
  * <b>-bits-per-interrupt.
@@ -78,6 +86,9 @@ static struct vgic_irq_rank *vgic_irq_rank(struct vcpu *v, int b, int n)
 int domain_vgic_init(struct domain *d)
 {
     int i;
+    int rc;
+    struct dt_device_node *node;
+    unsigned int num_vgics = 0;
 
     d->arch.vgic.ctlr = 0;
 
@@ -89,27 +100,28 @@ int domain_vgic_init(struct domain *d)
     else
         d->arch.vgic.nr_lines = 0; /* We don't need SPIs for the guest */
 
-    d->arch.vgic.shared_irqs =
-        xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
-    if ( d->arch.vgic.shared_irqs == NULL )
-        return -ENOMEM;
+    dt_for_each_device_node(dt_host, node)
+    {
+        rc = device_init(node, DEVICE_VGIC, NULL);
+        if ( !rc )
+            num_vgics++;
+    }
+
+    if ( !num_vgics )
+       panic("No compatible vgic found\n");
+
+    vgic_ops->vgic_domain_init(d);
 
     d->arch.vgic.pending_irqs =
         xzalloc_array(struct pending_irq, d->arch.vgic.nr_lines);
     if ( d->arch.vgic.pending_irqs == NULL )
-    {
-        xfree(d->arch.vgic.shared_irqs);
         return -ENOMEM;
-    }
 
-    for (i=0; i<d->arch.vgic.nr_lines; i++)
+    for ( i = 0; i < d->arch.vgic.nr_lines; i++ )
     {
         INIT_LIST_HEAD(&d->arch.vgic.pending_irqs[i].inflight);
         INIT_LIST_HEAD(&d->arch.vgic.pending_irqs[i].lr_queue);
     }
-    for (i=0; i<DOMAIN_NR_RANKS(d); i++)
-        spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
-    register_mmio_handler(d, &vgic_distr_mmio_handler);
     return 0;
 }
 
@@ -123,13 +135,10 @@ int vcpu_vgic_init(struct vcpu *v)
 {
     int i;
 
-    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
-    if ( v->arch.vgic.private_irqs == NULL )
-      return -ENOMEM;
-
-    memset(&v->arch.vgic.private_irqs, 0, sizeof(v->arch.vgic.private_irqs));
-
-    spin_lock_init(&v->arch.vgic.private_irqs->lock);
+    if ( vgic_ops )
+       vgic_ops->vgic_vcpu_init(v);
+    else
+       panic("No VGIC ops found\n");
 
     memset(&v->arch.vgic.pending_irqs, 0, sizeof(v->arch.vgic.pending_irqs));
     for (i = 0; i < 32; i++)
@@ -138,13 +147,6 @@ int vcpu_vgic_init(struct vcpu *v)
         INIT_LIST_HEAD(&v->arch.vgic.pending_irqs[i].lr_queue);
     }
 
-    /* For SGI and PPI the target is always this CPU */
-    for ( i = 0 ; i < 8 ; i++ )
-        v->arch.vgic.private_irqs->itargets[i] =
-              (1<<(v->vcpu_id+0))
-            | (1<<(v->vcpu_id+8))
-            | (1<<(v->vcpu_id+16))
-            | (1<<(v->vcpu_id+24));
     INIT_LIST_HEAD(&v->arch.vgic.inflight_irqs);
     INIT_LIST_HEAD(&v->arch.vgic.lr_pending);
     spin_lock_init(&v->arch.vgic.lock);
@@ -186,6 +188,13 @@ static void byte_write(uint32_t *reg, uint32_t var, int offset)
     *reg |= var;
 }
 
+static int vgic_read_priority(struct vcpu *v, int irq)
+{
+   int idx = irq >> 2;
+   struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
+   return byte_read(rank->ipriority[REG_RANK_INDEX(8, idx)], 0, irq & 0x3);
+}
+
 static int vgic_distr_mmio_read(struct vcpu *v, mmio_info_t *info)
 {
     struct hsr_dabt dabt = info->dabt;
@@ -719,9 +728,7 @@ void vgic_clear_pending_irqs(struct vcpu *v)
 
 void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq)
 {
-    int idx = irq >> 2, byte = irq & 0x3;
     uint8_t priority;
-    struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
     struct pending_irq *iter, *n = irq_to_pending(v, irq);
     unsigned long flags;
     bool_t running;
@@ -735,7 +742,7 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int irq)
         return;
     }
 
-    priority = byte_read(rank->ipriority[REG_RANK_INDEX(8, idx)], 0, byte);
+    priority = vgic_ops->read_priority(v, irq);
 
     n->irq = irq;
     set_bit(GIC_IRQ_GUEST_PENDING, &n->status);
@@ -768,6 +775,63 @@ out:
         smp_send_event_check_mask(cpumask_of(v->processor));
 }
 
+static int vgic_vcpu_init(struct vcpu *v)
+{
+    int i;
+
+    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
+    memset(v->arch.vgic.private_irqs, 0, sizeof(struct vgic_irq_rank));
+
+    spin_lock_init(&v->arch.vgic.private_irqs->lock);
+    /* For SGI and PPI the target is always this CPU */
+    for ( i = 0 ; i < 8 ; i++ )
+        v->arch.vgic.private_irqs->itargets[i] =
+              (1<<(v->vcpu_id+0))
+            | (1<<(v->vcpu_id+8))
+            | (1<<(v->vcpu_id+16))
+            | (1<<(v->vcpu_id+24));
+    return 0;
+}
+
+static int vgic_domain_init(struct domain *d)
+{
+    int i;
+
+    d->arch.vgic.shared_irqs =
+        xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
+
+    for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
+        spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
+
+    register_mmio_handler(d, &vgic_distr_mmio_handler);
+    return 0;
+}
+
+static struct vgic_ops ops = {
+    .vgic_vcpu_init   = vgic_vcpu_init,
+    .vgic_domain_init = vgic_domain_init,
+    .read_priority    = vgic_read_priority,
+};
+
+static int __init vgic_v2_init(struct dt_device_node *dev, const void *data)
+
+{
+    register_vgic_ops(&ops);
+    return 0;
+}
+
+static const char * const vgicv2_dt_compat[] __initconst =
+{
+    "arm,cortex-a15-gic",
+    "arm,cortex-a9-gic",
+    NULL
+};
+
+DT_DEVICE_START(gicv2, "VGIC", DEVICE_VGIC)
+        .compatible = vgicv2_dt_compat,
+        .init = vgic_v2_init,
+DT_DEVICE_END
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/device.h b/xen/include/asm-arm/device.h
index 61412e6..dd8ab54 100644
--- a/xen/include/asm-arm/device.h
+++ b/xen/include/asm-arm/device.h
@@ -7,7 +7,8 @@
 enum device_type
 {
     DEVICE_SERIAL,
-    DEVICE_GIC
+    DEVICE_GIC,
+    DEVICE_VGIC
 };
 
 struct device_desc {
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 27d2792..2abe23e 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -177,6 +177,13 @@ struct gic_hw_operations {
     void (*secondary_init)(void);
 };
 
+struct vgic_ops {
+    int (*vgic_vcpu_init)(struct vcpu *v);
+    int (*vgic_domain_init)(struct domain *d);
+    int (*read_priority)(struct vcpu *v, int irq);
+};
+
+void register_vgic_ops(struct vgic_ops *ops);
 void register_gic_ops(struct gic_hw_operations *ops);
 extern void update_cpu_lr_mask(void);
 #endif /* __ASSEMBLY__ */
-- 
1.7.9.5

Re: [PATCH v2 09/15] xen/arm: segregate VGIC low level functionality

[Julien Grall]

Hello Vijay,

Thank you for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> VGIC low level functionality is segregated into
> separate functions and are called using registered
> callback wherever required.
> 
> This helps to separate generic and hardware functionality
> later

Same remark as patch #5, is less clear to have 2 separate patches.

[..]

> +DT_DEVICE_START(gicv2, "VGIC", DEVICE_VGIC)
> +        .compatible = vgicv2_dt_compat,
> +        .init = vgic_v2_init,
> +DT_DEVICE_END
> +

hrm ... VGIC is not a device, if you don't plan to support different
VGIC per domain then you can directly return the vgic callbacks for the
gic drivers.

Regards,

-- 
Julien Grall

Re: [PATCH v2 09/15] xen/arm: segregate VGIC low level functionality

[Ian Campbell]

On Fri, 2014-04-04 at 15:13 +0100, Julien Grall wrote:
> > +DT_DEVICE_START(gicv2, "VGIC", DEVICE_VGIC)
> > +        .compatible = vgicv2_dt_compat,
> > +        .init = vgic_v2_init,
> > +DT_DEVICE_END
> > +
> 
> hrm ... VGIC is not a device, if you don't plan to support different
> VGIC per domain then you can directly return the vgic callbacks for the
> gic drivers.

Agreed, and if we were to support different VGIC per domain then it
would have to be driven by some sort of toolstack config option, domctl
etc, and not by the compat strings in the host DT.

So best to just register the vgic v2 ops when initialising gic v2 and
similar for v3 I think.

Ian.

[PATCH v2 10/15] xen/arm: split vgic driver into generic and vgic-v2 driver

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Existing vGIC driver has both generic code and hw specific
code. Segregate vGIC low level driver into vgic-v2.c and
keep generic code in existing vgic.c file

vGIC v2 driver registers required functions
to generic vGIC driver. This helps to plug in next version
of vGIC drivers like vGIC v3.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/Makefile     |    2 +-
 xen/arch/arm/vgic-v2.c    |  639 +++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic.c       |  601 +-----------------------------------------
 xen/include/asm-arm/gic.h |    2 +
 4 files changed, 644 insertions(+), 600 deletions(-)

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 969ee52..20f59f4 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -26,7 +26,7 @@ obj-y += smpboot.o
 obj-y += smp.o
 obj-y += shutdown.o
 obj-y += traps.o
-obj-y += vgic.o
+obj-y += vgic.o vgic-v2.o
 obj-y += vtimer.o
 obj-y += vuart.o
 obj-y += hvm.o
diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
new file mode 100644
index 0000000..2919665
--- /dev/null
+++ b/xen/arch/arm/vgic-v2.c
@@ -0,0 +1,639 @@
+/*
+ * xen/arch/arm/vgic.c
+ *
+ * ARM Virtual Generic Interrupt Controller support
+ *
+ * Ian Campbell <ian.campbell@citrix.com>
+ * Copyright (c) 2011 Citrix Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/bitops.h>
+#include <xen/config.h>
+#include <xen/lib.h>
+#include <xen/init.h>
+#include <xen/softirq.h>
+#include <xen/irq.h>
+#include <xen/sched.h>
+
+#include <asm/current.h>
+#include <asm/device.h>
+
+#include "io.h"
+#include <asm/gic_v2_defs.h>
+#include <asm/gic.h>
+
+#define REG(n) (n/4)
+
+/* Number of ranks of interrupt registers for a domain */
+#define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
+
+/*
+ * Rank containing GICD_<FOO><n> for GICD_<FOO> with
+ * <b>-bits-per-interrupt
+ */
+static inline int REG_RANK_NR(int b, uint32_t n)
+{
+    switch ( b )
+    {
+    case 8: return n >> 3;
+    case 4: return n >> 2;
+    case 2: return n >> 1;
+    case 1: return n;
+    default: BUG();
+    }
+}
+
+/*
+ * Offset of GICD_<FOO><n> with its rank, for GICD_<FOO> with
+ * <b>-bits-per-interrupt.
+ */
+#define REG_RANK_INDEX(b, n) ((n) & ((b)-1))
+
+/*
+ * Returns rank corresponding to a GICD_<FOO><n> register for
+ * GICD_<FOO> with <b>-bits-per-interrupt.
+ */
+static struct vgic_irq_rank *vgic_irq_rank(struct vcpu *v, int b, int n)
+{
+    int rank = REG_RANK_NR(b, n);
+
+    if ( rank == 0 )
+        return v->arch.vgic.private_irqs;
+    else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
+        return &v->domain->arch.vgic.shared_irqs[rank - 1];
+    else
+        return NULL;
+}
+
+#define vgic_lock(v)   spin_lock_irq(&(v)->domain->arch.vgic.lock)
+#define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
+
+#define vgic_lock_rank(v, r) spin_lock(&(r)->lock)
+#define vgic_unlock_rank(v, r) spin_unlock(&(r)->lock)
+
+static uint32_t byte_read(uint32_t val, int sign, int offset)
+{
+    int byte = offset & 0x3;
+
+    val = val >> (8*byte);
+    if ( sign && (val & 0x80) )
+        val |= 0xffffff00;
+    else
+        val &= 0x000000ff;
+    return val;
+}
+
+static void byte_write(uint32_t *reg, uint32_t var, int offset)
+{
+    int byte = offset & 0x3;
+
+    var &= (0xff << (8*byte));
+
+    *reg &= ~(0xff << (8*byte));
+    *reg |= var;
+}
+
+static int vgic_read_priority(struct vcpu *v, int irq)
+{
+   int idx = irq >> 2;
+   struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
+   return byte_read(rank->ipriority[REG_RANK_INDEX(8, idx)], 0, irq & 0x3);
+}
+
+static int vgic_distr_mmio_read(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    struct vgic_irq_rank *rank;
+    int offset = (int)(info->gpa - v->domain->arch.vgic.dbase);
+    int gicd_reg = REG(offset);
+
+    switch ( gicd_reg )
+    {
+    case GICD_CTLR:
+        if ( dabt.size != 2 ) goto bad_width;
+        vgic_lock(v);
+        *r = v->domain->arch.vgic.ctlr;
+        vgic_unlock(v);
+        return 1;
+    case GICD_TYPER:
+        if ( dabt.size != 2 ) goto bad_width;
+        /* No secure world support for guests. */
+        vgic_lock(v);
+        *r = ( (v->domain->max_vcpus<<5) & GICD_TYPE_CPUS )
+            |( ((v->domain->arch.vgic.nr_lines/32)) & GICD_TYPE_LINES );
+        vgic_unlock(v);
+        return 1;
+    case GICD_IIDR:
+        if ( dabt.size != 2 ) goto bad_width;
+        /*
+         * XXX Do we need a JEP106 manufacturer ID?
+         * Just use the physical h/w value for now
+         */
+        *r = 0x0000043b;
+        return 1;
+
+    /* Implementation defined -- read as zero */
+    case REG(0x020) ... REG(0x03c):
+        goto read_as_zero;
+
+    case GICD_IGROUPR ... GICD_IGROUPRN:
+        /* We do not implement security extensions for guests, read zero */
+        goto read_as_zero;
+
+    case GICD_ISENABLER ... GICD_ISENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISENABLER);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->ienable;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICENABLER ... GICD_ICENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICENABLER);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->ienable;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ISPENDR ... GICD_ISPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISPENDR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->ipend, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICPENDR ... GICD_ICPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICPENDR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->ipend, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ISACTIVER ... GICD_ISACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISACTIVER);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->iactive;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICACTIVER ... GICD_ICACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICACTIVER);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->iactive;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ITARGETSR ... GICD_ITARGETSRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_ITARGETSR);
+        if ( rank == NULL) goto read_as_zero;
+
+        vgic_lock_rank(v, rank);
+        *r = rank->itargets[REG_RANK_INDEX(8, gicd_reg - GICD_ITARGETSR)];
+        if ( dabt.size == 0 )
+            *r = byte_read(*r, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_IPRIORITYR ... GICD_IPRIORITYRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_IPRIORITYR);
+        if ( rank == NULL) goto read_as_zero;
+
+        vgic_lock_rank(v, rank);
+        *r = rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)];
+        if ( dabt.size == 0 )
+            *r = byte_read(*r, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICFGR ... GICD_ICFGRN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 2, gicd_reg - GICD_ICFGR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->icfg[REG_RANK_INDEX(2, gicd_reg - GICD_ICFGR)];
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_NSACR ... GICD_NSACRN:
+        /* We do not implement security extensions for guests, read zero */
+        goto read_as_zero;
+
+    case GICD_SGIR:
+        if ( dabt.size != 2 ) goto bad_width;
+        /* Write only -- read unknown */
+        *r = 0xdeadbeef;
+        return 1;
+
+    case GICD_CPENDSGIR ... GICD_CPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_CPENDSGIR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->pendsgi, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_SPENDSGIR ... GICD_SPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_SPENDSGIR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->pendsgi, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    /* Implementation defined -- read as zero */
+    case REG(0xfd0) ... REG(0xfe4):
+        goto read_as_zero;
+
+    case GICD_ICPIDR2:
+        if ( dabt.size != 2 ) goto bad_width;
+        printk("vGICD: unhandled read from ICPIDR2\n");
+        return 0;
+
+    /* Implementation defined -- read as zero */
+    case REG(0xfec) ... REG(0xffc):
+        goto read_as_zero;
+
+    /* Reserved -- read as zero */
+    case REG(0x00c) ... REG(0x01c):
+    case REG(0x040) ... REG(0x07c):
+    case REG(0x7fc):
+    case REG(0xbfc):
+    case REG(0xf04) ... REG(0xf0c):
+    case REG(0xf30) ... REG(0xfcc):
+        goto read_as_zero;
+
+    default:
+        printk("vGICD: unhandled read r%d offset %#08x\n",
+               dabt.reg, offset);
+        return 0;
+    }
+
+bad_width:
+    printk("vGICD: bad read width %d r%d offset %#08x\n",
+           dabt.size, dabt.reg, offset);
+    domain_crash_synchronous();
+    return 0;
+
+read_as_zero:
+    if ( dabt.size != 2 ) goto bad_width;
+    *r = 0;
+    return 1;
+}
+
+static inline int is_vcpu_running(struct domain *d, int vcpuid)
+{
+    struct vcpu *v;
+
+    if ( vcpuid >= d->max_vcpus )
+        return 0;
+
+    v = d->vcpu[vcpuid];
+    if ( v == NULL )
+        return 0;
+    if (test_bit(_VPF_down, &v->pause_flags) )
+        return 0;
+
+    return 1;
+}
+
+static int vgic_to_sgi(struct vcpu *v, register_t sgir)
+{
+    struct domain *d = v->domain;
+    int virtual_irq;
+    int filter;
+    int vcpuid;
+    int i;
+    unsigned long vcpu_mask = 0;
+
+    ASSERT(d->max_vcpus < 8*sizeof(vcpu_mask));
+
+    filter = (sgir & GICD_SGI_TARGET_LIST_MASK);
+    virtual_irq = (sgir & GICD_SGI_INTID_MASK);
+    ASSERT( virtual_irq < 16 );
+
+    switch ( filter )
+    {
+        case GICD_SGI_TARGET_LIST:
+            vcpu_mask = (sgir & GICD_SGI_TARGET_MASK) >> GICD_SGI_TARGET_SHIFT;
+            break;
+        case GICD_SGI_TARGET_OTHERS:
+            for ( i = 0; i < d->max_vcpus; i++ )
+            {
+                if ( i != current->vcpu_id && is_vcpu_running(d, i) )
+                    set_bit(i, &vcpu_mask);
+            }
+            break;
+        case GICD_SGI_TARGET_SELF:
+            set_bit(current->vcpu_id, &vcpu_mask);
+            break;
+        default:
+            gdprintk(XENLOG_WARNING, "vGICD: unhandled GICD_SGIR write %"PRIregister" with wrong TargetListFilter field\n",
+                     sgir);
+            return 0;
+    }
+
+    for_each_set_bit( vcpuid, &vcpu_mask, d->max_vcpus )
+    {
+        if ( !is_vcpu_running(d, vcpuid) )
+        {
+            gdprintk(XENLOG_WARNING, "vGICD: GICD_SGIR write r=%"PRIregister" vcpu_mask=%lx, wrong CPUTargetList\n",
+                     sgir, vcpu_mask);
+            continue;
+        }
+        vgic_vcpu_inject_irq(d->vcpu[vcpuid], virtual_irq);
+    }
+    return 1;
+}
+
+static int vgic_distr_mmio_write(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    struct vgic_irq_rank *rank;
+    int offset = (int)(info->gpa - v->domain->arch.vgic.dbase);
+    int gicd_reg = REG(offset);
+    uint32_t tr;
+
+    switch ( gicd_reg )
+    {
+    case GICD_CTLR:
+        if ( dabt.size != 2 ) goto bad_width;
+        /* Ignore all but the enable bit */
+        v->domain->arch.vgic.ctlr = (*r) & GICD_CTL_ENABLE;
+        return 1;
+
+    /* R/O -- write ignored */
+    case GICD_TYPER:
+    case GICD_IIDR:
+        goto write_ignore;
+
+    /* Implementation defined -- write ignored */
+    case REG(0x020) ... REG(0x03c):
+        goto write_ignore;
+
+    case GICD_IGROUPR ... GICD_IGROUPRN:
+        /* We do not implement security extensions for guests, write ignore */
+        goto write_ignore;
+
+    case GICD_ISENABLER ... GICD_ISENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISENABLER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        tr = rank->ienable;
+        rank->ienable |= *r;
+        vgic_unlock_rank(v, rank);
+        vgic_enable_irqs(v, (*r) & (~tr), gicd_reg - GICD_ISENABLER);
+        return 1;
+
+    case GICD_ICENABLER ... GICD_ICENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICENABLER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        tr = rank->ienable;
+        rank->ienable &= ~*r;
+        vgic_unlock_rank(v, rank);
+        vgic_disable_irqs(v, (*r) & tr, gicd_reg - GICD_ICENABLER);
+        return 1;
+
+    case GICD_ISPENDR ... GICD_ISPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        printk("vGICD: unhandled %s write %#"PRIregister" to ISPENDR%d\n",
+               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_ISPENDR);
+        return 0;
+
+    case GICD_ICPENDR ... GICD_ICPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        printk("vGICD: unhandled %s write %#"PRIregister" to ICPENDR%d\n",
+               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_ICPENDR);
+        return 0;
+
+    case GICD_ISACTIVER ... GICD_ISACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISACTIVER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->iactive &= ~*r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICACTIVER ... GICD_ICACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICACTIVER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->iactive &= ~*r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ITARGETSR ... GICD_ITARGETSR + 7:
+        /* SGI/PPI target is read only */
+        goto write_ignore;
+
+    case GICD_ITARGETSR + 8 ... GICD_ITARGETSRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_ITARGETSR);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        if ( dabt.size == 2 )
+            rank->itargets[REG_RANK_INDEX(8, gicd_reg - GICD_ITARGETSR)] = *r;
+        else
+            byte_write(&rank->itargets[REG_RANK_INDEX(8, gicd_reg - GICD_ITARGETSR)],
+                       *r, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_IPRIORITYR ... GICD_IPRIORITYRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_IPRIORITYR);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        if ( dabt.size == 2 )
+            rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)] = *r;
+        else
+            byte_write(&rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)],
+                       *r, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICFGR: /* SGIs */
+        goto write_ignore;
+    case GICD_ICFGR + 1: /* PPIs */
+        /* It is implementation defined if these are writeable. We chose not */
+        goto write_ignore;
+    case GICD_ICFGR + 2 ... GICD_ICFGRN: /* SPIs */
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 2, gicd_reg - GICD_ICFGR);
+        vgic_lock_rank(v, rank);
+        if ( rank == NULL) goto write_ignore;
+        rank->icfg[REG_RANK_INDEX(2, gicd_reg - GICD_ICFGR)] = *r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_NSACR ... GICD_NSACRN:
+        /* We do not implement security extensions for guests, write ignore */
+        goto write_ignore;
+
+    case GICD_SGIR:
+        if ( dabt.size != 2 )
+            goto bad_width;
+        return vgic_to_sgi(v, *r);
+
+    case GICD_CPENDSGIR ... GICD_CPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        printk("vGICD: unhandled %s write %#"PRIregister" to ICPENDSGIR%d\n",
+               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_CPENDSGIR);
+        return 0;
+
+    case GICD_SPENDSGIR ... GICD_SPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        printk("vGICD: unhandled %s write %#"PRIregister" to ISPENDSGIR%d\n",
+               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_SPENDSGIR);
+        return 0;
+
+    /* Implementation defined -- write ignored */
+    case REG(0xfd0) ... REG(0xfe4):
+        goto write_ignore;
+
+    /* R/O -- write ignore */
+    case GICD_ICPIDR2:
+        goto write_ignore;
+
+    /* Implementation defined -- write ignored */
+    case REG(0xfec) ... REG(0xffc):
+        goto write_ignore;
+
+    /* Reserved -- write ignored */
+    case REG(0x00c) ... REG(0x01c):
+    case REG(0x040) ... REG(0x07c):
+    case REG(0x7fc):
+    case REG(0xbfc):
+    case REG(0xf04) ... REG(0xf0c):
+    case REG(0xf30) ... REG(0xfcc):
+        goto write_ignore;
+
+    default:
+        printk("vGICD: unhandled write r%d=%"PRIregister" offset %#08x\n",
+               dabt.reg, *r, offset);
+        return 0;
+    }
+
+bad_width:
+    printk("vGICD: bad write width %d r%d=%"PRIregister" offset %#08x\n",
+           dabt.size, dabt.reg, *r, offset);
+    domain_crash_synchronous();
+    return 0;
+
+write_ignore:
+    if ( dabt.size != 2 ) goto bad_width;
+    return 1;
+}
+
+static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
+{
+    struct domain *d = v->domain;
+
+    return (addr >= (d->arch.vgic.dbase)) && (addr < (d->arch.vgic.dbase + PAGE_SIZE));
+}
+
+static struct mmio_handler vgic_distr_mmio_handler = {
+    .check_handler = vgic_distr_mmio_check,
+    .read_handler  = vgic_distr_mmio_read,
+    .write_handler = vgic_distr_mmio_write,
+};
+
+static int vgic_vcpu_init(struct vcpu *v)
+{
+    int i;
+
+    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
+    memset(v->arch.vgic.private_irqs, 0, sizeof(struct vgic_irq_rank));
+
+    spin_lock_init(&v->arch.vgic.private_irqs->lock);
+    /* For SGI and PPI the target is always this CPU */
+    for ( i = 0 ; i < 8 ; i++ )
+        v->arch.vgic.private_irqs->itargets[i] =
+              (1<<(v->vcpu_id+0))
+            | (1<<(v->vcpu_id+8))
+            | (1<<(v->vcpu_id+16))
+            | (1<<(v->vcpu_id+24));
+    return 0;
+}
+
+static int vgic_domain_init(struct domain *d)
+{
+    int i;
+
+    d->arch.vgic.shared_irqs =
+        xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
+
+    for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
+        spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
+
+    register_mmio_handler(d, &vgic_distr_mmio_handler);
+    return 0;
+}
+
+static struct vgic_ops ops = {
+    .vgic_vcpu_init   = vgic_vcpu_init,
+    .vgic_domain_init = vgic_domain_init,
+    .read_priority    = vgic_read_priority,
+};
+
+static int __init vgic_v2_init(struct dt_device_node *dev, const void *data)
+
+{
+    register_vgic_ops(&ops);
+    return 0;
+}
+
+static const char * const vgicv2_dt_compat[] __initconst =
+{
+    "arm,cortex-a15-gic",
+    "arm,cortex-a9-gic",
+    NULL
+};
+
+DT_DEVICE_START(gicv2, "VGIC", DEVICE_VGIC)
+        .compatible = vgicv2_dt_compat,
+        .init = vgic_v2_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 6c0189e..7516b48 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -29,60 +29,16 @@
 #include <asm/device.h>
 
 #include "io.h"
-#include <asm/gic_v2_defs.h>
 #include <asm/gic.h>
 
-#define REG(n) (n/4)
-
-/* Number of ranks of interrupt registers for a domain */
-#define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
 
 static struct vgic_ops *vgic_ops;
 
-static struct mmio_handler vgic_distr_mmio_handler;
-/*
- * Rank containing GICD_<FOO><n> for GICD_<FOO> with
- * <b>-bits-per-interrupt
- */
-static inline int REG_RANK_NR(int b, uint32_t n)
-{
-    switch ( b )
-    {
-    case 8: return n >> 3;
-    case 4: return n >> 2;
-    case 2: return n >> 1;
-    case 1: return n;
-    default: BUG();
-    }
-}
-
 void register_vgic_ops(struct vgic_ops *ops)
 {
    vgic_ops = ops;
 }
 
-/*
- * Offset of GICD_<FOO><n> with its rank, for GICD_<FOO> with
- * <b>-bits-per-interrupt.
- */
-#define REG_RANK_INDEX(b, n) ((n) & ((b)-1))
-
-/*
- * Returns rank corresponding to a GICD_<FOO><n> register for
- * GICD_<FOO> with <b>-bits-per-interrupt.
- */
-static struct vgic_irq_rank *vgic_irq_rank(struct vcpu *v, int b, int n)
-{
-    int rank = REG_RANK_NR(b, n);
-
-    if ( rank == 0 )
-        return v->arch.vgic.private_irqs;
-    else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
-        return &v->domain->arch.vgic.shared_irqs[rank - 1];
-    else
-        return NULL;
-}
-
 int domain_vgic_init(struct domain *d)
 {
     int i;
@@ -160,239 +116,7 @@ int vcpu_vgic_free(struct vcpu *v)
    return 0;
 }
 
-#define vgic_lock(v)   spin_lock_irq(&(v)->domain->arch.vgic.lock)
-#define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
-
-#define vgic_lock_rank(v, r) spin_lock(&(r)->lock)
-#define vgic_unlock_rank(v, r) spin_unlock(&(r)->lock)
-
-static uint32_t byte_read(uint32_t val, int sign, int offset)
-{
-    int byte = offset & 0x3;
-
-    val = val >> (8*byte);
-    if ( sign && (val & 0x80) )
-        val |= 0xffffff00;
-    else
-        val &= 0x000000ff;
-    return val;
-}
-
-static void byte_write(uint32_t *reg, uint32_t var, int offset)
-{
-    int byte = offset & 0x3;
-
-    var &= (0xff << (8*byte));
-
-    *reg &= ~(0xff << (8*byte));
-    *reg |= var;
-}
-
-static int vgic_read_priority(struct vcpu *v, int irq)
-{
-   int idx = irq >> 2;
-   struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, idx);
-   return byte_read(rank->ipriority[REG_RANK_INDEX(8, idx)], 0, irq & 0x3);
-}
-
-static int vgic_distr_mmio_read(struct vcpu *v, mmio_info_t *info)
-{
-    struct hsr_dabt dabt = info->dabt;
-    struct cpu_user_regs *regs = guest_cpu_user_regs();
-    register_t *r = select_user_reg(regs, dabt.reg);
-    struct vgic_irq_rank *rank;
-    int offset = (int)(info->gpa - v->domain->arch.vgic.dbase);
-    int gicd_reg = REG(offset);
-
-    switch ( gicd_reg )
-    {
-    case GICD_CTLR:
-        if ( dabt.size != 2 ) goto bad_width;
-        vgic_lock(v);
-        *r = v->domain->arch.vgic.ctlr;
-        vgic_unlock(v);
-        return 1;
-    case GICD_TYPER:
-        if ( dabt.size != 2 ) goto bad_width;
-        /* No secure world support for guests. */
-        vgic_lock(v);
-        *r = ( (v->domain->max_vcpus<<5) & GICD_TYPE_CPUS )
-            |( ((v->domain->arch.vgic.nr_lines/32)) & GICD_TYPE_LINES );
-        vgic_unlock(v);
-        return 1;
-    case GICD_IIDR:
-        if ( dabt.size != 2 ) goto bad_width;
-        /*
-         * XXX Do we need a JEP106 manufacturer ID?
-         * Just use the physical h/w value for now
-         */
-        *r = 0x0000043b;
-        return 1;
-
-    /* Implementation defined -- read as zero */
-    case REG(0x020) ... REG(0x03c):
-        goto read_as_zero;
-
-    case GICD_IGROUPR ... GICD_IGROUPRN:
-        /* We do not implement security extensions for guests, read zero */
-        goto read_as_zero;
-
-    case GICD_ISENABLER ... GICD_ISENABLERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISENABLER);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = rank->ienable;
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ICENABLER ... GICD_ICENABLERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICENABLER);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = rank->ienable;
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ISPENDR ... GICD_ISPENDRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISPENDR);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = byte_read(rank->ipend, dabt.sign, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ICPENDR ... GICD_ICPENDRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICPENDR);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = byte_read(rank->ipend, dabt.sign, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ISACTIVER ... GICD_ISACTIVERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISACTIVER);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = rank->iactive;
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ICACTIVER ... GICD_ICACTIVERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICACTIVER);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = rank->iactive;
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ITARGETSR ... GICD_ITARGETSRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_ITARGETSR);
-        if ( rank == NULL) goto read_as_zero;
-
-        vgic_lock_rank(v, rank);
-        *r = rank->itargets[REG_RANK_INDEX(8, gicd_reg - GICD_ITARGETSR)];
-        if ( dabt.size == 0 )
-            *r = byte_read(*r, dabt.sign, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_IPRIORITYR ... GICD_IPRIORITYRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_IPRIORITYR);
-        if ( rank == NULL) goto read_as_zero;
-
-        vgic_lock_rank(v, rank);
-        *r = rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)];
-        if ( dabt.size == 0 )
-            *r = byte_read(*r, dabt.sign, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ICFGR ... GICD_ICFGRN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 2, gicd_reg - GICD_ICFGR);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = rank->icfg[REG_RANK_INDEX(2, gicd_reg - GICD_ICFGR)];
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_NSACR ... GICD_NSACRN:
-        /* We do not implement security extensions for guests, read zero */
-        goto read_as_zero;
-
-    case GICD_SGIR:
-        if ( dabt.size != 2 ) goto bad_width;
-        /* Write only -- read unknown */
-        *r = 0xdeadbeef;
-        return 1;
-
-    case GICD_CPENDSGIR ... GICD_CPENDSGIRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_CPENDSGIR);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = byte_read(rank->pendsgi, dabt.sign, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_SPENDSGIR ... GICD_SPENDSGIRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_SPENDSGIR);
-        if ( rank == NULL) goto read_as_zero;
-        vgic_lock_rank(v, rank);
-        *r = byte_read(rank->pendsgi, dabt.sign, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    /* Implementation defined -- read as zero */
-    case REG(0xfd0) ... REG(0xfe4):
-        goto read_as_zero;
-
-    case GICD_ICPIDR2:
-        if ( dabt.size != 2 ) goto bad_width;
-        printk("vGICD: unhandled read from ICPIDR2\n");
-        return 0;
-
-    /* Implementation defined -- read as zero */
-    case REG(0xfec) ... REG(0xffc):
-        goto read_as_zero;
-
-    /* Reserved -- read as zero */
-    case REG(0x00c) ... REG(0x01c):
-    case REG(0x040) ... REG(0x07c):
-    case REG(0x7fc):
-    case REG(0xbfc):
-    case REG(0xf04) ... REG(0xf0c):
-    case REG(0xf30) ... REG(0xfcc):
-        goto read_as_zero;
-
-    default:
-        printk("vGICD: unhandled read r%d offset %#08x\n",
-               dabt.reg, offset);
-        return 0;
-    }
-
-bad_width:
-    printk("vGICD: bad read width %d r%d offset %#08x\n",
-           dabt.size, dabt.reg, offset);
-    domain_crash_synchronous();
-    return 0;
-
-read_as_zero:
-    if ( dabt.size != 2 ) goto bad_width;
-    *r = 0;
-    return 1;
-}
-
-static void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n)
+void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n)
 {
     const unsigned long mask = r;
     struct pending_irq *p;
@@ -410,7 +134,7 @@ static void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n)
     }
 }
 
-static void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n)
+void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n)
 {
     const unsigned long mask = r;
     struct pending_irq *p;
@@ -438,270 +162,6 @@ static void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n)
     }
 }
 
-static inline int is_vcpu_running(struct domain *d, int vcpuid)
-{
-    struct vcpu *v;
-
-    if ( vcpuid >= d->max_vcpus )
-        return 0;
-
-    v = d->vcpu[vcpuid];
-    if ( v == NULL )
-        return 0;
-    if (test_bit(_VPF_down, &v->pause_flags) )
-        return 0;
-
-    return 1;
-}
-
-static int vgic_to_sgi(struct vcpu *v, register_t sgir)
-{
-    struct domain *d = v->domain;
-    int virtual_irq;
-    int filter;
-    int vcpuid;
-    int i;
-    unsigned long vcpu_mask = 0;
-
-    ASSERT(d->max_vcpus < 8*sizeof(vcpu_mask));
-
-    filter = (sgir & GICD_SGI_TARGET_LIST_MASK);
-    virtual_irq = (sgir & GICD_SGI_INTID_MASK);
-    ASSERT( virtual_irq < 16 );
-
-    switch ( filter )
-    {
-        case GICD_SGI_TARGET_LIST:
-            vcpu_mask = (sgir & GICD_SGI_TARGET_MASK) >> GICD_SGI_TARGET_SHIFT;
-            break;
-        case GICD_SGI_TARGET_OTHERS:
-            for ( i = 0; i < d->max_vcpus; i++ )
-            {
-                if ( i != current->vcpu_id && is_vcpu_running(d, i) )
-                    set_bit(i, &vcpu_mask);
-            }
-            break;
-        case GICD_SGI_TARGET_SELF:
-            set_bit(current->vcpu_id, &vcpu_mask);
-            break;
-        default:
-            gdprintk(XENLOG_WARNING, "vGICD: unhandled GICD_SGIR write %"PRIregister" with wrong TargetListFilter field\n",
-                     sgir);
-            return 0;
-    }
-
-    for_each_set_bit( vcpuid, &vcpu_mask, d->max_vcpus )
-    {
-        if ( !is_vcpu_running(d, vcpuid) )
-        {
-            gdprintk(XENLOG_WARNING, "vGICD: GICD_SGIR write r=%"PRIregister" vcpu_mask=%lx, wrong CPUTargetList\n",
-                     sgir, vcpu_mask);
-            continue;
-        }
-        vgic_vcpu_inject_irq(d->vcpu[vcpuid], virtual_irq);
-    }
-    return 1;
-}
-
-static int vgic_distr_mmio_write(struct vcpu *v, mmio_info_t *info)
-{
-    struct hsr_dabt dabt = info->dabt;
-    struct cpu_user_regs *regs = guest_cpu_user_regs();
-    register_t *r = select_user_reg(regs, dabt.reg);
-    struct vgic_irq_rank *rank;
-    int offset = (int)(info->gpa - v->domain->arch.vgic.dbase);
-    int gicd_reg = REG(offset);
-    uint32_t tr;
-
-    switch ( gicd_reg )
-    {
-    case GICD_CTLR:
-        if ( dabt.size != 2 ) goto bad_width;
-        /* Ignore all but the enable bit */
-        v->domain->arch.vgic.ctlr = (*r) & GICD_CTL_ENABLE;
-        return 1;
-
-    /* R/O -- write ignored */
-    case GICD_TYPER:
-    case GICD_IIDR:
-        goto write_ignore;
-
-    /* Implementation defined -- write ignored */
-    case REG(0x020) ... REG(0x03c):
-        goto write_ignore;
-
-    case GICD_IGROUPR ... GICD_IGROUPRN:
-        /* We do not implement security extensions for guests, write ignore */
-        goto write_ignore;
-
-    case GICD_ISENABLER ... GICD_ISENABLERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISENABLER);
-        if ( rank == NULL) goto write_ignore;
-        vgic_lock_rank(v, rank);
-        tr = rank->ienable;
-        rank->ienable |= *r;
-        vgic_unlock_rank(v, rank);
-        vgic_enable_irqs(v, (*r) & (~tr), gicd_reg - GICD_ISENABLER);
-        return 1;
-
-    case GICD_ICENABLER ... GICD_ICENABLERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICENABLER);
-        if ( rank == NULL) goto write_ignore;
-        vgic_lock_rank(v, rank);
-        tr = rank->ienable;
-        rank->ienable &= ~*r;
-        vgic_unlock_rank(v, rank);
-        vgic_disable_irqs(v, (*r) & tr, gicd_reg - GICD_ICENABLER);
-        return 1;
-
-    case GICD_ISPENDR ... GICD_ISPENDRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        printk("vGICD: unhandled %s write %#"PRIregister" to ISPENDR%d\n",
-               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_ISPENDR);
-        return 0;
-
-    case GICD_ICPENDR ... GICD_ICPENDRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        printk("vGICD: unhandled %s write %#"PRIregister" to ICPENDR%d\n",
-               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_ICPENDR);
-        return 0;
-
-    case GICD_ISACTIVER ... GICD_ISACTIVERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISACTIVER);
-        if ( rank == NULL) goto write_ignore;
-        vgic_lock_rank(v, rank);
-        rank->iactive &= ~*r;
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ICACTIVER ... GICD_ICACTIVERN:
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICACTIVER);
-        if ( rank == NULL) goto write_ignore;
-        vgic_lock_rank(v, rank);
-        rank->iactive &= ~*r;
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ITARGETSR ... GICD_ITARGETSR + 7:
-        /* SGI/PPI target is read only */
-        goto write_ignore;
-
-    case GICD_ITARGETSR + 8 ... GICD_ITARGETSRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_ITARGETSR);
-        if ( rank == NULL) goto write_ignore;
-        vgic_lock_rank(v, rank);
-        if ( dabt.size == 2 )
-            rank->itargets[REG_RANK_INDEX(8, gicd_reg - GICD_ITARGETSR)] = *r;
-        else
-            byte_write(&rank->itargets[REG_RANK_INDEX(8, gicd_reg - GICD_ITARGETSR)],
-                       *r, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_IPRIORITYR ... GICD_IPRIORITYRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_IPRIORITYR);
-        if ( rank == NULL) goto write_ignore;
-        vgic_lock_rank(v, rank);
-        if ( dabt.size == 2 )
-            rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)] = *r;
-        else
-            byte_write(&rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)],
-                       *r, offset);
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_ICFGR: /* SGIs */
-        goto write_ignore;
-    case GICD_ICFGR + 1: /* PPIs */
-        /* It is implementation defined if these are writeable. We chose not */
-        goto write_ignore;
-    case GICD_ICFGR + 2 ... GICD_ICFGRN: /* SPIs */
-        if ( dabt.size != 2 ) goto bad_width;
-        rank = vgic_irq_rank(v, 2, gicd_reg - GICD_ICFGR);
-        vgic_lock_rank(v, rank);
-        if ( rank == NULL) goto write_ignore;
-        rank->icfg[REG_RANK_INDEX(2, gicd_reg - GICD_ICFGR)] = *r;
-        vgic_unlock_rank(v, rank);
-        return 1;
-
-    case GICD_NSACR ... GICD_NSACRN:
-        /* We do not implement security extensions for guests, write ignore */
-        goto write_ignore;
-
-    case GICD_SGIR:
-        if ( dabt.size != 2 )
-            goto bad_width;
-        return vgic_to_sgi(v, *r);
-
-    case GICD_CPENDSGIR ... GICD_CPENDSGIRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        printk("vGICD: unhandled %s write %#"PRIregister" to ICPENDSGIR%d\n",
-               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_CPENDSGIR);
-        return 0;
-
-    case GICD_SPENDSGIR ... GICD_SPENDSGIRN:
-        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
-        printk("vGICD: unhandled %s write %#"PRIregister" to ISPENDSGIR%d\n",
-               dabt.size ? "word" : "byte", *r, gicd_reg - GICD_SPENDSGIR);
-        return 0;
-
-    /* Implementation defined -- write ignored */
-    case REG(0xfd0) ... REG(0xfe4):
-        goto write_ignore;
-
-    /* R/O -- write ignore */
-    case GICD_ICPIDR2:
-        goto write_ignore;
-
-    /* Implementation defined -- write ignored */
-    case REG(0xfec) ... REG(0xffc):
-        goto write_ignore;
-
-    /* Reserved -- write ignored */
-    case REG(0x00c) ... REG(0x01c):
-    case REG(0x040) ... REG(0x07c):
-    case REG(0x7fc):
-    case REG(0xbfc):
-    case REG(0xf04) ... REG(0xf0c):
-    case REG(0xf30) ... REG(0xfcc):
-        goto write_ignore;
-
-    default:
-        printk("vGICD: unhandled write r%d=%"PRIregister" offset %#08x\n",
-               dabt.reg, *r, offset);
-        return 0;
-    }
-
-bad_width:
-    printk("vGICD: bad write width %d r%d=%"PRIregister" offset %#08x\n",
-           dabt.size, dabt.reg, *r, offset);
-    domain_crash_synchronous();
-    return 0;
-
-write_ignore:
-    if ( dabt.size != 2 ) goto bad_width;
-    return 1;
-}
-
-static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
-{
-    struct domain *d = v->domain;
-
-    return (addr >= (d->arch.vgic.dbase)) && (addr < (d->arch.vgic.dbase + PAGE_SIZE));
-}
-
-static struct mmio_handler vgic_distr_mmio_handler = {
-    .check_handler = vgic_distr_mmio_check,
-    .read_handler  = vgic_distr_mmio_read,
-    .write_handler = vgic_distr_mmio_write,
-};
-
 struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
 {
     struct pending_irq *n;
@@ -775,63 +235,6 @@ out:
         smp_send_event_check_mask(cpumask_of(v->processor));
 }
 
-static int vgic_vcpu_init(struct vcpu *v)
-{
-    int i;
-
-    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
-    memset(v->arch.vgic.private_irqs, 0, sizeof(struct vgic_irq_rank));
-
-    spin_lock_init(&v->arch.vgic.private_irqs->lock);
-    /* For SGI and PPI the target is always this CPU */
-    for ( i = 0 ; i < 8 ; i++ )
-        v->arch.vgic.private_irqs->itargets[i] =
-              (1<<(v->vcpu_id+0))
-            | (1<<(v->vcpu_id+8))
-            | (1<<(v->vcpu_id+16))
-            | (1<<(v->vcpu_id+24));
-    return 0;
-}
-
-static int vgic_domain_init(struct domain *d)
-{
-    int i;
-
-    d->arch.vgic.shared_irqs =
-        xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
-
-    for ( i = 0; i < DOMAIN_NR_RANKS(d); i++ )
-        spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
-
-    register_mmio_handler(d, &vgic_distr_mmio_handler);
-    return 0;
-}
-
-static struct vgic_ops ops = {
-    .vgic_vcpu_init   = vgic_vcpu_init,
-    .vgic_domain_init = vgic_domain_init,
-    .read_priority    = vgic_read_priority,
-};
-
-static int __init vgic_v2_init(struct dt_device_node *dev, const void *data)
-
-{
-    register_vgic_ops(&ops);
-    return 0;
-}
-
-static const char * const vgicv2_dt_compat[] __initconst =
-{
-    "arm,cortex-a15-gic",
-    "arm,cortex-a9-gic",
-    NULL
-};
-
-DT_DEVICE_START(gicv2, "VGIC", DEVICE_VGIC)
-        .compatible = vgicv2_dt_compat,
-        .init = vgic_v2_init,
-DT_DEVICE_END
-
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 2abe23e..290e821 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -86,6 +86,8 @@ extern void domain_vgic_free(struct domain *d);
 
 extern int vcpu_vgic_init(struct vcpu *v);
 extern int vcpu_vgic_free(struct vcpu *v);
+extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
+extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
 
 extern int vcpu_gic_init(struct vcpu *v);
 
-- 
1.7.9.5

Re: [PATCH v2 10/15] xen/arm: split vgic driver into generic and vgic-v2 driver

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> Existing vGIC driver has both generic code and hw specific
> code. Segregate vGIC low level driver into vgic-v2.c and
> keep generic code in existing vgic.c file

Just code motion? Please mention this in your commit logs so we know
what to expect (not just for this patch, but generally). If it is mostly
code motion and small small tweaks then it would be really helpful if
you could enumerate then. e.g.:

        This is all code motion except:
              * vgic_{enable,disable}_irqs become non-static so they can
                be called from the vgic-v2 driver
              * somew other change.

That would really help with review, since I can avoid looking too
closely at the code motion bits if I am confident they are really just
motion. (and I have scripts which can help automate it).

Ian.

[PATCH v2 11/15] xen/arm: make GIC context data version specific

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

GIC context data is dependent on hardware version
make the contents of gic context data structure
as version specific and access accordingly

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/gic-v2.c     |   15 ++++++++-------
 xen/include/asm-arm/gic.h |   11 ++++++++---
 2 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 06ed12b..4bcb392 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -96,6 +96,7 @@ static int gic_state_init(struct vcpu *v)
      v->arch.gic_state = xzalloc(struct gic_state_data);
      if ( !v->arch.gic_state )
         return -ENOMEM;
+     v->arch.gic_state->version = 2;
      return 0;
 }
 
@@ -108,9 +109,9 @@ static void save_state(struct vcpu *v)
      * accessed simultaneously by another pCPU.
      */
     for ( i = 0; i < nr_lrs; i++ )
-        v->arch.gic_state->gic_lr[i] = GICH[GICH_LR + i];
-    v->arch.gic_state->gic_apr = GICH[GICH_APR];
-    v->arch.gic_state->gic_vmcr = GICH[GICH_VMCR];
+        v->arch.gic_state->v2.gic_lr[i] = GICH[GICH_LR + i];
+    v->arch.gic_state->v2.gic_apr = GICH[GICH_APR];
+    v->arch.gic_state->v2.gic_vmcr = GICH[GICH_VMCR];
     /* Disable until next VCPU scheduled */
     GICH[GICH_HCR] = 0;
 }
@@ -120,9 +121,9 @@ static void restore_state(struct vcpu *v)
     int i;
 
     for ( i = 0; i < nr_lrs; i++ )
-        GICH[GICH_LR + i] = v->arch.gic_state->gic_lr[i];
-    GICH[GICH_APR] = v->arch.gic_state->gic_apr;
-    GICH[GICH_VMCR] = v->arch.gic_state->gic_vmcr;
+        GICH[GICH_LR + i] = v->arch.gic_state->v2.gic_lr[i];
+    GICH[GICH_APR] = v->arch.gic_state->v2.gic_apr;
+    GICH[GICH_VMCR] = v->arch.gic_state->v2.gic_vmcr;
     GICH[GICH_HCR] = GICH_HCR_EN;
 }
 
@@ -135,7 +136,7 @@ static void gic_dump_state(struct vcpu *v)
             printk("   HW_LR[%d]=%x\n", i, GICH[GICH_LR + i]);
     } else {
         for ( i = 0; i < nr_lrs; i++ )
-            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->gic_lr[i]);
+            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->v2.gic_lr[i]);
     }
 }
 
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 290e821..2156b55 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -67,9 +67,14 @@ struct vgic_irq_rank {
 };
 
 struct gic_state_data {
-    uint32_t gic_hcr, gic_vmcr;
-    uint32_t gic_apr;
-    uint32_t gic_lr[64];
+    uint8_t version;
+    union {
+        struct gic_v2 {
+            uint32_t gic_hcr, gic_vmcr;
+            uint32_t gic_apr;
+            uint32_t gic_lr[64];
+        }v2;
+    };
 };
 
 struct gic_lr {
-- 
1.7.9.5

Re: [PATCH v2 11/15] xen/arm: make GIC context data version specific

[Julien Grall]

Hello Vijaya,

Thank you for the patch.

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> GIC context data is dependent on hardware version
> make the contents of gic context data structure
> as version specific and access accordingly
> 
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> ---
>  xen/arch/arm/gic-v2.c     |   15 ++++++++-------
>  xen/include/asm-arm/gic.h |   11 ++++++++---
>  2 files changed, 16 insertions(+), 10 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 06ed12b..4bcb392 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -96,6 +96,7 @@ static int gic_state_init(struct vcpu *v)
>       v->arch.gic_state = xzalloc(struct gic_state_data);
>       if ( !v->arch.gic_state )
>          return -ENOMEM;
> +     v->arch.gic_state->version = 2;

I would prefer an enum rather than a fixed number.

Regards,

-- 
Julien Grall

Re: [PATCH v2 11/15] xen/arm: make GIC context data version specific

[Ian Campbell]

On Fri, 2014-04-04 at 15:09 +0100, Julien Grall wrote:
> Hello Vijaya,
> 
> Thank you for the patch.
> 
> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> > From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> > 
> > GIC context data is dependent on hardware version
> > make the contents of gic context data structure
> > as version specific and access accordingly
> > 
> > Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> > ---
> >  xen/arch/arm/gic-v2.c     |   15 ++++++++-------
> >  xen/include/asm-arm/gic.h |   11 ++++++++---
> >  2 files changed, 16 insertions(+), 10 deletions(-)
> > 
> > diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> > index 06ed12b..4bcb392 100644
> > --- a/xen/arch/arm/gic-v2.c
> > +++ b/xen/arch/arm/gic-v2.c
> > @@ -96,6 +96,7 @@ static int gic_state_init(struct vcpu *v)
> >       v->arch.gic_state = xzalloc(struct gic_state_data);
> >       if ( !v->arch.gic_state )
> >          return -ENOMEM;
> > +     v->arch.gic_state->version = 2;
> 
> I would prefer an enum rather than a fixed number.

How is this field actually used? I'd have thought the v2 and v3 specific
callbacks would just know which member of the union was "theirs" and
nothing outside of that should be poking at it anyway.

>+            printk("   VCPU_LR[%d]=%x\n", i, v->arch.gic_state->v2.gic_lr[i]);

Too many gic prefixes here, and the _state is redundant.
	v->arch.gic.v2.lr[i]
would be find IMHO.

Ian.

[PATCH v2 12/15] xen/arm: move GIC data to driver from domain structure

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Move driver specific GIC address information to driver
header file gic.h from domain.h.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/domain_build.c  |    8 ++++----
 xen/arch/arm/gic-v2.c        |   28 ++++++++++++++++++----------
 xen/arch/arm/vgic-v2.c       |    6 +++---
 xen/include/asm-arm/domain.h |    5 ++---
 xen/include/asm-arm/gic.h    |    5 +++++
 5 files changed, 32 insertions(+), 20 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index c9dda0e..66a98f1 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -577,12 +577,12 @@ static int make_gic_node(const struct domain *d, void *fdt,
 
     tmp = new_cells;
     DPRINT("  Set Distributor Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
-           d->arch.vgic.dbase, d->arch.vgic.dbase + PAGE_SIZE - 1);
-    dt_set_range(&tmp, node, d->arch.vgic.dbase, PAGE_SIZE);
+           d->arch.vgic.info->dbase, d->arch.vgic.info->dbase + PAGE_SIZE - 1);
+    dt_set_range(&tmp, node, d->arch.vgic.info->dbase, PAGE_SIZE);
 
     DPRINT("  Set Cpu Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
-           d->arch.vgic.cbase, d->arch.vgic.cbase + (PAGE_SIZE * 2) - 1);
-    dt_set_range(&tmp, node, d->arch.vgic.cbase, PAGE_SIZE * 2);
+           d->arch.vgic.info->cbase, d->arch.vgic.info->cbase + (PAGE_SIZE * 2) - 1);
+    dt_set_range(&tmp, node, d->arch.vgic.info->cbase, PAGE_SIZE * 2);
 
     res = fdt_property(fdt, "reg", new_cells, len);
     xfree(new_cells);
diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 4bcb392..6443bf4 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -413,19 +413,27 @@ static int gicv_init(struct domain *d)
 {
     int ret;
 
+    d->arch.vgic.info = xzalloc(struct vgic_info);
+    if ( !d->arch.vgic.info )
+    {
+        dprintk(XENLOG_ERR, "Failed to allocate memory for vgic_info\n");
+        return -ENOMEM;
+    }    
+    memset(d->arch.vgic.info, 0, sizeof(d->arch.vgic.info));
+
     /*
      * Domain 0 gets the hardware address.
      * Guests get the virtual platform layout.
      */
     if ( d->domain_id == 0 )
     {
-        d->arch.vgic.dbase = gic.dbase;
-        d->arch.vgic.cbase = gic.cbase;
+        d->arch.vgic.info->dbase = gic.dbase;
+        d->arch.vgic.info->cbase = gic.cbase;
     }
     else
     {
-        d->arch.vgic.dbase = GUEST_GICD_BASE;
-        d->arch.vgic.cbase = GUEST_GICC_BASE;
+        d->arch.vgic.info->dbase = GUEST_GICD_BASE;
+        d->arch.vgic.info->cbase = GUEST_GICC_BASE;
     }
 
     d->arch.vgic.nr_lines = 0;
@@ -437,19 +445,19 @@ static int gicv_init(struct domain *d)
      * The second page is always mapped at +4K irrespective of the
      * GIC_64K_STRIDE quirk. The DTB passed to the guest reflects this.
      */
-    ret = map_mmio_regions(d, d->arch.vgic.cbase,
-                           d->arch.vgic.cbase + PAGE_SIZE - 1,
+    ret = map_mmio_regions(d, d->arch.vgic.info->cbase,
+                           d->arch.vgic.info->cbase + PAGE_SIZE - 1,
                            gic.vbase);
     if ( ret )
         return ret;
 
     if ( !platform_has_quirk(PLATFORM_QUIRK_GIC_64K_STRIDE) )
-        ret = map_mmio_regions(d, d->arch.vgic.cbase + PAGE_SIZE,
-                               d->arch.vgic.cbase + (2 * PAGE_SIZE) - 1,
+        ret = map_mmio_regions(d, d->arch.vgic.info->cbase + PAGE_SIZE,
+                               d->arch.vgic.info->cbase + (2 * PAGE_SIZE) - 1,
                                gic.vbase + PAGE_SIZE);
     else
-        ret = map_mmio_regions(d, d->arch.vgic.cbase + PAGE_SIZE,
-                               d->arch.vgic.cbase + (2 * PAGE_SIZE) - 1,
+        ret = map_mmio_regions(d, d->arch.vgic.info->cbase + PAGE_SIZE,
+                               d->arch.vgic.info->cbase + (2 * PAGE_SIZE) - 1,
                                gic.vbase + 16*PAGE_SIZE);
 
     return ret;
diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index 2919665..d189f6b 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -116,7 +116,7 @@ static int vgic_distr_mmio_read(struct vcpu *v, mmio_info_t *info)
     struct cpu_user_regs *regs = guest_cpu_user_regs();
     register_t *r = select_user_reg(regs, dabt.reg);
     struct vgic_irq_rank *rank;
-    int offset = (int)(info->gpa - v->domain->arch.vgic.dbase);
+    int offset = (int)(info->gpa - v->domain->arch.vgic.info->dbase);
     int gicd_reg = REG(offset);
 
     switch ( gicd_reg )
@@ -378,7 +378,7 @@ static int vgic_distr_mmio_write(struct vcpu *v, mmio_info_t *info)
     struct cpu_user_regs *regs = guest_cpu_user_regs();
     register_t *r = select_user_reg(regs, dabt.reg);
     struct vgic_irq_rank *rank;
-    int offset = (int)(info->gpa - v->domain->arch.vgic.dbase);
+    int offset = (int)(info->gpa - v->domain->arch.vgic.info->dbase);
     int gicd_reg = REG(offset);
     uint32_t tr;
 
@@ -562,7 +562,7 @@ static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
 {
     struct domain *d = v->domain;
 
-    return (addr >= (d->arch.vgic.dbase)) && (addr < (d->arch.vgic.dbase + PAGE_SIZE));
+    return (addr >= (d->arch.vgic.info->dbase)) && (addr < (d->arch.vgic.info->dbase + PAGE_SIZE));
 }
 
 static struct mmio_handler vgic_distr_mmio_handler = {
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index c51b3cb..35d8bfa 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -148,9 +148,8 @@ struct arch_domain
          * struct arch_vcpu.
          */
         struct pending_irq *pending_irqs;
-        /* Base address for guest GIC */
-        paddr_t dbase; /* Distributor base address */
-        paddr_t cbase; /* CPU base address */
+        /* holds gic address information */
+        struct vgic_info *info;
     } vgic;
 
     struct vuart {
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 2156b55..b49dde1 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -66,6 +66,11 @@ struct vgic_irq_rank {
     uint32_t itargets[8];
 };
 
+struct vgic_info {
+    paddr_t dbase; /* Distributor base address */
+    paddr_t cbase; /* CPU base address */
+};
+
 struct gic_state_data {
     uint8_t version;
     union {
-- 
1.7.9.5

Re: [PATCH v2 12/15] xen/arm: move GIC data to driver from domain structure

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> Move driver specific GIC address information to driver
> header file gic.h from domain.h.

Like with the other gic state I think you can avoid dynamic allocation,
at least for now. (OTOH if you find the vcpu struct is getting too big,
please say)

Does the v3 gic have a different set of addresses?  Actually I see that
you just add dbase_size and rbase to the struct so I'm not sure it
really needs refactoring like this. Either a union or just
	struct ... {
		/* common */
		... pending_irq
		... dbase
		/* vgic v2 only */
		... cbase
		/* vgic v3 only */
		... rbase
	} vgic
would be fine here IMHO...

[PATCH v2 13/15] xen/arm: Add support for GIC v3

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Add support for GIC v3 specification
System register access(SRE) is enabled
to access cpu and virtual interface regiseters

This patch adds only basic v3 support.
Does not support Interrupt Translation support (ITS)

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/Makefile             |    7 +-
 xen/arch/arm/gic-v3.c             |  919 +++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/gic.h         |   10 +
 xen/include/asm-arm/gic_v3_defs.h |  187 ++++++++
 4 files changed, 1121 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 20f59f4..52f707c 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -10,7 +10,9 @@ obj-y += vpsci.o
 obj-y += domctl.o
 obj-y += sysctl.o
 obj-y += domain_build.o
-obj-y += gic.o gic-v2.o
+obj-y += gic.o
+obj-y += gic-v3.o
+obj-y += gic-v2.o
 obj-y += io.o
 obj-y += irq.o
 obj-y += kernel.o
@@ -26,7 +28,8 @@ obj-y += smpboot.o
 obj-y += smp.o
 obj-y += shutdown.o
 obj-y += traps.o
-obj-y += vgic.o vgic-v2.o
+obj-y += vgic.o
+obj-y += vgic-v2.o
 obj-y += vtimer.o
 obj-y += vuart.o
 obj-y += hvm.o
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
new file mode 100644
index 0000000..8cefdab
--- /dev/null
+++ b/xen/arch/arm/gic-v3.c
@@ -0,0 +1,919 @@
+/*
+ * xen/arch/arm/gic-v3.c
+ *
+ * ARM Generic Interrupt Controller support v3 version
+ * based on xen/arch/arm/gic-v2.c
+ * 
+ * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
+ * Copyright (c) 2014 Cavium Inc.
+ * 
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/config.h>
+#include <xen/lib.h>
+#include <xen/init.h>
+#include <xen/cpu.h>
+#include <xen/mm.h>
+#include <xen/irq.h>
+#include <xen/sched.h>
+#include <xen/errno.h>
+#include <xen/serial.h>
+#include <xen/softirq.h>
+#include <xen/list.h>
+#include <xen/device_tree.h>
+#include <asm/p2m.h>
+#include <asm/domain.h>
+#include <asm/platform.h>
+
+#include <asm/gic_v3_defs.h>
+#include <asm/gic.h>
+#include <asm/io.h>
+#include <asm/device.h>
+
+#define SZ_64K  0x00010000
+
+struct rdist_region {
+    paddr_t rdist_base;
+    paddr_t rdist_base_size;
+    void __iomem *map_rdist_base;
+};
+
+/* Global state */
+static struct {
+    paddr_t dbase;            /* Address of distributor registers */
+    paddr_t dbase_size;
+    void __iomem *map_dbase;  /* Mapped address of distributor registers */
+    struct rdist_region *rdist_regions;
+    u32  rdist_stride;
+    unsigned int rdist_count; /* Number of rdist regions count */
+    unsigned int lines;       /* Number of interrupts (SPIs + PPIs + SGIs) */
+    struct dt_irq maintenance;
+    unsigned int cpus;
+} gic;
+
+#define GICD (gic.map_dbase)
+
+/* Only one region is implemented which is enough for 0-31 cpus */
+#define GICR (gic.rdist_regions[0].map_rdist_base)
+
+/* per-cpu re-distributor base */
+static DEFINE_PER_CPU(paddr_t, rbase);
+
+static unsigned nr_lrs;
+static uint32_t nr_priorities;
+
+#define gic_data_rdist_rd_base()        (this_cpu(rbase))
+#define gic_data_rdist_sgi_base()       (gic_data_rdist_rd_base() + SZ_64K)
+
+static inline u64 read_cpuid_mpidr(void)
+{
+   return READ_SYSREG(MPIDR_EL1);
+}
+
+static u64 gich_read_lr(int lr)
+{
+    switch ( lr ) 
+    {
+    case 0: /* ICH_LRn is 64 bit */
+        return READ_SYSREG(ICH_LR0_EL2);
+        break;
+    case 1:
+        return READ_SYSREG(ICH_LR1_EL2);
+        break;
+    case 2:
+        return READ_SYSREG(ICH_LR2_EL2);
+        break;
+    case 3:
+        return READ_SYSREG(ICH_LR3_EL2);
+        break;
+    case 4:
+        return READ_SYSREG(ICH_LR4_EL2);
+        break;
+    case 5:
+        return READ_SYSREG(ICH_LR5_EL2);
+        break;
+    case 6:
+        return READ_SYSREG(ICH_LR6_EL2);
+        break;
+    case 7:
+        return READ_SYSREG(ICH_LR7_EL2);
+        break;
+    case 8:
+        return READ_SYSREG(ICH_LR8_EL2);
+        break;
+    case 9:
+        return READ_SYSREG(ICH_LR9_EL2);
+        break;
+    case 10:
+        return READ_SYSREG(ICH_LR10_EL2);
+        break;
+    case 11:
+        return READ_SYSREG(ICH_LR11_EL2);
+        break;
+    case 12:
+        return READ_SYSREG(ICH_LR12_EL2);
+        break;
+    case 13:
+        return READ_SYSREG(ICH_LR13_EL2);
+        break;
+    case 14:
+        return READ_SYSREG(ICH_LR14_EL2);
+        break;
+    case 15:
+        return READ_SYSREG(ICH_LR15_EL2);
+        break;
+    default:
+        return 0;
+    }
+}
+
+static void gich_write_lr(int lr, u64 val)
+{
+    switch ( lr ) 
+    {
+    case 0:
+        WRITE_SYSREG(val, ICH_LR0_EL2);
+        break;
+    case 1:
+        WRITE_SYSREG(val, ICH_LR1_EL2);
+        break;
+    case 2:
+        WRITE_SYSREG(val, ICH_LR2_EL2);
+        break;
+    case 3:
+        WRITE_SYSREG(val, ICH_LR3_EL2);
+        break;
+    case 4:
+        WRITE_SYSREG(val, ICH_LR4_EL2);
+        break;
+    case 5:
+        WRITE_SYSREG(val, ICH_LR5_EL2);
+        break;
+    case 6:
+        WRITE_SYSREG(val, ICH_LR6_EL2);
+        break;
+    case 7:
+        WRITE_SYSREG(val, ICH_LR7_EL2);
+        break;
+    case 8:
+        WRITE_SYSREG(val, ICH_LR8_EL2);
+        break;
+    case 9:
+        WRITE_SYSREG(val, ICH_LR9_EL2);
+        break;
+    case 10:
+        WRITE_SYSREG(val, ICH_LR10_EL2);
+        break;
+    case 11:
+        WRITE_SYSREG(val, ICH_LR11_EL2);
+        break;
+    case 12:
+        WRITE_SYSREG(val, ICH_LR12_EL2);
+        break;
+    case 13:
+        WRITE_SYSREG(val, ICH_LR13_EL2);
+        break;
+    case 14:
+        WRITE_SYSREG(val, ICH_LR14_EL2);
+        break;
+    case 15:
+        WRITE_SYSREG(val, ICH_LR15_EL2);
+        break;
+    default:
+        return;
+    }
+}
+
+/* Enable System register access (SRE) bit
+ * With this CPU & Virtual interface registers are 
+ * accessed as system registers instead of memory mapped
+ */
+static void gic_enable_sre(void)
+{
+    uint32_t val;
+
+    val = READ_SYSREG32(ICC_SRE_EL2);
+    val |= GICC_SRE_EL2_SRE | GICC_SRE_EL2_ENEL1 | GICC_SRE_EL2_DFB | GICC_SRE_EL2_DIB;
+    WRITE_SYSREG32(val, ICC_SRE_EL2);
+    isb();
+}
+
+/* Wait for completion of a distributor change */
+static void gic_do_wait_for_rwp(paddr_t base)
+{
+    u32 val;
+    do {
+        val = readl_relaxed((void *)base + GICD_CTLR);
+        val = readl_relaxed(GICD + GICD_CTLR);
+        cpu_relax();
+    } while ( val & GICD_CTLR_RWP );
+}
+
+static void gic_dist_wait_for_rwp(void)
+{
+    gic_do_wait_for_rwp((paddr_t)GICD);
+}
+
+static void gic_redist_wait_for_rwp(void)
+{
+    gic_do_wait_for_rwp(gic_data_rdist_rd_base());
+}
+
+static void gic_wait_for_rwp(int irq)
+{
+    if ( irq < 32 )
+         gic_redist_wait_for_rwp();
+    else
+         gic_dist_wait_for_rwp();
+}
+
+static unsigned int gic_mask_cpu(const cpumask_t *cpumask)
+{
+    unsigned int cpu;
+    cpumask_t possible_mask;
+
+    cpumask_and(&possible_mask, cpumask, &cpu_possible_map);
+    cpu = cpumask_any(&possible_mask);
+    return cpu;
+}
+
+static unsigned int gic_nr_lines(void)
+{
+    return gic.lines;
+}
+
+static unsigned int gic_nr_lrs(void)
+{
+    return nr_lrs;
+}
+
+static void write_aprn_regs(struct gic_state_data *d)
+{
+    switch ( nr_priorities )
+    {
+    case 7:
+        WRITE_SYSREG32(d->v3.gic_apr0[2], ICH_AP0R2_EL2);
+        WRITE_SYSREG32(d->v3.gic_apr1[2], ICH_AP1R2_EL2);
+    case 6:
+        WRITE_SYSREG32(d->v3.gic_apr0[1], ICH_AP0R1_EL2);
+        WRITE_SYSREG32(d->v3.gic_apr1[1], ICH_AP1R1_EL2);
+    case 5:
+        WRITE_SYSREG32(d->v3.gic_apr0[0], ICH_AP0R0_EL2);
+        WRITE_SYSREG32(d->v3.gic_apr1[0], ICH_AP1R0_EL2);
+        break;
+    default:
+        panic("Write Undefined active priorities \n");
+    }
+}
+
+static void read_aprn_regs(struct gic_state_data *d)
+{
+    switch ( nr_priorities )
+    {
+    case 7:
+        d->v3.gic_apr0[2] = READ_SYSREG32(ICH_AP0R2_EL2);
+        d->v3.gic_apr1[2] = READ_SYSREG32(ICH_AP1R2_EL2);
+    case 6:
+        d->v3.gic_apr0[1] = READ_SYSREG32(ICH_AP0R1_EL2);
+        d->v3.gic_apr1[1] = READ_SYSREG32(ICH_AP1R1_EL2);
+    case 5:
+        d->v3.gic_apr0[0] = READ_SYSREG32(ICH_AP0R0_EL2);
+        d->v3.gic_apr1[0] = READ_SYSREG32(ICH_AP1R0_EL2);
+        break;
+    default:
+        panic("Read Undefined active priorities \n");
+    }
+}
+
+static int gic_state_init(struct vcpu *v)
+{
+     v->arch.gic_state = xzalloc(struct gic_state_data);
+     if( !v->arch.gic_state )
+        return -ENOMEM;
+     return 0; 
+}
+
+static void save_state(struct vcpu *v)
+{
+    int i;
+
+    /* No need for spinlocks here because interrupts are disabled around
+     * this call and it only accesses struct vcpu fields that cannot be
+     * accessed simultaneously by another pCPU.
+     */
+    for ( i = 0; i < nr_lrs; i++)
+        v->arch.gic_state->v3.gic_lr[i] = gich_read_lr(i);
+
+    read_aprn_regs(v->arch.gic_state); 
+    v->arch.gic_state->v3.gic_vmcr = READ_SYSREG32(ICH_VMCR_EL2);
+}
+
+static void restore_state(struct vcpu *v)
+{
+    int i;
+
+    for ( i = 0; i < nr_lrs; i++)
+        gich_write_lr(i, v->arch.gic_state->v3.gic_lr[i]);
+
+    write_aprn_regs(v->arch.gic_state);
+    WRITE_SYSREG32(v->arch.gic_state->v3.gic_vmcr, ICH_VMCR_EL2);
+}
+
+static void gic_dump_state(struct vcpu *v)
+{
+    int i;
+    if ( v == current )
+    {
+        for ( i = 0; i < nr_lrs; i++ )
+            printk("   HW_LR[%d]=%lx\n", i, gich_read_lr(i));
+    }
+    else
+    {
+        for ( i = 0; i < nr_lrs; i++ )
+            printk("   VCPU_LR[%d]=%lx\n", i, v->arch.gic_state->v3.gic_lr[i]);
+    }
+}
+ 
+static void gic_enable_irq(int irq)
+{
+    uint32_t enabler;
+
+    /* Enable routing */
+    if ( irq > 31 )
+    {
+        enabler = readl_relaxed(GICD + GICD_ISENABLER + (irq / 32) * 4);
+        writel_relaxed(enabler | (1u << (irq % 32)), GICD + GICD_ISENABLER + (irq / 32) * 4);
+    }
+    else
+    {
+        enabler = readl_relaxed((void *)gic_data_rdist_sgi_base() + GICR_ISENABLER0);
+        writel_relaxed(enabler | (1u << irq), (void *)gic_data_rdist_sgi_base() + GICR_ISENABLER0);
+    }
+    gic_wait_for_rwp(irq);
+}
+
+static void gic_disable_irq(int irq)
+{
+    /* Disable routing */
+    if ( irq > 31 )
+        writel_relaxed(1u << (irq % 32), GICD + GICD_ICENABLER + ((irq / 32) * 4));
+    else
+        writel_relaxed(1u << irq, (void *)gic_data_rdist_sgi_base() + GICR_ICENABLER0);
+}
+
+static void gic_eoi_irq(int irq)
+{
+    /* Lower the priority */
+    WRITE_SYSREG32(irq, ICC_EOIR1_EL1);
+}
+
+static void gic_dir_irq(int irq)
+{
+    /* Deactivate */
+    WRITE_SYSREG32(irq, ICC_DIR_EL1);
+}
+
+static unsigned int gic_ack_irq(void)
+{
+    return (READ_SYSREG32(ICC_IAR1_EL1) & GICC_IA_IRQ);
+}
+
+static u64 gic_mpidr_to_affinity(u64 mpidr)
+{
+    /* Make sure we don't broadcast the interrupt */
+    return mpidr & ~GICD_IROUTER_SPI_MODE_ANY;
+}
+
+/*
+ * - needs to be called with gic.lock held
+ * - needs to be called with a valid cpu_mask, ie each cpu in the mask has
+ * already called gic_cpu_init
+ */
+static void gic_set_irq_property(unsigned int irq, bool_t level,
+                                   const cpumask_t *cpu_mask,
+                                   unsigned int priority)
+{
+    uint32_t cfg, edgebit;
+    u64 affinity;
+    unsigned int cpu = gic_mask_cpu(cpu_mask);
+    paddr_t rebase;
+
+
+    /* Set edge / level */
+    if ( irq < 16 )
+        /* SGI's are always edge-triggered not need to call GICD_ICFGR0 */
+        cfg = readl_relaxed((void *)gic_data_rdist_sgi_base() + GICR_ICFGR0);
+    else if ( irq < 32 )
+        cfg = readl_relaxed((void *)gic_data_rdist_sgi_base() + GICR_ICFGR1);
+    else
+        cfg = readl_relaxed(GICD + GICD_ICFGR + (irq / 16) * 4);
+
+    edgebit = 2u << (2 * (irq % 16));
+    if ( level )
+        cfg &= ~edgebit;
+    else
+        cfg |= edgebit;
+
+    if ( irq < 16 )
+       writel_relaxed(cfg, (void *)gic_data_rdist_sgi_base() + GICR_ICFGR0);
+    else if ( irq < 32 )
+       writel_relaxed(cfg, (void *)gic_data_rdist_sgi_base() + GICR_ICFGR1);
+    else
+       writel_relaxed(cfg, GICD + GICD_ICFGR + (irq / 16) * 4);
+
+    affinity = gic_mpidr_to_affinity(cpu_logical_map(cpu));
+    if ( irq > 31 )
+        writeq_relaxed(affinity, (GICD + GICD_IROUTER + irq * 8));
+
+    /* Set priority */
+    if ( irq < 32 )
+    {
+        rebase = gic_data_rdist_sgi_base();
+        writeb_relaxed(priority, (void *)rebase + GICR_IPRIORITYR0 + irq);
+    }
+    else 
+        writeb_relaxed(priority, GICD + GICD_IPRIORITYR + irq);
+}
+
+static void __init gic_dist_init(void)
+{
+    uint32_t type;
+    u64 affinity;
+    int i;
+
+    /* Disable the distributor */
+    writel_relaxed(0, GICD + GICD_CTLR);
+
+    type = readl_relaxed(GICD + GICD_TYPER);
+    gic.lines = 32 * ((type & GICD_TYPE_LINES) + 1);
+
+    printk("GIC: %d lines, (IID %8.8x).\n",
+           gic.lines, readl_relaxed(GICD + GICD_IIDR));
+
+    /* Default all global IRQs to level, active low */
+    for ( i = 32; i < gic.lines; i += 16 )
+        writel_relaxed(0, GICD + GICD_ICFGR + (i / 16) * 4);
+
+    /* Default priority for global interrupts */
+    for ( i = 32; i < gic.lines; i += 4 )
+        writel_relaxed((GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ), GICD + GICD_IPRIORITYR + (i / 4) * 4);
+
+    /* Disable all global interrupts */
+    for ( i = 32; i < gic.lines; i += 32 )
+        writel_relaxed(0xffffffff, GICD + GICD_ICENABLER + (i / 32) * 4);
+
+    gic_dist_wait_for_rwp();
+
+    /* Turn on the distributor */
+    writel_relaxed(GICD_CTL_ENABLE | GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1, GICD + GICD_CTLR);
+ 
+    /* Route all global IRQs to this CPU */
+    affinity = gic_mpidr_to_affinity(read_cpuid_mpidr());
+    for ( i = 32; i < gic.lines; i++ )
+        writeq_relaxed(affinity, GICD + GICD_IROUTER + i * 8);
+}
+
+static void gic_enable_redist(void)
+{
+    paddr_t rbase;
+    u32 val;
+
+    rbase = this_cpu(rbase);
+
+    /* Wake up this CPU redistributor */
+    val = readl_relaxed((void *)rbase + GICR_WAKER);
+    val &= ~GICR_WAKER_ProcessorSleep;
+    writel_relaxed(val, (void *)rbase + GICR_WAKER);
+
+    do {
+         val = readl_relaxed((void *)rbase + GICR_WAKER);
+         cpu_relax();
+    } while ( val & GICR_WAKER_ChildrenAsleep );
+}
+
+static int __init gic_populate_rdist(void)
+{
+    u64 mpidr = cpu_logical_map(smp_processor_id());
+    u64 typer;
+    u64 aff;
+    int i;
+    uint32_t reg;
+
+    aff  = mpidr & ((1 << 24) - 1);
+    aff |= (mpidr >> 8) & (0xffUL << 24);
+
+    for ( i = 0; i < gic.rdist_count; i++ )
+    {
+        uint32_t ptr = 0;
+
+        reg = readl_relaxed(GICR + ptr + GICR_PIDR0);
+        if ( (reg & 0xff) != GICR_PIDR0_GICv3 ) {
+            printk("No redistributor present @%x\n", ptr);
+            break;
+        }
+
+        do {
+            typer = readq_relaxed(GICR + ptr + GICR_TYPER);
+            if ( (typer >> 32) == aff )
+            {
+                this_cpu(rbase) = (u64)GICR + ptr;
+                printk("CPU%d: found redistributor %llx region %d\n",
+                            smp_processor_id(), (unsigned long long) mpidr, i);
+                return 0;
+            }
+
+            if ( gic.rdist_stride ) {
+                ptr += gic.rdist_stride;
+            } else {
+                ptr += SZ_64K * 2; /* Skip RD_base + SGI_base */
+                if ( typer & GICR_TYPER_VLPIS )
+                    ptr += SZ_64K * 2; /* Skip VLPI_base + reserved page */
+            }
+        } while ( !(typer & GICR_TYPER_LAST) );
+    }
+
+    /* We couldn't even deal with ourselves... */
+    printk("CPU%d: mpidr %lx has no re-distributor!\n",
+              smp_processor_id(), (unsigned long)mpidr);
+    return -ENODEV;
+}
+
+static void __cpuinit gic_cpu_init(void)
+{
+    int i;
+    paddr_t rbase_sgi;
+
+    /* Register ourselves with the rest of the world */
+    if ( gic_populate_rdist() )
+        return;
+
+    gic_enable_redist();
+
+    rbase_sgi = gic_data_rdist_sgi_base();
+
+    /*
+     * Set priority on PPI and SGI interrupts
+     */
+    for (i = 0; i < 16; i += 4)
+        writel_relaxed((GIC_PRI_IPI<<24 | GIC_PRI_IPI<<16 | GIC_PRI_IPI<<8 | GIC_PRI_IPI), (void *)rbase_sgi + GICR_IPRIORITYR0 + (i / 4) * 4);
+
+    for (i = 16; i < 32; i += 4)
+        writel_relaxed((GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ), (void *)rbase_sgi + GICR_IPRIORITYR0 + (i / 4) * 4);
+
+    /*
+     * Disable all PPI interrupts, ensure all SGI interrupts are
+     * enabled.
+     */
+    writel_relaxed(0xffff0000, (void *)rbase_sgi + GICR_ICENABLER0);
+    writel_relaxed(0x0000ffff, (void *)rbase_sgi + GICR_ISENABLER0);
+
+    gic_redist_wait_for_rwp();
+
+    /* Enable system registers */
+    gic_enable_sre();
+
+    WRITE_SYSREG32(0, ICC_BPR1_EL1);
+    /* Set priority mask register */
+    WRITE_SYSREG32(DEFAULT_PMR_VALUE, ICC_PMR_EL1);
+
+    /* EOI drops priority too (mode 0) */
+    WRITE_SYSREG32(GICC_CTLR_EL1_EOImode_drop, ICC_CTLR_EL1);
+
+    WRITE_SYSREG32(1, ICC_IGRPEN1_EL1);
+}
+
+static void gic_cpu_disable(void)
+{
+    WRITE_SYSREG32(0, ICC_CTLR_EL1);
+}
+
+static void __cpuinit gic_hyp_init(void)
+{
+    uint32_t vtr;
+
+    vtr = READ_SYSREG32(ICH_VTR_EL2);
+    nr_lrs  = (vtr & GICH_VTR_NRLRGS) + 1;
+    nr_priorities = ((vtr >> GICH_VTR_PRIBITS_SHIFT) & GICH_VTR_PRIBITS_MASK) + 1;
+
+    WRITE_SYSREG32(GICH_VMCR_EOI | GICH_VMCR_VENG1, ICH_VMCR_EL2);
+    WRITE_SYSREG32(GICH_HCR_EN, ICH_HCR_EL2);
+
+    update_cpu_lr_mask();
+    vtr = READ_SYSREG32(ICH_HCR_EL2);
+}
+
+/* Set up the per-CPU parts of the GIC for a secondary CPU */
+static void gic_secondary_cpu_init(void)
+{
+    gic_cpu_init();
+    gic_hyp_init();
+}
+
+static void __cpuinit gic_hyp_disable(void)
+{
+    uint32_t vtr;
+    vtr = READ_SYSREG32(ICH_HCR_EL2);
+    vtr &= ~0x1;
+    WRITE_SYSREG32( vtr, ICH_HCR_EL2);
+}
+
+static u16 gic_compute_target_list(int *base_cpu, const struct cpumask *mask,
+                                   u64 cluster_id)
+{
+    int cpu = *base_cpu;
+    u64 mpidr = cpu_logical_map(cpu);
+    u16 tlist = 0;
+
+    while ( cpu < nr_cpu_ids )
+    {
+        /*
+         * If we ever get a cluster of more than 16 CPUs, just
+         * scream and skip that CPU.
+         */
+        tlist |= 1 << (mpidr & 0xf);
+
+        cpu = cpumask_next(cpu, mask);
+        mpidr = cpu_logical_map(cpu);
+
+        if ( cluster_id != (mpidr & ~0xffUL) ) {
+            cpu--;
+            goto out;
+        }
+    }
+out:
+    *base_cpu = cpu;
+    return tlist;
+}
+
+static void send_sgi(u64 cluster_id, u16 tlist, unsigned int irq)
+{
+    u64 val;
+
+    val  = (cluster_id & 0xff00ff0000UL) << 16; /* Aff3 + Aff2 */
+    val |= (cluster_id & 0xff00) << 8;          /* Aff1 */
+    val |= irq << 24;
+    val |= tlist;
+
+    WRITE_SYSREG(val, ICC_SGI1R_EL1);   
+}
+
+static void gic_send_sgi(const cpumask_t *cpumask, enum gic_sgi sgi)
+{
+    int cpu = 0;
+
+    dsb(sy);
+
+    for_each_cpu(cpu, cpumask)
+    {
+        u64 cluster_id = cpu_logical_map(cpu) & ~0xffUL;
+        u16 tlist;
+
+        tlist = gic_compute_target_list(&cpu, cpumask, cluster_id);
+        send_sgi(cluster_id, tlist, sgi);
+    }
+}
+
+/* Shut down the per-CPU GIC interface */
+static void gic_disable_interface(void)
+{
+    gic_cpu_disable();
+    gic_hyp_disable();
+}
+
+static void gic_update_lr(int lr, struct pending_irq *p, unsigned int state)
+{
+    u64 grp = GICH_LR_GRP1;
+    u64 val = 0;
+
+    BUG_ON(lr >= nr_lrs);
+    BUG_ON(lr < 0);
+
+    val =  ((((u64)state) & 0x3) << GICH_LR_STATE_SHIFT) | grp |
+        ((((u64)p->priority) & 0xff) << GICH_LR_PRIORITY_SHIFT) |
+        (((u64)p->irq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT);
+
+   if ( p->desc != NULL )
+        val |= GICH_LR_HW |(((u64) p->desc->irq & GICH_LR_PHYSICAL_MASK) << GICH_LR_PHYSICAL_SHIFT);
+
+    gich_write_lr(lr, val);
+}
+
+static void gic_clear_lr(int lr)
+{
+    gich_write_lr(lr, 0);
+}
+
+static void gic_read_lr(int lr, struct gic_lr *lr_reg)
+{
+    u64 lrv;
+    lrv = gich_read_lr(lr);
+    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
+    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK; 
+    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
+    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
+    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
+    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
+}
+
+static void gic_write_lr(int lr, struct gic_lr *lr_reg)
+{
+    u64 lrv = 0;
+    lrv = ( ((u64)(lr_reg->pirq & GICH_LR_PHYSICAL_MASK) << GICH_LR_PHYSICAL_SHIFT)  |
+            ((u64)(lr_reg->virq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT)   |
+            ((u64)(lr_reg->priority & GICH_LR_PRIORITY_MASK) << GICH_LR_PRIORITY_SHIFT) |
+            ((u64)(lr_reg->state & GICH_LR_STATE_MASK) << GICH_LR_STATE_SHIFT) |
+            ((u64)(lr_reg->hw_status & GICH_LR_HW_MASK)  << GICH_LR_HW_SHIFT)  |
+            ((u64)(lr_reg->grp & GICH_LR_GRP_MASK) << GICH_LR_GRP_SHIFT) );
+    gich_write_lr(lr, lrv);
+}
+
+int gicv_init(struct domain *d)
+{
+
+    d->arch.vgic.info = xzalloc(struct vgic_info);
+    if ( !d->arch.vgic.info )
+    {
+        dprintk(XENLOG_ERR, "Failed to allocate memory for vgic_info\n");
+        return -ENOMEM;
+    }    
+    memset(d->arch.vgic.info, 0, sizeof(d->arch.vgic.info));
+    /*
+     * Domain 0 gets the hardware address.
+     * Guests get the virtual platform layout.
+     */
+    if ( d->domain_id == 0 )
+    {
+        d->arch.vgic.info->dbase = gic.dbase;
+        d->arch.vgic.info->dbase_size = gic.dbase_size;
+        d->arch.vgic.info->rbase = gic.rdist_regions[0].rdist_base;
+        d->arch.vgic.info->rbase_size = gic.rdist_regions[0].rdist_base_size;
+        d->arch.vgic.info->rdist_stride = gic.rdist_stride;
+    }
+    else
+    {
+        d->arch.vgic.info->dbase = GUEST_GICD_BASE;
+    }
+    d->arch.vgic.nr_lines = 0;
+    return 0;
+}
+
+static struct dt_irq * gic_maintenance_irq(void)
+{
+    return &gic.maintenance;
+}
+
+static void gic_hcr_status(uint32_t flag, uint8_t status)
+{
+    if ( status )
+      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) | flag), ICH_HCR_EL2);
+    else
+      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) & (~flag)), ICH_HCR_EL2);
+}
+
+static unsigned int gic_read_vmcr_priority(void)
+{
+   return ((READ_SYSREG32(ICH_VMCR_EL2) >> GICH_VMCR_PRIORITY_SHIFT) &
+            GICH_VMCR_PRIORITY_MASK);
+}
+
+static struct gic_hw_operations gic_ops = {
+    .nr_lines            = gic_nr_lines,
+    .nr_lrs              = gic_nr_lrs,
+    .get_maintenance_irq = gic_maintenance_irq,
+    .state_init          = gic_state_init,
+    .save_state          = save_state,
+    .restore_state       = restore_state,
+    .dump_state          = gic_dump_state,
+    .gicv_setup          = gicv_init,
+    .enable_irq          = gic_enable_irq,
+    .disable_irq         = gic_disable_irq,
+    .eoi_irq             = gic_eoi_irq,
+    .deactivate_irq      = gic_dir_irq,
+    .ack_irq             = gic_ack_irq,
+    .set_irq_property    = gic_set_irq_property,
+    .send_sgi            = gic_send_sgi,
+    .disable_interface   = gic_disable_interface,
+    .update_lr           = gic_update_lr,
+    .update_hcr_status   = gic_hcr_status,
+    .clear_lr            = gic_clear_lr,
+    .read_lr             = gic_read_lr,
+    .write_lr            = gic_write_lr,
+    .read_vmcr_priority  = gic_read_vmcr_priority,
+    .secondary_init      = gic_secondary_cpu_init,
+};
+
+/* Set up the GIC */
+static int __init gicv3_init(struct dt_device_node *node, const void *data)
+{
+    struct rdist_region *rdist_regs;
+    int res, i;
+    uint32_t reg;
+
+    dt_device_set_used_by(node, DOMID_XEN);
+
+    res = dt_device_get_address(node, 0, &gic.dbase, &gic.dbase_size);
+    if ( res || !gic.dbase  || (gic.dbase & ~PAGE_MASK) || (gic.dbase_size & ~PAGE_MASK) )
+        panic("GIC: Cannot find a valid address for the distributor");
+
+    gic.map_dbase = ioremap_nocache(gic.dbase, gic.dbase_size);
+    if ( !gic.map_dbase )
+    {
+        dprintk(XENLOG_ERR, "Unable to map GIC distributor\n");
+        return -EFAULT;
+    }
+
+    reg = readl_relaxed(GICD + GICD_PIDR0);
+    if ((reg & 0xff) != GICD_PIDR0_GICv3)
+        panic("GIC: no distributor detected, giving up\n"); 
+
+    if (!dt_property_read_u32(node, "#redistributor-regions", &gic.rdist_count))
+        gic.rdist_count = 1;
+
+    rdist_regs = xzalloc_array(struct rdist_region, gic.rdist_count);
+    if (!rdist_regs)
+        panic("GIC: no distributor detected, giving up\n");
+
+    for (i = 0; i < gic.rdist_count; i++) {
+        u64 rdist_base, rdist_size;
+
+        res = dt_device_get_address(node, 1 + i, &rdist_base, &rdist_size);
+        if ( res || !rdist_base)
+            printk("No rdist base found\n");
+
+        rdist_regs[i].rdist_base = rdist_base;
+        rdist_regs[i].rdist_base_size = rdist_size;
+    }
+
+    if(!dt_property_read_u32(node, "redistributor-stride", &gic.rdist_stride))
+        gic.rdist_stride = 0x0;
+
+    gic.rdist_regions= rdist_regs;
+ 
+    res = dt_device_get_irq(node, 0, &gic.maintenance);
+    if ( res )
+        panic("GIC: Cannot find the maintenance IRQ");
+
+    /* Set the GIC as the primary interrupt controller */
+    dt_interrupt_controller = node;
+
+    /* map dbase & rdist regions */
+    gic.rdist_regions[0].map_rdist_base = ioremap_nocache(gic.rdist_regions[0].
+                             rdist_base, gic.rdist_regions[0].rdist_base_size);
+    if ( !gic.rdist_regions[0].map_rdist_base )
+    {
+        dprintk(XENLOG_ERR, "Unable to map GIC re-distributor\n");
+        return -EFAULT;
+    }
+    printk("GIC initialization:\n"
+              "        gic_dist_addr=%"PRIpaddr"\n"
+              "        gic_dist_size=%"PRIpaddr"\n"
+              "        gic_dist_mapaddr=%"PRIpaddr"\n"
+              "        gic_rdist_regions=%d\n"
+              "        gic_rdist_stride=%x\n"
+              "        gic_rdist_base=%"PRIpaddr"\n"
+              "        gic_rdist_base_size=%"PRIpaddr"\n"
+              "        gic_rdist_base_mapaddr=%"PRIpaddr"\n"
+              "        gic_maintenance_irq=%u\n",
+              gic.dbase, gic.dbase_size, (u64)gic.map_dbase, gic.rdist_count,
+              gic.rdist_stride, gic.rdist_regions[0].rdist_base,
+              gic.rdist_regions[0].rdist_base_size,
+              (u64)gic.rdist_regions[0].map_rdist_base, gic.maintenance.irq);
+
+    /* Global settings: interrupt distributor */
+    gic_dist_init();
+    gic_cpu_init();
+    gic_hyp_init();
+
+    /* Register hw ops*/
+    register_gic_ops(&gic_ops);
+    return 0;
+}
+
+static const char * const gicv3_dt_compat[] __initconst =
+{
+    "arm,gic-v3",
+    NULL
+};
+
+DT_DEVICE_START(gicv3, "GIC", DEVICE_GIC)
+        .compatible = gicv3_dt_compat,
+        .init = gicv3_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index b49dde1..201f534 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -69,11 +69,21 @@ struct vgic_irq_rank {
 struct vgic_info {
     paddr_t dbase; /* Distributor base address */
     paddr_t cbase; /* CPU base address */
+    paddr_t dbase_size; /* Distributor base size */
+    paddr_t rbase;      /* Re-Distributor base address */
+    paddr_t rbase_size; /* Re-Distributor size */
+    uint32_t rdist_stride;
 };
 
 struct gic_state_data {
     uint8_t version;
     union {
+        struct gic_v3 {
+            uint32_t gic_hcr, gic_vmcr;
+            uint32_t gic_apr0[4];
+            uint32_t gic_apr1[4];
+            uint64_t gic_lr[16];
+        }v3;
         struct gic_v2 {
             uint32_t gic_hcr, gic_vmcr;
             uint32_t gic_apr;
diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
new file mode 100644
index 0000000..4b8a9d2
--- /dev/null
+++ b/xen/include/asm-arm/gic_v3_defs.h
@@ -0,0 +1,187 @@
+/*
+ * ARM Generic Interrupt Controller v3 definitions
+ *
+ * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
+ * Copyright (c) 2014 Cavium Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+
+#define GICD_CTLR       (0x000)
+#define GICD_TYPER      (0x004)
+#define GICD_IIDR       (0x008)
+#define GICD_STATUSR    (0x010)
+#define GICD_SETSPI_NSR (0x040)
+#define GICD_CLRSPI_NSR (0x048)
+#define GICD_SETSPI_SR  (0x050)
+#define GICD_CLRSPI_SR  (0x058)
+#define GICD_IGROUPR    (0x080)
+#define GICD_IGROUPRN   (0x0FC)
+#define GICD_ISENABLER  (0x100)
+#define GICD_ISENABLERN (0x17C)
+#define GICD_ICENABLER  (0x180)
+#define GICD_ICENABLERN (0x1fC)
+#define GICD_ISPENDR    (0x200)
+#define GICD_ISPENDRN   (0x27C)
+#define GICD_ICPENDR    (0x280)
+#define GICD_ICPENDRN   (0x2FC)
+#define GICD_ISACTIVER  (0x300)
+#define GICD_ISACTIVERN (0x37C)
+#define GICD_ICACTIVER  (0x380)
+#define GICD_ICACTIVERN (0x3FC)
+#define GICD_IPRIORITYR (0x400)
+#define GICD_IPRIORITYRN (0x7F8)
+#define GICD_ICFGR      (0xC00)
+#define GICD_ICFGRN     (0xCFC)
+#define GICD_NSACR      (0xE00)
+#define GICD_NSACRN     (0xEFC)
+#define GICD_SGIR       (0xF00)
+#define GICD_CPENDSGIR  (0xF10)
+#define GICD_CPENDSGIRN (0xF1C)
+#define GICD_SPENDSGIR  (0xF20)
+#define GICD_SPENDSGIRN (0xF2C)
+#define GICD_IROUTER    (0x6000)
+#define GICD_IROUTERN   (0x7FF8)
+#define GICD_PIDR0      (0xFFE0)
+#define GICD_PIDR7      (0xFFDC)
+
+#define GICD_SGI_TARGET_LIST_SHIFT   (24)
+#define GICD_SGI_TARGET_LIST_MASK    (0x3UL << GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_LIST         (0UL<<GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_OTHERS       (1UL<<GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_SELF         (2UL<<GICD_SGI_TARGET_LIST_SHIFT)
+#define GICD_SGI_TARGET_SHIFT        (16)
+#define GICD_SGI_TARGET_MASK         (0xFFUL<<GICD_SGI_TARGET_SHIFT)
+#define GICD_SGI_GROUP1              (1UL<<15)
+#define GICD_SGI_INTID_MASK          (0xFUL)
+
+#define GICC_SRE_EL2_SRE             (1UL << 0)
+#define GICC_SRE_EL2_DFB             (1UL << 1)
+#define GICC_SRE_EL2_DIB             (1UL << 2)
+#define GICC_SRE_EL2_ENEL1           (1UL << 3)
+
+#define GICD_PIDR0_GICv3                0x92
+
+#define GICD_CTLR_RWP                (1UL << 31)
+#define GICD_CTLR_ARE_NS             (1U << 4)
+#define GICD_CTLR_ENABLE_G1A         (1U << 1)
+#define GICD_CTLR_ENABLE_G1          (1U << 0)
+#define GICD_IROUTER_SPI_MODE_ONE    (0UL << 31)
+#define GICD_IROUTER_SPI_MODE_ANY    (1UL << 31)
+
+#define GICH_HCR_EN                  (1 << 0)
+
+#define GICC_CTLR_EL1_EOImode_drop_dir  (0U << 1)
+#define GICC_CTLR_EL1_EOImode_drop      (1U << 1)
+
+#define GICR_WAKER_ProcessorSleep       (1U << 1)
+#define GICR_WAKER_ChildrenAsleep       (1U << 2)
+#define GICR_PIDR0_GICv3                0x93
+
+#define GICR_CTLR       (0x0000)
+#define GICR_IIDR       (0x0004)
+#define GICR_TYPER      (0x0008)
+#define GICR_STATUSR    (0x0010)
+#define GICR_WAKER      (0x0014)
+#define GICR_SETLPIR    (0x0040)
+#define GICR_CLRLPIR    (0x0048)
+#define GICR_PROPBASER  (0x0070)
+#define GICR_PENDBASER  (0x0078)
+#define GICR_INVLPIR    (0x00A0)
+#define GICR_INVALLR    (0x00B0)
+#define GICR_SYNCR      (0x00C0)
+#define GICR_MOVLPIR    (0x100)
+#define GICR_MOVALLR    (0x0110)
+#define GICR_PIDR0      GICD_PIDR0
+#define GICR_PIDR7      GICD_PIDR7
+
+/* GICR for SGI's & PPI's */
+
+#define GICR_IGROUPR0    (0x0080)
+#define GICR_IGRPMODR0   (0x0F80)
+#define GICR_ISENABLER0  (0x0100)
+#define GICR_ICENABLER0  (0x0180)
+#define GICR_ISPENDR0    (0x0200)
+#define GICR_ICPENDR0    (0x0280)
+#define GICR_ISACTIVER0  (0x0300)
+#define GICR_ICACTIVER0  (0x0380)
+#define GICR_IPRIORITYR0 (0x0400)
+#define GICR_IPRIORITYR7 (0x041C)
+#define GICR_ICFGR0      (0x0C00)
+#define GICR_ICFGR1      (0x0C04)
+#define GICR_NSACR       (0x0E00)
+
+#define GICR_TYPER_PLPIS                (1U << 0)
+#define GICR_TYPER_VLPIS                (1U << 1)
+#define GICR_TYPER_LAST                 (1U << 4)
+
+/* Register bits */
+#define GICD_CTL_ENABLE 0x1
+
+#define GICD_TYPE_LINES 0x01f
+#define GICD_TYPE_CPUS  0x0e0
+#define GICD_TYPE_SEC   0x400
+
+#define GICC_CTL_ENABLE 0x1
+#define GICC_CTL_EOI    (0x1 << 9)
+
+#define GICC_IA_IRQ       0x03ff
+#define GICC_IA_CPU_MASK  0x1c00
+#define GICC_IA_CPU_SHIFT 10
+
+#define DEFAULT_PMR_VALUE 0xff
+
+#define GICH_HCR_TC       (1 << 10)
+
+#define GICH_MISR_EOI     (1 << 0)
+#define GICH_MISR_U       (1 << 1)
+#define GICH_MISR_LRENP   (1 << 2)
+#define GICH_MISR_NP      (1 << 3)
+#define GICH_MISR_VGRP0E  (1 << 4)
+#define GICH_MISR_VGRP0D  (1 << 5)
+#define GICH_MISR_VGRP1E  (1 << 6)
+#define GICH_MISR_VGRP1D  (1 << 7)
+
+#define GICH_VMCR_EOI     (1 << 9)
+#define GICH_VMCR_VENG1    (1 << 1)
+
+#define GICH_LR_VIRTUAL_MASK    0xffff
+#define GICH_LR_VIRTUAL_SHIFT   0
+#define GICH_LR_PHYSICAL_MASK   0x3ff
+#define GICH_LR_PHYSICAL_SHIFT  32
+#define GICH_LR_STATE_MASK      0x3
+#define GICH_LR_STATE_SHIFT     62
+#define GICH_LR_PRIORITY_MASK   0xff
+#define GICH_LR_PRIORITY_SHIFT  48
+#define GICH_LR_HW_MASK         0x1
+#define GICH_LR_HW_SHIFT        61
+#define GICH_LR_GRP_MASK        0x1
+#define GICH_LR_GRP_SHIFT       60
+#define GICH_LR_MAINTENANCE_IRQ (1UL<<41)
+#define GICH_LR_GRP1            (1UL<<60)
+#define GICH_LR_HW              (1UL<<61)
+
+#define GICH_VTR_NRLRGS         0x3f
+#define GICH_VTR_PRIBITS_MASK   0x7
+#define GICH_VTR_PRIBITS_SHIFT  29
+
+#define GICH_VMCR_PRIORITY_MASK   0xff
+#define GICH_VMCR_PRIORITY_SHIFT  24
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
1.7.9.5

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:

> +#define GICD_CTLR       (0x000)
> +#define GICD_TYPER      (0x004)
> +#define GICD_IIDR       (0x008)
> +#define GICD_STATUSR    (0x010)

A lot of these are identical to the GIV v2 definitions. Is a separate
set of #defines *really* necessary?

I'd be happy with:
        #define GICD_CTLR       (0x000)
        #define GICD_TYPER      (0x004)
        #define GICD_IIDR       (0x008)
        #define GICD_STATUSR    (0x010) /* GICv3 only */
        #define GICD_BLARGLE    (0x...) /* GICv2 only */
        etc.

Or if you really feel that separate defines are needed then at the least
they should incorporate v2 and v3 into their names.
 
Ian.

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:

> +#define gic_data_rdist_rd_base()        (this_cpu(rbase))
> +#define gic_data_rdist_sgi_base()       (gic_data_rdist_rd_base() + SZ_64K)

Macros should be SHOUTY and ideally a lot shorter.

Since we have GICD[...] etc how about GICR[...]? And for the SGI bit
just use GICR[GICR_SGI_...] (where GICR_SGI_ is 64K).

> +
> +static inline u64 read_cpuid_mpidr(void)
> +{
> +   return READ_SYSREG(MPIDR_EL1);
> +}

No need for this trivial wrapper.

> +/* Wait for completion of a distributor change */
> +static void gic_do_wait_for_rwp(paddr_t base)

base here is a virtual address not a physical one, no? Hence the cast
you have below.

> +{
> +    u32 val;
> +    do {
> +        val = readl_relaxed((void *)base + GICD_CTLR);
> +        val = readl_relaxed(GICD + GICD_CTLR);

???

> +        cpu_relax();

This doesn't currently yield, but in principal it could, which would
mean a delay even if the termination condition was true.

Perhaps
	for(;;)
	{
		val = readl...
		if (val & ... )
			break;
		cpu_relax();
	}

???

> +    } while ( val & GICD_CTLR_RWP );
> +}
> +
> +static void gic_dist_wait_for_rwp(void)
> +{
> +    gic_do_wait_for_rwp((paddr_t)GICD);

Ugly and seemingly unnecessary cast.

> +}
> +
> +static void gic_redist_wait_for_rwp(void)
> +{
> +    gic_do_wait_for_rwp(gic_data_rdist_rd_base());
> +}
> +
> +static void gic_wait_for_rwp(int irq)
> +{
> +    if ( irq < 32 )

Either NR_LOCAL_IRQS or suitable gic v3 specific #define please (and
rename NR_LOCAL_IRQS to a v2 name)

> +         gic_redist_wait_for_rwp();
> +    else
> +         gic_dist_wait_for_rwp();
> +}
> +
> +
> +static void write_aprn_regs(struct gic_state_data *d)
> +{
> +    switch ( nr_priorities )
> +    {
> +    case 7:
> +        WRITE_SYSREG32(d->v3.gic_apr0[2], ICH_AP0R2_EL2);
> +        WRITE_SYSREG32(d->v3.gic_apr1[2], ICH_AP1R2_EL2);
  +        /* Fall-thru */

when doing so deliberately please.

Is it harmful/illegal to write to AP0R2 etc if priorities < 7?

> +    case 6:
> +        WRITE_SYSREG32(d->v3.gic_apr0[1], ICH_AP0R1_EL2);
> +        WRITE_SYSREG32(d->v3.gic_apr1[1], ICH_AP1R1_EL2);
> +    case 5:
> +        WRITE_SYSREG32(d->v3.gic_apr0[0], ICH_AP0R0_EL2);
> +        WRITE_SYSREG32(d->v3.gic_apr1[0], ICH_AP1R0_EL2);
> +        break;
> +    default:
> +        panic("Write Undefined active priorities \n");

I think these limits should be checked at init time with a panic there
and this should be come an assertion.

> +    }
> +}
> +
> +static void read_aprn_regs(struct gic_state_data *d)
> +{

Same comments here as for write.

> +static void gic_enable_irq(int irq)
> +{
> +    uint32_t enabler;
> +
> +    /* Enable routing */
> +    if ( irq > 31 )
> +    {
> +        enabler = readl_relaxed(GICD + GICD_ISENABLER + (irq / 32) * 4);
> +        writel_relaxed(enabler | (1u << (irq % 32)), GICD + GICD_ISENABLER + (irq / 32) * 4);

&GICD[GICD_IS_ENABLER +...] ?

> +    }
> +    else
> +    {
> +        enabler = readl_relaxed((void *)gic_data_rdist_sgi_base() + GICR_ISENABLER0);
> +        writel_relaxed(enabler | (1u << irq), (void *)gic_data_rdist_sgi_base() + GICR_ISENABLER0);

No casts please, just get the type right to start with.

Per the comments at the macro definition this could be
&GIRC[GICR_SGI_ISENABLR0].

I think both halves of this if would benefit from using some temporary
variables for clarity, or at least 
	enabler = read...
	enabler |= thing
	writel(enabler, ...)

> +/*
> + * - needs to be called with gic.lock held
> + * - needs to be called with a valid cpu_mask, ie each cpu in the mask has
> + * already called gic_cpu_init
> + */
> +static void gic_set_irq_property(unsigned int irq, bool_t level,
> +                                   const cpumask_t *cpu_mask,
> +                                   unsigned int priority)
> +{
> +    uint32_t cfg, edgebit;
> +    u64 affinity;
> +    unsigned int cpu = gic_mask_cpu(cpu_mask);
> +    paddr_t rebase;
> +
> +
> +    /* Set edge / level */
> +    if ( irq < 16 )
> +        /* SGI's are always edge-triggered not need to call GICD_ICFGR0 */
> +        cfg = readl_relaxed((void *)gic_data_rdist_sgi_base() + GICR_ICFGR0);

casts and names throughout this function.
[...]

> +static void __init gic_dist_init(void)
> +{
> +    uint32_t type;
> +    u64 affinity;
> +    int i;
> +
> +    /* Disable the distributor */
> +    writel_relaxed(0, GICD + GICD_CTLR);

Does using readl/writel_relaxed buy you anything over using a GICD macro
with a volatile in it like the v2 code does?

> +
> +    type = readl_relaxed(GICD + GICD_TYPER);
> +    gic.lines = 32 * ((type & GICD_TYPE_LINES) + 1);
> +
> +    printk("GIC: %d lines, (IID %8.8x).\n",
> +           gic.lines, readl_relaxed(GICD + GICD_IIDR));
> +
> +    /* Default all global IRQs to level, active low */
> +    for ( i = 32; i < gic.lines; i += 16 )
> +        writel_relaxed(0, GICD + GICD_ICFGR + (i / 16) * 4);
> +
> +    /* Default priority for global interrupts */
> +    for ( i = 32; i < gic.lines; i += 4 )
> +        writel_relaxed((GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ), GICD + GICD_IPRIORITYR + (i / 4) * 4);

Watch out for long lines please, try and keep to 80 columns.

> +
> +    /* Disable all global interrupts */
> +    for ( i = 32; i < gic.lines; i += 32 )
> +        writel_relaxed(0xffffffff, GICD + GICD_ICENABLER + (i / 32) * 4);
> +
> +    gic_dist_wait_for_rwp();
> +
> +    /* Turn on the distributor */
> +    writel_relaxed(GICD_CTL_ENABLE | GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1, GICD + GICD_CTLR);
> + 
> +    /* Route all global IRQs to this CPU */
> +    affinity = gic_mpidr_to_affinity(read_cpuid_mpidr());
> +    for ( i = 32; i < gic.lines; i++ )
> +        writeq_relaxed(affinity, GICD + GICD_IROUTER + i * 8);
> +}
> +
> +static void gic_enable_redist(void)
> +{
> +    paddr_t rbase;
> +    u32 val;
> +
> +    rbase = this_cpu(rbase);
> +
> +    /* Wake up this CPU redistributor */
> +    val = readl_relaxed((void *)rbase + GICR_WAKER);
> +    val &= ~GICR_WAKER_ProcessorSleep;
> +    writel_relaxed(val, (void *)rbase + GICR_WAKER);
> +
> +    do {
> +         val = readl_relaxed((void *)rbase + GICR_WAKER);
> +         cpu_relax();
> +    } while ( val & GICR_WAKER_ChildrenAsleep );
> +}
> +
> +static int __init gic_populate_rdist(void)
> +{
> +    u64 mpidr = cpu_logical_map(smp_processor_id());
> +    u64 typer;
> +    u64 aff;
> +    int i;
> +    uint32_t reg;
> +
> +    aff  = mpidr & ((1 << 24) - 1);
> +    aff |= (mpidr >> 8) & (0xffUL << 24);
> +
> +    for ( i = 0; i < gic.rdist_count; i++ )
> +    {
> +        uint32_t ptr = 0;
> +
> +        reg = readl_relaxed(GICR + ptr + GICR_PIDR0);
> +        if ( (reg & 0xff) != GICR_PIDR0_GICv3 ) {
> +            printk("No redistributor present @%x\n", ptr);

Debug message?

> +            break;
> +        }
> +
> +        do {
> +            typer = readq_relaxed(GICR + ptr + GICR_TYPER);
> +            if ( (typer >> 32) == aff )
> +            {
> +                this_cpu(rbase) = (u64)GICR + ptr;

Cast?

> +                printk("CPU%d: found redistributor %llx region %d\n",
> +                            smp_processor_id(), (unsigned long long) mpidr, i);
> +                return 0;
> +            }
> +
> +            if ( gic.rdist_stride ) {
> +                ptr += gic.rdist_stride;
> +            } else {
> +                ptr += SZ_64K * 2; /* Skip RD_base + SGI_base */

Whatever initialises rdist_stride should default it to SZ_64K if that is
indeed the default.

> +                if ( typer & GICR_TYPER_VLPIS )
> +                    ptr += SZ_64K * 2; /* Skip VLPI_base + reserved page */
> +            }
> +        } while ( !(typer & GICR_TYPER_LAST) );
> +    }
> +
> +    /* We couldn't even deal with ourselves... */
> +    printk("CPU%d: mpidr %lx has no re-distributor!\n",
> +              smp_processor_id(), (unsigned long)mpidr);

No casts, either use PRIx64 or make the type correct.

(I'm going to stop mentioning casts, please eliminate them all or
explain why they are needed).

> +    /*
> +     * Set priority on PPI and SGI interrupts
> +     */
> +    for (i = 0; i < 16; i += 4)
> +        writel_relaxed((GIC_PRI_IPI<<24 | GIC_PRI_IPI<<16 | GIC_PRI_IPI<<8 | GIC_PRI_IPI), (void *)rbase_sgi + GICR_IPRIORITYR0 + (i / 4) * 4);

Long lines.

> +
> +static u16 gic_compute_target_list(int *base_cpu, const struct cpumask *mask,
> +                                   u64 cluster_id)
> +{
> +    int cpu = *base_cpu;
> +    u64 mpidr = cpu_logical_map(cpu);
> +    u16 tlist = 0;
> +
> +    while ( cpu < nr_cpu_ids )
> +    {
> +        /*
> +         * If we ever get a cluster of more than 16 CPUs, just
> +         * scream and skip that CPU.

I don't see any screaming. We would want to know if this was happening,
wouldn't we?

> +static void gic_update_lr(int lr, struct pending_irq *p, unsigned int state)
> +{
> +    u64 grp = GICH_LR_GRP1;
> +    u64 val = 0;
> +
> +    BUG_ON(lr >= nr_lrs);
> +    BUG_ON(lr < 0);
> +
> +    val =  ((((u64)state) & 0x3) << GICH_LR_STATE_SHIFT) | grp |
> +        ((((u64)p->priority) & 0xff) << GICH_LR_PRIORITY_SHIFT) |
> +        (((u64)p->irq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT);
> +
> +   if ( p->desc != NULL )
> +        val |= GICH_LR_HW |(((u64) p->desc->irq & GICH_LR_PHYSICAL_MASK) << GICH_LR_PHYSICAL_SHIFT);
> +
> +    gich_write_lr(lr, val);
> +}
> +
> +static void gic_clear_lr(int lr)
> +{
> +    gich_write_lr(lr, 0);
> +}
> +
> +static void gic_read_lr(int lr, struct gic_lr *lr_reg)

I can't find struct gic_lr anywhere.

> +{
> +    u64 lrv;
> +    lrv = gich_read_lr(lr);
> +    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
> +    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK; 
> +    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
> +    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
> +    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
> +    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;

If you want to be able to do field-by-field accesses then please do what
the page table code does and use a union and bit fields. See lpae_pt_t.

        typedef union __packed {
        	uint64_t bits;
        	struct {
        		unsigned long pirq:4;
        		unsugned long virq:4;
        	etc, including explicit padding
        	};
        } gicv3_lr;

Then:
	gicv3 lrv = {.bits = gich_read_lr(lr)};


> +static struct dt_irq * gic_maintenance_irq(void)
> +{
> +    return &gic.maintenance;
> +}
> +
> +static void gic_hcr_status(uint32_t flag, uint8_t status)
> +{
> +    if ( status )

status is actually "bool_t set" ?

> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) | flag), ICH_HCR_EL2);
> +    else
> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) & (~flag)), ICH_HCR_EL2);
> +}
> +
> +    rdist_regs = xzalloc_array(struct rdist_region, gic.rdist_count);

I thought I saw a comment at the top saying that only a single region
was supported? Shouldn't this be checked somewhere, or even better
fixed.

Is there a limit on gic.rdist_count? How large is it? Can rdist_regs be
a static array?

Ian.

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Julien Grall]

On 04/10/2014 11:00 AM, Ian Campbell wrote:
> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> 
>> +#define gic_data_rdist_rd_base()        (this_cpu(rbase))
>> +#define gic_data_rdist_sgi_base()       (gic_data_rdist_rd_base() + SZ_64K)
> 
> Macros should be SHOUTY and ideally a lot shorter.
> 
> Since we have GICD[...] etc how about GICR[...]? And for the SGI bit
> just use GICR[GICR_SGI_...] (where GICR_SGI_ is 64K).
> 
>> +
>> +static inline u64 read_cpuid_mpidr(void)
>> +{
>> +   return READ_SYSREG(MPIDR_EL1);
>> +}
> 
> No need for this trivial wrapper.

As said on V1:
MPDIR_EL1 is already replicated in current_cpu_data.mpidr.bits

>> +static void __init gic_dist_init(void)
>> +{
>> +    uint32_t type;
>> +    u64 affinity;
>> +    int i;
>> +
>> +    /* Disable the distributor */
>> +    writel_relaxed(0, GICD + GICD_CTLR);
> 
> Does using readl/writel_relaxed buy you anything over using a GICD macro
> with a volatile in it like the v2 code does?

As ioremap is used to map the GIC MMIO, IHMO it's better to use
readl/writel_relaxed to be consistent.

Regards,

-- 
Julien Grall

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Vijay Kilari]

On Thu, Apr 10, 2014 at 4:04 PM, Julien Grall <julien.grall@linaro.org> wrote:
> On 04/10/2014 11:00 AM, Ian Campbell wrote:
>> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>>
>>> +#define gic_data_rdist_rd_base()        (this_cpu(rbase))
>>> +#define gic_data_rdist_sgi_base()       (gic_data_rdist_rd_base() + SZ_64K)
>>
>> Macros should be SHOUTY and ideally a lot shorter.
>>
>> Since we have GICD[...] etc how about GICR[...]? And for the SGI bit
>> just use GICR[GICR_SGI_...] (where GICR_SGI_ is 64K).
>>
>>> +
>>> +static inline u64 read_cpuid_mpidr(void)
>>> +{
>>> +   return READ_SYSREG(MPIDR_EL1);
>>> +}
>>
>> No need for this trivial wrapper.
>
> As said on V1:
> MPDIR_EL1 is already replicated in current_cpu_data.mpidr.bits
>
  I find only MIDR register contents in current_cpu_data.midr.bits
but not MPIDR

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Julien Grall]

On 04/10/2014 12:06 PM, Vijay Kilari wrote:
> On Thu, Apr 10, 2014 at 4:04 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> On 04/10/2014 11:00 AM, Ian Campbell wrote:
>>> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>>>
>>>> +#define gic_data_rdist_rd_base()        (this_cpu(rbase))
>>>> +#define gic_data_rdist_sgi_base()       (gic_data_rdist_rd_base() + SZ_64K)
>>>
>>> Macros should be SHOUTY and ideally a lot shorter.
>>>
>>> Since we have GICD[...] etc how about GICR[...]? And for the SGI bit
>>> just use GICR[GICR_SGI_...] (where GICR_SGI_ is 64K).
>>>
>>>> +
>>>> +static inline u64 read_cpuid_mpidr(void)
>>>> +{
>>>> +   return READ_SYSREG(MPIDR_EL1);
>>>> +}
>>>
>>> No need for this trivial wrapper.
>>
>> As said on V1:
>> MPDIR_EL1 is already replicated in current_cpu_data.mpidr.bits
>>
>   I find only MIDR register contents in current_cpu_data.midr.bits
> but not MPIDR

Second field of the structure...
Or even better cpu_logical_map(mycpu) as you already use in
gic_set_irq_property.

That made me think about your function gic_mpidir_to_affinity (see
below). Why do you mask with GICD_IROUTER_SPI_MODE_ANY? It should be
valid anywhere.

If not you may send the SGI to a wrong CPU.

+static u64 gic_mpidr_to_affinity(u64 mpidr)
+{
+    /* Make sure we don't broadcast the interrupt */
+    return mpidr & ~GICD_IROUTER_SPI_MODE_ANY;
+}
+

-- 
Julien Grall

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Julien Grall]

On 04/10/2014 11:00 AM, Ian Campbell wrote:
>> +
>> +static u16 gic_compute_target_list(int *base_cpu, const struct cpumask *mask,
>> +                                   u64 cluster_id)
>> +{
>> +    int cpu = *base_cpu;
>> +    u64 mpidr = cpu_logical_map(cpu);
>> +    u16 tlist = 0;
>> +
>> +    while ( cpu < nr_cpu_ids )
>> +    {
>> +        /*
>> +         * If we ever get a cluster of more than 16 CPUs, just
>> +         * scream and skip that CPU.
> 
> I don't see any screaming. We would want to know if this was happening,
> wouldn't we?

It seems the code of GICv3 has been half-copied for Linux code :). I've
just found this function in Mark's series.

-- 
Julien Grall

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Vijay Kilari]

On Thu, Apr 10, 2014 at 3:30 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>
>> +static void write_aprn_regs(struct gic_state_data *d)
>> +{
>> +    switch ( nr_priorities )
>> +    {
>> +    case 7:
>> +        WRITE_SYSREG32(d->v3.gic_apr0[2], ICH_AP0R2_EL2);
>> +        WRITE_SYSREG32(d->v3.gic_apr1[2], ICH_AP1R2_EL2);
>   +        /* Fall-thru */
>
> when doing so deliberately please.
>
> Is it harmful/illegal to write to AP0R2 etc if priorities < 7?

The APR register is implementation defined. It is upto the platform
to support number of APR registers. So based on nr_priorities value
we read save and restore only platform supported registers

>> +
>> +static void gic_clear_lr(int lr)
>> +{
>> +    gich_write_lr(lr, 0);
>> +}
>> +
>> +static void gic_read_lr(int lr, struct gic_lr *lr_reg)
>
> I can't find struct gic_lr anywhere.
Already defined in previous patch
>
>> +{
>> +    u64 lrv;
>> +    lrv = gich_read_lr(lr);
>> +    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
>> +    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
>> +    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
>> +    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
>> +    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
>> +    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
>
> If you want to be able to do field-by-field accesses then please do what
> the page table code does and use a union and bit fields. See lpae_pt_t.
>
>         typedef union __packed {
>                 uint64_t bits;
>                 struct {
>                         unsigned long pirq:4;
>                         unsugned long virq:4;
>                 etc, including explicit padding
>                 };
>         } gicv3_lr;
>
> Then:
>         gicv3 lrv = {.bits = gich_read_lr(lr)};
>
The complexity is LR register is 64 bit in v3 and 32 bit v2.
Though I define this like above for v2 & v3, generic code still need to access
based on hw version. So I have make it without bit-fields

>
>> +static struct dt_irq * gic_maintenance_irq(void)
>> +{
>> +    return &gic.maintenance;
>> +}
>> +
>> +static void gic_hcr_status(uint32_t flag, uint8_t status)
>> +{
>> +    if ( status )
>
> status is actually "bool_t set" ?
>
>> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) | flag), ICH_HCR_EL2);
>> +    else
>> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) & (~flag)), ICH_HCR_EL2);
>> +}
>> +
>> +    rdist_regs = xzalloc_array(struct rdist_region, gic.rdist_count);
>
> I thought I saw a comment at the top saying that only a single region
> was supported? Shouldn't this be checked somewhere, or even better
> fixed.
Yes, only rdist_region[0] is read from dt, which supports upto 32 cores.
So one can add if more than 32 cores is required.

>
> Is there a limit on gic.rdist_count? How large is it? Can rdist_regs be
> a static array?
The rdist count is read from dt. so it is implementation defined
>
> Ian.
>
>

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Ian Campbell]

On Fri, 2014-04-11 at 18:29 +0530, Vijay Kilari wrote:
> On Thu, Apr 10, 2014 at 3:30 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> +
> >> +static void gic_clear_lr(int lr)
> >> +{
> >> +    gich_write_lr(lr, 0);
> >> +}
> >> +
> >> +static void gic_read_lr(int lr, struct gic_lr *lr_reg)
> >
> > I can't find struct gic_lr anywhere.
> Already defined in previous patch

Which? I looked at the time and couldn't find it.

> >
> >> +{
> >> +    u64 lrv;
> >> +    lrv = gich_read_lr(lr);
> >> +    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
> >> +    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
> >> +    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
> >> +    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
> >> +    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
> >> +    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
> >
> > If you want to be able to do field-by-field accesses then please do what
> > the page table code does and use a union and bit fields. See lpae_pt_t.
> >
> >         typedef union __packed {
> >                 uint64_t bits;
> >                 struct {
> >                         unsigned long pirq:4;
> >                         unsugned long virq:4;
> >                 etc, including explicit padding
> >                 };
> >         } gicv3_lr;
> >
> > Then:
> >         gicv3 lrv = {.bits = gich_read_lr(lr)};
> >
> The complexity is LR register is 64 bit in v3 and 32 bit v2.
> Though I define this like above for v2 & v3, generic code still need to access
> based on hw version. So I have make it without bit-fields

How does the generic code access without knowing the size? And if it
knows the size it can equally choose between gicv3_lr and gicv2_lr at
the appropriate point.

> >> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) | flag), ICH_HCR_EL2);
> >> +    else
> >> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) & (~flag)), ICH_HCR_EL2);
> >> +}
> >> +
> >> +    rdist_regs = xzalloc_array(struct rdist_region, gic.rdist_count);
> >
> > I thought I saw a comment at the top saying that only a single region
> > was supported? Shouldn't this be checked somewhere, or even better
> > fixed.
> Yes, only rdist_region[0] is read from dt, which supports upto 32 cores.
> So one can add if more than 32 cores is required.

Something should error out if more than 32 cores are present then. Does
it?

> > Is there a limit on gic.rdist_count? How large is it? Can rdist_regs be
> > a static array?
> The rdist count is read from dt. so it is implementation defined

Surely there must be some architectural limit which sets the maximum
size an implementation can use?

Ian.

Re: [PATCH v2 13/15] xen/arm: Add support for GIC v3

[Vijay Kilari]

On Mon, Apr 14, 2014 at 1:57 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-04-11 at 18:29 +0530, Vijay Kilari wrote:
>> On Thu, Apr 10, 2014 at 3:30 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >> +
>> >> +static void gic_clear_lr(int lr)
>> >> +{
>> >> +    gich_write_lr(lr, 0);
>> >> +}
>> >> +
>> >> +static void gic_read_lr(int lr, struct gic_lr *lr_reg)
>> >
>> > I can't find struct gic_lr anywhere.
>> Already defined in previous patch
>
> Which? I looked at the time and couldn't find it.
>
>> >
>> >> +{
>> >> +    u64 lrv;
>> >> +    lrv = gich_read_lr(lr);
>> >> +    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
>> >> +    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
>> >> +    lr_reg->priority = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
>> >> +    lr_reg->state    = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
>> >> +    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
>> >> +    lr_reg->grp = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
>> >
>> > If you want to be able to do field-by-field accesses then please do what
>> > the page table code does and use a union and bit fields. See lpae_pt_t.
>> >
>> >         typedef union __packed {
>> >                 uint64_t bits;
>> >                 struct {
>> >                         unsigned long pirq:4;
>> >                         unsugned long virq:4;
>> >                 etc, including explicit padding
>> >                 };
>> >         } gicv3_lr;
>> >
>> > Then:
>> >         gicv3 lrv = {.bits = gich_read_lr(lr)};
>> >
>> The complexity is LR register is 64 bit in v3 and 32 bit v2.
>> Though I define this like above for v2 & v3, generic code still need to access
>> based on hw version. So I have make it without bit-fields
>
> How does the generic code access without knowing the size? And if it
> knows the size it can equally choose between gicv3_lr and gicv2_lr at
> the appropriate point.
There is one common gic_lr as below where respective GIC {v2/v3}
will decode in the callback and generic code just access by its fields.

struct gic_lr {
   uint32_t pirq;
   uint32_t virq;
   uint8_t priority;
   uint8_t state;
   uint8_t hw_status;
   uint8_t grp;
};

>
>> >> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) | flag), ICH_HCR_EL2);
>> >> +    else
>> >> +      WRITE_SYSREG32((READ_SYSREG32(ICH_HCR_EL2) & (~flag)), ICH_HCR_EL2);
>> >> +}
>> >> +
>> >> +    rdist_regs = xzalloc_array(struct rdist_region, gic.rdist_count);
>> >
>> > I thought I saw a comment at the top saying that only a single region
>> > was supported? Shouldn't this be checked somewhere, or even better
>> > fixed.
>> Yes, only rdist_region[0] is read from dt, which supports upto 32 cores.
>> So one can add if more than 32 cores is required.
>
> Something should error out if more than 32 cores are present then. Does
> it?
>
>> > Is there a limit on gic.rdist_count? How large is it? Can rdist_regs be
>> > a static array?
>> The rdist count is read from dt. so it is implementation defined
>
> Surely there must be some architectural limit which sets the maximum
> size an implementation can use?

I fixed this in next version. GIC v3 driver will read all the
re-distributor regions
from dt. However, I set MAX_RDIST_COUNT to 4 because
of static allocation in domain struct to hold these addresses.
Check is made in GICv3 driver if number of regions in dt is more than 4.
If we go for dynamic allocation we can remove this constraint as well.

         paddr_t cbase; /* CPU base address */
+        /* GIC V3 addressing */
+        paddr_t dbase_size; /* Distributor base size */
+        paddr_t rbase[MAX_RDIST_COUNT];      /* Re-Distributor base address */
+        paddr_t rbase_size[MAX_RDIST_COUNT]; /* Re-Distributor size */
+        uint32_t rdist_stride;               /* Re-Distributor stride */
+        int rdist_count;                     /* No. of Re-Distributors */
     } vgic;


>
> Ian.
>

[PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Add vgic driver support for GIC v3 version.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/Makefile     |    1 +
 xen/arch/arm/vgic-v3.c    |  927 +++++++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/gic.h |    5 +-
 3 files changed, 932 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 52f707c..31c47a0 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -29,6 +29,7 @@ obj-y += smp.o
 obj-y += shutdown.o
 obj-y += traps.o
 obj-y += vgic.o
+obj-y += vgic-v3.o
 obj-y += vgic-v2.o
 obj-y += vtimer.o
 obj-y += vuart.o
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
new file mode 100644
index 0000000..e7ebad0
--- /dev/null
+++ b/xen/arch/arm/vgic-v3.c
@@ -0,0 +1,927 @@
+/*
+ * xen/arch/arm/vgic-v3.c
+ *
+ * ARM Virtual Generic Interrupt Controller v3 support
+ * based on xen/arch/arm/vgic.c
+ *
+ * Vijaya Kumar K <vijaya.kumar@caviumnetworks.com>
+ * Copyright (c) 2014 Cavium Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/bitops.h>
+#include <xen/config.h>
+#include <xen/lib.h>
+#include <xen/init.h>
+#include <xen/softirq.h>
+#include <xen/irq.h>
+#include <xen/sched.h>
+
+#include <asm/current.h>
+#include <asm/device.h>
+
+#include "io.h"
+#include <asm/gic_v3_defs.h>
+#include <asm/gic.h>
+
+#define SZ_64K  0x00010000
+#define REG(n) (n)
+
+/* Number of ranks of interrupt registers for a domain */
+#define DOMAIN_NR_RANKS(d) (((d)->arch.vgic.nr_lines+31)/32)
+
+/*
+ * Rank containing GICD_<FOO><n> for GICD_<FOO> with
+ * <b>-bits-per-interrupt
+ */
+static inline int REG_RANK_NR(int b, uint32_t n)
+{
+    switch ( b )
+    {
+    case 64: return n >> 6;
+    case 32: return n >> 5;
+    case 16: return n >> 4;
+    case 8: return n >> 3;
+    case 4: return n >> 2;
+    case 2: return n >> 1;
+    case 1: return n;
+    default: BUG();
+    }
+}
+
+/*
+ * Offset of GICD_<FOO><n> with its rank, for GICD_<FOO> with
+ * <b>-bits-per-interrupt.
+ */
+/* Shift n >> 2 to make it byte register diff */
+#define REG_RANK_INDEX(b, n) (((n) >> 2) & ((b)-1))
+
+/*
+ * Returns rank corresponding to a GICD_<FOO><n> register for
+ * GICD_<FOO> with <b>-bits-per-interrupt.
+ */
+static struct vgic_irq_rank *vgic_irq_rank(struct vcpu *v, int b, int n)
+{
+    int rank;
+
+    n = n >> 2;
+    rank  = REG_RANK_NR(b, n);
+
+    if ( rank == 0 ) /* Rank 0 is nothing but GICR registers in GICv3 */
+        return v->arch.vgic.private_irqs;
+    else if ( rank <= DOMAIN_NR_RANKS(v->domain) )
+        return &v->domain->arch.vgic.shared_irqs[rank - 1];
+    else
+        return NULL;
+}
+
+#define vgic_lock(v)   spin_lock_irq(&(v)->domain->arch.vgic.lock)
+#define vgic_unlock(v) spin_unlock_irq(&(v)->domain->arch.vgic.lock)
+
+#define vgic_lock_rank(v, r) spin_lock(&(r)->lock)
+#define vgic_unlock_rank(v, r) spin_unlock(&(r)->lock)
+
+static uint32_t byte_read(uint32_t val, int sign, int offset)
+{
+    int byte = offset & 0x3;
+
+    val = val >> (8*byte);
+    if ( sign && (val & 0x80) )
+        val |= 0xffffff00;
+    else
+        val &= 0x000000ff;
+    return val;
+}
+
+static void byte_write(uint32_t *reg, uint32_t var, int offset)
+{
+    int byte = offset & 0x3;
+
+    var &= (0xff << (8*byte));
+
+    *reg &= ~(0xff << (8*byte));
+    *reg |= var;
+}
+
+static int vgic_read_priority(struct vcpu *v, int irq)
+{
+   struct vgic_irq_rank *rank = vgic_irq_rank(v, 8, irq);
+   return byte_read(rank->ipriority[REG_RANK_INDEX(8, irq)], 0, irq & 0x3);
+}
+ 
+static int __vgic_rdistr_mmio_read(struct vcpu *v, mmio_info_t *info, uint32_t offset)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    int gicr_reg = REG(offset);
+    u64 mpidr;
+    u64 aff;
+
+    switch ( gicr_reg )
+    {
+    case GICR_CTLR:
+        /* We have implemented LPI's, read zero */
+        goto read_as_zero;
+    case GICR_IIDR:
+        *r = 0x34;
+        return 1;
+    case GICR_TYPER:
+        mpidr = cpu_logical_map(smp_processor_id());
+        aff  = mpidr & ((1 << 24) - 1);
+        aff |= (mpidr >> 8) & (0xffUL << 24);
+        aff = aff << 32;
+        aff = (aff | ((u64)smp_processor_id() << 8));
+        aff = (aff | (0x9UL << 0));
+        *r = aff;
+         return 1;
+    case GICR_STATUSR:
+        /* Not implemented */
+        goto read_as_zero;
+    case GICR_WAKER:
+        /* Not implemented */
+        goto read_as_zero;
+    case GICR_SETLPIR:
+    case GICR_CLRLPIR:
+        /* WO */
+        return 0;
+    case GICR_PROPBASER:
+        /* LPI's not implemented */
+        goto read_as_zero;
+    case GICR_PENDBASER:
+        /* LPI's not implemented */
+        goto read_as_zero;
+    case GICR_INVLPIR:
+    case GICR_INVALLR:
+        /* WO */
+        return 0;
+    case GICR_SYNCR:
+        if ( dabt.size != 2 ) goto bad_width;
+        /* RO */
+        /* Return as not busy */
+        *r = 1;
+        return 1;
+    case GICR_MOVLPIR:
+    case GICR_MOVALLR:
+        /* WO */
+        return 0;
+    case GICR_PIDR7... GICR_PIDR0:
+        *r = 0x93;
+         return 1;
+    default:
+        printk("vGICR: read r%d offset %#08x\n not found", dabt.reg, offset);
+       
+    }
+bad_width:
+    printk("vGICD: bad read width %d r%d offset %#08x\n",
+           dabt.size, dabt.reg, offset);
+    domain_crash_synchronous();
+    return 0;
+
+read_as_zero:
+    if ( dabt.size != 2 ) goto bad_width;
+    *r = 0;
+    return 1;
+}
+
+static int __vgic_rdistr_mmio_write(struct vcpu *v, mmio_info_t *info, uint32_t offset)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    int gicr_reg = REG(offset);
+
+    switch ( gicr_reg )
+    {
+    case GICR_CTLR:
+        /* We do not implement LPI's, read zero */
+        goto write_ignore;
+    case GICR_IIDR:
+    case GICR_TYPER:
+        /* RO */
+        goto write_ignore;
+    case GICR_STATUSR:
+        /* Not implemented */
+        goto write_ignore;
+    case GICR_WAKER:
+        /* Not implemented */
+        goto write_ignore;
+    case GICR_SETLPIR:
+        return 1;
+    case GICR_CLRLPIR:
+        return 1;
+    case GICR_PROPBASER:
+        /* LPI is not implemented */
+        goto write_ignore;
+    case GICR_PENDBASER:
+        /* LPI is not implemented */
+        goto write_ignore;
+    case GICR_INVLPIR:
+        return 1;
+    case GICR_INVALLR:
+        return 1;
+    case GICR_SYNCR:
+        /* RO */
+        goto write_ignore;
+    case GICR_MOVLPIR:
+        return 1;
+    case GICR_MOVALLR:
+        return 1;
+    case GICR_PIDR7... GICR_PIDR0:
+        /* RO */
+        goto write_ignore;
+    default:
+        printk("vGICR: write r%d offset %#08x\n not found", dabt.reg, offset);
+    }
+bad_width:
+    printk("vGICD: bad write width %d r%d=%"PRIregister" offset %#08x\n",
+           dabt.size, dabt.reg, *r, offset);
+    domain_crash_synchronous();
+    return 0;
+
+write_ignore:
+    if ( dabt.size != 2 ) goto bad_width;
+    return 1;
+}
+
+static int vgic_rdistr_sgi_mmio_read(struct vcpu *v, mmio_info_t *info, uint32_t offset)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    struct vgic_irq_rank *rank;
+    int gicr_reg = REG(offset);
+
+    switch ( gicr_reg )
+    {
+    case GICR_IGROUPR0:
+        /* We do not implement security extensions for guests, read zero */
+        goto read_as_zero;
+    case GICR_IGRPMODR0:
+        /* We do not implement security extensions for guests, read zero */
+        goto read_as_zero;
+    case GICR_ISENABLER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ISENABLER0);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->ienable;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ICENABLER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ICENABLER0);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->ienable;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ISPENDR0:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ISPENDR0);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->ipend, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ICPENDR0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ICPENDR0);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->ipend, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ISACTIVER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ISACTIVER0);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->iactive;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ICACTIVER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ICACTIVER0);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->iactive;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_IPRIORITYR0...GICR_IPRIORITYR7:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicr_reg - GICR_IPRIORITYR0);
+        if ( rank == NULL ) goto read_as_zero;
+
+        vgic_lock_rank(v, rank);
+        *r = rank->ipriority[REG_RANK_INDEX(8, gicr_reg - GICR_IPRIORITYR0)];
+        if ( dabt.size == 0 )
+            *r = byte_read(*r, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ICFGR0... GICR_ICFGR1:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 2, gicr_reg - GICR_ICFGR0);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->icfg[REG_RANK_INDEX(2, gicr_reg - GICR_ICFGR0)];
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_NSACR:
+        if ( dabt.size != 2 ) goto bad_width;
+        return 1;
+    default:
+        printk("vGICR: read r%d offset %#08x\n not found", dabt.reg, offset);
+       
+    }
+bad_width:
+    printk("vGICD: bad read width %d r%d offset %#08x\n",
+           dabt.size, dabt.reg, offset);
+    domain_crash_synchronous();
+    return 0;
+
+read_as_zero:
+    if ( dabt.size != 2 ) goto bad_width;
+    *r = 0;
+    return 1;
+}
+
+static int vgic_rdistr_sgi_mmio_write(struct vcpu *v, mmio_info_t *info, uint32_t offset)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    struct vgic_irq_rank *rank;
+    int gicr_reg = REG(offset);
+    uint32_t tr;
+
+    switch ( gicr_reg )
+    {
+    case GICR_IGROUPR0:
+        /* We do not implement security extensions for guests, write ignore */
+        goto write_ignore;
+    case GICR_IGRPMODR0:
+        /* We do not implement security extensions for guests, write ignore */
+        goto write_ignore;
+    case GICR_ISENABLER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ISENABLER0);
+        if ( rank == NULL ) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        tr = rank->ienable;
+        rank->ienable |= *r;
+        vgic_unlock_rank(v, rank);
+        vgic_enable_irqs(v, (*r) & (~tr), (gicr_reg - GICR_ISENABLER0) >> 2);
+        return 1;
+    case GICR_ICENABLER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ICENABLER0);
+        if ( rank == NULL ) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->ienable &= ~*r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ISPENDR0:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        return 0;
+    case GICR_ICPENDR0:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        return 0;
+    case GICR_ISACTIVER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ISACTIVER0);
+        if ( rank == NULL ) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->iactive &= ~*r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ICACTIVER0:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicr_reg - GICR_ICACTIVER0);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->iactive &= ~*r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_IPRIORITYR0...GICR_IPRIORITYR7:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicr_reg - GICR_IPRIORITYR0);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        if ( dabt.size == 2 )
+            rank->ipriority[REG_RANK_INDEX(8, gicr_reg - GICR_IPRIORITYR0)] = *r;
+        else
+            byte_write(&rank->ipriority[REG_RANK_INDEX(8, gicr_reg - GICR_IPRIORITYR0)],
+                       *r, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICR_ICFGR0:
+        /* RO */
+    case GICR_ICFGR1:
+        goto write_ignore;
+    case GICR_NSACR:
+        /* We do not implement security extensions for guests, write ignore */
+        goto write_ignore;
+    default:
+        printk("vGICR: write r%d offset %#08x\n not found", dabt.reg, offset);
+    }
+bad_width:
+    printk("vGICD: bad write width %d r%d=%"PRIregister" offset %#08x\n",
+           dabt.size, dabt.reg, *r, offset);
+    domain_crash_synchronous();
+    return 0;
+
+write_ignore:
+    if ( dabt.size != 2 ) goto bad_width;
+    return 1;
+}
+
+static int vgic_rdistr_mmio_read(struct vcpu *v, mmio_info_t *info)
+{
+    uint32_t offset;
+
+    if ( !v->domain->arch.vgic.info->rdist_stride )
+        offset = info->gpa & (v->domain->arch.vgic.info->rdist_stride - 1);
+    else
+        offset = info->gpa & 0x1FFFF;
+
+    if ( offset < SZ_64K )
+       return __vgic_rdistr_mmio_read(v, info, offset);
+    else if ( (offset - SZ_64K) < SZ_64K )
+       return vgic_rdistr_sgi_mmio_read(v, info, (offset - SZ_64K));
+    else 
+       printk("vGICR: wrong gpa read %"PRIpaddr"\n",info->gpa);
+    return 0;
+}
+      
+static int vgic_rdistr_mmio_write(struct vcpu *v, mmio_info_t *info)
+{
+    uint32_t offset;
+
+    if ( !v->domain->arch.vgic.info->rdist_stride )
+        offset = info->gpa & (v->domain->arch.vgic.info->rdist_stride - 1);
+    else
+        offset = info->gpa & 0x1FFFF;
+
+    if ( offset < SZ_64K )
+       return __vgic_rdistr_mmio_write(v, info, offset);
+    else if ( (offset - SZ_64K) < SZ_64K )
+       return vgic_rdistr_sgi_mmio_write(v, info, (offset - SZ_64K));
+    else
+       printk("vGICR: wrong gpa write %"PRIpaddr"\n",info->gpa);
+    return 0;
+}
+
+static int vgic_distr_mmio_read(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    struct vgic_irq_rank *rank;
+    int offset = (int)(info->gpa - v->domain->arch.vgic.info->dbase);
+    int gicd_reg = REG(offset);
+
+    switch ( gicd_reg )
+    {
+    case GICD_CTLR:
+        if ( dabt.size != 2 ) goto bad_width;
+        vgic_lock(v);
+        *r = v->domain->arch.vgic.ctlr;
+        vgic_unlock(v);
+        return 1;
+    case GICD_TYPER:
+        if ( dabt.size != 2 ) goto bad_width;
+        /* No secure world support for guests. */
+        vgic_lock(v);
+        *r = ( (v->domain->max_vcpus<<5) & GICD_TYPE_CPUS )
+            |( ((v->domain->arch.vgic.nr_lines/32)) & GICD_TYPE_LINES );
+        vgic_unlock(v);
+        return 1;
+    case GICD_STATUSR:
+        /* Not implemented */
+        goto read_as_zero;
+    case GICD_IIDR:
+        if ( dabt.size != 2 ) goto bad_width;
+        /*
+         * XXX Do we need a JEP106 manufacturer ID?
+         * Just use the physical h/w value for now
+         */
+        *r = 0x0000034c;
+        return 1;
+    /* Implementation defined -- read as zero */
+    case REG(0x020) ... REG(0x03c):
+        goto read_as_zero;
+    case GICD_IGROUPR ... GICD_IGROUPRN:
+        /* We do not implement security extensions for guests, read zero */
+        goto read_as_zero;
+    case GICD_ISENABLER ... GICD_ISENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISENABLER);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->ienable;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_ICENABLER ... GICD_ICENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICENABLER);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->ienable;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_ISPENDR ... GICD_ISPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISPENDR);
+        if ( rank == NULL ) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->ipend, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_ICPENDR ... GICD_ICPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICPENDR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->ipend, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_ISACTIVER ... GICD_ISACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISACTIVER);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->iactive;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_ICACTIVER ... GICD_ICACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICACTIVER);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->iactive;
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_IROUTER ... GICD_IROUTERN:
+        rank = vgic_irq_rank(v, 64, gicd_reg - GICD_IROUTERN);
+        if ( rank == NULL) goto read_as_zero;
+
+        vgic_lock_rank(v, rank);
+        /* IROUTER is 64 bit so, to make it byte size right shift by 3. 
+           Here once. macro REG_RANK_INDEX will do it twice */
+        *r = rank->irouter[REG_RANK_INDEX(64, (gicd_reg - GICD_IROUTER)>>1)];
+        if ( dabt.size == 0 )
+            *r = byte_read(*r, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_IPRIORITYR ... GICD_IPRIORITYRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_IPRIORITYR);
+        if ( rank == NULL) goto read_as_zero;
+
+        vgic_lock_rank(v, rank);
+        *r = rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)];
+        if ( dabt.size == 0 )
+            *r = byte_read(*r, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_ICFGR ... GICD_ICFGRN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 2, gicd_reg - GICD_ICFGR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = rank->icfg[REG_RANK_INDEX(2, gicd_reg - GICD_ICFGR)];
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_NSACR ... GICD_NSACRN:
+        /* We do not implement security extensions for guests, read zero */
+        goto read_as_zero;
+    case GICD_SGIR:
+        if ( dabt.size != 2 ) goto bad_width;
+        /* Write only -- read unknown */
+        *r = 0xdeadbeef;
+        return 1;
+    case GICD_CPENDSGIR ... GICD_CPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_CPENDSGIR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->pendsgi, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_SPENDSGIR ... GICD_SPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_SPENDSGIR);
+        if ( rank == NULL) goto read_as_zero;
+        vgic_lock_rank(v, rank);
+        *r = byte_read(rank->pendsgi, dabt.sign, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+    case GICD_PIDR7... GICD_PIDR0:
+        /* GICv3 identification value */
+        *r = 0x92;
+        return 1;
+    case REG(0x00c):
+    case REG(0x044):
+    case REG(0x04c):
+    case REG(0x05c) ... REG(0x07c):
+    case REG(0xf30) ... REG(0x5fcc):
+    case REG(0x8000) ... REG(0xbfcc):
+    case REG(0xc000) ... REG(0xffcc):
+        printk("vGICD: read unknown 0x00c .. 0xfcc r%d offset %#08x\n", dabt.reg, offset);
+        goto read_as_zero;
+
+    default:
+        printk("vGICD: unhandled read r%d offset %#08x\n",
+               dabt.reg, offset);
+        return 0;
+    }
+
+bad_width:
+    printk("vGICD: bad read width %d r%d offset %#08x\n",
+           dabt.size, dabt.reg, offset);
+    domain_crash_synchronous();
+    return 0;
+
+read_as_zero:
+    printk("vGICD: read as zero %d r%d offset %#08x\n",
+           dabt.size, dabt.reg, offset);
+    if ( dabt.size != 2 ) goto bad_width;
+    *r = 0;
+    return 1;
+}
+
+static int vgic_distr_mmio_write(struct vcpu *v, mmio_info_t *info)
+{
+    struct hsr_dabt dabt = info->dabt;
+    struct cpu_user_regs *regs = guest_cpu_user_regs();
+    register_t *r = select_user_reg(regs, dabt.reg);
+    struct vgic_irq_rank *rank;
+    int offset = (int)(info->gpa - v->domain->arch.vgic.info->dbase);
+    int gicd_reg = REG(offset);
+    uint32_t tr;
+
+    switch ( gicd_reg )
+    {
+    case GICD_CTLR:
+        if ( dabt.size != 2 ) goto bad_width;
+        /* Ignore all but the enable bit */
+        v->domain->arch.vgic.ctlr = (*r) & GICD_CTL_ENABLE;
+        return 1;
+
+    /* R/O -- write ignored */
+    case GICD_TYPER:
+    case GICD_IIDR:
+        goto write_ignore;
+
+    case GICD_STATUSR:
+        goto write_ignore;
+    case GICD_SETSPI_NSR:
+        goto write_ignore;
+    case GICD_CLRSPI_NSR:
+        goto write_ignore;
+    case GICD_SETSPI_SR:
+        goto write_ignore;
+    case GICD_CLRSPI_SR:
+        goto write_ignore;
+    /* Implementation defined -- write ignored */
+    case REG(0x020) ... REG(0x03c):
+        printk("vGICD: write unknown 0x020 - 0x03c r%d offset %#08x\n", dabt.reg, offset);
+        goto write_ignore;
+
+    case GICD_IGROUPR ... GICD_IGROUPRN:
+        /* We do not implement security extensions for guests, write ignore */
+        goto write_ignore;
+
+    case GICD_ISENABLER ... GICD_ISENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISENABLER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        tr = rank->ienable;
+        rank->ienable |= *r;
+        vgic_unlock_rank(v, rank);
+        vgic_enable_irqs(v, (*r) & (~tr), (gicd_reg - GICD_ISENABLER) >> 2);
+        return 1;
+
+    case GICD_ICENABLER ... GICD_ICENABLERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICENABLER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        tr = rank->ienable;
+        rank->ienable &= ~*r;
+        vgic_unlock_rank(v, rank);
+        vgic_disable_irqs(v, (*r) & tr, gicd_reg - GICD_ICENABLER);
+        return 1;
+
+    case GICD_ISPENDR ... GICD_ISPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        return 0;
+
+    case GICD_ICPENDR ... GICD_ICPENDRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        return 0;
+
+    case GICD_ISACTIVER ... GICD_ISACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ISACTIVER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->iactive &= ~*r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICACTIVER ... GICD_ICACTIVERN:
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 1, gicd_reg - GICD_ICACTIVER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->iactive &= ~*r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_IROUTER ... GICD_IROUTER + 8*7:
+        /* SGI/PPI target is read only */
+        goto write_ignore;
+
+    case GICD_IROUTER + 8*8 ... GICD_IROUTERN:
+        rank = vgic_irq_rank(v, 64, gicd_reg - GICD_IROUTER);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        rank->irouter[REG_RANK_INDEX(64, (gicd_reg - GICD_IROUTER)>>1)] = *r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_IPRIORITYR ... GICD_IPRIORITYRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 8, gicd_reg - GICD_IPRIORITYR);
+        if ( rank == NULL) goto write_ignore;
+        vgic_lock_rank(v, rank);
+        if ( dabt.size == 2 )
+            rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)] = *r;
+        else
+            byte_write(&rank->ipriority[REG_RANK_INDEX(8, gicd_reg - GICD_IPRIORITYR)],
+                       *r, offset);
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_ICFGR: /* SGIs */
+        goto write_ignore;
+    case GICD_ICFGR + 4: /* PPIs */
+        /* It is implementation defined if these are writeable. We chose not */
+        goto write_ignore;
+    case GICD_ICFGR + 8 ... GICD_ICFGRN: /* SPIs */
+        if ( dabt.size != 2 ) goto bad_width;
+        rank = vgic_irq_rank(v, 2, gicd_reg - GICD_ICFGR);
+        vgic_lock_rank(v, rank);
+        if ( rank == NULL) goto write_ignore;
+        rank->icfg[REG_RANK_INDEX(2, gicd_reg - GICD_ICFGR)] = *r;
+        vgic_unlock_rank(v, rank);
+        return 1;
+
+    case GICD_NSACR ... GICD_NSACRN:
+        /* We do not implement security extensions for guests, write ignore */
+        goto write_ignore;
+
+    case GICD_SGIR:
+        /* it is accessed as system register */
+        goto write_ignore;
+
+    case GICD_CPENDSGIR ... GICD_CPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        return 0;
+
+    case GICD_SPENDSGIR ... GICD_SPENDSGIRN:
+        if ( dabt.size != 0 && dabt.size != 2 ) goto bad_width;
+        return 0;
+
+    case REG(0x00c):
+    case REG(0x044):
+    case REG(0x04c):
+    case REG(0x05c) ... REG(0x07c):
+    case REG(0xf30) ... REG(0x5fcc):
+    case REG(0x8000) ... REG(0xbfcc):
+    case REG(0xc000) ... REG(0xffcc):
+        printk("vGICD: write unknown 0x00c 0xfcc  r%d offset %#08x\n", dabt.reg, offset);
+        goto write_ignore;
+
+    default:
+        printk("vGICD: unhandled write r%d=%"PRIregister" offset %#08x\n",
+               dabt.reg, *r, offset);
+        return 0;
+    }
+
+bad_width:
+    printk("vGICD: bad write width %d r%d=%"PRIregister" offset %#08x\n",
+           dabt.size, dabt.reg, *r, offset);
+    domain_crash_synchronous();
+    return 0;
+
+write_ignore:
+    if ( dabt.size != 2 ) goto bad_width;
+    return 1;
+}
+
+
+static int vgic_rdistr_mmio_check(struct vcpu *v, paddr_t addr)
+{
+    struct domain *d = v->domain;
+
+    /* Compare with global rdist base. the addr should fall in this region */
+    return (addr >= (d->arch.vgic.info->rbase)) && (addr < (d->arch.vgic.info->rbase + d->arch.vgic.info->rbase_size));
+}
+
+static struct mmio_handler vgic_rdistr_mmio_handler = {
+    .check_handler = vgic_rdistr_mmio_check,
+    .read_handler  = vgic_rdistr_mmio_read,
+    .write_handler = vgic_rdistr_mmio_write,
+};
+
+static int vgic_distr_mmio_check(struct vcpu *v, paddr_t addr)
+{
+    struct domain *d = v->domain;
+
+    return (addr >= (d->arch.vgic.info->dbase)) && (addr < (d->arch.vgic.info->dbase + d->arch.vgic.info->dbase_size));
+}
+
+static struct mmio_handler vgic_distr_mmio_handler = {
+    .check_handler = vgic_distr_mmio_check,
+    .read_handler  = vgic_distr_mmio_read,
+    .write_handler = vgic_distr_mmio_write,
+};
+
+
+static int vgic_vcpu_init(struct vcpu *v)
+{
+    int i;
+    u64 affinity;
+
+    v->arch.vgic.private_irqs = xzalloc(struct vgic_irq_rank);
+
+    memset(v->arch.vgic.private_irqs, 0, sizeof(struct vgic_irq_rank));
+
+    spin_lock_init(&v->arch.vgic.private_irqs->lock);
+
+    /* For SGI and PPI the target is always this CPU */
+    affinity = cpu_logical_map(v->vcpu_id);
+    for ( i = 0 ; i < 32 ; i++ )
+        v->arch.vgic.private_irqs->irouter[i] = affinity;
+    return 0;
+}
+
+static int vgic_domain_init(struct domain *d)
+{
+    int i;
+
+    d->arch.vgic.shared_irqs =
+        xzalloc_array(struct vgic_irq_rank, DOMAIN_NR_RANKS(d));
+
+    for ( i = 0; i < DOMAIN_NR_RANKS(d); i++)
+        spin_lock_init(&d->arch.vgic.shared_irqs[i].lock);
+
+    register_mmio_handler(d, &vgic_distr_mmio_handler);
+    register_mmio_handler(d, &vgic_rdistr_mmio_handler);
+    return 0;
+}
+
+static struct vgic_ops ops = {
+    .vgic_vcpu_init   = vgic_vcpu_init,
+    .vgic_domain_init = vgic_domain_init,
+    .read_priority = vgic_read_priority,
+};
+
+static int __init vgic_v3_init(struct dt_device_node *dev, const void *data)
+{
+    register_vgic_ops(&ops);
+    return 0;
+}
+
+static const char * const vgicv3_dt_compat[] __initconst =
+{
+    "arm,gic-v3",
+    NULL 
+};
+
+DT_DEVICE_START(vgicv3, "VGIC", DEVICE_VGIC)
+        .compatible = vgicv3_dt_compat,
+        .init = vgic_v3_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 201f534..eadbfcc 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -63,7 +63,10 @@ struct vgic_irq_rank {
     uint32_t ienable, iactive, ipend, pendsgi;
     uint32_t icfg[2];
     uint32_t ipriority[8];
-    uint32_t itargets[8];
+    union {
+        uint32_t itargets[8];
+        uint64_t irouter[32];
+    };
 };
 
 struct vgic_info {
-- 
1.7.9.5

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> Add vgic driver support for GIC v3 version.

There is an awful lot of code duplication with the v2 stuff here,
especially with the distributor, but even the redistrubotr stuff looks
like it could share common implementation with the v2 distro for at
least some functionality.

I think we should investigate a sinlge driver here, or at least
refactoring anything which is common into a library of helper functions
which both can use.

> +    case GICR_SETLPIR:
> +    case GICR_CLRLPIR:
> +        /* WO */
> +        return 0;

Would the real hardware fault here? Or would it read unknown or zero or
os "WO" short for "write ignore"? (in which case goto write_ignore)


> +    case GICR_PROPBASER:
> +        /* LPI's not implemented */
> +        goto read_as_zero;
> +    case GICR_PENDBASER:
> +        /* LPI's not implemented */
> +        goto read_as_zero;
> +    case GICR_INVLPIR:
> +    case GICR_INVALLR:
> +        /* WO */
> +        return 0;
> +    case GICR_SYNCR:
> +        if ( dabt.size != 2 ) goto bad_width;
> +        /* RO */
> +        /* Return as not busy */
> +        *r = 1;

#define for the 1 please.

> +        return 1;
> +    case GICR_MOVLPIR:
> +    case GICR_MOVALLR:
> +        /* WO */
> +        return 0;
> +    case GICR_PIDR7... GICR_PIDR0:
> +        *r = 0x93;

Magic number?

> +         return 1;
> +    default:
> +        printk("vGICR: read r%d offset %#08x\n not found", dabt.reg, offset);

missing return 0.
> +
> +static int __vgic_rdistr_mmio_write(struct vcpu *v, mmio_info_t *info, uint32_t offset)
> +{
> +    struct hsr_dabt dabt = info->dabt;
> +    struct cpu_user_regs *regs = guest_cpu_user_regs();
> +    register_t *r = select_user_reg(regs, dabt.reg);
> +    int gicr_reg = REG(offset);
> +
> +    switch ( gicr_reg )
> +    {
> +    case GICR_CTLR:
> +        /* We do not implement LPI's, read zero */
> +        goto write_ignore;
> +    case GICR_IIDR:
> +    case GICR_TYPER:
> +        /* RO */
> +        goto write_ignore;
> +    case GICR_STATUSR:
> +        /* Not implemented */
> +        goto write_ignore;
> +    case GICR_WAKER:
> +        /* Not implemented */
> +        goto write_ignore;
> +    case GICR_SETLPIR:

Missing a comment? (Not implemented? Write ignore?) (several others
here)

> +        return 1;
> +    case GICR_CLRLPIR:
> +        return 1;
> +    case GICR_PROPBASER:
> +        /* LPI is not implemented */
> +        goto write_ignore;
> +    case GICR_PENDBASER:
> +        /* LPI is not implemented */
> +        goto write_ignore;
> +    case GICR_INVLPIR:
> +        return 1;
> +    case GICR_INVALLR:
> +        return 1;
> +    case GICR_SYNCR:
> +        /* RO */
> +        goto write_ignore;
> +    case GICR_MOVLPIR:
> +        return 1;
> +    case GICR_MOVALLR:
> +        return 1;
> +    case GICR_PIDR7... GICR_PIDR0:
> +        /* RO */
> +        goto write_ignore;
> +    default:
> +        printk("vGICR: write r%d offset %#08x\n not found", dabt.reg, offset);

missing return.

> +    }
> +bad_width:
> +    printk("vGICD: bad write width %d r%d=%"PRIregister" offset %#08x\n",
> +           dabt.size, dabt.reg, *r, offset);
> +    domain_crash_synchronous();
> +    return 0;
> +
> +write_ignore:
> +    if ( dabt.size != 2 ) goto bad_width;
> +    return 1;
> +}
> +
> +
> +static int vgic_rdistr_mmio_read(struct vcpu *v, mmio_info_t *info)
> +{
> +    uint32_t offset;
> +
> +    if ( !v->domain->arch.vgic.info->rdist_stride )
> +        offset = info->gpa & (v->domain->arch.vgic.info->rdist_stride - 1);
> +    else
> +        offset = info->gpa & 0x1FFFF;
> +
> +    if ( offset < SZ_64K )
> +       return __vgic_rdistr_mmio_read(v, info, offset);
> +    else if ( (offset - SZ_64K) < SZ_64K )
> +       return vgic_rdistr_sgi_mmio_read(v, info, (offset - SZ_64K));

I think you should either register two mmio handlers or maybe just
handle this all at once in a single callback.

I didn't see any system register handling, I suppose that is all handled
by the hardware GICV stuff?

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Vijay Kilari]

On Thu, Apr 10, 2014 at 3:53 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>
>> Add vgic driver support for GIC v3 version.
>
> There is an awful lot of code duplication with the v2 stuff here,
> especially with the distributor, but even the redistrubotr stuff looks
> like it could share common implementation with the v2 distro for at
> least some functionality.
>
> I think we should investigate a sinlge driver here, or at least
> refactoring anything which is common into a library of helper functions
> which both can use.

Yes, only distributor code is common which is just 2 functions i.e
read & write functions.
But the whole complexity is v2 gic & vgic driver is it runs with byte
offset that is the reason
why in gic_v2_defs all the register addresses definitions are divided by 4.
So this make it quite difficult to make common code.

With GICv2 HW constraint, I could not take it up. May be we can take
this as separate task.

>> +    case GICR_SETLPIR:
> I think you should either register two mmio handlers or maybe just
> handle this all at once in a single callback.
>
   There is only one address rdistributore base and size extracted from dt.
So we register one handle for this entire region and this single callback will
take access either GICR or SGI's GICR region.

> I didn't see any system register handling, I suppose that is all handled
> by the hardware GICV stuff?
   For all system register accesses in Dom's, HW treats are virtual access.
The memory mapped GICD & GICR registers are virtualized by VGIC
>

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Ian Campbell]

On Thu, 2014-04-10 at 16:13 +0530, Vijay Kilari wrote:
> On Thu, Apr 10, 2014 at 3:53 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> >> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> >>
> >> Add vgic driver support for GIC v3 version.
> >
> > There is an awful lot of code duplication with the v2 stuff here,
> > especially with the distributor, but even the redistrubotr stuff looks
> > like it could share common implementation with the v2 distro for at
> > least some functionality.
> >
> > I think we should investigate a sinlge driver here, or at least
> > refactoring anything which is common into a library of helper functions
> > which both can use.
> 
> Yes, only distributor code is common which is just 2 functions i.e
> read & write functions.
> But the whole complexity is v2 gic & vgic driver is it runs with byte
> offset that is the reason
> why in gic_v2_defs all the register addresses definitions are divided by 4.
> So this make it quite difficult to make common code.

It sounds trivial to make this work with appropriate accessor macros to
me.

In fact I'm not sure why readl(&GICD[GICD_FOO]) won't work for you just
as it does for v2 (e.g. type of GICD is uint32_t and GICD_FOO is /4).

> With GICv2 HW constraint, I could not take it up. May be we can take
> this as separate task.

What do you mean?

> >> +    case GICR_SETLPIR:
> > I think you should either register two mmio handlers or maybe just
> > handle this all at once in a single callback.
> >
>    There is only one address rdistributore base and size extracted from dt.
> So we register one handle for this entire region and this single callback will
> take access either GICR or SGI's GICR region.

Having a single region in DT doesn't stop you from doing the same
calculation you do now in the callback at init time and registering two
regions.

> > I didn't see any system register handling, I suppose that is all handled
> > by the hardware GICV stuff?
>    For all system register accesses in Dom's, HW treats are virtual access.
> The memory mapped GICD & GICR registers are virtualized by VGIC

I was looking for some register configuration which causes EL1 accesses
to the system registers to go to the "GICV" equivalent, but I didn't see
it. Can you point me to it please?

Ian.

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Vijay Kilari]

On Thu, Apr 10, 2014 at 4:21 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-04-10 at 16:13 +0530, Vijay Kilari wrote:
>> On Thu, Apr 10, 2014 at 3:53 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
>> >> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>> >>
>> >> Add vgic driver support for GIC v3 version.
>> >
>> > There is an awful lot of code duplication with the v2 stuff here,
>> > especially with the distributor, but even the redistrubotr stuff looks
>> > like it could share common implementation with the v2 distro for at
>> > least some functionality.
>> >
>> > I think we should investigate a sinlge driver here, or at least
>> > refactoring anything which is common into a library of helper functions
>> > which both can use.
>>
>> Yes, only distributor code is common which is just 2 functions i.e
>> read & write functions.
>> But the whole complexity is v2 gic & vgic driver is it runs with byte
>> offset that is the reason
>> why in gic_v2_defs all the register addresses definitions are divided by 4.
>> So this make it quite difficult to make common code.
>
> It sounds trivial to make this work with appropriate accessor macros to
> me.
>
> In fact I'm not sure why readl(&GICD[GICD_FOO]) won't work for you just
> as it does for v2 (e.g. type of GICD is uint32_t and GICD_FOO is /4).
>
>> With GICv2 HW constraint, I could not take it up. May be we can take
>> this as separate task.
>
> What do you mean?
>
I mean, I don't have V2 hardware to test any trivial changes that I do to
GIC v2. In all my patches, I am not making any changes to functionality of
v2 except code movement and changing names

>> >> +    case GICR_SETLPIR:
>> > I think you should either register two mmio handlers or maybe just
>> > handle this all at once in a single callback.
>> >
>>    There is only one address rdistributore base and size extracted from dt.
>> So we register one handle for this entire region and this single callback will
>> take access either GICR or SGI's GICR region.
>
> Having a single region in DT doesn't stop you from doing the same
> calculation you do now in the callback at init time and registering two
> regions.
>
SGI GICR region is per cpu, there is no single address with which
we can register a callback. So we register one callback for entire
re-distributor region and if it falls in that region we consider it
re-distributor
access.

This callback will take re-distributor stride offset and finds offset.

Also, this reduces the additional check for sgi re-distributor region

>> > I didn't see any system register handling, I suppose that is all handled
>> > by the hardware GICV stuff?
>>    For all system register accesses in Dom's, HW treats are virtual access.
>> The memory mapped GICD & GICR registers are virtualized by VGIC
>
> I was looking for some register configuration which causes EL1 accesses
> to the system registers to go to the "GICV" equivalent, but I didn't see
> it. Can you point me to it please?
>
You mean ICC_SRE_EL2 register.  GICv3 by default enables SRE bit.
> Ian.
>
>

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Ian Campbell]

On Thu, 2014-04-10 at 16:49 +0530, Vijay Kilari wrote:

> I mean, I don't have V2 hardware to test any trivial changes that I do to
> GIC v2. In all my patches, I am not making any changes to functionality of
> v2 except code movement and changing names

I can test it if necessary. Do you have access to fast models or
anything similar?

> >> >> +    case GICR_SETLPIR:
> >> > I think you should either register two mmio handlers or maybe just
> >> > handle this all at once in a single callback.
> >> >
> >>    There is only one address rdistributore base and size extracted from dt.
> >> So we register one handle for this entire region and this single callback will
> >> take access either GICR or SGI's GICR region.
> >
> > Having a single region in DT doesn't stop you from doing the same
> > calculation you do now in the callback at init time and registering two
> > regions.
> >
> SGI GICR region is per cpu, there is no single address with which
> we can register a callback. So we register one callback for entire
> re-distributor region and if it falls in that region we consider it
> re-distributor
> access.

Ah ok, that makes sense. Thanks.

> This callback will take re-distributor stride offset and finds offset.
> 
> Also, this reduces the additional check for sgi re-distributor region
> 
> >> > I didn't see any system register handling, I suppose that is all handled
> >> > by the hardware GICV stuff?
> >>    For all system register accesses in Dom's, HW treats are virtual access.
> >> The memory mapped GICD & GICR registers are virtualized by VGIC
> >
> > I was looking for some register configuration which causes EL1 accesses
> > to the system registers to go to the "GICV" equivalent, but I didn't see
> > it. Can you point me to it please?
> >
> You mean ICC_SRE_EL2 register.  GICv3 by default enables SRE bit.

Excellent, thanks.

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Vijay Kilari]

On Thu, Apr 10, 2014 at 4:56 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-04-10 at 16:49 +0530, Vijay Kilari wrote:
>
>> I mean, I don't have V2 hardware to test any trivial changes that I do to
>> GIC v2. In all my patches, I am not making any changes to functionality of
>> v2 except code movement and changing names
>
> I can test it if necessary. Do you have access to fast models or
> anything similar?

Currently I don't have one. I can setup.
Is it ok if I investigate on this later separately?

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Ian Campbell]

On Thu, 2014-04-10 at 17:08 +0530, Vijay Kilari wrote:
> On Thu, Apr 10, 2014 at 4:56 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Thu, 2014-04-10 at 16:49 +0530, Vijay Kilari wrote:
> >
> >> I mean, I don't have V2 hardware to test any trivial changes that I do to
> >> GIC v2. In all my patches, I am not making any changes to functionality of
> >> v2 except code movement and changing names
> >
> > I can test it if necessary. Do you have access to fast models or
> > anything similar?
> 
> Currently I don't have one. I can setup.
> Is it ok if I investigate on this later separately?

I'm afraid that I don't want to add duplicated code to be cleaned up
later. If this stuff can be shared then it should be done from the
start. (These things have a habit of not happening otherwise).

Ian.

Re: [PATCH v2 14/15] xen/arm: Add vgic support for GIC v3

[Vijay Kilari]

On Thu, Apr 10, 2014 at 5:38 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-04-10 at 17:08 +0530, Vijay Kilari wrote:
>> On Thu, Apr 10, 2014 at 4:56 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Thu, 2014-04-10 at 16:49 +0530, Vijay Kilari wrote:
>> >
>> >> I mean, I don't have V2 hardware to test any trivial changes that I do to
>> >> GIC v2. In all my patches, I am not making any changes to functionality of
>> >> v2 except code movement and changing names
>> >
>> > I can test it if necessary. Do you have access to fast models or
>> > anything similar?
>>
>> Currently I don't have one. I can setup.
>> Is it ok if I investigate on this later separately?
>
> I'm afraid that I don't want to add duplicated code to be cleaned up
> later. If this stuff can be shared then it should be done from the
> start. (These things have a habit of not happening otherwise).
>
Let me attempt and send a patch.
However I feel except vgic_distr_mmio_read & vgic_distr_mmio_write of
gicv2 code,
we can move all other functions to either header file (new file
vgic.h) or common code file (vgic.c)
which can be shared with gicv3.This is the simplest approach I think.

Also, in GICD, few register are changed between v2 & v3 ex: GICD_IROUTER

> Ian.
>

[PATCH v2 15/15] xen/arm: update GIC dt node with GIC v3 information

[vijay.kilari]

From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>

Update GIC device tree node for DOM0 with GICv3
information.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
---
 xen/arch/arm/domain_build.c |   41 ++++++++++++++++++++++++++++++++++-------
 xen/arch/arm/gic-v2.c       |    8 ++++++++
 xen/arch/arm/gic-v3.c       |    8 ++++++++
 xen/arch/arm/gic.c          |    5 +++++
 xen/include/asm-arm/gic.h   |    5 +++++
 5 files changed, 60 insertions(+), 7 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 66a98f1..5a75403 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -532,6 +532,8 @@ static int make_gic_node(const struct domain *d, void *fdt,
     u32 len;
     __be32 *new_cells, *tmp;
     int res = 0;
+    int hw_type = GIC_VERSION_V2;
+    u32 rd_stride = 0;
 
     /*
      * Xen currently supports only a single GIC. Discard any secondary
@@ -545,6 +547,8 @@ static int make_gic_node(const struct domain *d, void *fdt,
 
     DPRINT("Create gic node\n");
 
+    hw_type = gic_hw_version();
+
     compatible = dt_get_property(gic, "compatible", &len);
     if ( !compatible )
     {
@@ -552,6 +556,12 @@ static int make_gic_node(const struct domain *d, void *fdt,
         return -FDT_ERR_XEN(ENOENT);
     }
 
+    if ( hw_type == GIC_VERSION_V3 )
+    {
+        res = dt_property_read_u32(gic, "redistributor-stride", &rd_stride);
+        if ( !res )
+            rd_stride = 0;
+    }
     res = fdt_begin_node(fdt, "interrupt-controller");
     if ( res )
         return res;
@@ -576,14 +586,31 @@ static int make_gic_node(const struct domain *d, void *fdt,
         return -FDT_ERR_XEN(ENOMEM);
 
     tmp = new_cells;
-    DPRINT("  Set Distributor Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
-           d->arch.vgic.info->dbase, d->arch.vgic.info->dbase + PAGE_SIZE - 1);
-    dt_set_range(&tmp, node, d->arch.vgic.info->dbase, PAGE_SIZE);
-
-    DPRINT("  Set Cpu Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
-           d->arch.vgic.info->cbase, d->arch.vgic.info->cbase + (PAGE_SIZE * 2) - 1);
-    dt_set_range(&tmp, node, d->arch.vgic.info->cbase, PAGE_SIZE * 2);
+    if ( hw_type == GIC_VERSION_V3 )
+    {
+        res = fdt_property_cell(fdt, "redistributor-stride", rd_stride);
+        if ( res )
+            return res;
+        DPRINT("  gicv3 rd stride 0x%x\n", rd_stride);
+    
+        DPRINT("  Set Distributor Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
+        d->arch.vgic.info->dbase, d->arch.vgic.info->dbase + d->arch.vgic.info->dbase_size - 1);
+        dt_set_range(&tmp, node, d->arch.vgic.info->dbase, d->arch.vgic.info->dbase_size);
+
+        DPRINT("  Set  Re-distributor Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
+        d->arch.vgic.info->rbase, d->arch.vgic.info->rbase + d->arch.vgic.info->rbase_size - 1);
+        dt_set_range(&tmp, node, d->arch.vgic.info->rbase, d->arch.vgic.info->rbase_size);
+    }
+    else
+    {
+        DPRINT("  Set Distributor Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
+        d->arch.vgic.info->dbase, d->arch.vgic.info->dbase + PAGE_SIZE - 1);
+        dt_set_range(&tmp, node, d->arch.vgic.info->dbase, PAGE_SIZE);
 
+        DPRINT("  Set Cpu Base 0x%"PRIpaddr"-0x%"PRIpaddr"\n",
+        d->arch.vgic.info->cbase, d->arch.vgic.info->cbase + (PAGE_SIZE * 2) - 1);
+        dt_set_range(&tmp, node, d->arch.vgic.info->cbase, PAGE_SIZE * 2);
+    }
     res = fdt_property(fdt, "reg", new_cells, len);
     xfree(new_cells);
 
diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 6443bf4..de3e709 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -43,6 +43,7 @@
 
 /* Global state */
 static struct {
+    int hw_version;
     paddr_t dbase;       /* Address of distributor registers */
     paddr_t cbase;       /* Address of CPU interface registers */
     paddr_t hbase;       /* Address of virtual interface registers */
@@ -351,6 +352,7 @@ int __init gicv2_init(struct dt_device_node *node, const void *data)
     gic_cpu_init();
     gic_hyp_init();
 
+    gic.hw_version = GIC_VERSION_V2;
     register_gic_ops(&gic_ops);
     return 0;
 }
@@ -361,6 +363,11 @@ static void gic_secondary_cpu_init(void)
     gic_hyp_init();
 }
 
+static int gic_hw_type(void)
+{
+    return gic.hw_version;
+}
+
 static struct dt_irq * gic_maintenance_irq(void)
 {
     return &gic.maintenance;
@@ -504,6 +511,7 @@ static unsigned int gic_read_vmcr_priority(void)
 }
 
 static struct gic_hw_operations gic_ops = {
+    .gic_type            = gic_hw_type,
     .nr_lines            = gic_nr_lines,
     .nr_lrs              = gic_nr_lrs,
     .secondary_init      = gic_secondary_cpu_init,
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 8cefdab..0fe1059 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -49,6 +49,7 @@ struct rdist_region {
 
 /* Global state */
 static struct {
+    int hw_version;
     paddr_t dbase;            /* Address of distributor registers */
     paddr_t dbase_size;
     void __iomem *map_dbase;  /* Mapped address of distributor registers */
@@ -766,6 +767,11 @@ int gicv_init(struct domain *d)
     return 0;
 }
 
+static int gic_hw_type(void)
+{
+    return gic.hw_version;
+}
+
 static struct dt_irq * gic_maintenance_irq(void)
 {
     return &gic.maintenance;
@@ -786,6 +792,7 @@ static unsigned int gic_read_vmcr_priority(void)
 }
 
 static struct gic_hw_operations gic_ops = {
+    .gic_type            = gic_hw_type,
     .nr_lines            = gic_nr_lines,
     .nr_lrs              = gic_nr_lrs,
     .get_maintenance_irq = gic_maintenance_irq,
@@ -893,6 +900,7 @@ static int __init gicv3_init(struct dt_device_node *node, const void *data)
     gic_cpu_init();
     gic_hyp_init();
 
+    gic.hw_version = GIC_VERSION_V3;
     /* Register hw ops*/
     register_gic_ops(&gic_ops);
     return 0;
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 87a7ad0..e09d5a5 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -58,6 +58,11 @@ void update_cpu_lr_mask(void)
 
 static void gic_clear_one_lr(struct vcpu *v, int i);
 
+int gic_hw_version(void)
+{
+    return gic_hw_ops->gic_type();
+}
+
 unsigned int gic_number_lines(void)
 {
     return gic_hw_ops->nr_lines();
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index eadbfcc..32212f9 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -51,6 +51,9 @@
 #define GICH_HCR_VGRP1EIE (1 << 6)
 #define GICH_HCR_VGRP1DIE (1 << 7)
 
+#define GIC_VERSION_V3 0x3
+#define GIC_VERSION_V2 0x2
+
 #ifndef __ASSEMBLY__
 #include <xen/device_tree.h>
 
@@ -175,6 +178,7 @@ int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
 void gic_clear_lrs(struct vcpu *v);
 
 struct gic_hw_operations {
+    int (*gic_type)(void);
     struct dt_irq * (*get_maintenance_irq)(void);
     unsigned int (*nr_lines)(void);
     unsigned int (*nr_lrs)(void);
@@ -211,6 +215,7 @@ struct vgic_ops {
 void register_vgic_ops(struct vgic_ops *ops);
 void register_gic_ops(struct gic_hw_operations *ops);
 extern void update_cpu_lr_mask(void);
+extern int gic_hw_version(void);
 #endif /* __ASSEMBLY__ */
 #endif
 
-- 
1.7.9.5

Re: [PATCH v2 15/15] xen/arm: update GIC dt node with GIC v3 information

[Julien Grall]

Hello Vijaya,

Thank you for the patch.

The name of this patch is still wrong. Which GIC DT node are you updating?

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> +    if ( hw_type == GIC_VERSION_V3 )
> +    {
> +        res = dt_property_read_u32(gic, "redistributor-stride", &rd_stride);
> +        if ( !res )
> +            rd_stride = 0;
> +    }


You have skipped some of my remarks on V1. I would definitely prefer to
have a callback in your {v,}GIC structure which add the specific GIC
properties following the version supported by DOM0.

[..]

>  static struct dt_irq * gic_maintenance_irq(void)
>  {
>      return &gic.maintenance;
> @@ -786,6 +792,7 @@ static unsigned int gic_read_vmcr_priority(void)
>  }
>  
>  static struct gic_hw_operations gic_ops = {
> +    .gic_type            = gic_hw_type,

This should go in patch #5. As I said on V1, you don't need a callback
for return a fixed number (that you've initialized in gicv3_init). You
can store a number in this structure.

At the same time what is the difference between this number and the one
in GIC state?

>      .nr_lines            = gic_nr_lines,
>      .nr_lrs              = gic_nr_lrs,
>      .get_maintenance_irq = gic_maintenance_irq,
> @@ -893,6 +900,7 @@ static int __init gicv3_init(struct dt_device_node *node, const void *data)
>      gic_cpu_init();
>      gic_hyp_init();
>  
> +    gic.hw_version = GIC_VERSION_V3;
>      /* Register hw ops*/
>      register_gic_ops(&gic_ops);
>      return 0;
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 87a7ad0..e09d5a5 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -58,6 +58,11 @@ void update_cpu_lr_mask(void)
>  
>  static void gic_clear_one_lr(struct vcpu *v, int i);
>  
> +int gic_hw_version(void)
> +{
> +    return gic_hw_ops->gic_type();
> +}
> +
>  unsigned int gic_number_lines(void)
>  {
>      return gic_hw_ops->nr_lines();
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index eadbfcc..32212f9 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -51,6 +51,9 @@
>  #define GICH_HCR_VGRP1EIE (1 << 6)
>  #define GICH_HCR_VGRP1DIE (1 << 7)
>  
> +#define GIC_VERSION_V3 0x3
> +#define GIC_VERSION_V2 0x2
> +

Should not be defined earlier?

-- 
Julien Grall

Re: [PATCH v2 15/15] xen/arm: update GIC dt node with GIC v3 information

[Vijay Kilari]

On Fri, Apr 4, 2014 at 7:52 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hello Vijaya,
>
> Thank you for the patch.
>
> The name of this patch is still wrong. Which GIC DT node are you updating?
>
> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>> +    if ( hw_type == GIC_VERSION_V3 )
>> +    {
>> +        res = dt_property_read_u32(gic, "redistributor-stride", &rd_stride);
>> +        if ( !res )
>> +            rd_stride = 0;
>> +    }
>
>
> You have skipped some of my remarks on V1. I would definitely prefer to
> have a callback in your {v,}GIC structure which add the specific GIC
> properties following the version supported by DOM0.
>
Yes, I recognized your comment. My understanding is that while
creating this gic node for
dom0, we should follow the order in creating cells.
Also, the cells are allocated here and these should be passed to gic
driver to fill it.
So looks ugly.

I prefer completely move this code to gic.c and from gic.c based on
registered callback, let gic-{v2/v3}.c manage version specific dt cells update.
Is this OK?
+
> --
> Julien Grall

Re: [PATCH v2 15/15] xen/arm: update GIC dt node with GIC v3 information

[Julien Grall]

Hello Vijaya,

On 04/04/2014 04:45 PM, Vijay Kilari wrote:
> On Fri, Apr 4, 2014 at 7:52 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>> +    if ( hw_type == GIC_VERSION_V3 )
>>> +    {
>>> +        res = dt_property_read_u32(gic, "redistributor-stride", &rd_stride);
>>> +        if ( !res )
>>> +            rd_stride = 0;
>>> +    }
>>
>>
>> You have skipped some of my remarks on V1. I would definitely prefer to
>> have a callback in your {v,}GIC structure which add the specific GIC
>> properties following the version supported by DOM0.
>>
> Yes, I recognized your comment. My understanding is that while
> creating this gic node for
> dom0, we should follow the order in creating cells.
> Also, the cells are allocated here and these should be passed to gic
> driver to fill it.
> So looks ugly.

I don't understand the issue with the order and the allocation...

You have bunch of code in if (hw_type == GICV3) than can be easily move
to the GIC driver.

You could have something like:

fdt_set_property(fdt, "mycommonprop1",..)
fdt_set_property(fdt, "mycommonprop2",..)
gic_node_specific_properties();
fdt_end();

> I prefer completely move this code to gic.c and from gic.c based on
> registered callback, let gic-{v2/v3}.c manage version specific dt cells update.
> Is this OK?

What do you mean?

Regards,

-- 
Julien Grall

Re: [PATCH v2 15/15] xen/arm: update GIC dt node with GIC v3 information

[Vijay Kilari]

On Fri, Apr 4, 2014 at 9:30 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hello Vijaya,
>
> On 04/04/2014 04:45 PM, Vijay Kilari wrote:
>> On Fri, Apr 4, 2014 at 7:52 PM, Julien Grall <julien.grall@linaro.org> wrote:
>>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>>> +    if ( hw_type == GIC_VERSION_V3 )
>>>> +    {
>>>> +        res = dt_property_read_u32(gic, "redistributor-stride", &rd_stride);
>>>> +        if ( !res )
>>>> +            rd_stride = 0;
>>>> +    }
>>>
>>>
>>> You have skipped some of my remarks on V1. I would definitely prefer to
>>> have a callback in your {v,}GIC structure which add the specific GIC
>>> properties following the version supported by DOM0.
>>>
>> Yes, I recognized your comment. My understanding is that while
>> creating this gic node for
>> dom0, we should follow the order in creating cells.
>> Also, the cells are allocated here and these should be passed to gic
>> driver to fill it.
>> So looks ugly.
>
> I don't understand the issue with the order and the allocation...
>
   I mean if we change the order of calling fdt_set_property for
various properties, the node creation goes wrong.  something like
this
  fdt_set_property(fdt, "mycommonprop2",..)
 fdt_set_property(fdt, "mycommonprop1",..)

I have given a try and dom0 fails to boot.

> You have bunch of code in if (hw_type == GICV3) than can be easily move
> to the GIC driver.
>
> You could have something like:
>
> fdt_set_property(fdt, "mycommonprop1",..)
> fdt_set_property(fdt, "mycommonprop2",..)
> gic_node_specific_properties();
> fdt_end();
>
>> I prefer completely move this code to gic.c and from gic.c based on
>> registered callback, let gic-{v2/v3}.c manage version specific dt cells update.
>> Is this OK?
>
> What do you mean?
>
 I mean why can't we move, why can't we move complete make_gic_node
to gic.c? as below and let it handle hw specific things?

fdt_set_property(fdt, "mycommonprop1",..)
fdt_set_property(fdt, "mycommonprop2",..)
gic_node_specific_properties();
fdt_end();

Anyways, I will relook and see if I can simplify as you suggested

> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 15/15] xen/arm: update GIC dt node with GIC v3 information

[Julien Grall]

On 04/04/2014 05:13 PM, Vijay Kilari wrote:
> On Fri, Apr 4, 2014 at 9:30 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> Hello Vijaya,
>>
>> On 04/04/2014 04:45 PM, Vijay Kilari wrote:
>>> On Fri, Apr 4, 2014 at 7:52 PM, Julien Grall <julien.grall@linaro.org> wrote:
>>>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>>>> +    if ( hw_type == GIC_VERSION_V3 )
>>>>> +    {
>>>>> +        res = dt_property_read_u32(gic, "redistributor-stride", &rd_stride);
>>>>> +        if ( !res )
>>>>> +            rd_stride = 0;
>>>>> +    }
>>>>
>>>>
>>>> You have skipped some of my remarks on V1. I would definitely prefer to
>>>> have a callback in your {v,}GIC structure which add the specific GIC
>>>> properties following the version supported by DOM0.
>>>>
>>> Yes, I recognized your comment. My understanding is that while
>>> creating this gic node for
>>> dom0, we should follow the order in creating cells.
>>> Also, the cells are allocated here and these should be passed to gic
>>> driver to fill it.
>>> So looks ugly.
>>
>> I don't understand the issue with the order and the allocation...
>>
>    I mean if we change the order of calling fdt_set_property for
> various properties, the node creation goes wrong.  something like
> this
>   fdt_set_property(fdt, "mycommonprop2",..)
>  fdt_set_property(fdt, "mycommonprop1",..)
> 
> I have given a try and dom0 fails to boot.

Are you sure? DOM0 doesn't care about the order of the properties... it
only care about the name and the content.

>> What do you mean?
>>
>  I mean why can't we move, why can't we move complete make_gic_node
> to gic.c? as below and let it handle hw specific things?
> 
> fdt_set_property(fdt, "mycommonprop1",..)
> fdt_set_property(fdt, "mycommonprop2",..)
> gic_node_specific_properties();
> fdt_end();

That's the solution I've asked on V1...

-- 
Julien Grall

Re: [PATCH v2 15/15] xen/arm: update GIC dt node with GIC v3 information

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> Update GIC device tree node for DOM0 with GICv3
> information.

Perhaps this should just be a callback in the vigc ops struct?

> 
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> ---
>  xen/arch/arm/domain_build.c |   41 ++++++++++++++++++++++++++++++++++-------
>  xen/arch/arm/gic-v2.c       |    8 ++++++++
>  xen/arch/arm/gic-v3.c       |    8 ++++++++
>  xen/arch/arm/gic.c          |    5 +++++
>  xen/include/asm-arm/gic.h   |    5 +++++
>  5 files changed, 60 insertions(+), 7 deletions(-)
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 66a98f1..5a75403 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -532,6 +532,8 @@ static int make_gic_node(const struct domain *d, void *fdt,
>      u32 len;
>      __be32 *new_cells, *tmp;
>      int res = 0;
> +    int hw_type = GIC_VERSION_V2;

an enum, perhaps?

> +    u32 rd_stride = 0;
>  
>      /*
>       * Xen currently supports only a single GIC. Discard any secondary
> @@ -545,6 +547,8 @@ static int make_gic_node(const struct domain *d, void *fdt,
>  
>      DPRINT("Create gic node\n");
>  
> +    hw_type = gic_hw_version();
> +
>      compatible = dt_get_property(gic, "compatible", &len);
>      if ( !compatible )
>      {
> @@ -552,6 +556,12 @@ static int make_gic_node(const struct domain *d, void *fdt,
>          return -FDT_ERR_XEN(ENOENT);
>      }
>  
> +    if ( hw_type == GIC_VERSION_V3 )

switch over a hw_type enum? With an empty v2 case. Likewise elsewhere,
Then the compiler will help us catch missing cases.

> +    {
> +        res = dt_property_read_u32(gic, "redistributor-stride", &rd_stride);
> +        if ( !res )
> +            rd_stride = 0;
> +    }
>      res = fdt_begin_node(fdt, "interrupt-controller");
>      if ( res )
>          return res;
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 8cefdab..0fe1059 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -49,6 +49,7 @@ struct rdist_region {
>  
>  /* Global state */
>  static struct {
> +    int hw_version;
>      paddr_t dbase;            /* Address of distributor registers */
>      paddr_t dbase_size;
>      void __iomem *map_dbase;  /* Mapped address of distributor registers */
> @@ -766,6 +767,11 @@ int gicv_init(struct domain *d)
>      return 0;
>  }
>  
> +static int gic_hw_type(void)
> +{
> +    return gic.hw_version;

Wouldn't this just be hardcoded as v3?

Ian.

Re: [PATCH v2 00/15] xen/arm: Add GICv3 support

[Julien Grall]

Hi Vijay

On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> Made changes to existing gic and vgic drivers
> to make it generic and added support for GIC v3
> hw version
> 
> Tested with ARM64 simulator with single core

What about multiple core? On V1 you said you were unable to use SMP with
GICv3, did you find the issue?

BTW, do you have a git tree with these patch series? I'd like to give a
try on GICv2 platform.

Regards,

-- 
Julien Grall

Re: [PATCH v2 00/15] xen/arm: Add GICv3 support

[Vijay Kilari]

On Fri, Apr 4, 2014 at 6:31 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Vijay
>
> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>
>> Made changes to existing gic and vgic drivers
>> to make it generic and added support for GIC v3
>> hw version
>>
>> Tested with ARM64 simulator with single core
>
> What about multiple core? On V1 you said you were unable to use SMP with
> GICv3, did you find the issue?
>
   There is no issue with hardware now. Xen can boot with multicore (2
cores) but dom0 single core.
limited using bootargs. I started working on Dom0 boot with multicore.

In GICv3, On SGI at EL1 (kernel), it traps to EL2. Xen does not handle
this trap.
So need to implement it. I started working on it and also trying to
understand EL1
 SGI routing in vgic driver. BTW do you know how SGI is handled in
ARMv7 with GICv2?
I think vgic_to_sgi() in vgic driver is managing it.

> BTW, do you have a git tree with these patch series? I'd like to give a
> try on GICv2 platform.
>
No, I don't have any public git tree to share the patches

> Regards,
>
> --
> Julien Grall

Re: [PATCH v2 00/15] xen/arm: Add GICv3 support

[Julien Grall]

On 04/04/2014 04:56 PM, Vijay Kilari wrote:
> On Fri, Apr 4, 2014 at 6:31 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> Hi Vijay
>>
>> On 04/04/2014 12:56 PM, vijay.kilari@gmail.com wrote:
>>> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
>>>
>>> Made changes to existing gic and vgic drivers
>>> to make it generic and added support for GIC v3
>>> hw version
>>>
>>> Tested with ARM64 simulator with single core
>>
>> What about multiple core? On V1 you said you were unable to use SMP with
>> GICv3, did you find the issue?
>>
>    There is no issue with hardware now. Xen can boot with multicore (2
> cores) but dom0 single core.
> limited using bootargs. I started working on Dom0 boot with multicore.
> 
> In GICv3, On SGI at EL1 (kernel), it traps to EL2. Xen does not handle
> this trap.
> So need to implement it. I started working on it and also trying to
> understand EL1
>  SGI routing in vgic driver. BTW do you know how SGI is handled in
> ARMv7 with GICv2?
> I think vgic_to_sgi() in vgic driver is managing it.

Right, EL1 raises an SGI via the distributor.

Regards,

-- 
Julien Grall

Re: [PATCH v2 00/15] xen/arm: Add GICv3 support

[Ian Campbell]

On Fri, 2014-04-04 at 17:26 +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
> 
> Made changes to existing gic and vgic drivers
> to make it generic and added support for GIC v3
> hw version

Which Xen changeset is this based on? I tried to apply it to staging but
there were some conflicts in gic.h. I just wanted to have a look at the
final code to help me understand the trajectory of one of the patches...