[PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Jess Frazelle]

Marked msi_domain_ops structs as __ro_after_init when called only during init.
This protects the data structure from accidental corruption.

Suggested-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Jess Frazelle <me@jessfraz.com>
---
 drivers/pci/host/pci-hyperv.c | 2 +-
 drivers/pci/host/vmd.c        | 2 +-
 drivers/pci/msi.c             | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
index 3efcc7bdc5fb..f05b93689d8f 100644
--- a/drivers/pci/host/pci-hyperv.c
+++ b/drivers/pci/host/pci-hyperv.c
@@ -958,7 +958,7 @@ static irq_hw_number_t hv_msi_domain_ops_get_hwirq(struct msi_domain_info *info,
 	return arg->msi_hwirq;
 }

-static struct msi_domain_ops hv_msi_ops = {
+static struct msi_domain_ops hv_msi_ops __ro_after_init = {
 	.get_hwirq	= hv_msi_domain_ops_get_hwirq,
 	.msi_prepare	= pci_msi_prepare,
 	.set_desc	= pci_msi_set_desc,
diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
index 18ef1a93c10a..152c461538e4 100644
--- a/drivers/pci/host/vmd.c
+++ b/drivers/pci/host/vmd.c
@@ -253,7 +253,7 @@ static void vmd_set_desc(msi_alloc_info_t *arg, struct msi_desc *desc)
 	arg->desc = desc;
 }

-static struct msi_domain_ops vmd_msi_domain_ops = {
+static struct msi_domain_ops vmd_msi_domain_ops __ro_after_init = {
 	.get_hwirq	= vmd_get_hwirq,
 	.msi_init	= vmd_msi_init,
 	.msi_free	= vmd_msi_free,
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 50c5003295ca..93141d5e2d1c 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -1413,7 +1413,7 @@ static void pci_msi_domain_set_desc(msi_alloc_info_t *arg,
 #define pci_msi_domain_set_desc		NULL
 #endif

-static struct msi_domain_ops pci_msi_domain_ops_default = {
+static struct msi_domain_ops pci_msi_domain_ops_default __ro_after_init = {
 	.set_desc	= pci_msi_domain_set_desc,
 	.msi_check	= pci_msi_domain_check_cap,
 	.handle_error	= pci_msi_domain_handle_error,
--
2.11.0

RE: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[KY Srinivasan]

> -----Original Message-----
> From: Jess Frazelle [mailto:me@jessfraz.com]
> Sent: Friday, February 10, 2017 5:38 PM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; Bjorn Helgaas <bhelgaas@google.com>; Keith
> Busch <keith.busch@intel.com>; open list:Hyper-V CORE AND DRIVERS
> <devel@linuxdriverproject.org>; open list:PCI SUBSYSTEM <linux-
> pci@vger.kernel.org>; open list <linux-kernel@vger.kernel.org>
> Cc: kernel-hardening@lists.openwall.com; Jess Frazelle <me@jessfraz.com>
> Subject: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init
>=20
> Marked msi_domain_ops structs as __ro_after_init when called only during
> init.
> This protects the data structure from accidental corruption.
>=20
> Suggested-by: Kees Cook <keescook@chromium.org>
> Signed-off-by: Jess Frazelle <me@jessfraz.com>

Acked-by: K. Y. Srinivasan <kys@microsoft.com>

K. Y

> ---
>  drivers/pci/host/pci-hyperv.c | 2 +-
>  drivers/pci/host/vmd.c        | 2 +-
>  drivers/pci/msi.c             | 2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)
>=20
> diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.=
c
> index 3efcc7bdc5fb..f05b93689d8f 100644
> --- a/drivers/pci/host/pci-hyperv.c
> +++ b/drivers/pci/host/pci-hyperv.c
> @@ -958,7 +958,7 @@ static irq_hw_number_t
> hv_msi_domain_ops_get_hwirq(struct msi_domain_info *info,
>  	return arg->msi_hwirq;
>  }
>=20
> -static struct msi_domain_ops hv_msi_ops =3D {
> +static struct msi_domain_ops hv_msi_ops __ro_after_init =3D {
>  	.get_hwirq	=3D hv_msi_domain_ops_get_hwirq,
>  	.msi_prepare	=3D pci_msi_prepare,
>  	.set_desc	=3D pci_msi_set_desc,
> diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
> index 18ef1a93c10a..152c461538e4 100644
> --- a/drivers/pci/host/vmd.c
> +++ b/drivers/pci/host/vmd.c
> @@ -253,7 +253,7 @@ static void vmd_set_desc(msi_alloc_info_t *arg,
> struct msi_desc *desc)
>  	arg->desc =3D desc;
>  }
>=20
> -static struct msi_domain_ops vmd_msi_domain_ops =3D {
> +static struct msi_domain_ops vmd_msi_domain_ops __ro_after_init =3D {
>  	.get_hwirq	=3D vmd_get_hwirq,
>  	.msi_init	=3D vmd_msi_init,
>  	.msi_free	=3D vmd_msi_free,
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 50c5003295ca..93141d5e2d1c 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -1413,7 +1413,7 @@ static void
> pci_msi_domain_set_desc(msi_alloc_info_t *arg,
>  #define pci_msi_domain_set_desc		NULL
>  #endif
>=20
> -static struct msi_domain_ops pci_msi_domain_ops_default =3D {
> +static struct msi_domain_ops pci_msi_domain_ops_default __ro_after_init
> =3D {
>  	.set_desc	=3D pci_msi_domain_set_desc,
>  	.msi_check	=3D pci_msi_domain_check_cap,
>  	.handle_error	=3D pci_msi_domain_handle_error,
> --
> 2.11.0

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Keith Busch]

On Fri, Feb 10, 2017 at 05:37:56PM -0800, Jess Frazelle wrote:
> Marked msi_domain_ops structs as __ro_after_init when called only during init.
> This protects the data structure from accidental corruption.
> 
> Suggested-by: Kees Cook <keescook@chromium.org>
> Signed-off-by: Jess Frazelle <me@jessfraz.com>
>
>  drivers/pci/host/pci-hyperv.c | 2 +-
>  drivers/pci/host/vmd.c        | 2 +-

Ok for vmd driver.

Acked-by: Keith Busch <keith.busch@intel.com>

>  drivers/pci/msi.c             | 2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Bjorn Helgaas]

[+cc Kees, Thomas, Marc]

Hi Jess,

Thanks for the patch!

On Fri, Feb 10, 2017 at 05:37:56PM -0800, Jess Frazelle wrote:
> Marked msi_domain_ops structs as __ro_after_init when called only during init.
> This protects the data structure from accidental corruption.
> 
> Suggested-by: Kees Cook <keescook@chromium.org>
> Signed-off-by: Jess Frazelle <me@jessfraz.com>
> ---
>  drivers/pci/host/pci-hyperv.c | 2 +-
>  drivers/pci/host/vmd.c        | 2 +-
>  drivers/pci/msi.c             | 2 +-

I understand the value of __ro_after_init, but I'm not certain about
sprinkling it around in seemingly random places because it's hard to
know where to put it and whether we put it in all the right places.

How did you choose these three files to change?  There are other
instances of struct msi_domain_ops that use MSI_FLAG_USE_DEF_DOM_OPS.
Should they be changed, too?  If not, is there a rule to figure out
which ones should be made __ro_after_init?

I wonder if adding __ro_after_init is really the best solution.  I
looked at VMD to see how vmd_msi_domain_ops is updated.  Here are the
definitions:

  static struct msi_domain_ops vmd_msi_domain_ops = {
    .get_hwirq = vmd_get_hwirq,
    .msi_init  = vmd_msi_init,
    ...
  };

  static struct msi_domain_info vmd_msi_domain_info = {
    .flags = MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
             MSI_FLAG_PCI_MSIX,
    .ops = &vmd_msi_domain_ops,
    ...
  };

Both vmd_msi_domain_ops and vmd_msi_domain_info are statically
initialized, but not completely.  Then we pass a pointer to
pci_msi_create_irq_domain(), which fills in defaults for some of the
function pointers that weren't statically initialized:

  vmd_enable_domain()
    pci_msi_create_irq_domain(NULL, &vmd_msi_domain_info, ...)
      if (info->flags & MSI_FLAG_USE_DEF_DOM_OPS)
        pci_msi_domain_update_dom_ops(info)
	  if (ops->set_desc == NULL)
	    ops->msi_check = pci_msi_domain_check_cap

We know at build-time what all the function pointers will be, so in
principle we should be able to make the struct const, which would be
even better than __ro_after_init.

For example, we could require that callers set every function pointer
before calling pci_msi_create_irq_domain(), using the default ones
(pci_msi_domain_set_desc, pci_msi_domain_check_cap,
pci_msi_domain_handle_error) if it doesn't need to override them,
e.g.,

  static struct msi_domain_ops vmd_msi_domain_ops = {
    .get_hwirq = vmd_get_hwirq,
    .msi_check = pci_msi_domain_check_cap,
  };

Or we could leave NULL pointers in the structure and have the code
that calls through the function pointers check for NULL and call the
default itself, e.g.,

  if (ops->msi_check)
    ops->msi_check(...)
  else
    pci_msi_domain_check_cap(...)

It looks like the "USE_DEF_OPS" framework was added by Jiang Liu with
the commits below.  I would CC: him for his thoughts, but I don't
have a current email address.

  aeeb59657c35 ("genirq: Provide default callbacks for msi_domain_ops")
  3878eaefb89a ("PCI/MSI: Enhance core to support hierarchy irqdomain")

Bjorn

>  3 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
> index 3efcc7bdc5fb..f05b93689d8f 100644
> --- a/drivers/pci/host/pci-hyperv.c
> +++ b/drivers/pci/host/pci-hyperv.c
> @@ -958,7 +958,7 @@ static irq_hw_number_t hv_msi_domain_ops_get_hwirq(struct msi_domain_info *info,
>  	return arg->msi_hwirq;
>  }
> 
> -static struct msi_domain_ops hv_msi_ops = {
> +static struct msi_domain_ops hv_msi_ops __ro_after_init = {
>  	.get_hwirq	= hv_msi_domain_ops_get_hwirq,
>  	.msi_prepare	= pci_msi_prepare,
>  	.set_desc	= pci_msi_set_desc,
> diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
> index 18ef1a93c10a..152c461538e4 100644
> --- a/drivers/pci/host/vmd.c
> +++ b/drivers/pci/host/vmd.c
> @@ -253,7 +253,7 @@ static void vmd_set_desc(msi_alloc_info_t *arg, struct msi_desc *desc)
>  	arg->desc = desc;
>  }
> 
> -static struct msi_domain_ops vmd_msi_domain_ops = {
> +static struct msi_domain_ops vmd_msi_domain_ops __ro_after_init = {
>  	.get_hwirq	= vmd_get_hwirq,
>  	.msi_init	= vmd_msi_init,
>  	.msi_free	= vmd_msi_free,
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 50c5003295ca..93141d5e2d1c 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -1413,7 +1413,7 @@ static void pci_msi_domain_set_desc(msi_alloc_info_t *arg,
>  #define pci_msi_domain_set_desc		NULL
>  #endif
> 
> -static struct msi_domain_ops pci_msi_domain_ops_default = {
> +static struct msi_domain_ops pci_msi_domain_ops_default __ro_after_init = {
>  	.set_desc	= pci_msi_domain_set_desc,
>  	.msi_check	= pci_msi_domain_check_cap,
>  	.handle_error	= pci_msi_domain_handle_error,
> --
> 2.11.0
> 

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Kees Cook]

On Wed, Feb 15, 2017 at 12:33 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> [+cc Kees, Thomas, Marc]
>
> Hi Jess,
>
> Thanks for the patch!
>
> On Fri, Feb 10, 2017 at 05:37:56PM -0800, Jess Frazelle wrote:
>> Marked msi_domain_ops structs as __ro_after_init when called only during init.
>> This protects the data structure from accidental corruption.
>>
>> Suggested-by: Kees Cook <keescook@chromium.org>
>> Signed-off-by: Jess Frazelle <me@jessfraz.com>
>> ---
>>  drivers/pci/host/pci-hyperv.c | 2 +-
>>  drivers/pci/host/vmd.c        | 2 +-
>>  drivers/pci/msi.c             | 2 +-
>
> I understand the value of __ro_after_init, but I'm not certain about
> sprinkling it around in seemingly random places because it's hard to
> know where to put it and whether we put it in all the right places.
>
> How did you choose these three files to change?  There are other
> instances of struct msi_domain_ops that use MSI_FLAG_USE_DEF_DOM_OPS.
> Should they be changed, too?  If not, is there a rule to figure out
> which ones should be made __ro_after_init?
>
> I wonder if adding __ro_after_init is really the best solution.  I
> looked at VMD to see how vmd_msi_domain_ops is updated.  Here are the
> definitions:
>
>   static struct msi_domain_ops vmd_msi_domain_ops = {
>     .get_hwirq = vmd_get_hwirq,
>     .msi_init  = vmd_msi_init,
>     ...
>   };
>
>   static struct msi_domain_info vmd_msi_domain_info = {
>     .flags = MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
>              MSI_FLAG_PCI_MSIX,
>     .ops = &vmd_msi_domain_ops,
>     ...
>   };
>
> Both vmd_msi_domain_ops and vmd_msi_domain_info are statically
> initialized, but not completely.  Then we pass a pointer to
> pci_msi_create_irq_domain(), which fills in defaults for some of the
> function pointers that weren't statically initialized:
>
>   vmd_enable_domain()
>     pci_msi_create_irq_domain(NULL, &vmd_msi_domain_info, ...)
>       if (info->flags & MSI_FLAG_USE_DEF_DOM_OPS)
>         pci_msi_domain_update_dom_ops(info)
>           if (ops->set_desc == NULL)
>             ops->msi_check = pci_msi_domain_check_cap
>
> We know at build-time what all the function pointers will be, so in
> principle we should be able to make the struct const, which would be
> even better than __ro_after_init.
>
> For example, we could require that callers set every function pointer
> before calling pci_msi_create_irq_domain(), using the default ones
> (pci_msi_domain_set_desc, pci_msi_domain_check_cap,
> pci_msi_domain_handle_error) if it doesn't need to override them,
> e.g.,
>
>   static struct msi_domain_ops vmd_msi_domain_ops = {
>     .get_hwirq = vmd_get_hwirq,
>     .msi_check = pci_msi_domain_check_cap,
>   };
>
> Or we could leave NULL pointers in the structure and have the code
> that calls through the function pointers check for NULL and call the
> default itself, e.g.,
>
>   if (ops->msi_check)
>     ops->msi_check(...)
>   else
>     pci_msi_domain_check_cap(...)
>
> It looks like the "USE_DEF_OPS" framework was added by Jiang Liu with
> the commits below.  I would CC: him for his thoughts, but I don't
> have a current email address.
>
>   aeeb59657c35 ("genirq: Provide default callbacks for msi_domain_ops")
>   3878eaefb89a ("PCI/MSI: Enhance core to support hierarchy irqdomain")

If we can do const, that would be preferred. That's generally easier
to reason about. I ended up doing this to the cdrom ops structure just
the other day:

http://www.openwall.com/lists/kernel-hardening/2017/02/14/2

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Thomas Gleixner]

On Wed, 15 Feb 2017, Bjorn Helgaas wrote:
> We know at build-time what all the function pointers will be, so in
> principle we should be able to make the struct const, which would be
> even better than __ro_after_init.

Not everywhere unfortunately. In some instances it's a runtime decision, but
yes, they could be fixed. But there is a downside in doing this. See below.

> For example, we could require that callers set every function pointer
> before calling pci_msi_create_irq_domain(), using the default ones
> (pci_msi_domain_set_desc, pci_msi_domain_check_cap,
> pci_msi_domain_handle_error) if it doesn't need to override them,
> e.g.,
> 
>   static struct msi_domain_ops vmd_msi_domain_ops = {
>     .get_hwirq = vmd_get_hwirq,
>     .msi_check = pci_msi_domain_check_cap,
>   };
> 
> Or we could leave NULL pointers in the structure and have the code
> that calls through the function pointers check for NULL and call the
> default itself, e.g.,
> 
>   if (ops->msi_check)
>     ops->msi_check(...)
>   else
>     pci_msi_domain_check_cap(...)
> 
> It looks like the "USE_DEF_OPS" framework was added by Jiang Liu with
> the commits below.  I would CC: him for his thoughts, but I don't
> have a current email address.

Me neither :(

I think I suggested to Jiang to do that 'update with default functions' to

- avoid exporting the world and some more

- have the flexibility to add new functions to the ops w/o updating a
  gazillion of existing usage sites, which has saved us lots of chaising in
  the last years

- avoid the if (ops->ptr) ops->ptr(); else default_fn(); constructs all
  over the place.

I admit I did not think about the fact that this makes the structs non
const.

Mopping that up by exporting the default functions and setting all the
function pointers is tedious and requires a full tree sweep when we add new
stuff. There's also code shared between PCI/platform/DT based stuff, so
that becomes interesting.

Doing the if (ops->ptr) ops->ptr() else default_fn(); dance should be
simpler to pull off. There are not that many sites to look at, but then we
have some of the GICv3 code using the domain ops out of core.

For now doing the __ro_after_init is definitely the simplest and fastest
solution to tighten these statically allocated structures.

I have a look with Marc, what can be done in the long run.

Thanks,

	tglx

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Bjorn Helgaas]

On Wed, Feb 15, 2017 at 10:16:32PM +0100, Thomas Gleixner wrote:

> I think I suggested to Jiang to do that 'update with default functions' to
> 
> - avoid exporting the world and some more
> 
> - have the flexibility to add new functions to the ops w/o updating a
>   gazillion of existing usage sites, which has saved us lots of chaising in
>   the last years
> 
> - avoid the if (ops->ptr) ops->ptr(); else default_fn(); constructs all
>   over the place.
> 
> I admit I did not think about the fact that this makes the structs non
> const.
> 
> Mopping that up by exporting the default functions and setting all the
> function pointers is tedious and requires a full tree sweep when we add new
> stuff. There's also code shared between PCI/platform/DT based stuff, so
> that becomes interesting.

It's legal to initialize a field multiple times, and the last one
takes precedence, so doing this might at least avoid the full tree
sweeps:

  static struct msi_domain_ops vmd_msi_domain_ops = {
    MSI_DOMAIN_DEFAULT_OPS,
    .get_hwirq = vmd_get_hwirq,
  };

The functions referenced by MSI_DOMAIN_DEFAULT_OPS would still have to
be exported, though.

> Doing the if (ops->ptr) ops->ptr() else default_fn(); dance should be
> simpler to pull off. There are not that many sites to look at, but then we
> have some of the GICv3 code using the domain ops out of core.
> 
> For now doing the __ro_after_init is definitely the simplest and fastest
> solution to tighten these statically allocated structures.

I'm OK with __ro_after_init, at least as an interim solution.

I do think it would be good to audit all the uses of
MSI_FLAG_USE_DEF_DOM_OPS and MSI_FLAG_USE_DEF_CHIP_OPS, since they
seem to be the primary indicator of when the struct might be modified.
I suspect we could add __ro_after_init to more than just pci-hyperv.c,
vmd.c, and msi.c

Bjorn

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Thomas Gleixner]

On Thu, 16 Feb 2017, Bjorn Helgaas wrote:
> On Wed, Feb 15, 2017 at 10:16:32PM +0100, Thomas Gleixner wrote:
> 
> > I think I suggested to Jiang to do that 'update with default functions' to
> > 
> > - avoid exporting the world and some more
> > 
> > - have the flexibility to add new functions to the ops w/o updating a
> >   gazillion of existing usage sites, which has saved us lots of chaising in
> >   the last years
> > 
> > - avoid the if (ops->ptr) ops->ptr(); else default_fn(); constructs all
> >   over the place.
> > 
> > I admit I did not think about the fact that this makes the structs non
> > const.
> > 
> > Mopping that up by exporting the default functions and setting all the
> > function pointers is tedious and requires a full tree sweep when we add new
> > stuff. There's also code shared between PCI/platform/DT based stuff, so
> > that becomes interesting.
> 
> It's legal to initialize a field multiple times, and the last one
> takes precedence, so doing this might at least avoid the full tree
> sweeps:
> 
>   static struct msi_domain_ops vmd_msi_domain_ops = {
>     MSI_DOMAIN_DEFAULT_OPS,
>     .get_hwirq = vmd_get_hwirq,
>   };
> 
> The functions referenced by MSI_DOMAIN_DEFAULT_OPS would still have to
> be exported, though.

Hmm, that'd work. Though it will fall apart for those pieces where we share
code across backends. But I did not yet go through all the places and check
them.

> > Doing the if (ops->ptr) ops->ptr() else default_fn(); dance should be
> > simpler to pull off. There are not that many sites to look at, but then we
> > have some of the GICv3 code using the domain ops out of core.
> > 
> > For now doing the __ro_after_init is definitely the simplest and fastest
> > solution to tighten these statically allocated structures.
> 
> I'm OK with __ro_after_init, at least as an interim solution.
> 
> I do think it would be good to audit all the uses of
> MSI_FLAG_USE_DEF_DOM_OPS and MSI_FLAG_USE_DEF_CHIP_OPS, since they
> seem to be the primary indicator of when the struct might be modified.
> I suspect we could add __ro_after_init to more than just pci-hyperv.c,
> vmd.c, and msi.c

Agreed. I have it on my radar.

Thanks,

	tglx

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Bjorn Helgaas]

On Thu, Feb 16, 2017 at 03:38:05PM +0100, Thomas Gleixner wrote:
> On Thu, 16 Feb 2017, Bjorn Helgaas wrote:
> > On Wed, Feb 15, 2017 at 10:16:32PM +0100, Thomas Gleixner wrote:
> > 
> > > I think I suggested to Jiang to do that 'update with default functions' to
> > > 
> > > - avoid exporting the world and some more
> > > 
> > > - have the flexibility to add new functions to the ops w/o updating a
> > >   gazillion of existing usage sites, which has saved us lots of chaising in
> > >   the last years
> > > 
> > > - avoid the if (ops->ptr) ops->ptr(); else default_fn(); constructs all
> > >   over the place.
> > > 
> > > I admit I did not think about the fact that this makes the structs non
> > > const.
> > > 
> > > Mopping that up by exporting the default functions and setting all the
> > > function pointers is tedious and requires a full tree sweep when we add new
> > > stuff. There's also code shared between PCI/platform/DT based stuff, so
> > > that becomes interesting.
> > 
> > It's legal to initialize a field multiple times, and the last one
> > takes precedence, so doing this might at least avoid the full tree
> > sweeps:
> > 
> >   static struct msi_domain_ops vmd_msi_domain_ops = {
> >     MSI_DOMAIN_DEFAULT_OPS,
> >     .get_hwirq = vmd_get_hwirq,
> >   };
> > 
> > The functions referenced by MSI_DOMAIN_DEFAULT_OPS would still have to
> > be exported, though.
> 
> Hmm, that'd work. Though it will fall apart for those pieces where we share
> code across backends. But I did not yet go through all the places and check
> them.
> 
> > > Doing the if (ops->ptr) ops->ptr() else default_fn(); dance should be
> > > simpler to pull off. There are not that many sites to look at, but then we
> > > have some of the GICv3 code using the domain ops out of core.
> > > 
> > > For now doing the __ro_after_init is definitely the simplest and fastest
> > > solution to tighten these statically allocated structures.
> > 
> > I'm OK with __ro_after_init, at least as an interim solution.
> > 
> > I do think it would be good to audit all the uses of
> > MSI_FLAG_USE_DEF_DOM_OPS and MSI_FLAG_USE_DEF_CHIP_OPS, since they
> > seem to be the primary indicator of when the struct might be modified.
> > I suspect we could add __ro_after_init to more than just pci-hyperv.c,
> > vmd.c, and msi.c
> 
> Agreed. I have it on my radar.

This seems like a worthwhile change, so I don't want to just drop this
patch.  But if we're going to do something, I'd like to do it
everywhere that it makes sense, all at the same time.

It looks like the v2 series was split up by subsystem, which is fine
with me.  I'll happily apply the PCI parts (or ack them, since it
might make sense to apply all of them via the same non-PCI tree).

But I *would* like to include the following users of
MSI_FLAG_USE_DEF_DOM_OPS and MSI_FLAG_USE_DEF_CHIP_OPS at the same
time (or explain why __ro_after_init won't work for them):

  pci-xgene-msi.c
  pcie-altera-msi.c
  pcie-iproc-msi.c
  pcie-xilinx-nwl.c

Bjorn

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Jessica Frazelle]

I can update the patch series, sorry haven't had much time to devote
to this the past few weeks, but will update in the next day.

On Tue, Mar 7, 2017 at 11:07 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Thu, Feb 16, 2017 at 03:38:05PM +0100, Thomas Gleixner wrote:
>> On Thu, 16 Feb 2017, Bjorn Helgaas wrote:
>> > On Wed, Feb 15, 2017 at 10:16:32PM +0100, Thomas Gleixner wrote:
>> >
>> > > I think I suggested to Jiang to do that 'update with default functions' to
>> > >
>> > > - avoid exporting the world and some more
>> > >
>> > > - have the flexibility to add new functions to the ops w/o updating a
>> > >   gazillion of existing usage sites, which has saved us lots of chaising in
>> > >   the last years
>> > >
>> > > - avoid the if (ops->ptr) ops->ptr(); else default_fn(); constructs all
>> > >   over the place.
>> > >
>> > > I admit I did not think about the fact that this makes the structs non
>> > > const.
>> > >
>> > > Mopping that up by exporting the default functions and setting all the
>> > > function pointers is tedious and requires a full tree sweep when we add new
>> > > stuff. There's also code shared between PCI/platform/DT based stuff, so
>> > > that becomes interesting.
>> >
>> > It's legal to initialize a field multiple times, and the last one
>> > takes precedence, so doing this might at least avoid the full tree
>> > sweeps:
>> >
>> >   static struct msi_domain_ops vmd_msi_domain_ops = {
>> >     MSI_DOMAIN_DEFAULT_OPS,
>> >     .get_hwirq = vmd_get_hwirq,
>> >   };
>> >
>> > The functions referenced by MSI_DOMAIN_DEFAULT_OPS would still have to
>> > be exported, though.
>>
>> Hmm, that'd work. Though it will fall apart for those pieces where we share
>> code across backends. But I did not yet go through all the places and check
>> them.
>>
>> > > Doing the if (ops->ptr) ops->ptr() else default_fn(); dance should be
>> > > simpler to pull off. There are not that many sites to look at, but then we
>> > > have some of the GICv3 code using the domain ops out of core.
>> > >
>> > > For now doing the __ro_after_init is definitely the simplest and fastest
>> > > solution to tighten these statically allocated structures.
>> >
>> > I'm OK with __ro_after_init, at least as an interim solution.
>> >
>> > I do think it would be good to audit all the uses of
>> > MSI_FLAG_USE_DEF_DOM_OPS and MSI_FLAG_USE_DEF_CHIP_OPS, since they
>> > seem to be the primary indicator of when the struct might be modified.
>> > I suspect we could add __ro_after_init to more than just pci-hyperv.c,
>> > vmd.c, and msi.c
>>
>> Agreed. I have it on my radar.
>
> This seems like a worthwhile change, so I don't want to just drop this
> patch.  But if we're going to do something, I'd like to do it
> everywhere that it makes sense, all at the same time.
>
> It looks like the v2 series was split up by subsystem, which is fine
> with me.  I'll happily apply the PCI parts (or ack them, since it
> might make sense to apply all of them via the same non-PCI tree).
>
> But I *would* like to include the following users of
> MSI_FLAG_USE_DEF_DOM_OPS and MSI_FLAG_USE_DEF_CHIP_OPS at the same
> time (or explain why __ro_after_init won't work for them):
>
>   pci-xgene-msi.c
>   pcie-altera-msi.c
>   pcie-iproc-msi.c
>   pcie-xilinx-nwl.c
>
> Bjorn



-- 


Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC  511E 18F3 685C 0022 BFF3
pgp.mit.edu

Re: [PATCH v2 3/5] pci: set msi_domain_ops as __ro_after_init

[Bjorn Helgaas]

On Tue, Mar 14, 2017 at 11:50:50AM -0700, Jessica Frazelle wrote:
> I can update the patch series, sorry haven't had much time to devote
> to this the past few weeks, but will update in the next day.

Thanks, Jessica!  No problem, I know the feeling :)

Bjorn