[PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Alex Williamson]

The IRQ resource for a device is established when pci_enabled_device()
is called on a fully disabled device (ie. enable_cnt == 0).  With
commit b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ
resources") this same IRQ resource is released when the driver is
unbound from the device, regardless of the enable_cnt.  This presents
the situation that an ill-behaved driver can now make a device
unusable to subsequent drivers by an imbalance in their use of
pci_enable/disable_device().  It's one thing to break your own device
if you're one of these ill-behaved drivers, but it's a serious
regression for secondary drivers like vfio-pci, which are innocent
of the transgressions of the previous driver.

Resolve by pushing the device to a fully disabled state before
releasing the IRQ resource.

Fixes: b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ resources")
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Cc: Jiang Liu <jiang.liu@linux.intel.com>
---
 arch/x86/pci/common.c |   13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
index 3d2612b..4810194 100644
--- a/arch/x86/pci/common.c
+++ b/arch/x86/pci/common.c
@@ -527,8 +527,19 @@ static int pci_irq_notifier(struct notifier_block *nb, unsigned long action,
 	if (action != BUS_NOTIFY_UNBOUND_DRIVER)
 		return NOTIFY_DONE;
 
-	if (pcibios_disable_irq)
+	if (pcibios_disable_irq) {
+		/*
+		 * Broken drivers may allow a device to be .remove()'d while
+		 * still enabled.  pci_enable_device() will only re-establish
+		 * dev->irq if the devices is fully disabled.  So if we want
+		 * to release the IRQ, we need to make sure the next driver
+		 * can re-establish it using pci_enable_device().
+		 */
+		while (pci_is_enabled(dev))
+			pci_disable_device(dev);
+
 		pcibios_disable_irq(dev);
+	}
 
 	return NOTIFY_OK;
 }

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Jiang Liu]

On 2015/3/6 5:06, Alex Williamson wrote:
> The IRQ resource for a device is established when pci_enabled_device()
> is called on a fully disabled device (ie. enable_cnt == 0).  With
> commit b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ
> resources") this same IRQ resource is released when the driver is
> unbound from the device, regardless of the enable_cnt.  This presents
> the situation that an ill-behaved driver can now make a device
> unusable to subsequent drivers by an imbalance in their use of
> pci_enable/disable_device().  It's one thing to break your own device
> if you're one of these ill-behaved drivers, but it's a serious
> regression for secondary drivers like vfio-pci, which are innocent
> of the transgressions of the previous driver.
> 
> Resolve by pushing the device to a fully disabled state before
> releasing the IRQ resource.
> 
> Fixes: b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ resources")
> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> ---
>  arch/x86/pci/common.c |   13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> index 3d2612b..4810194 100644
> --- a/arch/x86/pci/common.c
> +++ b/arch/x86/pci/common.c
> @@ -527,8 +527,19 @@ static int pci_irq_notifier(struct notifier_block *nb, unsigned long action,
>  	if (action != BUS_NOTIFY_UNBOUND_DRIVER)
>  		return NOTIFY_DONE;
>  
> -	if (pcibios_disable_irq)
> +	if (pcibios_disable_irq) {
> +		/*
> +		 * Broken drivers may allow a device to be .remove()'d while
> +		 * still enabled.  pci_enable_device() will only re-establish
> +		 * dev->irq if the devices is fully disabled.  So if we want
> +		 * to release the IRQ, we need to make sure the next driver
> +		 * can re-establish it using pci_enable_device().
> +		 */
> +		while (pci_is_enabled(dev))
> +			pci_disable_device(dev);
> +
>  		pcibios_disable_irq(dev);
> +	}
Hi Alex,
	Thanks for debugging and fixing it.
	Will it be feasible to give a debug message to remind those
driver authors to correctly disable PCI when unbinding?
Regards!
Gerry

>  
>  	return NOTIFY_OK;
>  }
> 

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Alex Williamson]

On Fri, 2015-03-06 at 09:49 +0800, Jiang Liu wrote:
> On 2015/3/6 5:06, Alex Williamson wrote:
> > The IRQ resource for a device is established when pci_enabled_device()
> > is called on a fully disabled device (ie. enable_cnt == 0).  With
> > commit b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ
> > resources") this same IRQ resource is released when the driver is
> > unbound from the device, regardless of the enable_cnt.  This presents
> > the situation that an ill-behaved driver can now make a device
> > unusable to subsequent drivers by an imbalance in their use of
> > pci_enable/disable_device().  It's one thing to break your own device
> > if you're one of these ill-behaved drivers, but it's a serious
> > regression for secondary drivers like vfio-pci, which are innocent
> > of the transgressions of the previous driver.
> > 
> > Resolve by pushing the device to a fully disabled state before
> > releasing the IRQ resource.
> > 
> > Fixes: b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ resources")
> > Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> > Cc: Jiang Liu <jiang.liu@linux.intel.com>
> > ---
> >  arch/x86/pci/common.c |   13 ++++++++++++-
> >  1 file changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> > index 3d2612b..4810194 100644
> > --- a/arch/x86/pci/common.c
> > +++ b/arch/x86/pci/common.c
> > @@ -527,8 +527,19 @@ static int pci_irq_notifier(struct notifier_block *nb, unsigned long action,
> >  	if (action != BUS_NOTIFY_UNBOUND_DRIVER)
> >  		return NOTIFY_DONE;
> >  
> > -	if (pcibios_disable_irq)
> > +	if (pcibios_disable_irq) {
> > +		/*
> > +		 * Broken drivers may allow a device to be .remove()'d while
> > +		 * still enabled.  pci_enable_device() will only re-establish
> > +		 * dev->irq if the devices is fully disabled.  So if we want
> > +		 * to release the IRQ, we need to make sure the next driver
> > +		 * can re-establish it using pci_enable_device().
> > +		 */
> > +		while (pci_is_enabled(dev))
> > +			pci_disable_device(dev);
> > +
> >  		pcibios_disable_irq(dev);
> > +	}
> Hi Alex,
> 	Thanks for debugging and fixing it.
> 	Will it be feasible to give a debug message to remind those
> driver authors to correctly disable PCI when unbinding?

I can certainly add a warning to the loop, it loses a bit of its teeth
here though since we can't specify which driver to blame at this point.
Maybe that warning and perhaps this enabling roll-back should happen in
drivers/pci/pci-driver.c:pci_device_remove().  Bjorn, would you prefer
it be done generically there?  Thanks,

Alex

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Alex Williamson]

On Thu, 2015-03-05 at 20:51 -0700, Alex Williamson wrote:
> On Fri, 2015-03-06 at 09:49 +0800, Jiang Liu wrote:
> > On 2015/3/6 5:06, Alex Williamson wrote:
> > > The IRQ resource for a device is established when pci_enabled_device()
> > > is called on a fully disabled device (ie. enable_cnt == 0).  With
> > > commit b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ
> > > resources") this same IRQ resource is released when the driver is
> > > unbound from the device, regardless of the enable_cnt.  This presents
> > > the situation that an ill-behaved driver can now make a device
> > > unusable to subsequent drivers by an imbalance in their use of
> > > pci_enable/disable_device().  It's one thing to break your own device
> > > if you're one of these ill-behaved drivers, but it's a serious
> > > regression for secondary drivers like vfio-pci, which are innocent
> > > of the transgressions of the previous driver.
> > > 
> > > Resolve by pushing the device to a fully disabled state before
> > > releasing the IRQ resource.
> > > 
> > > Fixes: b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ resources")
> > > Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> > > Cc: Jiang Liu <jiang.liu@linux.intel.com>
> > > ---
> > >  arch/x86/pci/common.c |   13 ++++++++++++-
> > >  1 file changed, 12 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> > > index 3d2612b..4810194 100644
> > > --- a/arch/x86/pci/common.c
> > > +++ b/arch/x86/pci/common.c
> > > @@ -527,8 +527,19 @@ static int pci_irq_notifier(struct notifier_block *nb, unsigned long action,
> > >  	if (action != BUS_NOTIFY_UNBOUND_DRIVER)
> > >  		return NOTIFY_DONE;
> > >  
> > > -	if (pcibios_disable_irq)
> > > +	if (pcibios_disable_irq) {
> > > +		/*
> > > +		 * Broken drivers may allow a device to be .remove()'d while
> > > +		 * still enabled.  pci_enable_device() will only re-establish
> > > +		 * dev->irq if the devices is fully disabled.  So if we want
> > > +		 * to release the IRQ, we need to make sure the next driver
> > > +		 * can re-establish it using pci_enable_device().
> > > +		 */
> > > +		while (pci_is_enabled(dev))
> > > +			pci_disable_device(dev);
> > > +
> > >  		pcibios_disable_irq(dev);
> > > +	}
> > Hi Alex,
> > 	Thanks for debugging and fixing it.
> > 	Will it be feasible to give a debug message to remind those
> > driver authors to correctly disable PCI when unbinding?
> 
> I can certainly add a warning to the loop, it loses a bit of its teeth
> here though since we can't specify which driver to blame at this point.
> Maybe that warning and perhaps this enabling roll-back should happen in
> drivers/pci/pci-driver.c:pci_device_remove().  Bjorn, would you prefer
> it be done generically there?  Thanks,

Unfortunately there's a long standing comment in pci_device_remove():

        /*
         * We would love to complain here if pci_dev->is_enabled is set, that
         * the driver should have called pci_disable_device(), but the
         * unfortunate fact is there are too many odd BIOS and bridge setups
         * that don't like drivers doing that all of the time.
         * Oh well, we can dream of sane hardware when we sleep, no matter how
         * horrible the crap we have to deal with is when we are awake...
         */

So, unless we can somehow ignore that comment, I suspect forcing the
device to be disabled on driver remove, whether done from pci-core or
from x86/pci, is going to cause all sorts of breakage.  Are the
expectations set by b4b55cda5874 really valid?  It seems like something
needs to be done to allow the IRQ to be automatically re-established on
x86 regardless of the driver doing the right thing when releasing the
device.  We're still looking at a regression for v4.0 as a result of
b4b55cda5874.  Thanks,

Alex

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Rafael J. Wysocki]

On Wednesday, March 11, 2015 10:47:30 AM Alex Williamson wrote:
> On Thu, 2015-03-05 at 20:51 -0700, Alex Williamson wrote:
> > On Fri, 2015-03-06 at 09:49 +0800, Jiang Liu wrote:
> > > On 2015/3/6 5:06, Alex Williamson wrote:
> > > > The IRQ resource for a device is established when pci_enabled_device()
> > > > is called on a fully disabled device (ie. enable_cnt == 0).  With
> > > > commit b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ
> > > > resources") this same IRQ resource is released when the driver is
> > > > unbound from the device, regardless of the enable_cnt.  This presents
> > > > the situation that an ill-behaved driver can now make a device
> > > > unusable to subsequent drivers by an imbalance in their use of
> > > > pci_enable/disable_device().  It's one thing to break your own device
> > > > if you're one of these ill-behaved drivers, but it's a serious
> > > > regression for secondary drivers like vfio-pci, which are innocent
> > > > of the transgressions of the previous driver.
> > > > 
> > > > Resolve by pushing the device to a fully disabled state before
> > > > releasing the IRQ resource.
> > > > 
> > > > Fixes: b4b55cda5874 ("x86/PCI: Refine the way to release PCI IRQ resources")
> > > > Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> > > > Cc: Jiang Liu <jiang.liu@linux.intel.com>
> > > > ---
> > > >  arch/x86/pci/common.c |   13 ++++++++++++-
> > > >  1 file changed, 12 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> > > > index 3d2612b..4810194 100644
> > > > --- a/arch/x86/pci/common.c
> > > > +++ b/arch/x86/pci/common.c
> > > > @@ -527,8 +527,19 @@ static int pci_irq_notifier(struct notifier_block *nb, unsigned long action,
> > > >  	if (action != BUS_NOTIFY_UNBOUND_DRIVER)
> > > >  		return NOTIFY_DONE;
> > > >  
> > > > -	if (pcibios_disable_irq)
> > > > +	if (pcibios_disable_irq) {
> > > > +		/*
> > > > +		 * Broken drivers may allow a device to be .remove()'d while
> > > > +		 * still enabled.  pci_enable_device() will only re-establish
> > > > +		 * dev->irq if the devices is fully disabled.  So if we want
> > > > +		 * to release the IRQ, we need to make sure the next driver
> > > > +		 * can re-establish it using pci_enable_device().
> > > > +		 */
> > > > +		while (pci_is_enabled(dev))
> > > > +			pci_disable_device(dev);
> > > > +
> > > >  		pcibios_disable_irq(dev);
> > > > +	}
> > > Hi Alex,
> > > 	Thanks for debugging and fixing it.
> > > 	Will it be feasible to give a debug message to remind those
> > > driver authors to correctly disable PCI when unbinding?
> > 
> > I can certainly add a warning to the loop, it loses a bit of its teeth
> > here though since we can't specify which driver to blame at this point.
> > Maybe that warning and perhaps this enabling roll-back should happen in
> > drivers/pci/pci-driver.c:pci_device_remove().  Bjorn, would you prefer
> > it be done generically there?  Thanks,
> 
> Unfortunately there's a long standing comment in pci_device_remove():
> 
>         /*
>          * We would love to complain here if pci_dev->is_enabled is set, that
>          * the driver should have called pci_disable_device(), but the
>          * unfortunate fact is there are too many odd BIOS and bridge setups
>          * that don't like drivers doing that all of the time.
>          * Oh well, we can dream of sane hardware when we sleep, no matter how
>          * horrible the crap we have to deal with is when we are awake...
>          */
> 
> So, unless we can somehow ignore that comment, I suspect forcing the
> device to be disabled on driver remove, whether done from pci-core or
> from x86/pci, is going to cause all sorts of breakage.  Are the
> expectations set by b4b55cda5874 really valid?  It seems like something
> needs to be done to allow the IRQ to be automatically re-established on
> x86 regardless of the driver doing the right thing when releasing the
> device.  We're still looking at a regression for v4.0 as a result of
> b4b55cda5874.

In which case we probably should revert commit b4b55cda5874 for the time being.

At least I'd be very nervous about any ad-hoc fixes at this stage of the cycle.

Gerry?


-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

RE: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Luck, Tony]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1647 bytes --]

>> Unfortunately there's a long standing comment in pci_device_remove():
>> 
>>         /*
>>          * We would love to complain here if pci_dev->is_enabled is set, that
>>          * the driver should have called pci_disable_device(), but the
>>          * unfortunate fact is there are too many odd BIOS and bridge setups
>>          * that don't like drivers doing that all of the time.
>>          * Oh well, we can dream of sane hardware when we sleep, no matter how
>>          * horrible the crap we have to deal with is when we are awake...
>>          */
>> 
>> So, unless we can somehow ignore that comment, I suspect forcing the
>> device to be disabled on driver remove, whether done from pci-core or
>> from x86/pci, is going to cause all sorts of breakage.  Are the
>> expectations set by b4b55cda5874 really valid?  It seems like something
>> needs to be done to allow the IRQ to be automatically re-established on
>> x86 regardless of the driver doing the right thing when releasing the
>> device.  We're still looking at a regression for v4.0 as a result of
>> b4b55cda5874.
>
> In which case we probably should revert commit b4b55cda5874 for the time being.
>
> At least I'd be very nervous about any ad-hoc fixes at this stage of the cycle.

The comment goes back to the dawn of "git" time ... not sure how much further
back.

Is this actually still an issue on modern systems?  Maybe we need a black list
or white list to separate the good from bad systems?

-Tony
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Rafael J. Wysocki]

On Wednesday, March 11, 2015 10:04:42 PM Luck, Tony wrote:
> >> Unfortunately there's a long standing comment in pci_device_remove():
> >> 
> >>         /*
> >>          * We would love to complain here if pci_dev->is_enabled is set, that
> >>          * the driver should have called pci_disable_device(), but the
> >>          * unfortunate fact is there are too many odd BIOS and bridge setups
> >>          * that don't like drivers doing that all of the time.
> >>          * Oh well, we can dream of sane hardware when we sleep, no matter how
> >>          * horrible the crap we have to deal with is when we are awake...
> >>          */
> >> 
> >> So, unless we can somehow ignore that comment, I suspect forcing the
> >> device to be disabled on driver remove, whether done from pci-core or
> >> from x86/pci, is going to cause all sorts of breakage.  Are the
> >> expectations set by b4b55cda5874 really valid?  It seems like something
> >> needs to be done to allow the IRQ to be automatically re-established on
> >> x86 regardless of the driver doing the right thing when releasing the
> >> device.  We're still looking at a regression for v4.0 as a result of
> >> b4b55cda5874.
> >
> > In which case we probably should revert commit b4b55cda5874 for the time being.
> >
> > At least I'd be very nervous about any ad-hoc fixes at this stage of the cycle.
> 
> The comment goes back to the dawn of "git" time ... not sure how much further
> back.
> 
> Is this actually still an issue on modern systems?  Maybe we need a black list
> or white list to separate the good from bad systems?

The answer to that is "We don't know" and in my not so humble opinion it is too
risky to try to find out at the end of the cycle.


-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Jiang Liu]

On 2015/3/12 9:17, Rafael J. Wysocki wrote:
> On Wednesday, March 11, 2015 10:04:42 PM Luck, Tony wrote:
>>>> Unfortunately there's a long standing comment in pci_device_remove():
>>>>
>>>>         /*
>>>>          * We would love to complain here if pci_dev->is_enabled is set, that
>>>>          * the driver should have called pci_disable_device(), but the
>>>>          * unfortunate fact is there are too many odd BIOS and bridge setups
>>>>          * that don't like drivers doing that all of the time.
>>>>          * Oh well, we can dream of sane hardware when we sleep, no matter how
>>>>          * horrible the crap we have to deal with is when we are awake...
>>>>          */
>>>>
>>>> So, unless we can somehow ignore that comment, I suspect forcing the
>>>> device to be disabled on driver remove, whether done from pci-core or
>>>> from x86/pci, is going to cause all sorts of breakage.  Are the
>>>> expectations set by b4b55cda5874 really valid?  It seems like something
>>>> needs to be done to allow the IRQ to be automatically re-established on
>>>> x86 regardless of the driver doing the right thing when releasing the
>>>> device.  We're still looking at a regression for v4.0 as a result of
>>>> b4b55cda5874.
>>>
>>> In which case we probably should revert commit b4b55cda5874 for the time being.
>>>
>>> At least I'd be very nervous about any ad-hoc fixes at this stage of the cycle.
>>
>> The comment goes back to the dawn of "git" time ... not sure how much further
>> back.
>>
>> Is this actually still an issue on modern systems?  Maybe we need a black list
>> or white list to separate the good from bad systems?
> 
> The answer to that is "We don't know" and in my not so humble opinion it is too
> risky to try to find out at the end of the cycle.
Hi Rafael and Alex,
How about a patch which:
1) gives a warning if PCI device is still enabled when unloading driver
2) release PCI interrupt only if PCI device is disabled.
By this way, we could support IOAPIC hot-removal on latest platforms and
avoid regressions on old platforms.
Thanks!
Gerry

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Rafael J. Wysocki]

On Thursday, March 12, 2015 09:41:21 AM Jiang Liu wrote:
> On 2015/3/12 9:17, Rafael J. Wysocki wrote:
> > On Wednesday, March 11, 2015 10:04:42 PM Luck, Tony wrote:
> >>>> Unfortunately there's a long standing comment in pci_device_remove():
> >>>>
> >>>>         /*
> >>>>          * We would love to complain here if pci_dev->is_enabled is set, that
> >>>>          * the driver should have called pci_disable_device(), but the
> >>>>          * unfortunate fact is there are too many odd BIOS and bridge setups
> >>>>          * that don't like drivers doing that all of the time.
> >>>>          * Oh well, we can dream of sane hardware when we sleep, no matter how
> >>>>          * horrible the crap we have to deal with is when we are awake...
> >>>>          */
> >>>>
> >>>> So, unless we can somehow ignore that comment, I suspect forcing the
> >>>> device to be disabled on driver remove, whether done from pci-core or
> >>>> from x86/pci, is going to cause all sorts of breakage.  Are the
> >>>> expectations set by b4b55cda5874 really valid?  It seems like something
> >>>> needs to be done to allow the IRQ to be automatically re-established on
> >>>> x86 regardless of the driver doing the right thing when releasing the
> >>>> device.  We're still looking at a regression for v4.0 as a result of
> >>>> b4b55cda5874.
> >>>
> >>> In which case we probably should revert commit b4b55cda5874 for the time being.
> >>>
> >>> At least I'd be very nervous about any ad-hoc fixes at this stage of the cycle.
> >>
> >> The comment goes back to the dawn of "git" time ... not sure how much further
> >> back.
> >>
> >> Is this actually still an issue on modern systems?  Maybe we need a black list
> >> or white list to separate the good from bad systems?
> > 
> > The answer to that is "We don't know" and in my not so humble opinion it is too
> > risky to try to find out at the end of the cycle.
> Hi Rafael and Alex,
> How about a patch which:
> 1) gives a warning if PCI device is still enabled when unloading driver

That may become sort of noisy.  I really would prefer to introduce things like
that by the beginning of the cycle, not by the end of it.

> 2) release PCI interrupt only if PCI device is disabled.
> By this way, we could support IOAPIC hot-removal on latest platforms and
> avoid regressions on old platforms.

Well, please submit a patch for discussion.

I would like to know Bjorn's opinion about that too at least.

Rafael

Re: [PATCH] x86/PCI: Fully disable devices before releasing IRQ resource

[Jiang Liu]

On 2015/3/13 0:08, Rafael J. Wysocki wrote:
> On Thursday, March 12, 2015 09:41:21 AM Jiang Liu wrote:
>> On 2015/3/12 9:17, Rafael J. Wysocki wrote:
>>> On Wednesday, March 11, 2015 10:04:42 PM Luck, Tony wrote:
>>>>>> Unfortunately there's a long standing comment in pci_device_remove():
>>>>>>
>>>>>>         /*
>>>>>>          * We would love to complain here if pci_dev->is_enabled is set, that
>>>>>>          * the driver should have called pci_disable_device(), but the
>>>>>>          * unfortunate fact is there are too many odd BIOS and bridge setups
>>>>>>          * that don't like drivers doing that all of the time.
>>>>>>          * Oh well, we can dream of sane hardware when we sleep, no matter how
>>>>>>          * horrible the crap we have to deal with is when we are awake...
>>>>>>          */
>>>>>>
>>>>>> So, unless we can somehow ignore that comment, I suspect forcing the
>>>>>> device to be disabled on driver remove, whether done from pci-core or
>>>>>> from x86/pci, is going to cause all sorts of breakage.  Are the
>>>>>> expectations set by b4b55cda5874 really valid?  It seems like something
>>>>>> needs to be done to allow the IRQ to be automatically re-established on
>>>>>> x86 regardless of the driver doing the right thing when releasing the
>>>>>> device.  We're still looking at a regression for v4.0 as a result of
>>>>>> b4b55cda5874.
>>>>>
>>>>> In which case we probably should revert commit b4b55cda5874 for the time being.
>>>>>
>>>>> At least I'd be very nervous about any ad-hoc fixes at this stage of the cycle.
>>>>
>>>> The comment goes back to the dawn of "git" time ... not sure how much further
>>>> back.
>>>>
>>>> Is this actually still an issue on modern systems?  Maybe we need a black list
>>>> or white list to separate the good from bad systems?
>>>
>>> The answer to that is "We don't know" and in my not so humble opinion it is too
>>> risky to try to find out at the end of the cycle.
>> Hi Rafael and Alex,
>> How about a patch which:
>> 1) gives a warning if PCI device is still enabled when unloading driver
> 
> That may become sort of noisy.  I really would prefer to introduce things like
> that by the beginning of the cycle, not by the end of it.
Will try this on next merging window.

>> 2) release PCI interrupt only if PCI device is disabled.
>> By this way, we could support IOAPIC hot-removal on latest platforms and
>> avoid regressions on old platforms.
> 
> Well, please submit a patch for discussion.
> 
> I would like to know Bjorn's opinion about that too at least.
Still testing the patch, will send it out soon.

> 
> Rafael
> 

[Bugfix] x86/PCI: Release PCI IRQ resource only if PCI device is disabled when unbinding

[Jiang Liu]

To support IOAPIC hot-removal, we need to release PCI interrupt resource
when unbinding PCI device driver. But due to historical reason,
/*
 * We would love to complain here if pci_dev->is_enabled is set, that
 * the driver should have called pci_disable_device(), but the
 * unfortunate fact is there are too many odd BIOS and bridge setups
 * that don't like drivers doing that all of the time.
 * Oh well, we can dream of sane hardware when we sleep, no matter how
 * horrible the crap we have to deal with is when we are awake...
 */
some drivers don't call pci_disable_device() when unloading, which
prevents us from reallocating PCI interrupt resource on reloading
PCI driver and causes regressions.

So release PCI interrupt resource only if PCI device is disabled when
unbinding. By this way, we could support IOAPIC hot-removal on latest
platforms and avoid regressions on old platforms.

Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
---
 arch/x86/pci/common.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
index 3d2612b68694..8d792142cb2a 100644
--- a/arch/x86/pci/common.c
+++ b/arch/x86/pci/common.c
@@ -527,7 +527,7 @@ static int pci_irq_notifier(struct notifier_block *nb, unsigned long action,
 	if (action != BUS_NOTIFY_UNBOUND_DRIVER)
 		return NOTIFY_DONE;
 
-	if (pcibios_disable_irq)
+	if (!pci_is_enabled(dev) && pcibios_disable_irq)
 		pcibios_disable_irq(dev);
 
 	return NOTIFY_OK;
-- 
1.7.10.4

Re: [Bugfix] x86/PCI: Release PCI IRQ resource only if PCI device is disabled when unbinding

[Rafael J. Wysocki]

On Friday, March 13, 2015 10:06:43 AM Jiang Liu wrote:
> To support IOAPIC hot-removal, we need to release PCI interrupt resource
> when unbinding PCI device driver. But due to historical reason,
> /*
>  * We would love to complain here if pci_dev->is_enabled is set, that
>  * the driver should have called pci_disable_device(), but the
>  * unfortunate fact is there are too many odd BIOS and bridge setups
>  * that don't like drivers doing that all of the time.
>  * Oh well, we can dream of sane hardware when we sleep, no matter how
>  * horrible the crap we have to deal with is when we are awake...
>  */
> some drivers don't call pci_disable_device() when unloading, which
> prevents us from reallocating PCI interrupt resource on reloading
> PCI driver and causes regressions.
> 
> So release PCI interrupt resource only if PCI device is disabled when
> unbinding. By this way, we could support IOAPIC hot-removal on latest
> platforms and avoid regressions on old platforms.
> 
> Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>

OK, I can agree with that.

Bjorn, what do you think?

> ---
>  arch/x86/pci/common.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> index 3d2612b68694..8d792142cb2a 100644
> --- a/arch/x86/pci/common.c
> +++ b/arch/x86/pci/common.c
> @@ -527,7 +527,7 @@ static int pci_irq_notifier(struct notifier_block *nb, unsigned long action,
>  	if (action != BUS_NOTIFY_UNBOUND_DRIVER)
>  		return NOTIFY_DONE;
>  
> -	if (pcibios_disable_irq)
> +	if (!pci_is_enabled(dev) && pcibios_disable_irq)
>  		pcibios_disable_irq(dev);
>  
>  	return NOTIFY_OK;
> 

-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.