[RFC PATCH] nvme: retain split access workaround for capability reads

[RFC PATCH] nvme: retain split access workaround for capability reads

[Ard Biesheuvel]

Recent changes to the NVMe core have re-introduced an issue that we
have attempted to work around in the past, in commit a310acd7a7ea
("NVMe: use split lo_hi_{read,write}q").

The problem is that some PCIe NVMe controllers do not implement 64-bit
outbound accesses correctly, which is why the commit above switched
to using lo_hi_[read|write]q for all 64-bit BAR accesses.

In the mean time, the NVMe subsystem has been refactored, and now calls
into the PCIe support layer for NVMe via a .reg_read64() method, which
fails to use lo_hi_readq(), and thus reintroduces the problem that the
commit above aimed to address.

Given that, at the moment, .reg_read64() is only used to read the
capability register [which is known to tolerate split reads, which is
not guaranteed in the general case, given that the NVMe BAR may be
non-prefetchable], let's switch .reg_read64() to lo_hi_readq() as
well.

To ensure that we will spot any changes that will start using the
.reg_read64() method for other purposes, WARN() if the requested
offset != NVME_REG_CAP.

This fixes a boot issue on some ARM boxes with NVMe behind a
Synopsys DesignWare PCIe host controller.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
Broken since v5.3, so if this gets fixed one way or the other, please
add cc: stable.

Given that reg_read64() is only used in a single place to read the
capability register, it would be cleaner to just drop it and add a
.reg_readcap() method instead, but this is a more invasive change.

 drivers/nvme/host/pci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index c0808f9eb8ab..bb012075fbb2 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2672,7 +2672,8 @@ static int nvme_pci_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val)
 
 static int nvme_pci_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val)
 {
-	*val = readq(to_nvme_dev(ctrl)->bar + off);
+	WARN_ON(off != NVME_REG_CAP);
+	*val = lo_hi_readq(to_nvme_dev(ctrl)->bar + off);
 	return 0;
 }
 
-- 
2.20.1


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [RFC PATCH] nvme: retain split access workaround for capability reads

[Nadolski, Edmund]

On 10/2/2019 1:36 AM, Ard Biesheuvel wrote:
> Recent changes to the NVMe core have re-introduced an issue that we
> have attempted to work around in the past, in commit a310acd7a7ea
> ("NVMe: use split lo_hi_{read,write}q").
> 
> The problem is that some PCIe NVMe controllers do not implement 64-bit
> outbound accesses correctly, which is why the commit above switched
> to using lo_hi_[read|write]q for all 64-bit BAR accesses.
> 
> In the mean time, the NVMe subsystem has been refactored, and now calls
> into the PCIe support layer for NVMe via a .reg_read64() method, which
> fails to use lo_hi_readq(), and thus reintroduces the problem that the
> commit above aimed to address.
> 
> Given that, at the moment, .reg_read64() is only used to read the
> capability register [which is known to tolerate split reads, which is
> not guaranteed in the general case, given that the NVMe BAR may be
> non-prefetchable], let's switch .reg_read64() to lo_hi_readq() as
> well.

Might be good to include this as a comment, for clarity and so it won't
get refactored out again.

> To ensure that we will spot any changes that will start using the
> .reg_read64() method for other purposes, WARN() if the requested
> offset != NVME_REG_CAP.

Would WARN_ONCE() suffice?

Ed

> This fixes a boot issue on some ARM boxes with NVMe behind a
> Synopsys DesignWare PCIe host controller.
> 
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> Broken since v5.3, so if this gets fixed one way or the other, please
> add cc: stable.
> 
> Given that reg_read64() is only used in a single place to read the
> capability register, it would be cleaner to just drop it and add a
> .reg_readcap() method instead, but this is a more invasive change.
> 
>   drivers/nvme/host/pci.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index c0808f9eb8ab..bb012075fbb2 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -2672,7 +2672,8 @@ static int nvme_pci_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val)
>   
>   static int nvme_pci_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val)
>   {
> -	*val = readq(to_nvme_dev(ctrl)->bar + off);
> +	WARN_ON(off != NVME_REG_CAP);
> +	*val = lo_hi_readq(to_nvme_dev(ctrl)->bar + off);
>   	return 0;
>   }
>   
> 


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [RFC PATCH] nvme: retain split access workaround for capability reads

[Keith Busch]

On Wed, Oct 02, 2019 at 09:36:43AM +0200, Ard Biesheuvel wrote:
> Recent changes to the NVMe core have re-introduced an issue that we
> have attempted to work around in the past, in commit a310acd7a7ea
> ("NVMe: use split lo_hi_{read,write}q").
> 
> The problem is that some PCIe NVMe controllers do not implement 64-bit
> outbound accesses correctly, which is why the commit above switched
> to using lo_hi_[read|write]q for all 64-bit BAR accesses.
> 
> In the mean time, the NVMe subsystem has been refactored, and now calls
> into the PCIe support layer for NVMe via a .reg_read64() method, which
> fails to use lo_hi_readq(), and thus reintroduces the problem that the
> commit above aimed to address.
> 
> Given that, at the moment, .reg_read64() is only used to read the
> capability register [which is known to tolerate split reads, which is
> not guaranteed in the general case, given that the NVMe BAR may be
> non-prefetchable], let's switch .reg_read64() to lo_hi_readq() as
> well.
> 
> To ensure that we will spot any changes that will start using the
> .reg_read64() method for other purposes, WARN() if the requested
> offset != NVME_REG_CAP.
> 
> This fixes a boot issue on some ARM boxes with NVMe behind a
> Synopsys DesignWare PCIe host controller.
> 
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> Broken since v5.3, so if this gets fixed one way or the other, please
> add cc: stable.

Since 5.3?! 'git blame' says the code has been this way since 7fd8930f26be4,
which was from 2015 during the 4.4 development cycle.

Please add a:

  Fixes: 7fd8930f26be4 ("nvme: add a common helper to read Identify Controller data")

to your change log above your sign-off, and remove the WARN_ON().

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [RFC PATCH] nvme: retain split access workaround for capability reads

[Ard Biesheuvel]

On Thu, 3 Oct 2019 at 13:49, Keith Busch <kbusch@kernel.org> wrote:
>
> On Wed, Oct 02, 2019 at 09:36:43AM +0200, Ard Biesheuvel wrote:
> > Recent changes to the NVMe core have re-introduced an issue that we
> > have attempted to work around in the past, in commit a310acd7a7ea
> > ("NVMe: use split lo_hi_{read,write}q").
> >
> > The problem is that some PCIe NVMe controllers do not implement 64-bit
> > outbound accesses correctly, which is why the commit above switched
> > to using lo_hi_[read|write]q for all 64-bit BAR accesses.
> >
> > In the mean time, the NVMe subsystem has been refactored, and now calls
> > into the PCIe support layer for NVMe via a .reg_read64() method, which
> > fails to use lo_hi_readq(), and thus reintroduces the problem that the
> > commit above aimed to address.
> >
> > Given that, at the moment, .reg_read64() is only used to read the
> > capability register [which is known to tolerate split reads, which is
> > not guaranteed in the general case, given that the NVMe BAR may be
> > non-prefetchable], let's switch .reg_read64() to lo_hi_readq() as
> > well.
> >
> > To ensure that we will spot any changes that will start using the
> > .reg_read64() method for other purposes, WARN() if the requested
> > offset != NVME_REG_CAP.
> >
> > This fixes a boot issue on some ARM boxes with NVMe behind a
> > Synopsys DesignWare PCIe host controller.
> >
> > Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> > ---
> > Broken since v5.3, so if this gets fixed one way or the other, please
> > add cc: stable.
>
> Since 5.3?! 'git blame' says the code has been this way since 7fd8930f26be4,
> which was from 2015 during the 4.4 development cycle.
>

That may be true, but the box in question happily boots a v5.2 kernel.

> Please add a:
>
>   Fixes: 7fd8930f26be4 ("nvme: add a common helper to read Identify Controller data")
>
> to your change log above your sign-off, and remove the WARN_ON().

Sure.

Thanks,
Ard.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [RFC PATCH] nvme: retain split access workaround for capability reads

[Keith Busch]

On Thu, Oct 03, 2019 at 01:51:03PM +0200, Ard Biesheuvel wrote:
> On Thu, 3 Oct 2019 at 13:49, Keith Busch <kbusch@kernel.org> wrote:
> > On Wed, Oct 02, 2019 at 09:36:43AM +0200, Ard Biesheuvel wrote:
> > > Broken since v5.3, so if this gets fixed one way or the other, please
> > > add cc: stable.
> >
> > Since 5.3?! 'git blame' says the code has been this way since 7fd8930f26be4,
> > which was from 2015 during the 4.4 development cycle.
> >
> 
> That may be true, but the box in question happily boots a v5.2 kernel.

Looks like your observation was introduced with c0f2f45be2976abe.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [RFC PATCH] nvme: retain split access workaround for capability reads

[Ard Biesheuvel]

On Thu, 3 Oct 2019 at 14:03, Keith Busch <kbusch@kernel.org> wrote:
>
> On Thu, Oct 03, 2019 at 01:51:03PM +0200, Ard Biesheuvel wrote:
> > On Thu, 3 Oct 2019 at 13:49, Keith Busch <kbusch@kernel.org> wrote:
> > > On Wed, Oct 02, 2019 at 09:36:43AM +0200, Ard Biesheuvel wrote:
> > > > Broken since v5.3, so if this gets fixed one way or the other, please
> > > > add cc: stable.
> > >
> > > Since 5.3?! 'git blame' says the code has been this way since 7fd8930f26be4,
> > > which was from 2015 during the 4.4 development cycle.
> > >
> >
> > That may be true, but the box in question happily boots a v5.2 kernel.
>
> Looks like your observation was introduced with c0f2f45be2976abe.

Indeed. The corrupted read produces a bogus minimum page size, and the
driver gives up.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [RFC PATCH] nvme: retain split access workaround for capability reads

[Sagi Grimberg]

>>>>> Broken since v5.3, so if this gets fixed one way or the other, please
>>>>> add cc: stable.
>>>>
>>>> Since 5.3?! 'git blame' says the code has been this way since 7fd8930f26be4,
>>>> which was from 2015 during the 4.4 development cycle.
>>>>
>>>
>>> That may be true, but the box in question happily boots a v5.2 kernel.
>>
>> Looks like your observation was introduced with c0f2f45be2976abe.
> 
> Indeed. The corrupted read produces a bogus minimum page size, and the
> driver gives up.

Oops... Sorry... But in all fairness that was bound to happen at some
point...

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme