[PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Stefan Agner]

There might be hardware configurations where 64-bit data accesses
to NVMe registers are not supported properly.  This patch removes
the readq/writeq so always two 32-bit accesses are used to read/write
64-bit NVMe registers, similarly as it is done in Linux kernel.

This patch fixes operation of NVMe devices on RPi4 Broadcom
BCM2711 SoC based board, where the VL805 USB XHCI controller is
connected to the PCIe Root Complex, which is attached to the system
through the SCB bridge.

Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
the 64-bit wide register accesses initiated by the CPU are not properly
translated to a sequence of 32-bit PCIe accesses.
nvme_readq(), for example, always returns same value in upper and lower
32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
probing.

This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
32-bit accesses in xhci_writeq/xhci_readq").

Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
Cc: Matthias Brugger <mbrugger@suse.com>
Signed-off-by: Stefan Agner <stefan@agner.ch>
---

 drivers/nvme/nvme.h | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/drivers/nvme/nvme.h b/drivers/nvme/nvme.h
index 0e8cb221a7..aa4b3bac67 100644
--- a/drivers/nvme/nvme.h
+++ b/drivers/nvme/nvme.h
@@ -535,28 +535,20 @@ struct nvme_completion {
  */
 static inline u64 nvme_readq(__le64 volatile *regs)
 {
-#if BITS_PER_LONG == 64
-	return readq(regs);
-#else
 	__u32 *ptr = (__u32 *)regs;
 	u64 val_lo = readl(ptr);
 	u64 val_hi = readl(ptr + 1);
 
 	return val_lo + (val_hi << 32);
-#endif
 }
 
 static inline void nvme_writeq(const u64 val, __le64 volatile *regs)
 {
-#if BITS_PER_LONG == 64
-	writeq(val, regs);
-#else
 	__u32 *ptr = (__u32 *)regs;
 	u32 val_lo = lower_32_bits(val);
 	u32 val_hi = upper_32_bits(val);
 	writel(val_lo, ptr);
 	writel(val_hi, ptr + 1);
-#endif
 }
 
 struct nvme_bar {
-- 
2.29.2

[PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Stefan Agner]

There might be hardware configurations where 64-bit data accesses
to NVMe registers are not supported properly.  This patch removes
the readq/writeq so always two 32-bit accesses are used to read/write
64-bit NVMe registers, similarly as it is done in Linux kernel.

This patch fixes operation of NVMe devices on RPi4 Broadcom
BCM2711 SoC based board, where the VL805 USB XHCI controller is
connected to the PCIe Root Complex, which is attached to the system
through the SCB bridge.

Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
the 64-bit wide register accesses initiated by the CPU are not properly
translated to a sequence of 32-bit PCIe accesses.
nvme_readq(), for example, always returns same value in upper and lower
32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
probing.

This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
32-bit accesses in xhci_writeq/xhci_readq").

Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
Cc: Matthias Brugger <mbrugger@suse.com>
Signed-off-by: Stefan Agner <stefan@agner.ch>
---

 drivers/nvme/nvme.h | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/drivers/nvme/nvme.h b/drivers/nvme/nvme.h
index 0e8cb221a7..aa4b3bac67 100644
--- a/drivers/nvme/nvme.h
+++ b/drivers/nvme/nvme.h
@@ -535,28 +535,20 @@ struct nvme_completion {
  */
 static inline u64 nvme_readq(__le64 volatile *regs)
 {
-#if BITS_PER_LONG == 64
-	return readq(regs);
-#else
 	__u32 *ptr = (__u32 *)regs;
 	u64 val_lo = readl(ptr);
 	u64 val_hi = readl(ptr + 1);
 
 	return val_lo + (val_hi << 32);
-#endif
 }
 
 static inline void nvme_writeq(const u64 val, __le64 volatile *regs)
 {
-#if BITS_PER_LONG == 64
-	writeq(val, regs);
-#else
 	__u32 *ptr = (__u32 *)regs;
 	u32 val_lo = lower_32_bits(val);
 	u32 val_hi = upper_32_bits(val);
 	writel(val_lo, ptr);
 	writel(val_hi, ptr + 1);
-#endif
 }
 
 struct nvme_bar {
-- 
2.29.2

[PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Bin Meng]

Hi Stefan,

On Wed, Dec 30, 2020 at 3:37 AM Stefan Agner <stefan@agner.ch> wrote:

The tag is wrong. Should be nvme:

>
> There might be hardware configurations where 64-bit data accesses
> to NVMe registers are not supported properly.  This patch removes
> the readq/writeq so always two 32-bit accesses are used to read/write
> 64-bit NVMe registers, similarly as it is done in Linux kernel.
>
> This patch fixes operation of NVMe devices on RPi4 Broadcom
> BCM2711 SoC based board, where the VL805 USB XHCI controller is
> connected to the PCIe Root Complex, which is attached to the system
> through the SCB bridge.
>
> Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
> the 64-bit wide register accesses initiated by the CPU are not properly
> translated to a sequence of 32-bit PCIe accesses.
> nvme_readq(), for example, always returns same value in upper and lower
> 32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
> probing.
>
> This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
> 32-bit accesses in xhci_writeq/xhci_readq").
>
> Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
> Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
> Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> Cc: Matthias Brugger <mbrugger@suse.com>
> Signed-off-by: Stefan Agner <stefan@agner.ch>
> ---
>
>  drivers/nvme/nvme.h | 8 --------
>  1 file changed, 8 deletions(-)
>

Otherwise, LGTM:

Reviewed-by: Bin Meng <bmeng.cn@gmail.com>

Re: [PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Bin Meng]

Hi Stefan,

On Wed, Dec 30, 2020 at 3:37 AM Stefan Agner <stefan@agner.ch> wrote:

The tag is wrong. Should be nvme:

>
> There might be hardware configurations where 64-bit data accesses
> to NVMe registers are not supported properly.  This patch removes
> the readq/writeq so always two 32-bit accesses are used to read/write
> 64-bit NVMe registers, similarly as it is done in Linux kernel.
>
> This patch fixes operation of NVMe devices on RPi4 Broadcom
> BCM2711 SoC based board, where the VL805 USB XHCI controller is
> connected to the PCIe Root Complex, which is attached to the system
> through the SCB bridge.
>
> Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
> the 64-bit wide register accesses initiated by the CPU are not properly
> translated to a sequence of 32-bit PCIe accesses.
> nvme_readq(), for example, always returns same value in upper and lower
> 32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
> probing.
>
> This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
> 32-bit accesses in xhci_writeq/xhci_readq").
>
> Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
> Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
> Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> Cc: Matthias Brugger <mbrugger@suse.com>
> Signed-off-by: Stefan Agner <stefan@agner.ch>
> ---
>
>  drivers/nvme/nvme.h | 8 --------
>  1 file changed, 8 deletions(-)
>

Otherwise, LGTM:

Reviewed-by: Bin Meng <bmeng.cn@gmail.com>

[PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Stefan Roese]

Hi Stefan,

On 29.12.20 20:37, Stefan Agner wrote:
> There might be hardware configurations where 64-bit data accesses
> to NVMe registers are not supported properly.  This patch removes
> the readq/writeq so always two 32-bit accesses are used to read/write
> 64-bit NVMe registers, similarly as it is done in Linux kernel.
> 
> This patch fixes operation of NVMe devices on RPi4 Broadcom
> BCM2711 SoC based board, where the VL805 USB XHCI controller is
> connected to the PCIe Root Complex, which is attached to the system
> through the SCB bridge.
> 
> Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
> the 64-bit wide register accesses initiated by the CPU are not properly
> translated to a sequence of 32-bit PCIe accesses.
> nvme_readq(), for example, always returns same value in upper and lower
> 32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
> probing.
> 
> This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
> 32-bit accesses in xhci_writeq/xhci_readq").

I see. This is where you cloned the patch subject line from then. ;)

Please change the subject line and add my:

Reviewed-by: Stefan Roese <sr@denx.de>

Thanks,
Stefan

> Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
> Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
> Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> Cc: Matthias Brugger <mbrugger@suse.com>
> Signed-off-by: Stefan Agner <stefan@agner.ch>
> ---
> 
>   drivers/nvme/nvme.h | 8 --------
>   1 file changed, 8 deletions(-)
> 
> diff --git a/drivers/nvme/nvme.h b/drivers/nvme/nvme.h
> index 0e8cb221a7..aa4b3bac67 100644
> --- a/drivers/nvme/nvme.h
> +++ b/drivers/nvme/nvme.h
> @@ -535,28 +535,20 @@ struct nvme_completion {
>    */
>   static inline u64 nvme_readq(__le64 volatile *regs)
>   {
> -#if BITS_PER_LONG == 64
> -	return readq(regs);
> -#else
>   	__u32 *ptr = (__u32 *)regs;
>   	u64 val_lo = readl(ptr);
>   	u64 val_hi = readl(ptr + 1);
>   
>   	return val_lo + (val_hi << 32);
> -#endif
>   }
>   
>   static inline void nvme_writeq(const u64 val, __le64 volatile *regs)
>   {
> -#if BITS_PER_LONG == 64
> -	writeq(val, regs);
> -#else
>   	__u32 *ptr = (__u32 *)regs;
>   	u32 val_lo = lower_32_bits(val);
>   	u32 val_hi = upper_32_bits(val);
>   	writel(val_lo, ptr);
>   	writel(val_hi, ptr + 1);
> -#endif
>   }
>   
>   struct nvme_bar {
> 


Viele Gr??e,
Stefan

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-51 Fax: (+49)-8142-66989-80 Email: sr at denx.de

Re: [PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Stefan Roese]

Hi Stefan,

On 29.12.20 20:37, Stefan Agner wrote:
> There might be hardware configurations where 64-bit data accesses
> to NVMe registers are not supported properly.  This patch removes
> the readq/writeq so always two 32-bit accesses are used to read/write
> 64-bit NVMe registers, similarly as it is done in Linux kernel.
> 
> This patch fixes operation of NVMe devices on RPi4 Broadcom
> BCM2711 SoC based board, where the VL805 USB XHCI controller is
> connected to the PCIe Root Complex, which is attached to the system
> through the SCB bridge.
> 
> Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
> the 64-bit wide register accesses initiated by the CPU are not properly
> translated to a sequence of 32-bit PCIe accesses.
> nvme_readq(), for example, always returns same value in upper and lower
> 32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
> probing.
> 
> This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
> 32-bit accesses in xhci_writeq/xhci_readq").

I see. This is where you cloned the patch subject line from then. ;)

Please change the subject line and add my:

Reviewed-by: Stefan Roese <sr@denx.de>

Thanks,
Stefan

> Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
> Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
> Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
> Cc: Matthias Brugger <mbrugger@suse.com>
> Signed-off-by: Stefan Agner <stefan@agner.ch>
> ---
> 
>   drivers/nvme/nvme.h | 8 --------
>   1 file changed, 8 deletions(-)
> 
> diff --git a/drivers/nvme/nvme.h b/drivers/nvme/nvme.h
> index 0e8cb221a7..aa4b3bac67 100644
> --- a/drivers/nvme/nvme.h
> +++ b/drivers/nvme/nvme.h
> @@ -535,28 +535,20 @@ struct nvme_completion {
>    */
>   static inline u64 nvme_readq(__le64 volatile *regs)
>   {
> -#if BITS_PER_LONG == 64
> -	return readq(regs);
> -#else
>   	__u32 *ptr = (__u32 *)regs;
>   	u64 val_lo = readl(ptr);
>   	u64 val_hi = readl(ptr + 1);
>   
>   	return val_lo + (val_hi << 32);
> -#endif
>   }
>   
>   static inline void nvme_writeq(const u64 val, __le64 volatile *regs)
>   {
> -#if BITS_PER_LONG == 64
> -	writeq(val, regs);
> -#else
>   	__u32 *ptr = (__u32 *)regs;
>   	u32 val_lo = lower_32_bits(val);
>   	u32 val_hi = upper_32_bits(val);
>   	writel(val_lo, ptr);
>   	writel(val_hi, ptr + 1);
> -#endif
>   }
>   
>   struct nvme_bar {
> 


Viele Grüße,
Stefan

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-51 Fax: (+49)-8142-66989-80 Email: sr@denx.de

[PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Stefan Agner]

On 2020-12-30 02:36, Bin Meng wrote:
> Hi Stefan,
> 
> On Wed, Dec 30, 2020 at 3:37 AM Stefan Agner <stefan@agner.ch> wrote:
> 
> The tag is wrong. Should be nvme:

Thanks for pointing out. Will send a v2.

FWIW, the send to amlogic ML was accidentially.

--
Stefan

> 
>>
>> There might be hardware configurations where 64-bit data accesses
>> to NVMe registers are not supported properly.  This patch removes
>> the readq/writeq so always two 32-bit accesses are used to read/write
>> 64-bit NVMe registers, similarly as it is done in Linux kernel.
>>
>> This patch fixes operation of NVMe devices on RPi4 Broadcom
>> BCM2711 SoC based board, where the VL805 USB XHCI controller is
>> connected to the PCIe Root Complex, which is attached to the system
>> through the SCB bridge.
>>
>> Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
>> the 64-bit wide register accesses initiated by the CPU are not properly
>> translated to a sequence of 32-bit PCIe accesses.
>> nvme_readq(), for example, always returns same value in upper and lower
>> 32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
>> probing.
>>
>> This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
>> 32-bit accesses in xhci_writeq/xhci_readq").
>>
>> Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
>> Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
>> Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
>> Cc: Matthias Brugger <mbrugger@suse.com>
>> Signed-off-by: Stefan Agner <stefan@agner.ch>
>> ---
>>
>>  drivers/nvme/nvme.h | 8 --------
>>  1 file changed, 8 deletions(-)
>>
> 
> Otherwise, LGTM:
> 
> Reviewed-by: Bin Meng <bmeng.cn@gmail.com>

Re: [PATCH] usb: xhci: Use only 32-bit accesses in nvme_writeq/nvme_readq

[Stefan Agner]

On 2020-12-30 02:36, Bin Meng wrote:
> Hi Stefan,
> 
> On Wed, Dec 30, 2020 at 3:37 AM Stefan Agner <stefan@agner.ch> wrote:
> 
> The tag is wrong. Should be nvme:

Thanks for pointing out. Will send a v2.

FWIW, the send to amlogic ML was accidentially.

--
Stefan

> 
>>
>> There might be hardware configurations where 64-bit data accesses
>> to NVMe registers are not supported properly.  This patch removes
>> the readq/writeq so always two 32-bit accesses are used to read/write
>> 64-bit NVMe registers, similarly as it is done in Linux kernel.
>>
>> This patch fixes operation of NVMe devices on RPi4 Broadcom
>> BCM2711 SoC based board, where the VL805 USB XHCI controller is
>> connected to the PCIe Root Complex, which is attached to the system
>> through the SCB bridge.
>>
>> Even though the architecture is 64-bit the PCIe BAR is 32-bit and likely
>> the 64-bit wide register accesses initiated by the CPU are not properly
>> translated to a sequence of 32-bit PCIe accesses.
>> nvme_readq(), for example, always returns same value in upper and lower
>> 32-bits, e.g. 0x3c033fff3c033fff which lead to NVMe devices to fail
>> probing.
>>
>> This fix is analogous to commit 8e2ab05000ab ("usb: xhci: Use only
>> 32-bit accesses in xhci_writeq/xhci_readq").
>>
>> Cc: Sylwester Nawrocki <s.nawrocki@samsung.com>
>> Cc: Zhikang Zhang <zhikang.zhang@nxp.com>
>> Cc: Nicolas Saenz Julienne <nsaenzjulienne@suse.de>
>> Cc: Matthias Brugger <mbrugger@suse.com>
>> Signed-off-by: Stefan Agner <stefan@agner.ch>
>> ---
>>
>>  drivers/nvme/nvme.h | 8 --------
>>  1 file changed, 8 deletions(-)
>>
> 
> Otherwise, LGTM:
> 
> Reviewed-by: Bin Meng <bmeng.cn@gmail.com>