[PATCH 2/2] [IPoIB] Handle sending of unicast RARP responses

[PATCH 2/2] [IPoIB] Handle sending of unicast RARP responses

[Roland Dreier]

From: Hal Rosenstock <halr@voltaire.com>

RARP replies are another valid case where IPoIB may need to send a
unicast packet with no neighbour structure.

Signed-off-by: Hal Rosenstock <halr@voltaire.com>
Signed-off-by: Roland Dreier <rolandd@cisco.com>
---

 drivers/infiniband/ulp/ipoib/ipoib_main.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

0dca0f7bf82face7b700890318d5550fd542cabf
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -600,9 +600,10 @@ static int ipoib_start_xmit(struct sk_bu
 
 			ipoib_mcast_send(dev, (union ib_gid *) (phdr->hwaddr + 4), skb);
 		} else {
-			/* unicast GID -- should be ARP reply */
+			/* unicast GID -- should be ARP or RARP reply */
 
-			if (be16_to_cpup((u16 *) skb->data) != ETH_P_ARP) {
+			if ((be16_to_cpup((__be16 *) skb->data) != ETH_P_ARP) &&
+			    (be16_to_cpup((__be16 *) skb->data) != ETH_P_RARP)) {
 				ipoib_warn(priv, "Unicast, no %s: type %04x, QPN %06x "
 					   IPOIB_GID_FMT "\n",
 					   skb->dst ? "neigh" : "dst",

[PATCH 0/2] REALLY final InfiniBand updates for 2.6.13

[Roland Dreier]

Here are two last-minute fixes for InfiniBand: one to avoid pain in
releasing 2.6.13 with an incorrect constant and then having to rename
the enum later, and one to fix RARP on IP-over-InfiniBand.

After Greg's scolding, I'm sending them as patches, but they're also
available in the git tree at

    rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

(pending mirror propagation)

The patches update the following files:

 include/ib_cm.h        |    3 ++-
 ulp/ipoib/ipoib_main.c |    5 +++--
 2 files changed, 5 insertions(+), 3 deletions(-)

through the following changes:

commit 0dca0f7bf82face7b700890318d5550fd542cabf
Author: Hal Rosenstock <halr@voltaire.com>
Date:   Thu Jul 28 13:17:26 2005 -0700

    [PATCH] [IPoIB] Handle sending of unicast RARP responses
    
    RARP replies are another valid case where IPoIB may need to send a
    unicast packet with no neighbour structure.
    
    Signed-off-by: Hal Rosenstock <halr@voltaire.com>
    Signed-off-by: Roland Dreier <rolandd@cisco.com>

commit 4e38d36d88ead4e56f3155573976da84d5df18b3
Author: Roland Dreier <roland@eddore.topspincom.com>
Date:   Thu Jul 28 13:16:30 2005 -0700

    [PATCH] [IB/cm]: Correct CM port redirect reject codes
    
    Reject code 24 is port and CM redirection, not just port redirection.
    Port redirection alone is code 25.
    
    Therefore we should rename code 24 to IB_CM_REJ_PORT_CM_REDIRECT and
    use IB_CM_REJ_PORT_REDIRECT for code 25.
    
    Signed-off-by: Roland Dreier <rolandd@cisco.com>

Thanks,
  Roland

[PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes

[Roland Dreier]

Reject code 24 is port and CM redirection, not just port redirection.
Port redirection alone is code 25.

Therefore we should rename code 24 to IB_CM_REJ_PORT_CM_REDIRECT and
use IB_CM_REJ_PORT_REDIRECT for code 25.

Signed-off-by: Roland Dreier <rolandd@cisco.com>
---

 drivers/infiniband/include/ib_cm.h |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

4e38d36d88ead4e56f3155573976da84d5df18b3
diff --git a/drivers/infiniband/include/ib_cm.h b/drivers/infiniband/include/ib_cm.h
--- a/drivers/infiniband/include/ib_cm.h
+++ b/drivers/infiniband/include/ib_cm.h
@@ -169,7 +169,8 @@ enum ib_cm_rej_reason {
 	IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS	= __constant_htons(21),
 	IB_CM_REJ_INVALID_ALT_HOP_LIMIT		= __constant_htons(22),
 	IB_CM_REJ_INVALID_ALT_PACKET_RATE	= __constant_htons(23),
-	IB_CM_REJ_PORT_REDIRECT			= __constant_htons(24),
+	IB_CM_REJ_PORT_CM_REDIRECT		= __constant_htons(24),
+	IB_CM_REJ_PORT_REDIRECT			= __constant_htons(25),
 	IB_CM_REJ_INVALID_MTU			= __constant_htons(26),
 	IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES	= __constant_htons(27),
 	IB_CM_REJ_CONSUMER_DEFINED		= __constant_htons(28),

Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes

[yhlu]

Roland,

In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB
driver in Kernel can not be loaded.

YH

PCI: 00:18.0 1c1 <- [0x0000001000 - 0x0000003fff] io <node 0 link 1>
PCI: 00:18.0 1b9 <- [0xfce0000000 - 0xfcf07fffff] prefmem <node 0 link 1>
PCI: 00:18.0 1b1 <- [0x00fc000000 - 0x00fd2fffff] mem <node 0 link 1>           
PCI: 01:0f.0 24 <- [0xfce0000000 - 0xfcf07fffff] bus 4 prefmem
PCI: 01:0f.0 20 <- [0x00fd100000 - 0x00fd1fffff] bus 4 mem
PCI: 04:00.0 10 <- [0x00fd100000 - 0x00fd1fffff] mem64
PCI: 04:00.0 18 <- [0xfcf0000000 - 0xfcf07fffff] prefmem64
PCI: 04:00.0 20 <- [0xfce0000000 - 0xfcefffffff] prefmem64                      

ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor c)
ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.
ib_mthca: probe of 0000:04:00.0 failed with error -16

Re: [openib-general] Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes

[Grant Grundler]

On Wed, Aug 03, 2005 at 05:58:11PM -0700, yhlu wrote:
> Roland,
> 
> In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB
> driver in Kernel can not be loaded.

Can you provide a few more details about the configuration?
o kernel version
o architecture (i386 or x86-64)
o post the full console output from power up?

Recent email on linux-pci raised awareness that 32-bit kernel
can not support 64-bit PCI MMIO addresses. struct resource (defined in
include/linux/ioport.h) defines the start/end field as "unsigned long".
That's only 32-bit on i386 kernels.

> PCI: 04:00.0 18 <- [0xfcf0000000 - 0xfcf07fffff] prefmem64
> PCI: 04:00.0 20 <- [0xfce0000000 - 0xfcefffffff] prefmem64                      
I have to wonder if those BARs are truly prefetchable.
Does Mellanox assume CPU is the only one to write the 3rd BAR (RAM)
and the CPU implements a write-through cache (vs write back)?

I'm just guessing because I don't understand exactly how the
256MB of onboard RAM is accessed.

hth,
grant

> 
> ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
> ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor c)
> ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.
> ib_mthca: probe of 0000:04:00.0 failed with error -16
> _______________________________________________
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

And I have to wonder if those BARs truly are prefetchable.
It would imply only the CPU writes them and 

mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes)

[Roland Dreier]

    yhlu> In LinuxBIOS, If I enable the prefmem64 to use real 64
    yhlu> range. the IB driver in Kernel can not be loaded.

What does it mean to "enable the prefmem64 to use real 64 range"?

Does the driver work if you don't do this?

    yhlu> ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.

Can you add printk()s to mthca_qp.c::mthca_init_qp_table() to find out
how far the function gets before it fails?

It would also be useful for you to build with CONFIG_INFINIBAND_MTHCA_DEBUG=y
and send the kernel output you get with that.

 - Roland

Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes

[Michael S. Tsirkin]

Quoting r. yhlu <yhlu.kernel@gmail.com>:
> Subject: Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes
> 
> Roland,
> 
> In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB
> driver in Kernel can not be loaded.

Are you using the latest firmware on the HCA card?

-- 
MST

Re: mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes)

[yhlu]

YES.

I will send you the output message later about "CONFIG_INFINIBAND_MTHCA_DEBUG=y"

YH

On 8/3/05, Roland Dreier <rolandd@cisco.com> wrote:
>     yhlu> In LinuxBIOS, If I enable the prefmem64 to use real 64
>     yhlu> range. the IB driver in Kernel can not be loaded.
> 
> What does it mean to "enable the prefmem64 to use real 64 range"?
> 
> Does the driver work if you don't do this?
> 
>     yhlu> ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.
> 
> Can you add printk()s to mthca_qp.c::mthca_init_qp_table() to find out
> how far the function gets before it fails?
> 
> It would also be useful for you to build with CONFIG_INFINIBAND_MTHCA_DEBUG=y
> and send the kernel output you get with that.
> 
>  - Roland
>

Re: mthca and LinuxBIOS

[Roland Dreier]

>>>>> "yhlu" == yhlu  <yhlu.kernel@gmail.com> writes:

    yhlu> YES.  I will send you the output message later about
    yhlu> "CONFIG_INFINIBAND_MTHCA_DEBUG=y"

Thanks.  In the meantime, can you explain what it means to "enable the
prefmem64 to use real 64 range"?  What is the difference between this
and the configuration that works?

 - R.

Re: mthca and LinuxBIOS

[yhlu]

The mellanox can use prefmem64, but the BIOS could only allocate the
some range below 4G, So 32 bit OS still can use the IB cards.
but for 64bit OS, We could allocate range above 4G (0xfc00000000), So
the mmio below 4G can be smaller. ( for example from 512M to 128M, the
user can get back some RAM back if Opteron don't have hardware memhole
support).

YH



On 8/4/05, Roland Dreier <rolandd@cisco.com> wrote:
> >>>>> "yhlu" == yhlu  <yhlu.kernel@gmail.com> writes:
> 
>     yhlu> YES.  I will send you the output message later about
>     yhlu> "CONFIG_INFINIBAND_MTHCA_DEBUG=y"
> 
> Thanks.  In the meantime, can you explain what it means to "enable the
> prefmem64 to use real 64 range"?  What is the difference between this
> and the configuration that works?
> 
>  - R.
>

Re: mthca and LinuxBIOS

[yhlu]

i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y

I didn't get any more debug info, is that depend other setting?

YH

On 8/4/05, yhlu <yhlu.kernel@gmail.com> wrote:
> The mellanox can use prefmem64, but the BIOS could only allocate the
> some range below 4G, So 32 bit OS still can use the IB cards.
> but for 64bit OS, We could allocate range above 4G (0xfc00000000), So
> the mmio below 4G can be smaller. ( for example from 512M to 128M, the
> user can get back some RAM back if Opteron don't have hardware memhole
> support).
> 
> YH
> 
> 
> 
> On 8/4/05, Roland Dreier <rolandd@cisco.com> wrote:
> > >>>>> "yhlu" == yhlu  <yhlu.kernel@gmail.com> writes:
> >
> >     yhlu> YES.  I will send you the output message later about
> >     yhlu> "CONFIG_INFINIBAND_MTHCA_DEBUG=y"
> >
> > Thanks.  In the meantime, can you explain what it means to "enable the
> > prefmem64 to use real 64 range"?  What is the difference between this
> > and the configuration that works?
> >
> >  - R.
> >
>

Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes

[yhlu]

Yes.

On 8/3/05, Michael S. Tsirkin <mst@mellanox.co.il> wrote:
> Quoting r. yhlu <yhlu.kernel@gmail.com>:
> > Subject: Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes
> >
> > Roland,
> >
> > In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB
> > driver in Kernel can not be loaded.
> 
> Are you using the latest firmware on the HCA card?
> 
> --
> MST
>

Re: mthca and LinuxBIOS

[Roland Dreier]

    yhlu> i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any
    yhlu> more debug info, is that depend other setting?

It shouldn't depend on anything.  mthca_dbg() gets turned into
dev_dbg(), which just does dev_printk(KERN_DEBUG,...).  Perhaps you
have to change your console level to see KERN_DEBUG messages?

Since you're getting to the call to mthca_init_qp_table(), there are
mthca_dbg() calls that you should definitely be getting output from.

 - R.

Re: mthca and LinuxBIOS

[yhlu]

ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex
(Tavor compatibility mode) (0000:04:00.0)
ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64
ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff)
ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000,
end fcefffffff)
ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256
ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64
ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64
ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16
ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1
ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0
ib_mthca 0000:04:00.0: Flags: 00370347
ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x      fce0000000 (size 0x 4000000)
ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x      fce4000000 (size 0x 1000000)
ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x      fce5000000 (size 0x  800000)
ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x      fce5800000 (size 0x  800000)
ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x      fce6000000 (size 0x  400000)
ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x      fce6400000 (size 0x  200000)
ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x      fce6600000 (size 0x  100000)
ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x      fce6700000 (size 0x   80000)
ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x      fce6780000 (size 0x   10000)
ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x      fce6790000 (size 0x     800)
ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB
(149950 KB free)
ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries
ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries
ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries
ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2
ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3
ib_mthca 0000:04:00.0: NOP command IRQ test passed
<------------------------------------------------------------------------------------------------------stuck
30s
ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.
ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2
ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3
ib_mthca: probe of 0000:04:00.0 failed with error -16


On 8/4/05, Roland Dreier <rolandd@cisco.com> wrote:
>     yhlu> i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any
>     yhlu> more debug info, is that depend other setting?
> 
> It shouldn't depend on anything.  mthca_dbg() gets turned into
> dev_dbg(), which just does dev_printk(KERN_DEBUG,...).  Perhaps you
> have to change your console level to see KERN_DEBUG messages?
> 
> Since you're getting to the call to mthca_init_qp_table(), there are
> mthca_dbg() calls that you should definitely be getting output from.
> 
>  - R.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: mthca and LinuxBIOS

[Roland Dreier]

Hmm, that output all looks fine.  Can you run with the patch below to
see exactly where the QP table initialization fails?  (I haven't
actually compiled this patch so you may have to fix a typo or two)

I'm guessing that the CONF_SPECIAL_QP command is failing, but let's
make sure.

Thanks,
  Roland

diff --git a/drivers/infiniband/hw/mthca/mthca_qp.c b/drivers/infiniband/hw/mthca/mthca_qp.c
--- a/drivers/infiniband/hw/mthca/mthca_qp.c
+++ b/drivers/infiniband/hw/mthca/mthca_qp.c
@@ -2214,13 +2214,16 @@ int __devinit mthca_init_qp_table(struct
 			       (1 << 24) - 1,
 			       dev->qp_table.sqp_start +
 			       MTHCA_MAX_PORTS * 2);
-	if (err)
+	if (err) {
+		mthca_err(dev, "mthca_init_qp_table: mthca_alloc_init failed (%d)\n", err);
 		return err;
+	}
 
 	err = mthca_array_init(&dev->qp_table.qp,
 			       dev->limits.num_qps);
 	if (err) {
 		mthca_alloc_cleanup(&dev->qp_table.alloc);
+		mthca_err(dev, "mthca_init_qp_table: mthca_array_init failed (%d)\n", err);
 		return err;
 	}
 
@@ -2228,8 +2231,10 @@ int __devinit mthca_init_qp_table(struct
 		err = mthca_CONF_SPECIAL_QP(dev, i ? IB_QPT_GSI : IB_QPT_SMI,
 					    dev->qp_table.sqp_start + i * 2,
 					    &status);
-		if (err)
+		if (err) {
+			mthca_err(dev, "mthca_init_qp_table: mthca_CONF_SPECIAL_QP failed for %d/%d (%d)\n", i, dev->qp_table.sqp_start + i * 2, err);
 			goto err_out;
+		}
 		if (status) {
 			mthca_warn(dev, "CONF_SPECIAL_QP returned "
 				   "status %02x, aborting.\n",

Re: mthca and LinuxBIOS

[yhlu]

You are right. CONG_SPECIAL_QP

ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex
(Tavor compatibility mode) (0000:04:00.0)
ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64
ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff)
ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000,
end fcefffffff)
ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256
ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64
ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64
ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16
ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1
ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0
ib_mthca 0000:04:00.0: Flags: 00370347
ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x      fce0000000 (size 0x 4000000)
ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x      fce4000000 (size 0x 1000000)
ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x      fce5000000 (size 0x  800000)
ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x      fce5800000 (size 0x  800000)
ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x      fce6000000 (size 0x  400000)
ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x      fce6400000 (size 0x  200000)
ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x      fce6600000 (size 0x  100000)
ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x      fce6700000 (size 0x   80000)
ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x      fce6780000 (size 0x   10000)
ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x      fce6790000 (size 0x     800)
ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB
(149950 KB free)
ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries
ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries
ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries
ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2
ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3
ib_mthca 0000:04:00.0: NOP command IRQ test passed
ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP
failed for 0/1024 (-16)
ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.
ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2
ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3
ib_mthca: probe of 0000:04:00.0 failed with error -16

Re: mthca and LinuxBIOS

[yhlu]

ps.
some kernel pci code patch broke sth yesterday night.
it mask out bit [32-39]
ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x        e0000000 (size 0x 4000000)
ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x        e4000000 (size 0x 1000000)
ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x        e5000000 (size 0x  800000)
ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x        e5800000 (size 0x  800000)
ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x        e6000000 (size 0x  400000)
ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x        e6400000 (size 0x  200000)
ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x        e6600000 (size 0x  100000)
ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x        e6700000 (size 0x   80000)
ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x        e6780000 (size 0x   10000)
ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x        e6790000 (size
0x     800)

YH 

On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote:
> You are right. CONG_SPECIAL_QP
> 
> ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
> ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex
> (Tavor compatibility mode) (0000:04:00.0)
> ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64
> ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff)
> ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000,
> end fcefffffff)
> ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256
> ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64
> ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64
> ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16
> ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1
> ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0
> ib_mthca 0000:04:00.0: Flags: 00370347
> ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x      fce0000000 (size 0x 4000000)
> ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x      fce4000000 (size 0x 1000000)
> ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x      fce5000000 (size 0x  800000)
> ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x      fce5800000 (size 0x  800000)
> ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x      fce6000000 (size 0x  400000)
> ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x      fce6400000 (size 0x  200000)
> ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x      fce6600000 (size 0x  100000)
> ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x      fce6700000 (size 0x   80000)
> ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x      fce6780000 (size 0x   10000)
> ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x      fce6790000 (size 0x     800)
> ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB
> (149950 KB free)
> ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries
> ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries
> ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries
> ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2
> ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3
> ib_mthca 0000:04:00.0: NOP command IRQ test passed
> ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP
> failed for 0/1024 (-16)
> ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.
> ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2
> ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3
> ib_mthca: probe of 0000:04:00.0 failed with error -16
>

Re: mthca and LinuxBIOS

[Roland Dreier]

    > ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64

This is FW 4.6.2 -- 4.7.0 has been released, so it might be worth
trying that.

    > ib_mthca 0000:04:00.0: NOP command IRQ test passed
    > ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP failed for 0/1024 (-16)

Hmm, looks like CONF_SPECIAL_QP is timing out.

MST (or any Mellanox people), any idea why this might happening?  The
NOP command is working fine with interrupts, but CONF_SPECIAL_QP is
timing out.  The difference from the working setup is that the HCA's
local memory is mapped above 4 GB.

 - R.

Re: mthca and LinuxBIOS

[Roland Dreier]

    yhlu> ps.  some kernel pci code patch broke sth yesterday night.
    yhlu> it mask out bit [32-39]

Is it possible that all your problems are coming from the PCI setup
code incorrectly assigning BARs?

 - R.

Re: mthca and LinuxBIOS

[yhlu]

before I do the cg-update this morning, it didn't mask out the upper 8 bit.

YH

On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote:
>     yhlu> ps.  some kernel pci code patch broke sth yesterday night.
>     yhlu> it mask out bit [32-39]
> 
> Is it possible that all your problems are coming from the PCI setup
> code incorrectly assigning BARs?
> 
>  - R.
>

Re: mthca and LinuxBIOS

[yhlu]

pci_restore_bars cause that.
it didn't restore that according to if resource is 64 bit or not. So
it overwirte upper 32 bit with 0.

YH

file:1b34fc56067ed8ae0ba9b32f46679e13068bb86c ->
file:65ea7d25f6911d7396e19afbf4bb2738906376f7
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -222,6 +222,37 @@ pci_find_parent_resource(const struct pc
}
/**
+ * pci_restore_bars - restore a devices BAR values (e.g. after wake-up)
+ * @dev: PCI device to have its BARs restored
+ *
+ * Restore the BAR values for a given device, so as to make it
+ * accessible by its driver.
+ */
+void
+pci_restore_bars(struct pci_dev *dev)
+{
+ int i, numres;
+
+ switch (dev->hdr_type) {
+ case PCI_HEADER_TYPE_NORMAL:
+ numres = 6;
+ break;
+ case PCI_HEADER_TYPE_BRIDGE:
+ numres = 2;
+ break;
+ case PCI_HEADER_TYPE_CARDBUS:
+ numres = 1;
+ break;
+ default:
+ /* Should never get here, but just in case... */
+ return;
+ }
+
+ for (i = 0; i < numres; i ++)
+ pci_update_resource(dev, &dev->resource[i], i);
+}
+
+/**

On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote:
> before I do the cg-update this morning, it didn't mask out the upper 8 bit.
> 
> YH
> 
> On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote:
> >     yhlu> ps.  some kernel pci code patch broke sth yesterday night.
> >     yhlu> it mask out bit [32-39]
> >
> > Is it possible that all your problems are coming from the PCI setup
> > code incorrectly assigning BARs?
> >
> >  - R.
> >
>

Re: mthca and LinuxBIOS

[yhlu]

in drivers/pci/setup-res.c: pci_update_resource()

why????
                new = 0; /* currently everyone zeros the high address */

        if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) ==
            (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) {
                new = 0; /* currently everyone zeros the high address */
                pci_write_config_dword(dev, reg + 4, new);
                pci_read_config_dword(dev, reg + 4, &check);
                if (check != new) {
                        printk(KERN_ERR "PCI: Error updating region "
                               "%s/%d (high %08x != %08x)\n",
                               pci_name(dev), resno, new, check);
                }
        }


On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote:
> pci_restore_bars cause that.
> it didn't restore that according to if resource is 64 bit or not. So
> it overwirte upper 32 bit with 0.
> 
> YH
> 
> file:1b34fc56067ed8ae0ba9b32f46679e13068bb86c ->
> file:65ea7d25f6911d7396e19afbf4bb2738906376f7
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -222,6 +222,37 @@ pci_find_parent_resource(const struct pc
> }
> /**
> + * pci_restore_bars - restore a devices BAR values (e.g. after wake-up)
> + * @dev: PCI device to have its BARs restored
> + *
> + * Restore the BAR values for a given device, so as to make it
> + * accessible by its driver.
> + */
> +void
> +pci_restore_bars(struct pci_dev *dev)
> +{
> + int i, numres;
> +
> + switch (dev->hdr_type) {
> + case PCI_HEADER_TYPE_NORMAL:
> + numres = 6;
> + break;
> + case PCI_HEADER_TYPE_BRIDGE:
> + numres = 2;
> + break;
> + case PCI_HEADER_TYPE_CARDBUS:
> + numres = 1;
> + break;
> + default:
> + /* Should never get here, but just in case... */
> + return;
> + }
> +
> + for (i = 0; i < numres; i ++)
> + pci_update_resource(dev, &dev->resource[i], i);
> +}
> +
> +/**
> 
> On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote:
> > before I do the cg-update this morning, it didn't mask out the upper 8 bit.
> >
> > YH
> >
> > On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote:
> > >     yhlu> ps.  some kernel pci code patch broke sth yesterday night.
> > >     yhlu> it mask out bit [32-39]
> > >
> > > Is it possible that all your problems are coming from the PCI setup
> > > code incorrectly assigning BARs?
> > >
> > >  - R.
> > >
> >
>

Re: mthca and LinuxBIOS

[yhlu]

please check the patch for fix overwrite upper 32bit 

YH

--- drivers/pci/setup-res.c.orig        2005-08-05 10:08:45.000000000 -0700
+++ drivers/pci/setup-res.c     2005-08-05 13:25:06.000000000 -0700
@@ -33,6 +33,18 @@
        u32 new, check, mask;
        int reg;

+        if (resno < 6) {
+                reg = PCI_BASE_ADDRESS_0 + 4 * resno;
+                if((resno & 1)==1) {
+                        /* check if previous reg is 64 mem */
+                        pci_read_config_dword(dev, reg-4, &check );
+                        if ((check &
(PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) ==
+                           
(PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64))
+                                return;
+                }
+
+        }
+
        pcibios_resource_to_bus(dev, &region, res);

        pr_debug("  got res [%lx:%lx] bus [%lx:%lx] flags %lx for "
@@ -67,7 +79,7 @@

        if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) ==
            (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) {
-               new = 0; /* currently everyone zeros the high address */
+               new = region.start >> 32 ;
                pci_write_config_dword(dev, reg + 4, new);
                pci_read_config_dword(dev, reg + 4, &check);
                if (check != new) {

Re: mthca and LinuxBIOS

[Linus Torvalds]

Hmm.. This looks half-way sane, but too ugly for words.

I'd much rather see that when we detect a 64-bit resource, we always mark 
the next resource as being reserved some way, and then we just make 
pci_update_resource() ignore such reserved resources.

The

		if((resno & 1)==1) {
			/* check if previous reg is 64 mem */
			..

stuff is really too ugly.

Greg? Ivan?

		Linus


On Fri, 5 Aug 2005, yhlu wrote:
>
> please check the patch for fix overwrite upper 32bit 
> 
> YH
> 
> --- drivers/pci/setup-res.c.orig        2005-08-05 10:08:45.000000000 -0700
> +++ drivers/pci/setup-res.c     2005-08-05 13:25:06.000000000 -0700
> @@ -33,6 +33,18 @@
>         u32 new, check, mask;
>         int reg;
> 
> +        if (resno < 6) {
> +                reg = PCI_BASE_ADDRESS_0 + 4 * resno;
> +                if((resno & 1)==1) {
> +                        /* check if previous reg is 64 mem */
> +                        pci_read_config_dword(dev, reg-4, &check );
> +                        if ((check &
> (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) ==
> +                           
> (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64))
> +                                return;
> +                }
> +
> +        }
> +
>         pcibios_resource_to_bus(dev, &region, res);
> 
>         pr_debug("  got res [%lx:%lx] bus [%lx:%lx] flags %lx for "
> @@ -67,7 +79,7 @@
> 
>         if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) ==
>             (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) {
> -               new = 0; /* currently everyone zeros the high address */
> +               new = region.start >> 32 ;
>                 pci_write_config_dword(dev, reg + 4, new);
>                 pci_read_config_dword(dev, reg + 4, &check);
>                 if (check != new) {
> 

Re: mthca and LinuxBIOS

[Greg KH]

On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote:
> 
> Hmm.. This looks half-way sane, but too ugly for words.
> 
> I'd much rather see that when we detect a 64-bit resource, we always mark 
> the next resource as being reserved some way, and then we just make 
> pci_update_resource() ignore such reserved resources.
> 
> The
> 
> 		if((resno & 1)==1) {
> 			/* check if previous reg is 64 mem */
> 			..
> 
> stuff is really too ugly.

Yeah, that's not nice.

But what's the real problem we are trying to fix here?  I seem to have
missed that in the email thread somehow.

> Greg? Ivan?

Ivan's the pci resource guru, any thoughts as to how to do this in a
nicer way?

thanks,

greg k-h

Re: mthca and LinuxBIOS

[yhlu]

In LinuxBIOS, We can allocate 64 bit region ( 0xfc0000000....) to the
mellanox Infiniband card. Some range above 4G.  So the mmio below 4G
is some smaller only 128M, Otherwise need 512M. If 4 IB cards are
used, the mmio will be 2G. For new opteron E stepping, We could use
hareware memhole support. But if the CPU is before opteron E, We only
can use SW mem mapping ( will lose some performance) or lose (2G RAM).
at such case We need 64bit pref mem. We only lose 128M even four IB
card are installed.

yesterday, someone add pci_restore_bars...., that will call
pci_update_resource, and it will overwirte upper 32 bit of BAR2 and
BAR4 of IB card.

So the patch make pci_restore_resource
1. don't touch BAR3, and BAR5, if BAR2, and BAR4 are 64 bit MEM IO
2. not assume BAR2 and BAR4 upper 32bit is 0 if if BAR2, and BAR4 are
64 bit MEM IO

YH



On 8/5/05, Greg KH <gregkh@suse.de> wrote:
> On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote:
> >
> > Hmm.. This looks half-way sane, but too ugly for words.
> >
> > I'd much rather see that when we detect a 64-bit resource, we always mark
> > the next resource as being reserved some way, and then we just make
> > pci_update_resource() ignore such reserved resources.
> >
> > The
> >
> >               if((resno & 1)==1) {
> >                       /* check if previous reg is 64 mem */
> >                       ..
> >
> > stuff is really too ugly.
> 
> Yeah, that's not nice.
> 
> But what's the real problem we are trying to fix here?  I seem to have
> missed that in the email thread somehow.
> 
> > Greg? Ivan?
> 
> Ivan's the pci resource guru, any thoughts as to how to do this in a
> nicer way?
> 
> thanks,
> 
> greg k-h
>

Re: mthca and LinuxBIOS

[Greg KH]

On Fri, Aug 05, 2005 at 03:25:02PM -0700, yhlu wrote:
> In LinuxBIOS, We can allocate 64 bit region ( 0xfc0000000....) to the
> mellanox Infiniband card. Some range above 4G.  So the mmio below 4G
> is some smaller only 128M, Otherwise need 512M. If 4 IB cards are
> used, the mmio will be 2G. For new opteron E stepping, We could use
> hareware memhole support. But if the CPU is before opteron E, We only
> can use SW mem mapping ( will lose some performance) or lose (2G RAM).
> at such case We need 64bit pref mem. We only lose 128M even four IB
> card are installed.
> 
> yesterday, someone add pci_restore_bars...., that will call
> pci_update_resource, and it will overwirte upper 32 bit of BAR2 and
> BAR4 of IB card.

Hm, perhaps that change should not do this?

Dominik, care to weigh in here?  That was your patch...

> So the patch make pci_restore_resource
> 1. don't touch BAR3, and BAR5, if BAR2, and BAR4 are 64 bit MEM IO
> 2. not assume BAR2 and BAR4 upper 32bit is 0 if if BAR2, and BAR4 are
> 64 bit MEM IO


thanks,

greg k-h

Re: mthca and LinuxBIOS

[Linus Torvalds]

On Fri, 5 Aug 2005, Greg KH wrote:
> On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote:
> 
> But what's the real problem we are trying to fix here?

We're screwing up the top 32 bits of the BAR when you resume it. Look at
the patch, you'll see the fix (the other part of the patch looks fine, but
then in order to not overwrite the upper bits with zero again when doing
the _next_ - nonexistent - BAR update, we need to have something that 
avoids writing the next BAR).

Remember: a 64-bit BAR puts the upper 32 bits in what would otherwise be 
the low 32 bits of the next BAR. Which is why we need to mark the next BAR 
resource as _not_ being valid some way - so that we don't try to 
(incorrectly) "restore" it and overwrite the high bits of the previous 
BAR.

Of course, this only hits the (very few) people who not only have 64-bit 
PCI devices, but literally have them mapped in the 4GB+ region. Quite 
uncommon.

		Linus

Re: [openib-general] Re: mthca and LinuxBIOS

[Grant Grundler]

On Fri, Aug 05, 2005 at 04:06:06PM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 5 Aug 2005, Greg KH wrote:
> > On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote:
> > 
> > But what's the real problem we are trying to fix here?
> 
> We're screwing up the top 32 bits of the BAR when you resume it. Look at
> the patch, you'll see the fix (the other part of the patch looks fine, but
> then in order to not overwrite the upper bits with zero again when doing
> the _next_ - nonexistent - BAR update, we need to have something that 
> avoids writing the next BAR).

ISTR making comments before about the offending patch on linux-pci mailing
list.  Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ?
That's not true for 64-bit bars.

> Remember: a 64-bit BAR puts the upper 32 bits in what would otherwise be 
> the low 32 bits of the next BAR. Which is why we need to mark the next BAR 
> resource as _not_ being valid some way - so that we don't try to 
> (incorrectly) "restore" it and overwrite the high bits of the previous 
> BAR.

> Of course, this only hits the (very few) people who not only have 64-bit 
> PCI devices, but literally have them mapped in the 4GB+ region.

*lots* of PCI device now have 64-bit BAR.
The first I'm aware of was LSI 53c896 card (Ultra 2 SCSI).

> Quite uncommon.

Assigning 4GB+ regions is uncommon because too often
either the HW, the OS, or the driver would break.
firmware keeps having to worry about legacy OSs.

grant

Re: mthca and LinuxBIOS

[yhlu]

Roland,

what is the -16 mean?

is it
        /* Attempt to modify a QP/EE which is not in the presumed state: */
        MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10,

YH

On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote:
> You are right. CONG_SPECIAL_QP
> 
> ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005)
> ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex
> (Tavor compatibility mode) (0000:04:00.0)
> ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64
> ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff)
> ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000,
> end fcefffffff)
> ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256
> ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64
> ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64
> ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16
> ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1
> ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0
> ib_mthca 0000:04:00.0: Flags: 00370347
> ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x      fce0000000 (size 0x 4000000)
> ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x      fce4000000 (size 0x 1000000)
> ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x      fce5000000 (size 0x  800000)
> ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x      fce5800000 (size 0x  800000)
> ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x      fce6000000 (size 0x  400000)
> ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x      fce6400000 (size 0x  200000)
> ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x      fce6600000 (size 0x  100000)
> ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x      fce6700000 (size 0x   80000)
> ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x      fce6780000 (size 0x   10000)
> ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x      fce6790000 (size 0x     800)
> ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB
> (149950 KB free)
> ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries
> ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries
> ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries
> ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2
> ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3
> ib_mthca 0000:04:00.0: NOP command IRQ test passed
> ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP
> failed for 0/1024 (-16)
> ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting.
> ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2
> ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3
> ib_mthca: probe of 0000:04:00.0 failed with error -16
>

Re: mthca and LinuxBIOS

[Roland Dreier]

    yhlu> Roland, what is the -16 mean?

    yhlu> is it /* Attempt to modify a QP/EE which is not in the
    yhlu> presumed state: */ MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10,

No, -16 is just -EBUSY.  You could put a printk in event_timeout() in
mthca_cmd.c to make sure, but I'm pretty sure that's where it's coming
from.  In other words we issue the CONF_SPECIAL_QP firmware command
and don't ever get a response back from the HCA.

 - R.

Re: mthca and LinuxBIOS

[yhlu]

I remember last year when I used IBGOLD 0.5 with PCI-X IB card, it
seems that it could support 64 bit pref mem.

I will try IBGOLD 1.7 .....

YH

On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote:
>    yhlu> Roland, what is the -16 mean?
> 
>    yhlu> is it /* Attempt to modify a QP/EE which is not in the
>    yhlu> presumed state: */ MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10,
> 
> No, -16 is just -EBUSY.  You could put a printk in event_timeout() in
> mthca_cmd.c to make sure, but I'm pretty sure that's where it's coming
> from.  In other words we issue the CONF_SPECIAL_QP firmware command
> and don't ever get a response back from the HCA.
> 
>  - R.
>

Re: [openib-general] Re: mthca and LinuxBIOS

[Grant Grundler]

On Fri, Aug 05, 2005 at 04:59:37PM -0700, Grant Grundler wrote:
> ISTR making comments before about the offending patch on linux-pci mailing
> list.  Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ?

I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i].
not that the two are equivalent.

grant

Re: [openib-general] Re: mthca and LinuxBIOS

[Grant Grundler]

On Fri, Aug 05, 2005 at 04:03:00PM -0700, Greg KH wrote:
...
> > yesterday, someone add pci_restore_bars...., that will call
> > pci_update_resource, and it will overwirte upper 32 bit of BAR2 and
> > BAR4 of IB card.
> 
> Hm, perhaps that change should not do this?
> 
> Dominik, care to weigh in here?  That was your patch...

Was the origin of this patch the following thread started
by John Linville:
	http://lkml.org/lkml/2005/6/23/257

I pointed out it would have issues with 64-bit BARs.
And I suggested some solutions to JohnL's patch here:
	http://lkml.org/lkml/2005/7/2/14

In any case same issues apply to pci_update_resource().

grant

Re: [openib-general] Re: mthca and LinuxBIOS

[yhlu]

In LinuxBIOS internal structure for resource, We have index member in resource.

So the resource will be count from 0, ....7 or etc, but index member
will point to real BAR position.

I would like to see Kernel has simmliar definintion.
in LinuxBIOS
typedef uint64_t resource_t;
struct resource {
	resource_t base;	/* Base address of the resource */
	resource_t size;	/* Size of the resource */
	resource_t limit;	/* Largest valid value base + size -1 */
	unsigned long flags;	/* Descriptions of the kind of resource */
	unsigned long index;	/* Bus specific per device resource id */
	unsigned char align;	/* Required alignment (log 2) of the resource */
	unsigned char gran;	/* Granularity (log 2) of the resource */
	/* Alignment must be >= the granularity of the resource */
};



YH

On 8/5/05, Grant Grundler <iod00d@hp.com> wrote:
> On Fri, Aug 05, 2005 at 04:59:37PM -0700, Grant Grundler wrote:
> > ISTR making comments before about the offending patch on linux-pci mailing
> > list.  Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ?
> 
> I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i].
> not that the two are equivalent.
> 
> grant
>

Re: [openib-general] Re: mthca and LinuxBIOS

[Ivan Kokshaysky]

On Fri, Aug 05, 2005 at 09:33:54PM -0700, Grant Grundler wrote:
> > ISTR making comments before about the offending patch on linux-pci mailing
> > list.  Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ?
> 
> I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i].
> not that the two are equivalent.

This is correct assumption. For 64-bit BAR[i] only pci_dev->resource[i] is
valid, pci_dev->resource[i+1] slot is unused and contains zeroes in all
fields. So all we need is just to check that we're going to update a _valid_
resource.
[Though, if we ever want to support >4Gb bus allocations on 32-bit
architectures we need to make resource start and end fields u64.]

Ivan.

--- 2.6.13-rc5-git4/drivers/pci/setup-res.c	Sun Aug  7 12:08:23 2005
+++ linux/drivers/pci/setup-res.c	Sun Aug  7 13:27:54 2005
@@ -33,6 +33,11 @@ pci_update_resource(struct pci_dev *dev,
 	u32 new, check, mask;
 	int reg;
 
+	/* Ignore resources for unimplemented BARs and unused resource slots
+	   for 64 bit BARs. */
+	if (!res->flags)
+		return;
+
 	pcibios_resource_to_bus(dev, &region, res);
 
 	pr_debug("  got res [%lx:%lx] bus [%lx:%lx] flags %lx for "
@@ -67,7 +72,7 @@ pci_update_resource(struct pci_dev *dev,
 
 	if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) ==
 	    (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) {
-		new = 0; /* currently everyone zeros the high address */
+		new = region.start >> 32;
 		pci_write_config_dword(dev, reg + 4, new);
 		pci_read_config_dword(dev, reg + 4, &check);
 		if (check != new) {

Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes

[Michael S. Tsirkin]

Quoting r. yhlu <yhlu.kernel@gmail.com>:
> On 8/3/05, Michael S. Tsirkin <mst@mellanox.co.il> wrote:
> > Quoting yhlu <yhlu.kernel@gmail.com>:
> > > Subject: Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes
> > >
> > > Roland,
> > >
> > > In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB
> > > driver in Kernel can not be loaded.

Could you please test with latest firmware 4.7.0?
Thanks,

-- 
MST