* [PATCH 0/2] REALLY final InfiniBand updates for 2.6.13
@ 2005-07-28 20:31 Roland Dreier
2005-07-28 20:31 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Roland Dreier
0 siblings, 1 reply; 37+ messages in thread
From: Roland Dreier @ 2005-07-28 20:31 UTC (permalink / raw)
To: akpm; +Cc: linux-kernel, openib-general
Here are two last-minute fixes for InfiniBand: one to avoid pain in
releasing 2.6.13 with an incorrect constant and then having to rename
the enum later, and one to fix RARP on IP-over-InfiniBand.
After Greg's scolding, I'm sending them as patches, but they're also
available in the git tree at
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus
(pending mirror propagation)
The patches update the following files:
include/ib_cm.h | 3 ++-
ulp/ipoib/ipoib_main.c | 5 +++--
2 files changed, 5 insertions(+), 3 deletions(-)
through the following changes:
commit 0dca0f7bf82face7b700890318d5550fd542cabf
Author: Hal Rosenstock <halr@voltaire.com>
Date: Thu Jul 28 13:17:26 2005 -0700
[PATCH] [IPoIB] Handle sending of unicast RARP responses
RARP replies are another valid case where IPoIB may need to send a
unicast packet with no neighbour structure.
Signed-off-by: Hal Rosenstock <halr@voltaire.com>
Signed-off-by: Roland Dreier <rolandd@cisco.com>
commit 4e38d36d88ead4e56f3155573976da84d5df18b3
Author: Roland Dreier <roland@eddore.topspincom.com>
Date: Thu Jul 28 13:16:30 2005 -0700
[PATCH] [IB/cm]: Correct CM port redirect reject codes
Reject code 24 is port and CM redirection, not just port redirection.
Port redirection alone is code 25.
Therefore we should rename code 24 to IB_CM_REJ_PORT_CM_REDIRECT and
use IB_CM_REJ_PORT_REDIRECT for code 25.
Signed-off-by: Roland Dreier <rolandd@cisco.com>
Thanks,
Roland
^ permalink raw reply [flat|nested] 37+ messages in thread* [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes 2005-07-28 20:31 [PATCH 0/2] REALLY final InfiniBand updates for 2.6.13 Roland Dreier @ 2005-07-28 20:31 ` Roland Dreier 2005-07-28 20:31 ` [PATCH 2/2] [IPoIB] Handle sending of unicast RARP responses Roland Dreier 2005-08-04 0:58 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes yhlu 0 siblings, 2 replies; 37+ messages in thread From: Roland Dreier @ 2005-07-28 20:31 UTC (permalink / raw) To: akpm; +Cc: linux-kernel, openib-general Reject code 24 is port and CM redirection, not just port redirection. Port redirection alone is code 25. Therefore we should rename code 24 to IB_CM_REJ_PORT_CM_REDIRECT and use IB_CM_REJ_PORT_REDIRECT for code 25. Signed-off-by: Roland Dreier <rolandd@cisco.com> --- drivers/infiniband/include/ib_cm.h | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) 4e38d36d88ead4e56f3155573976da84d5df18b3 diff --git a/drivers/infiniband/include/ib_cm.h b/drivers/infiniband/include/ib_cm.h --- a/drivers/infiniband/include/ib_cm.h +++ b/drivers/infiniband/include/ib_cm.h @@ -169,7 +169,8 @@ enum ib_cm_rej_reason { IB_CM_REJ_INVALID_ALT_TRAFFIC_CLASS = __constant_htons(21), IB_CM_REJ_INVALID_ALT_HOP_LIMIT = __constant_htons(22), IB_CM_REJ_INVALID_ALT_PACKET_RATE = __constant_htons(23), - IB_CM_REJ_PORT_REDIRECT = __constant_htons(24), + IB_CM_REJ_PORT_CM_REDIRECT = __constant_htons(24), + IB_CM_REJ_PORT_REDIRECT = __constant_htons(25), IB_CM_REJ_INVALID_MTU = __constant_htons(26), IB_CM_REJ_INSUFFICIENT_RESP_RESOURCES = __constant_htons(27), IB_CM_REJ_CONSUMER_DEFINED = __constant_htons(28), ^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH 2/2] [IPoIB] Handle sending of unicast RARP responses 2005-07-28 20:31 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Roland Dreier @ 2005-07-28 20:31 ` Roland Dreier 2005-08-04 0:58 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes yhlu 1 sibling, 0 replies; 37+ messages in thread From: Roland Dreier @ 2005-07-28 20:31 UTC (permalink / raw) To: akpm; +Cc: linux-kernel, openib-general From: Hal Rosenstock <halr@voltaire.com> RARP replies are another valid case where IPoIB may need to send a unicast packet with no neighbour structure. Signed-off-by: Hal Rosenstock <halr@voltaire.com> Signed-off-by: Roland Dreier <rolandd@cisco.com> --- drivers/infiniband/ulp/ipoib/ipoib_main.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) 0dca0f7bf82face7b700890318d5550fd542cabf diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -600,9 +600,10 @@ static int ipoib_start_xmit(struct sk_bu ipoib_mcast_send(dev, (union ib_gid *) (phdr->hwaddr + 4), skb); } else { - /* unicast GID -- should be ARP reply */ + /* unicast GID -- should be ARP or RARP reply */ - if (be16_to_cpup((u16 *) skb->data) != ETH_P_ARP) { + if ((be16_to_cpup((__be16 *) skb->data) != ETH_P_ARP) && + (be16_to_cpup((__be16 *) skb->data) != ETH_P_RARP)) { ipoib_warn(priv, "Unicast, no %s: type %04x, QPN %06x " IPOIB_GID_FMT "\n", skb->dst ? "neigh" : "dst", ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes 2005-07-28 20:31 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Roland Dreier 2005-07-28 20:31 ` [PATCH 2/2] [IPoIB] Handle sending of unicast RARP responses Roland Dreier @ 2005-08-04 0:58 ` yhlu 2005-08-04 1:39 ` [openib-general] " Grant Grundler ` (2 more replies) 1 sibling, 3 replies; 37+ messages in thread From: yhlu @ 2005-08-04 0:58 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general Roland, In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB driver in Kernel can not be loaded. YH PCI: 00:18.0 1c1 <- [0x0000001000 - 0x0000003fff] io <node 0 link 1> PCI: 00:18.0 1b9 <- [0xfce0000000 - 0xfcf07fffff] prefmem <node 0 link 1> PCI: 00:18.0 1b1 <- [0x00fc000000 - 0x00fd2fffff] mem <node 0 link 1> PCI: 01:0f.0 24 <- [0xfce0000000 - 0xfcf07fffff] bus 4 prefmem PCI: 01:0f.0 20 <- [0x00fd100000 - 0x00fd1fffff] bus 4 mem PCI: 04:00.0 10 <- [0x00fd100000 - 0x00fd1fffff] mem64 PCI: 04:00.0 18 <- [0xfcf0000000 - 0xfcf07fffff] prefmem64 PCI: 04:00.0 20 <- [0xfce0000000 - 0xfcefffffff] prefmem64 ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor c) ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. ib_mthca: probe of 0000:04:00.0 failed with error -16 ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [openib-general] Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes 2005-08-04 0:58 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes yhlu @ 2005-08-04 1:39 ` Grant Grundler 2005-08-04 4:44 ` mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes) Roland Dreier 2005-08-04 6:42 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Michael S. Tsirkin 2 siblings, 0 replies; 37+ messages in thread From: Grant Grundler @ 2005-08-04 1:39 UTC (permalink / raw) To: yhlu; +Cc: Roland Dreier, linux-kernel, openib-general On Wed, Aug 03, 2005 at 05:58:11PM -0700, yhlu wrote: > Roland, > > In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB > driver in Kernel can not be loaded. Can you provide a few more details about the configuration? o kernel version o architecture (i386 or x86-64) o post the full console output from power up? Recent email on linux-pci raised awareness that 32-bit kernel can not support 64-bit PCI MMIO addresses. struct resource (defined in include/linux/ioport.h) defines the start/end field as "unsigned long". That's only 32-bit on i386 kernels. > PCI: 04:00.0 18 <- [0xfcf0000000 - 0xfcf07fffff] prefmem64 > PCI: 04:00.0 20 <- [0xfce0000000 - 0xfcefffffff] prefmem64 I have to wonder if those BARs are truly prefetchable. Does Mellanox assume CPU is the only one to write the 3rd BAR (RAM) and the CPU implements a write-through cache (vs write back)? I'm just guessing because I don't understand exactly how the 256MB of onboard RAM is accessed. hth, grant > > ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) > ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor c) > ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. > ib_mthca: probe of 0000:04:00.0 failed with error -16 > _______________________________________________ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general And I have to wonder if those BARs truly are prefetchable. It would imply only the CPU writes them and ^ permalink raw reply [flat|nested] 37+ messages in thread
* mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes) 2005-08-04 0:58 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes yhlu 2005-08-04 1:39 ` [openib-general] " Grant Grundler @ 2005-08-04 4:44 ` Roland Dreier 2005-08-04 16:33 ` yhlu 2005-08-04 6:42 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Michael S. Tsirkin 2 siblings, 1 reply; 37+ messages in thread From: Roland Dreier @ 2005-08-04 4:44 UTC (permalink / raw) To: yhlu; +Cc: linux-kernel, openib-general yhlu> In LinuxBIOS, If I enable the prefmem64 to use real 64 yhlu> range. the IB driver in Kernel can not be loaded. What does it mean to "enable the prefmem64 to use real 64 range"? Does the driver work if you don't do this? yhlu> ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. Can you add printk()s to mthca_qp.c::mthca_init_qp_table() to find out how far the function gets before it fails? It would also be useful for you to build with CONFIG_INFINIBAND_MTHCA_DEBUG=y and send the kernel output you get with that. - Roland ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes) 2005-08-04 4:44 ` mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes) Roland Dreier @ 2005-08-04 16:33 ` yhlu 2005-08-04 16:36 ` mthca and LinuxBIOS Roland Dreier 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-04 16:33 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general YES. I will send you the output message later about "CONFIG_INFINIBAND_MTHCA_DEBUG=y" YH On 8/3/05, Roland Dreier <rolandd@cisco.com> wrote: > yhlu> In LinuxBIOS, If I enable the prefmem64 to use real 64 > yhlu> range. the IB driver in Kernel can not be loaded. > > What does it mean to "enable the prefmem64 to use real 64 range"? > > Does the driver work if you don't do this? > > yhlu> ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. > > Can you add printk()s to mthca_qp.c::mthca_init_qp_table() to find out > how far the function gets before it fails? > > It would also be useful for you to build with CONFIG_INFINIBAND_MTHCA_DEBUG=y > and send the kernel output you get with that. > > - Roland > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-04 16:33 ` yhlu @ 2005-08-04 16:36 ` Roland Dreier 2005-08-04 17:23 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: Roland Dreier @ 2005-08-04 16:36 UTC (permalink / raw) To: yhlu; +Cc: linux-kernel, openib-general >>>>> "yhlu" == yhlu <yhlu.kernel@gmail.com> writes: yhlu> YES. I will send you the output message later about yhlu> "CONFIG_INFINIBAND_MTHCA_DEBUG=y" Thanks. In the meantime, can you explain what it means to "enable the prefmem64 to use real 64 range"? What is the difference between this and the configuration that works? - R. ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-04 16:36 ` mthca and LinuxBIOS Roland Dreier @ 2005-08-04 17:23 ` yhlu 2005-08-04 18:01 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-04 17:23 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general The mellanox can use prefmem64, but the BIOS could only allocate the some range below 4G, So 32 bit OS still can use the IB cards. but for 64bit OS, We could allocate range above 4G (0xfc00000000), So the mmio below 4G can be smaller. ( for example from 512M to 128M, the user can get back some RAM back if Opteron don't have hardware memhole support). YH On 8/4/05, Roland Dreier <rolandd@cisco.com> wrote: > >>>>> "yhlu" == yhlu <yhlu.kernel@gmail.com> writes: > > yhlu> YES. I will send you the output message later about > yhlu> "CONFIG_INFINIBAND_MTHCA_DEBUG=y" > > Thanks. In the meantime, can you explain what it means to "enable the > prefmem64 to use real 64 range"? What is the difference between this > and the configuration that works? > > - R. > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-04 17:23 ` yhlu @ 2005-08-04 18:01 ` yhlu 2005-08-04 18:35 ` Roland Dreier 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-04 18:01 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any more debug info, is that depend other setting? YH On 8/4/05, yhlu <yhlu.kernel@gmail.com> wrote: > The mellanox can use prefmem64, but the BIOS could only allocate the > some range below 4G, So 32 bit OS still can use the IB cards. > but for 64bit OS, We could allocate range above 4G (0xfc00000000), So > the mmio below 4G can be smaller. ( for example from 512M to 128M, the > user can get back some RAM back if Opteron don't have hardware memhole > support). > > YH > > > > On 8/4/05, Roland Dreier <rolandd@cisco.com> wrote: > > >>>>> "yhlu" == yhlu <yhlu.kernel@gmail.com> writes: > > > > yhlu> YES. I will send you the output message later about > > yhlu> "CONFIG_INFINIBAND_MTHCA_DEBUG=y" > > > > Thanks. In the meantime, can you explain what it means to "enable the > > prefmem64 to use real 64 range"? What is the difference between this > > and the configuration that works? > > > > - R. > > > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-04 18:01 ` yhlu @ 2005-08-04 18:35 ` Roland Dreier 2005-08-04 19:30 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: Roland Dreier @ 2005-08-04 18:35 UTC (permalink / raw) To: yhlu; +Cc: linux-kernel, openib-general yhlu> i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any yhlu> more debug info, is that depend other setting? It shouldn't depend on anything. mthca_dbg() gets turned into dev_dbg(), which just does dev_printk(KERN_DEBUG,...). Perhaps you have to change your console level to see KERN_DEBUG messages? Since you're getting to the call to mthca_init_qp_table(), there are mthca_dbg() calls that you should definitely be getting output from. - R. ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-04 18:35 ` Roland Dreier @ 2005-08-04 19:30 ` yhlu 2005-08-05 3:47 ` Roland Dreier 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-04 19:30 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (0000:04:00.0) ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64 ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff) ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000, end fcefffffff) ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256 ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64 ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64 ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16 ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1 ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0 ib_mthca 0000:04:00.0: Flags: 00370347 ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x fce0000000 (size 0x 4000000) ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x fce4000000 (size 0x 1000000) ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x fce5000000 (size 0x 800000) ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x fce5800000 (size 0x 800000) ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x fce6000000 (size 0x 400000) ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x fce6400000 (size 0x 200000) ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x fce6600000 (size 0x 100000) ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x fce6700000 (size 0x 80000) ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x fce6780000 (size 0x 10000) ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x fce6790000 (size 0x 800) ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB (149950 KB free) ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2 ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3 ib_mthca 0000:04:00.0: NOP command IRQ test passed <------------------------------------------------------------------------------------------------------stuck 30s ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2 ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3 ib_mthca: probe of 0000:04:00.0 failed with error -16 On 8/4/05, Roland Dreier <rolandd@cisco.com> wrote: > yhlu> i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any > yhlu> more debug info, is that depend other setting? > > It shouldn't depend on anything. mthca_dbg() gets turned into > dev_dbg(), which just does dev_printk(KERN_DEBUG,...). Perhaps you > have to change your console level to see KERN_DEBUG messages? > > Since you're getting to the call to mthca_init_qp_table(), there are > mthca_dbg() calls that you should definitely be getting output from. > > - R. > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-04 19:30 ` yhlu @ 2005-08-05 3:47 ` Roland Dreier 2005-08-05 18:03 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: Roland Dreier @ 2005-08-05 3:47 UTC (permalink / raw) To: yhlu; +Cc: linux-kernel, openib-general Hmm, that output all looks fine. Can you run with the patch below to see exactly where the QP table initialization fails? (I haven't actually compiled this patch so you may have to fix a typo or two) I'm guessing that the CONF_SPECIAL_QP command is failing, but let's make sure. Thanks, Roland diff --git a/drivers/infiniband/hw/mthca/mthca_qp.c b/drivers/infiniband/hw/mthca/mthca_qp.c --- a/drivers/infiniband/hw/mthca/mthca_qp.c +++ b/drivers/infiniband/hw/mthca/mthca_qp.c @@ -2214,13 +2214,16 @@ int __devinit mthca_init_qp_table(struct (1 << 24) - 1, dev->qp_table.sqp_start + MTHCA_MAX_PORTS * 2); - if (err) + if (err) { + mthca_err(dev, "mthca_init_qp_table: mthca_alloc_init failed (%d)\n", err); return err; + } err = mthca_array_init(&dev->qp_table.qp, dev->limits.num_qps); if (err) { mthca_alloc_cleanup(&dev->qp_table.alloc); + mthca_err(dev, "mthca_init_qp_table: mthca_array_init failed (%d)\n", err); return err; } @@ -2228,8 +2231,10 @@ int __devinit mthca_init_qp_table(struct err = mthca_CONF_SPECIAL_QP(dev, i ? IB_QPT_GSI : IB_QPT_SMI, dev->qp_table.sqp_start + i * 2, &status); - if (err) + if (err) { + mthca_err(dev, "mthca_init_qp_table: mthca_CONF_SPECIAL_QP failed for %d/%d (%d)\n", i, dev->qp_table.sqp_start + i * 2, err); goto err_out; + } if (status) { mthca_warn(dev, "CONF_SPECIAL_QP returned " "status %02x, aborting.\n", ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 3:47 ` Roland Dreier @ 2005-08-05 18:03 ` yhlu 2005-08-05 18:07 ` yhlu ` (2 more replies) 0 siblings, 3 replies; 37+ messages in thread From: yhlu @ 2005-08-05 18:03 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general You are right. CONG_SPECIAL_QP ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (0000:04:00.0) ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64 ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff) ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000, end fcefffffff) ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256 ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64 ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64 ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16 ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1 ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0 ib_mthca 0000:04:00.0: Flags: 00370347 ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x fce0000000 (size 0x 4000000) ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x fce4000000 (size 0x 1000000) ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x fce5000000 (size 0x 800000) ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x fce5800000 (size 0x 800000) ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x fce6000000 (size 0x 400000) ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x fce6400000 (size 0x 200000) ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x fce6600000 (size 0x 100000) ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x fce6700000 (size 0x 80000) ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x fce6780000 (size 0x 10000) ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x fce6790000 (size 0x 800) ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB (149950 KB free) ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2 ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3 ib_mthca 0000:04:00.0: NOP command IRQ test passed ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP failed for 0/1024 (-16) ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2 ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3 ib_mthca: probe of 0000:04:00.0 failed with error -16 ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 18:03 ` yhlu @ 2005-08-05 18:07 ` yhlu 2005-08-05 18:13 ` Roland Dreier 2005-08-05 18:11 ` Roland Dreier 2005-08-06 0:57 ` yhlu 2 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-05 18:07 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general ps. some kernel pci code patch broke sth yesterday night. it mask out bit [32-39] ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x e0000000 (size 0x 4000000) ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x e4000000 (size 0x 1000000) ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x e5000000 (size 0x 800000) ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x e5800000 (size 0x 800000) ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x e6000000 (size 0x 400000) ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x e6400000 (size 0x 200000) ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x e6600000 (size 0x 100000) ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x e6700000 (size 0x 80000) ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x e6780000 (size 0x 10000) ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x e6790000 (size 0x 800) YH On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote: > You are right. CONG_SPECIAL_QP > > ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) > ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex > (Tavor compatibility mode) (0000:04:00.0) > ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64 > ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff) > ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000, > end fcefffffff) > ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256 > ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64 > ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64 > ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16 > ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1 > ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0 > ib_mthca 0000:04:00.0: Flags: 00370347 > ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x fce0000000 (size 0x 4000000) > ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x fce4000000 (size 0x 1000000) > ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x fce5000000 (size 0x 800000) > ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x fce5800000 (size 0x 800000) > ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x fce6000000 (size 0x 400000) > ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x fce6400000 (size 0x 200000) > ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x fce6600000 (size 0x 100000) > ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x fce6700000 (size 0x 80000) > ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x fce6780000 (size 0x 10000) > ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x fce6790000 (size 0x 800) > ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB > (149950 KB free) > ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries > ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries > ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries > ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2 > ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3 > ib_mthca 0000:04:00.0: NOP command IRQ test passed > ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP > failed for 0/1024 (-16) > ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. > ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2 > ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3 > ib_mthca: probe of 0000:04:00.0 failed with error -16 > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 18:07 ` yhlu @ 2005-08-05 18:13 ` Roland Dreier 2005-08-05 18:26 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: Roland Dreier @ 2005-08-05 18:13 UTC (permalink / raw) To: yhlu; +Cc: linux-kernel, openib-general yhlu> ps. some kernel pci code patch broke sth yesterday night. yhlu> it mask out bit [32-39] Is it possible that all your problems are coming from the PCI setup code incorrectly assigning BARs? - R. ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 18:13 ` Roland Dreier @ 2005-08-05 18:26 ` yhlu 2005-08-05 19:25 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-05 18:26 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general before I do the cg-update this morning, it didn't mask out the upper 8 bit. YH On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote: > yhlu> ps. some kernel pci code patch broke sth yesterday night. > yhlu> it mask out bit [32-39] > > Is it possible that all your problems are coming from the PCI setup > code incorrectly assigning BARs? > > - R. > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 18:26 ` yhlu @ 2005-08-05 19:25 ` yhlu 2005-08-05 19:45 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-05 19:25 UTC (permalink / raw) To: Roland Dreier, linville, gregkh, torvalds; +Cc: linux-kernel, openib-general pci_restore_bars cause that. it didn't restore that according to if resource is 64 bit or not. So it overwirte upper 32 bit with 0. YH file:1b34fc56067ed8ae0ba9b32f46679e13068bb86c -> file:65ea7d25f6911d7396e19afbf4bb2738906376f7 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -222,6 +222,37 @@ pci_find_parent_resource(const struct pc } /** + * pci_restore_bars - restore a devices BAR values (e.g. after wake-up) + * @dev: PCI device to have its BARs restored + * + * Restore the BAR values for a given device, so as to make it + * accessible by its driver. + */ +void +pci_restore_bars(struct pci_dev *dev) +{ + int i, numres; + + switch (dev->hdr_type) { + case PCI_HEADER_TYPE_NORMAL: + numres = 6; + break; + case PCI_HEADER_TYPE_BRIDGE: + numres = 2; + break; + case PCI_HEADER_TYPE_CARDBUS: + numres = 1; + break; + default: + /* Should never get here, but just in case... */ + return; + } + + for (i = 0; i < numres; i ++) + pci_update_resource(dev, &dev->resource[i], i); +} + +/** On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote: > before I do the cg-update this morning, it didn't mask out the upper 8 bit. > > YH > > On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote: > > yhlu> ps. some kernel pci code patch broke sth yesterday night. > > yhlu> it mask out bit [32-39] > > > > Is it possible that all your problems are coming from the PCI setup > > code incorrectly assigning BARs? > > > > - R. > > > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 19:25 ` yhlu @ 2005-08-05 19:45 ` yhlu 2005-08-05 20:28 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-05 19:45 UTC (permalink / raw) To: Roland Dreier, linville, gregkh, torvalds; +Cc: linux-kernel, openib-general in drivers/pci/setup-res.c: pci_update_resource() why???? new = 0; /* currently everyone zeros the high address */ if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) == (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) { new = 0; /* currently everyone zeros the high address */ pci_write_config_dword(dev, reg + 4, new); pci_read_config_dword(dev, reg + 4, &check); if (check != new) { printk(KERN_ERR "PCI: Error updating region " "%s/%d (high %08x != %08x)\n", pci_name(dev), resno, new, check); } } On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote: > pci_restore_bars cause that. > it didn't restore that according to if resource is 64 bit or not. So > it overwirte upper 32 bit with 0. > > YH > > file:1b34fc56067ed8ae0ba9b32f46679e13068bb86c -> > file:65ea7d25f6911d7396e19afbf4bb2738906376f7 > --- a/drivers/pci/pci.c > +++ b/drivers/pci/pci.c > @@ -222,6 +222,37 @@ pci_find_parent_resource(const struct pc > } > /** > + * pci_restore_bars - restore a devices BAR values (e.g. after wake-up) > + * @dev: PCI device to have its BARs restored > + * > + * Restore the BAR values for a given device, so as to make it > + * accessible by its driver. > + */ > +void > +pci_restore_bars(struct pci_dev *dev) > +{ > + int i, numres; > + > + switch (dev->hdr_type) { > + case PCI_HEADER_TYPE_NORMAL: > + numres = 6; > + break; > + case PCI_HEADER_TYPE_BRIDGE: > + numres = 2; > + break; > + case PCI_HEADER_TYPE_CARDBUS: > + numres = 1; > + break; > + default: > + /* Should never get here, but just in case... */ > + return; > + } > + > + for (i = 0; i < numres; i ++) > + pci_update_resource(dev, &dev->resource[i], i); > +} > + > +/** > > On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote: > > before I do the cg-update this morning, it didn't mask out the upper 8 bit. > > > > YH > > > > On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote: > > > yhlu> ps. some kernel pci code patch broke sth yesterday night. > > > yhlu> it mask out bit [32-39] > > > > > > Is it possible that all your problems are coming from the PCI setup > > > code incorrectly assigning BARs? > > > > > > - R. > > > > > > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 19:45 ` yhlu @ 2005-08-05 20:28 ` yhlu 2005-08-05 20:38 ` Linus Torvalds 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-05 20:28 UTC (permalink / raw) To: Roland Dreier, linville, gregkh, torvalds; +Cc: linux-kernel, openib-general please check the patch for fix overwrite upper 32bit YH --- drivers/pci/setup-res.c.orig 2005-08-05 10:08:45.000000000 -0700 +++ drivers/pci/setup-res.c 2005-08-05 13:25:06.000000000 -0700 @@ -33,6 +33,18 @@ u32 new, check, mask; int reg; + if (resno < 6) { + reg = PCI_BASE_ADDRESS_0 + 4 * resno; + if((resno & 1)==1) { + /* check if previous reg is 64 mem */ + pci_read_config_dword(dev, reg-4, &check ); + if ((check & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) == + (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) + return; + } + + } + pcibios_resource_to_bus(dev, ®ion, res); pr_debug(" got res [%lx:%lx] bus [%lx:%lx] flags %lx for " @@ -67,7 +79,7 @@ if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) == (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) { - new = 0; /* currently everyone zeros the high address */ + new = region.start >> 32 ; pci_write_config_dword(dev, reg + 4, new); pci_read_config_dword(dev, reg + 4, &check); if (check != new) { ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 20:28 ` yhlu @ 2005-08-05 20:38 ` Linus Torvalds 2005-08-05 22:00 ` Greg KH 0 siblings, 1 reply; 37+ messages in thread From: Linus Torvalds @ 2005-08-05 20:38 UTC (permalink / raw) To: yhlu Cc: Roland Dreier, linville, Greg KH, Ivan Kokshaysky, Linux Kernel Mailing List, openib-general Hmm.. This looks half-way sane, but too ugly for words. I'd much rather see that when we detect a 64-bit resource, we always mark the next resource as being reserved some way, and then we just make pci_update_resource() ignore such reserved resources. The if((resno & 1)==1) { /* check if previous reg is 64 mem */ .. stuff is really too ugly. Greg? Ivan? Linus On Fri, 5 Aug 2005, yhlu wrote: > > please check the patch for fix overwrite upper 32bit > > YH > > --- drivers/pci/setup-res.c.orig 2005-08-05 10:08:45.000000000 -0700 > +++ drivers/pci/setup-res.c 2005-08-05 13:25:06.000000000 -0700 > @@ -33,6 +33,18 @@ > u32 new, check, mask; > int reg; > > + if (resno < 6) { > + reg = PCI_BASE_ADDRESS_0 + 4 * resno; > + if((resno & 1)==1) { > + /* check if previous reg is 64 mem */ > + pci_read_config_dword(dev, reg-4, &check ); > + if ((check & > (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) == > + > (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) > + return; > + } > + > + } > + > pcibios_resource_to_bus(dev, ®ion, res); > > pr_debug(" got res [%lx:%lx] bus [%lx:%lx] flags %lx for " > @@ -67,7 +79,7 @@ > > if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) == > (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) { > - new = 0; /* currently everyone zeros the high address */ > + new = region.start >> 32 ; > pci_write_config_dword(dev, reg + 4, new); > pci_read_config_dword(dev, reg + 4, &check); > if (check != new) { > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 20:38 ` Linus Torvalds @ 2005-08-05 22:00 ` Greg KH 2005-08-05 22:25 ` yhlu 2005-08-05 23:06 ` Linus Torvalds 0 siblings, 2 replies; 37+ messages in thread From: Greg KH @ 2005-08-05 22:00 UTC (permalink / raw) To: Linus Torvalds Cc: yhlu, Roland Dreier, linville, Ivan Kokshaysky, Linux Kernel Mailing List, openib-general On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote: > > Hmm.. This looks half-way sane, but too ugly for words. > > I'd much rather see that when we detect a 64-bit resource, we always mark > the next resource as being reserved some way, and then we just make > pci_update_resource() ignore such reserved resources. > > The > > if((resno & 1)==1) { > /* check if previous reg is 64 mem */ > .. > > stuff is really too ugly. Yeah, that's not nice. But what's the real problem we are trying to fix here? I seem to have missed that in the email thread somehow. > Greg? Ivan? Ivan's the pci resource guru, any thoughts as to how to do this in a nicer way? thanks, greg k-h ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 22:00 ` Greg KH @ 2005-08-05 22:25 ` yhlu 2005-08-05 23:03 ` Greg KH 2005-08-05 23:06 ` Linus Torvalds 1 sibling, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-05 22:25 UTC (permalink / raw) To: Greg KH Cc: Linus Torvalds, Roland Dreier, linville, Ivan Kokshaysky, Linux Kernel Mailing List, openib-general In LinuxBIOS, We can allocate 64 bit region ( 0xfc0000000....) to the mellanox Infiniband card. Some range above 4G. So the mmio below 4G is some smaller only 128M, Otherwise need 512M. If 4 IB cards are used, the mmio will be 2G. For new opteron E stepping, We could use hareware memhole support. But if the CPU is before opteron E, We only can use SW mem mapping ( will lose some performance) or lose (2G RAM). at such case We need 64bit pref mem. We only lose 128M even four IB card are installed. yesterday, someone add pci_restore_bars...., that will call pci_update_resource, and it will overwirte upper 32 bit of BAR2 and BAR4 of IB card. So the patch make pci_restore_resource 1. don't touch BAR3, and BAR5, if BAR2, and BAR4 are 64 bit MEM IO 2. not assume BAR2 and BAR4 upper 32bit is 0 if if BAR2, and BAR4 are 64 bit MEM IO YH On 8/5/05, Greg KH <gregkh@suse.de> wrote: > On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote: > > > > Hmm.. This looks half-way sane, but too ugly for words. > > > > I'd much rather see that when we detect a 64-bit resource, we always mark > > the next resource as being reserved some way, and then we just make > > pci_update_resource() ignore such reserved resources. > > > > The > > > > if((resno & 1)==1) { > > /* check if previous reg is 64 mem */ > > .. > > > > stuff is really too ugly. > > Yeah, that's not nice. > > But what's the real problem we are trying to fix here? I seem to have > missed that in the email thread somehow. > > > Greg? Ivan? > > Ivan's the pci resource guru, any thoughts as to how to do this in a > nicer way? > > thanks, > > greg k-h > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 22:25 ` yhlu @ 2005-08-05 23:03 ` Greg KH 2005-08-06 5:29 ` [openib-general] " Grant Grundler 0 siblings, 1 reply; 37+ messages in thread From: Greg KH @ 2005-08-05 23:03 UTC (permalink / raw) To: yhlu, Dominik Brodowski Cc: Linus Torvalds, Roland Dreier, linville, Ivan Kokshaysky, Linux Kernel Mailing List, openib-general On Fri, Aug 05, 2005 at 03:25:02PM -0700, yhlu wrote: > In LinuxBIOS, We can allocate 64 bit region ( 0xfc0000000....) to the > mellanox Infiniband card. Some range above 4G. So the mmio below 4G > is some smaller only 128M, Otherwise need 512M. If 4 IB cards are > used, the mmio will be 2G. For new opteron E stepping, We could use > hareware memhole support. But if the CPU is before opteron E, We only > can use SW mem mapping ( will lose some performance) or lose (2G RAM). > at such case We need 64bit pref mem. We only lose 128M even four IB > card are installed. > > yesterday, someone add pci_restore_bars...., that will call > pci_update_resource, and it will overwirte upper 32 bit of BAR2 and > BAR4 of IB card. Hm, perhaps that change should not do this? Dominik, care to weigh in here? That was your patch... > So the patch make pci_restore_resource > 1. don't touch BAR3, and BAR5, if BAR2, and BAR4 are 64 bit MEM IO > 2. not assume BAR2 and BAR4 upper 32bit is 0 if if BAR2, and BAR4 are > 64 bit MEM IO thanks, greg k-h ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [openib-general] Re: mthca and LinuxBIOS 2005-08-05 23:03 ` Greg KH @ 2005-08-06 5:29 ` Grant Grundler 0 siblings, 0 replies; 37+ messages in thread From: Grant Grundler @ 2005-08-06 5:29 UTC (permalink / raw) To: Greg KH Cc: yhlu, Dominik Brodowski, linville, openib-general, Linux Kernel Mailing List, Linus Torvalds, Ivan Kokshaysky On Fri, Aug 05, 2005 at 04:03:00PM -0700, Greg KH wrote: ... > > yesterday, someone add pci_restore_bars...., that will call > > pci_update_resource, and it will overwirte upper 32 bit of BAR2 and > > BAR4 of IB card. > > Hm, perhaps that change should not do this? > > Dominik, care to weigh in here? That was your patch... Was the origin of this patch the following thread started by John Linville: http://lkml.org/lkml/2005/6/23/257 I pointed out it would have issues with 64-bit BARs. And I suggested some solutions to JohnL's patch here: http://lkml.org/lkml/2005/7/2/14 In any case same issues apply to pci_update_resource(). grant ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 22:00 ` Greg KH 2005-08-05 22:25 ` yhlu @ 2005-08-05 23:06 ` Linus Torvalds 2005-08-05 23:59 ` [openib-general] " Grant Grundler 1 sibling, 1 reply; 37+ messages in thread From: Linus Torvalds @ 2005-08-05 23:06 UTC (permalink / raw) To: Greg KH Cc: yhlu, Roland Dreier, linville, Ivan Kokshaysky, Linux Kernel Mailing List, openib-general On Fri, 5 Aug 2005, Greg KH wrote: > On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote: > > But what's the real problem we are trying to fix here? We're screwing up the top 32 bits of the BAR when you resume it. Look at the patch, you'll see the fix (the other part of the patch looks fine, but then in order to not overwrite the upper bits with zero again when doing the _next_ - nonexistent - BAR update, we need to have something that avoids writing the next BAR). Remember: a 64-bit BAR puts the upper 32 bits in what would otherwise be the low 32 bits of the next BAR. Which is why we need to mark the next BAR resource as _not_ being valid some way - so that we don't try to (incorrectly) "restore" it and overwrite the high bits of the previous BAR. Of course, this only hits the (very few) people who not only have 64-bit PCI devices, but literally have them mapped in the 4GB+ region. Quite uncommon. Linus ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [openib-general] Re: mthca and LinuxBIOS 2005-08-05 23:06 ` Linus Torvalds @ 2005-08-05 23:59 ` Grant Grundler 2005-08-06 4:33 ` Grant Grundler 0 siblings, 1 reply; 37+ messages in thread From: Grant Grundler @ 2005-08-05 23:59 UTC (permalink / raw) To: Linus Torvalds Cc: Greg KH, Linux Kernel Mailing List, openib-general, linville, Ivan Kokshaysky, yhlu On Fri, Aug 05, 2005 at 04:06:06PM -0700, Linus Torvalds wrote: > > > On Fri, 5 Aug 2005, Greg KH wrote: > > On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote: > > > > But what's the real problem we are trying to fix here? > > We're screwing up the top 32 bits of the BAR when you resume it. Look at > the patch, you'll see the fix (the other part of the patch looks fine, but > then in order to not overwrite the upper bits with zero again when doing > the _next_ - nonexistent - BAR update, we need to have something that > avoids writing the next BAR). ISTR making comments before about the offending patch on linux-pci mailing list. Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ? That's not true for 64-bit bars. > Remember: a 64-bit BAR puts the upper 32 bits in what would otherwise be > the low 32 bits of the next BAR. Which is why we need to mark the next BAR > resource as _not_ being valid some way - so that we don't try to > (incorrectly) "restore" it and overwrite the high bits of the previous > BAR. > Of course, this only hits the (very few) people who not only have 64-bit > PCI devices, but literally have them mapped in the 4GB+ region. *lots* of PCI device now have 64-bit BAR. The first I'm aware of was LSI 53c896 card (Ultra 2 SCSI). > Quite uncommon. Assigning 4GB+ regions is uncommon because too often either the HW, the OS, or the driver would break. firmware keeps having to worry about legacy OSs. grant ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [openib-general] Re: mthca and LinuxBIOS 2005-08-05 23:59 ` [openib-general] " Grant Grundler @ 2005-08-06 4:33 ` Grant Grundler 2005-08-06 23:52 ` yhlu 2005-08-07 9:49 ` Ivan Kokshaysky 0 siblings, 2 replies; 37+ messages in thread From: Grant Grundler @ 2005-08-06 4:33 UTC (permalink / raw) To: Grant Grundler Cc: Linus Torvalds, Greg KH, Linux Kernel Mailing List, openib-general, linville, Ivan Kokshaysky, yhlu On Fri, Aug 05, 2005 at 04:59:37PM -0700, Grant Grundler wrote: > ISTR making comments before about the offending patch on linux-pci mailing > list. Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ? I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i]. not that the two are equivalent. grant ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [openib-general] Re: mthca and LinuxBIOS 2005-08-06 4:33 ` Grant Grundler @ 2005-08-06 23:52 ` yhlu 2005-08-07 9:49 ` Ivan Kokshaysky 1 sibling, 0 replies; 37+ messages in thread From: yhlu @ 2005-08-06 23:52 UTC (permalink / raw) To: Grant Grundler Cc: Linus Torvalds, Greg KH, Linux Kernel Mailing List, openib-general, linville, Ivan Kokshaysky In LinuxBIOS internal structure for resource, We have index member in resource. So the resource will be count from 0, ....7 or etc, but index member will point to real BAR position. I would like to see Kernel has simmliar definintion. in LinuxBIOS typedef uint64_t resource_t; struct resource { resource_t base; /* Base address of the resource */ resource_t size; /* Size of the resource */ resource_t limit; /* Largest valid value base + size -1 */ unsigned long flags; /* Descriptions of the kind of resource */ unsigned long index; /* Bus specific per device resource id */ unsigned char align; /* Required alignment (log 2) of the resource */ unsigned char gran; /* Granularity (log 2) of the resource */ /* Alignment must be >= the granularity of the resource */ }; YH On 8/5/05, Grant Grundler <iod00d@hp.com> wrote: > On Fri, Aug 05, 2005 at 04:59:37PM -0700, Grant Grundler wrote: > > ISTR making comments before about the offending patch on linux-pci mailing > > list. Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ? > > I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i]. > not that the two are equivalent. > > grant > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [openib-general] Re: mthca and LinuxBIOS 2005-08-06 4:33 ` Grant Grundler 2005-08-06 23:52 ` yhlu @ 2005-08-07 9:49 ` Ivan Kokshaysky 1 sibling, 0 replies; 37+ messages in thread From: Ivan Kokshaysky @ 2005-08-07 9:49 UTC (permalink / raw) To: Grant Grundler Cc: Linus Torvalds, Greg KH, Linux Kernel Mailing List, openib-general, linville, yhlu On Fri, Aug 05, 2005 at 09:33:54PM -0700, Grant Grundler wrote: > > ISTR making comments before about the offending patch on linux-pci mailing > > list. Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ? > > I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i]. > not that the two are equivalent. This is correct assumption. For 64-bit BAR[i] only pci_dev->resource[i] is valid, pci_dev->resource[i+1] slot is unused and contains zeroes in all fields. So all we need is just to check that we're going to update a _valid_ resource. [Though, if we ever want to support >4Gb bus allocations on 32-bit architectures we need to make resource start and end fields u64.] Ivan. --- 2.6.13-rc5-git4/drivers/pci/setup-res.c Sun Aug 7 12:08:23 2005 +++ linux/drivers/pci/setup-res.c Sun Aug 7 13:27:54 2005 @@ -33,6 +33,11 @@ pci_update_resource(struct pci_dev *dev, u32 new, check, mask; int reg; + /* Ignore resources for unimplemented BARs and unused resource slots + for 64 bit BARs. */ + if (!res->flags) + return; + pcibios_resource_to_bus(dev, ®ion, res); pr_debug(" got res [%lx:%lx] bus [%lx:%lx] flags %lx for " @@ -67,7 +72,7 @@ pci_update_resource(struct pci_dev *dev, if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) == (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) { - new = 0; /* currently everyone zeros the high address */ + new = region.start >> 32; pci_write_config_dword(dev, reg + 4, new); pci_read_config_dword(dev, reg + 4, &check); if (check != new) { ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 18:03 ` yhlu 2005-08-05 18:07 ` yhlu @ 2005-08-05 18:11 ` Roland Dreier 2005-08-06 0:57 ` yhlu 2 siblings, 0 replies; 37+ messages in thread From: Roland Dreier @ 2005-08-05 18:11 UTC (permalink / raw) To: yhlu, mst; +Cc: linux-kernel, openib-general > ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64 This is FW 4.6.2 -- 4.7.0 has been released, so it might be worth trying that. > ib_mthca 0000:04:00.0: NOP command IRQ test passed > ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP failed for 0/1024 (-16) Hmm, looks like CONF_SPECIAL_QP is timing out. MST (or any Mellanox people), any idea why this might happening? The NOP command is working fine with interrupts, but CONF_SPECIAL_QP is timing out. The difference from the working setup is that the HCA's local memory is mapped above 4 GB. - R. ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-05 18:03 ` yhlu 2005-08-05 18:07 ` yhlu 2005-08-05 18:11 ` Roland Dreier @ 2005-08-06 0:57 ` yhlu 2005-08-06 1:30 ` Roland Dreier 2 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-06 0:57 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general Roland, what is the -16 mean? is it /* Attempt to modify a QP/EE which is not in the presumed state: */ MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10, YH On 8/5/05, yhlu <yhlu.kernel@gmail.com> wrote: > You are right. CONG_SPECIAL_QP > > ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) > ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex > (Tavor compatibility mode) (0000:04:00.0) > ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64 > ib_mthca 0000:04:00.0: FW size 6143 KB (start fcefa00000, end fcefffffff) > ib_mthca 0000:04:00.0: HCA memory size 262143 KB (start fce0000000, > end fcefffffff) > ib_mthca 0000:04:00.0: Max QPs: 16777216, reserved QPs: 1024, entry size: 256 > ib_mthca 0000:04:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64 > ib_mthca 0000:04:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64 > ib_mthca 0000:04:00.0: reserved MPTs: 16, reserved MTTs: 16 > ib_mthca 0000:04:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1 > ib_mthca 0000:04:00.0: Max QP/MCG: 16777216, reserved MGMs: 0 > ib_mthca 0000:04:00.0: Flags: 00370347 > ib_mthca 0000:04:00.0: profile[ 0]--10/20 @ 0x fce0000000 (size 0x 4000000) > ib_mthca 0000:04:00.0: profile[ 1]-- 0/16 @ 0x fce4000000 (size 0x 1000000) > ib_mthca 0000:04:00.0: profile[ 2]-- 7/18 @ 0x fce5000000 (size 0x 800000) > ib_mthca 0000:04:00.0: profile[ 3]-- 9/17 @ 0x fce5800000 (size 0x 800000) > ib_mthca 0000:04:00.0: profile[ 4]-- 3/16 @ 0x fce6000000 (size 0x 400000) > ib_mthca 0000:04:00.0: profile[ 5]-- 4/16 @ 0x fce6400000 (size 0x 200000) > ib_mthca 0000:04:00.0: profile[ 6]--12/15 @ 0x fce6600000 (size 0x 100000) > ib_mthca 0000:04:00.0: profile[ 7]-- 8/13 @ 0x fce6700000 (size 0x 80000) > ib_mthca 0000:04:00.0: profile[ 8]--11/11 @ 0x fce6780000 (size 0x 10000) > ib_mthca 0000:04:00.0: profile[ 9]-- 6/ 5 @ 0x fce6790000 (size 0x 800) > ib_mthca 0000:04:00.0: HCA memory: allocated 106050 KB/256000 KB > (149950 KB free) > ib_mthca 0000:04:00.0: Allocated EQ 1 with 65536 entries > ib_mthca 0000:04:00.0: Allocated EQ 2 with 128 entries > ib_mthca 0000:04:00.0: Allocated EQ 3 with 128 entries > ib_mthca 0000:04:00.0: Setting mask 00000000000f43fe for eqn 2 > ib_mthca 0000:04:00.0: Setting mask 0000000000000400 for eqn 3 > ib_mthca 0000:04:00.0: NOP command IRQ test passed > ib_mthca 0000:04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP > failed for 0/1024 (-16) > ib_mthca 0000:04:00.0: Failed to initialize queue pair table, aborting. > ib_mthca 0000:04:00.0: Clearing mask 00000000000f43fe for eqn 2 > ib_mthca 0000:04:00.0: Clearing mask 0000000000000400 for eqn 3 > ib_mthca: probe of 0000:04:00.0 failed with error -16 > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-06 0:57 ` yhlu @ 2005-08-06 1:30 ` Roland Dreier 2005-08-06 2:47 ` yhlu 0 siblings, 1 reply; 37+ messages in thread From: Roland Dreier @ 2005-08-06 1:30 UTC (permalink / raw) To: yhlu; +Cc: linux-kernel, openib-general yhlu> Roland, what is the -16 mean? yhlu> is it /* Attempt to modify a QP/EE which is not in the yhlu> presumed state: */ MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10, No, -16 is just -EBUSY. You could put a printk in event_timeout() in mthca_cmd.c to make sure, but I'm pretty sure that's where it's coming from. In other words we issue the CONF_SPECIAL_QP firmware command and don't ever get a response back from the HCA. - R. ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: mthca and LinuxBIOS 2005-08-06 1:30 ` Roland Dreier @ 2005-08-06 2:47 ` yhlu 0 siblings, 0 replies; 37+ messages in thread From: yhlu @ 2005-08-06 2:47 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general I remember last year when I used IBGOLD 0.5 with PCI-X IB card, it seems that it could support 64 bit pref mem. I will try IBGOLD 1.7 ..... YH On 8/5/05, Roland Dreier <rolandd@cisco.com> wrote: > yhlu> Roland, what is the -16 mean? > > yhlu> is it /* Attempt to modify a QP/EE which is not in the > yhlu> presumed state: */ MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10, > > No, -16 is just -EBUSY. You could put a printk in event_timeout() in > mthca_cmd.c to make sure, but I'm pretty sure that's where it's coming > from. In other words we issue the CONF_SPECIAL_QP firmware command > and don't ever get a response back from the HCA. > > - R. > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes 2005-08-04 0:58 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes yhlu 2005-08-04 1:39 ` [openib-general] " Grant Grundler 2005-08-04 4:44 ` mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes) Roland Dreier @ 2005-08-04 6:42 ` Michael S. Tsirkin 2005-08-04 18:22 ` yhlu 2 siblings, 1 reply; 37+ messages in thread From: Michael S. Tsirkin @ 2005-08-04 6:42 UTC (permalink / raw) To: yhlu; +Cc: Roland Dreier, linux-kernel, openib-general Quoting r. yhlu <yhlu.kernel@gmail.com>: > Subject: Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes > > Roland, > > In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB > driver in Kernel can not be loaded. Are you using the latest firmware on the HCA card? -- MST ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes 2005-08-04 6:42 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Michael S. Tsirkin @ 2005-08-04 18:22 ` yhlu 2005-08-08 12:21 ` Michael S. Tsirkin 0 siblings, 1 reply; 37+ messages in thread From: yhlu @ 2005-08-04 18:22 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: Roland Dreier, linux-kernel, openib-general Yes. On 8/3/05, Michael S. Tsirkin <mst@mellanox.co.il> wrote: > Quoting r. yhlu <yhlu.kernel@gmail.com>: > > Subject: Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes > > > > Roland, > > > > In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB > > driver in Kernel can not be loaded. > > Are you using the latest firmware on the HCA card? > > -- > MST > ^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes 2005-08-04 18:22 ` yhlu @ 2005-08-08 12:21 ` Michael S. Tsirkin 0 siblings, 0 replies; 37+ messages in thread From: Michael S. Tsirkin @ 2005-08-08 12:21 UTC (permalink / raw) To: yhlu; +Cc: Roland Dreier, linux-kernel, openib-general Quoting r. yhlu <yhlu.kernel@gmail.com>: > On 8/3/05, Michael S. Tsirkin <mst@mellanox.co.il> wrote: > > Quoting yhlu <yhlu.kernel@gmail.com>: > > > Subject: Re: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes > > > > > > Roland, > > > > > > In LinuxBIOS, If I enable the prefmem64 to use real 64 range. the IB > > > driver in Kernel can not be loaded. Could you please test with latest firmware 4.7.0? Thanks, -- MST ^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2005-08-08 12:21 UTC | newest] Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2005-07-28 20:31 [PATCH 0/2] REALLY final InfiniBand updates for 2.6.13 Roland Dreier 2005-07-28 20:31 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Roland Dreier 2005-07-28 20:31 ` [PATCH 2/2] [IPoIB] Handle sending of unicast RARP responses Roland Dreier 2005-08-04 0:58 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes yhlu 2005-08-04 1:39 ` [openib-general] " Grant Grundler 2005-08-04 4:44 ` mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes) Roland Dreier 2005-08-04 16:33 ` yhlu 2005-08-04 16:36 ` mthca and LinuxBIOS Roland Dreier 2005-08-04 17:23 ` yhlu 2005-08-04 18:01 ` yhlu 2005-08-04 18:35 ` Roland Dreier 2005-08-04 19:30 ` yhlu 2005-08-05 3:47 ` Roland Dreier 2005-08-05 18:03 ` yhlu 2005-08-05 18:07 ` yhlu 2005-08-05 18:13 ` Roland Dreier 2005-08-05 18:26 ` yhlu 2005-08-05 19:25 ` yhlu 2005-08-05 19:45 ` yhlu 2005-08-05 20:28 ` yhlu 2005-08-05 20:38 ` Linus Torvalds 2005-08-05 22:00 ` Greg KH 2005-08-05 22:25 ` yhlu 2005-08-05 23:03 ` Greg KH 2005-08-06 5:29 ` [openib-general] " Grant Grundler 2005-08-05 23:06 ` Linus Torvalds 2005-08-05 23:59 ` [openib-general] " Grant Grundler 2005-08-06 4:33 ` Grant Grundler 2005-08-06 23:52 ` yhlu 2005-08-07 9:49 ` Ivan Kokshaysky 2005-08-05 18:11 ` Roland Dreier 2005-08-06 0:57 ` yhlu 2005-08-06 1:30 ` Roland Dreier 2005-08-06 2:47 ` yhlu 2005-08-04 6:42 ` [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes Michael S. Tsirkin 2005-08-04 18:22 ` yhlu 2005-08-08 12:21 ` Michael S. Tsirkin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).