ib-diags: compatability issue with ibstat

ib-diags: compatability issue with ibstat

[Hefty, Sean]

commit 1344cb3feacafc462440dabfa5997c5205486d83 added support for FDR10 in a way that is not compatible with Windows support.  Windows does not use files to read attribute information.

I will probably need to obtain the necessary information using ibverbs on windows by reading port attributes.  I don't think FDR10 support is available through ibverbs on linux yet, but would this be acceptable?  Is there some other way that you'd like to handle this?  The other option I can think of is moving is_fdr10() and sys_read_string() out of ibstat.c and into a linux specific file, so that windows can provide its own implementation.  Thoughts?

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Ira Weiny]

On Wed, 21 Sep 2011 12:06:12 -0700
"Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> commit 1344cb3feacafc462440dabfa5997c5205486d83 added support for FDR10 in a way that is not compatible with Windows support.  Windows does not use files to read attribute information.
> 

Ok, I think you meant commit 1344cb3feacafc462440dabfa5997c5205486d83

> I will probably need to obtain the necessary information using ibverbs on windows by reading port attributes.  I don't think FDR10 support is available through ibverbs on linux yet, but would this be acceptable?  Is there some other way that you'd like to handle this?  The other option I can think of is moving is_fdr10() and sys_read_string() out of ibstat.c and into a linux specific file, so that windows can provide its own implementation.  Thoughts?
> 

Does this mean "ibstatus" does not work on Windows?

I was contemplating getting rid of ibstat as ibstatus (a script which uses sysfs exclusively) gave the same information without the dependency on libibumad.  With this patch we were trying to not break libibumad's ABI with this addition.

How are you proposing the addition to ibverbs?  It seems this would break ABI there.

Does windows have sysfs like capabilities which would make the separate layer clean?

Ira

> - Sean


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ib-diags: compatability issue with ibstat

[Hefty, Sean]

> Does this mean "ibstatus" does not work on Windows?

We do not support any of the scripts on windows.  As far as I could tell, the scripts look like they just do post-processing of available output.
 
> How are you proposing the addition to ibverbs?  It seems this would break ABI
> there.

On windows, libibumad uses libibverbs to obtain whatever data it needs.  I'm assuming that a non-MAD application will eventually be able to use ibverbs to determine the rate, without needing to parse data from some file.

> Does windows have sysfs like capabilities which would make the separate layer
> clean?

There's nothing like sysfs on windows.  If the data is available from ibverbs, windows could pull the data from there, even if the diags do not want to rely on ibverbs on the linux side.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Ira Weiny]

On Wed, 21 Sep 2011 15:23:45 -0700
"Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> > Does this mean "ibstatus" does not work on Windows?
> 
> We do not support any of the scripts on windows.  As far as I could tell, the scripts look like they just do post-processing of available output.
>  

Good to know, thanks.

> > How are you proposing the addition to ibverbs?  It seems this would break ABI
> > there.
> 
> On windows, libibumad uses libibverbs to obtain whatever data it needs.  I'm assuming that a non-MAD application will eventually be able to use ibverbs to determine the rate, without needing to parse data from some file.
> 
> > Does windows have sysfs like capabilities which would make the separate layer
> > clean?
> 
> There's nothing like sysfs on windows.  If the data is available from ibverbs, windows could pull the data from there, even if the diags do not want to rely on ibverbs on the linux side.
> 

To be clear I am not against using ibverbs on the linux side.  It sounds like that would be the best move going forward.

Ira

> - Sean


-- 
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Jason Gunthorpe]

On Wed, Sep 21, 2011 at 03:32:54PM -0700, Ira Weiny wrote:

> To be clear I am not against using ibverbs on the linux side.  It
> sounds like that would be the best move going forward.

Technically umad already needs to use verbs on Linux because that is
the only way to get the subnet timeout value that is needed to compute
the MAD RPC timeout. Currently umad ignores this part of IBA ...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Ira Weiny]

On Wed, 21 Sep 2011 15:32:54 -0700
Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:

> On Wed, 21 Sep 2011 15:23:45 -0700
> "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> > > Does this mean "ibstatus" does not work on Windows?
> > 
> > We do not support any of the scripts on windows.  As far as I could tell, the scripts look like they just do post-processing of available output.
> >  
> 
> Good to know, thanks.
> 
> > > How are you proposing the addition to ibverbs?  It seems this would break ABI
> > > there.
> > 
> > On windows, libibumad uses libibverbs to obtain whatever data it needs.  I'm assuming that a non-MAD application will eventually be able to use ibverbs to determine the rate, without needing to parse data from some file.
> > 
> > > Does windows have sysfs like capabilities which would make the separate layer
> > > clean?
> > 
> > There's nothing like sysfs on windows.  If the data is available from ibverbs, windows could pull the data from there, even if the diags do not want to rely on ibverbs on the linux side.
> > 
> 
> To be clear I am not against using ibverbs on the linux side.  It sounds like that would be the best move going forward.
> 

Honestly, ibverbs supplies similar functionality in ibv_devinfo.  :-/

It seems both ibstat and ibstatus should be dropped and ibv_devinfo enhanced to supply, Rate, Physical state, Capability Mask, and the transport (on a per port basis)[*].  There might be some other functionality as well.  Roland, ibv_devinfo, is in the examples directory.  Is there any reason this should not be used as the "official" tool?  If not I think we should use the code from ibv_devinfo as a basis for ibstat.  But I hate to see 2 implementations so close.

Ira

[*] There are cards which support this.  However, ibverbs does not have a transport field for each port.  Example:

15:54:10 > ibv_devinfo -d mlx4_1 | grep trans
        transport:                      InfiniBand (0)

15:55:22 > ibstat mlx4_1 | grep Link
                Physical state: LinkUp
                Link layer: InfiniBand
                Link layer: Ethernet


Full output from ibstat:

15:55:32 > ibstat mlx4_1
CA 'mlx4_1'
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.9.1000
        Hardware version: b0
        Node GUID: 0x0002c9030008e7f0
        System image GUID: 0x0002c9030008e7f3
        Port 1:
                State: Initializing
                Physical state: LinkUp
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02510868
                Port GUID: 0x0002c9030008e7f1
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x0202c9fffe08e7f1
                Link layer: Ethernet


> Ira
> 
> > - Sean
> 
> 
> -- 
> Ira Weiny
> Math Programmer/Computer Scientist
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Hal Rosenstock]

On Wed, Sep 21, 2011 at 6:35 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, Sep 21, 2011 at 03:32:54PM -0700, Ira Weiny wrote:
>
>> To be clear I am not against using ibverbs on the linux side.  It
>> sounds like that would be the best move going forward.
>
> Technically umad already needs to use verbs on Linux because that is
> the only way to get the subnet timeout value that is needed to compute
> the MAD RPC timeout. Currently umad ignores this part of IBA ...

Determining the MAD timeout is more complicated than this. Subnet
timeout is insufficient (so you would need more than just this from
verbs - things that would not be part of verbs) if this mechanism were
to be made compliant rather than a configurable timeout as it is now.

-- Hal

> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ofw] ib-diags: compatability issue with ibstat

[Hal Rosenstock]

On Wed, Sep 21, 2011 at 8:14 PM, Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> On Wed, 21 Sep 2011 15:32:54 -0700
> Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
>
>> On Wed, 21 Sep 2011 15:23:45 -0700
>> "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>>
>> > > Does this mean "ibstatus" does not work on Windows?
>> >
>> > We do not support any of the scripts on windows.  As far as I could tell, the scripts look like they just do post-processing of available output.
>> >
>>
>> Good to know, thanks.
>>
>> > > How are you proposing the addition to ibverbs?  It seems this would break ABI
>> > > there.
>> >
>> > On windows, libibumad uses libibverbs to obtain whatever data it needs.  I'm assuming that a non-MAD application will eventually be able to use ibverbs to determine the rate, without needing to parse data from some file.
>> >
>> > > Does windows have sysfs like capabilities which would make the separate layer
>> > > clean?
>> >
>> > There's nothing like sysfs on windows.  If the data is available from ibverbs, windows could pull the data from there, even if the diags do not want to rely on ibverbs on the linux side.
>> >
>>
>> To be clear I am not against using ibverbs on the linux side.  It sounds like that would be the best move going forward.
>>
>
> Honestly, ibverbs supplies similar functionality in ibv_devinfo.  :-/
>
> It seems both ibstat and ibstatus should be dropped and ibv_devinfo enhanced to supply, Rate, Physical state, Capability Mask, and the transport (on a per port basis)[*].  There might be some other functionality as well.  Roland, ibv_devinfo, is in the examples directory.  Is there any reason this should not be used as the "official" tool?  If not I think we should use the code from ibv_devinfo as a basis for ibstat.  But I hate to see 2 implementations so close.

Even if ibv_devinfo is updated to include the additional information,
do we want to require libibverbs, etc. on any IB management machine
just for this ? That's not the case today on IB management nodes.

-- Hal

>
> Ira
>
> [*] There are cards which support this.  However, ibverbs does not have a transport field for each port.  Example:
>
> 15:54:10 > ibv_devinfo -d mlx4_1 | grep trans
>        transport:                      InfiniBand (0)
>
> 15:55:22 > ibstat mlx4_1 | grep Link
>                Physical state: LinkUp
>                Link layer: InfiniBand
>                Link layer: Ethernet
>
>
> Full output from ibstat:
>
> 15:55:32 > ibstat mlx4_1
> CA 'mlx4_1'
>        CA type: MT26428
>        Number of ports: 2
>        Firmware version: 2.9.1000
>        Hardware version: b0
>        Node GUID: 0x0002c9030008e7f0
>        System image GUID: 0x0002c9030008e7f3
>        Port 1:
>                State: Initializing
>                Physical state: LinkUp
>                Rate: 10
>                Base lid: 0
>                LMC: 0
>                SM lid: 0
>                Capability mask: 0x02510868
>                Port GUID: 0x0002c9030008e7f1
>                Link layer: InfiniBand
>        Port 2:
>                State: Down
>                Physical state: Disabled
>                Rate: 10
>                Base lid: 0
>                LMC: 0
>                SM lid: 0
>                Capability mask: 0x00010000
>                Port GUID: 0x0202c9fffe08e7f1
>                Link layer: Ethernet
>
>
>> Ira
>>
>> > - Sean
>>
>>
>> --
>> Ira Weiny
>> Math Programmer/Computer Scientist
>> Lawrence Livermore National Lab
>> 925-423-8008
>> weiny2-i2BcT+NCU+M@public.gmane.org
>
>
> --
> Ira Weiny
> Member of Technical Staff
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org
> _______________________________________________
> ofw mailing list
> ofw-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ofw] ib-diags: compatability issue with ibstat

[Hal Rosenstock]

On Wed, Sep 21, 2011 at 3:06 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> commit 1344cb3feacafc462440dabfa5997c5205486d83 added support for FDR10 in a way that is not compatible with Windows support.  Windows does not use files to read attribute information.
>
> I will probably need to obtain the necessary information using ibverbs on windows by reading port attributes.  I don't think FDR10 support is available through ibverbs on linux yet, but would this be acceptable?  Is there some other way that you'd like to handle this?  The other option I can think of is moving is_fdr10() and sys_read_string() out of ibstat.c and into a linux specific file, so that windows can provide its own implementation.  Thoughts?

I think there are 2 (related but somewhat separate) issues here:
1. How to obtain is_fdr10
2. Whether ibverbs should be used for this (in windows, linux, both)

The only way to determine whether fdr10 is active or not is via the
vendor proprietary MAD. That info may be reflected in some other API
(and/or file) so that MAD does not need to be reissued. In a separate
thread on linux-rdma, there was discussion on a couple of different
ways to do that from verbs and in this thread that there's no sysfs
equivalent in Windows. You've already stated that the Windows support
is using libibverbs for libibumad support so it seems appropriate to
me to do the same here (in Windows at least).

-- Hal

>
> - Sean
> _______________________________________________
> ofw mailing list
> ofw-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ofw] ib-diags: compatability issue with ibstat

[Hal Rosenstock]

On Fri, Sep 23, 2011 at 9:28 AM, Hal Rosenstock
<hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Sep 21, 2011 at 3:06 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> commit 1344cb3feacafc462440dabfa5997c5205486d83 added support for FDR10 in a way that is not compatible with Windows support.  Windows does not use files to read attribute information.
>>
>> I will probably need to obtain the necessary information using ibverbs on windows by reading port attributes.  I don't think FDR10 support is available through ibverbs on linux yet, but would this be acceptable?  Is there some other way that you'd like to handle this?  The other option I can think of is moving is_fdr10() and sys_read_string() out of ibstat.c and into a linux specific file, so that windows can provide its own implementation.  Thoughts?
>
> I think there are 2 (related but somewhat separate) issues here:
> 1. How to obtain is_fdr10
> 2. Whether ibverbs should be used for this (in windows, linux, both)
>
> The only way to determine whether fdr10 is active or not is via the
> vendor proprietary MAD. That info may be reflected in some other API
> (and/or file) so that MAD does not need to be reissued. In a separate
> thread on linux-rdma, there was discussion on a couple of different
> ways to do that from verbs and in this thread that there's no sysfs
> equivalent in Windows. You've already stated that the Windows support
> is using libibverbs for libibumad support so it seems appropriate to
> me to do the same here (in Windows at least).

The proper place for is_fdr10 is in libibumad (and then we wouldn't be
discussing libibverbs w/ibstat) but that was not done to avoid a
change to the umad port structure.

-- Hal

>
> -- Hal
>
>>
>> - Sean
>> _______________________________________________
>> ofw mailing list
>> ofw-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Hefty, Sean]

> Even if ibv_devinfo is updated to include the additional information,
> do we want to require libibverbs, etc. on any IB management machine
> just for this ? That's not the case today on IB management nodes.

IMO, it's acceptable for ibverbs to be the basic requirement for any IB userspace application.  I think you'll find the ibumad duplicates functionally that ibverbs provides, and that's why windows already has the dependency built in.

- Sean

RE: [ofw] ib-diags: compatability issue with ibstat

[Hefty, Sean]

> > The only way to determine whether fdr10 is active or not is via the
> > vendor proprietary MAD. That info may be reflected in some other API
> > (and/or file) so that MAD does not need to be reissued. In a separate
> > thread on linux-rdma, there was discussion on a couple of different
> > ways to do that from verbs and in this thread that there's no sysfs
> > equivalent in Windows. You've already stated that the Windows support
> > is using libibverbs for libibumad support so it seems appropriate to
> > me to do the same here (in Windows at least).
> 
> The proper place for is_fdr10 is in libibumad (and then we wouldn't be
> discussing libibverbs w/ibstat) but that was not done to avoid a
> change to the umad port structure.

ibstat determines is_fdr10 by reading a file.  That same data could just as easily be exported as a port attribute, which would make it belong to ibverbs, rather than umad.

If the only point of exporting is_fdr10 from the kernel is for an ib management diag to display the value, then it's not a useful value for verbs applications.

Re: ib-diags: compatability issue with ibstat

[Hal Rosenstock]

[-- Attachment #1.1: Type: text/plain, Size: 1357 bytes --]

On Fri, Sep 23, 2011 at 12:21 PM, Hefty, Sean <sean.hefty@intel.com> wrote:

> > > The only way to determine whether fdr10 is active or not is via the
> > > vendor proprietary MAD. That info may be reflected in some other API
> > > (and/or file) so that MAD does not need to be reissued. In a separate
> > > thread on linux-rdma, there was discussion on a couple of different
> > > ways to do that from verbs and in this thread that there's no sysfs
> > > equivalent in Windows. You've already stated that the Windows support
> > > is using libibverbs for libibumad support so it seems appropriate to
> > > me to do the same here (in Windows at least).
> >
> > The proper place for is_fdr10 is in libibumad (and then we wouldn't be
> > discussing libibverbs w/ibstat) but that was not done to avoid a
> > change to the umad port structure.
>
> ibstat determines is_fdr10 by reading a file.  That same data could just as
> easily be exported as a port attribute, which would make it belong to
> ibverbs, rather than umad.
>
> If the only point of exporting is_fdr10 from the kernel is for an ib
> management diag to display the value, then it's not a useful value for verbs
> applications.
>

fdr10 is in the same category as active_speed. What's the use for that other
than display ? Anyhow, display seems useful to me to know the local port
speed.

-- Hal

[-- Attachment #1.2: Type: text/html, Size: 1840 bytes --]

[-- Attachment #2: Type: text/plain, Size: 148 bytes --]

_______________________________________________
ofw mailing list
ofw@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw

Re: ib-diags: compatability issue with ibstat

[Hal Rosenstock]

[-- Attachment #1.1: Type: text/plain, Size: 1588 bytes --]

On Fri, Sep 23, 2011 at 12:33 PM, Hal Rosenstock
<hal.rosenstock@gmail.com>wrote:

>
>
> On Fri, Sep 23, 2011 at 12:21 PM, Hefty, Sean <sean.hefty@intel.com>wrote:
>
>> > > The only way to determine whether fdr10 is active or not is via the
>> > > vendor proprietary MAD. That info may be reflected in some other API
>> > > (and/or file) so that MAD does not need to be reissued. In a separate
>> > > thread on linux-rdma, there was discussion on a couple of different
>> > > ways to do that from verbs and in this thread that there's no sysfs
>> > > equivalent in Windows. You've already stated that the Windows support
>> > > is using libibverbs for libibumad support so it seems appropriate to
>> > > me to do the same here (in Windows at least).
>> >
>> > The proper place for is_fdr10 is in libibumad (and then we wouldn't be
>> > discussing libibverbs w/ibstat) but that was not done to avoid a
>> > change to the umad port structure.
>>
>> ibstat determines is_fdr10 by reading a file.
>
>
Yes, same file that active_speed comes from (which umad already parses).


> That same data could just as easily be exported as a port attribute, which
>> would make it belong to ibverbs, rather than umad.
>>
>
Sure; that's another way.


>
>> If the only point of exporting is_fdr10 from the kernel is for an ib
>> management diag to display the value, then it's not a useful value for verbs
>> applications.
>>
>
> fdr10 is in the same category as active_speed. What's the use for that
> other than display ? Anyhow, display seems useful to me to know the local
> port speed.
>
> -- Hal
>

[-- Attachment #1.2: Type: text/html, Size: 3381 bytes --]

[-- Attachment #2: Type: text/plain, Size: 148 bytes --]

_______________________________________________
ofw mailing list
ofw@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw

RE: [ofw] ib-diags: compatability issue with ibstat

[Hefty, Sean]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 757 bytes --]

So far, it seems that the choices are:

1. Remove ibstat and use ibv_devinfo instead
2. Change ibstat to obtain the fdr10 information using ibverbs
3. Move the is_fdr10 functionality into OS specific files or code sections of ibstat
4. Change ibstat to obtain fdr10 data using MADs or some other OS independent mechanism
5. Change ibumad so that it provides the fdr10 data
   Note: umad has OS dependent implementation anyway, but the windows umad would
         depend on the data being available from ibverbs
6. ???

Can someone explain why the information can't come from the umad_port::rate field?

- Sean
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: ib-diags: compatability issue with ibstat

[Hal Rosenstock]

[-- Attachment #1.1: Type: text/plain, Size: 874 bytes --]

On Fri, Sep 23, 2011 at 12:40 PM, Hefty, Sean <sean.hefty@intel.com> wrote:

> So far, it seems that the choices are:
>
> 1. Remove ibstat and use ibv_devinfo instead
> 2. Change ibstat to obtain the fdr10 information using ibverbs
> 3. Move the is_fdr10 functionality into OS specific files or code sections
> of ibstat
> 4. Change ibstat to obtain fdr10 data using MADs or some other OS
> independent mechanism
> 5. Change ibumad so that it provides the fdr10 data
>   Note: umad has OS dependent implementation anyway, but the windows umad
> would
>         depend on the data being available from ibverbs
> 6. ???
>
> Can someone explain why the information can't come from the umad_port::rate
> field?
>

Because fdr10 is non standard, there's no rate for this; it overloads QDR
based rates as both are 10 Gbps but have different link encodings.

-- Hal


>
> - Sean
>

[-- Attachment #1.2: Type: text/html, Size: 1504 bytes --]

[-- Attachment #2: Type: text/plain, Size: 148 bytes --]

_______________________________________________
ofw mailing list
ofw@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw

Re: ib-diags: compatability issue with ibstat

[Jason Gunthorpe]

On Fri, Sep 23, 2011 at 08:57:12AM -0400, Hal Rosenstock wrote:

> > Technically umad already needs to use verbs on Linux because that is
> > the only way to get the subnet timeout value that is needed to compute
> > the MAD RPC timeout. Currently umad ignores this part of IBA ...
> 
> Determining the MAD timeout is more complicated than this. Subnet
> timeout is insufficient (so you would need more than just this from
> verbs - things that would not be part of verbs) if this mechanism were
> to be made compliant rather than a configurable timeout as it is now.

I didn't have any problem with this in python-rdma. The only wonky bit
was that verbs was needed for subnet timeout because it isn't in
sysfs. Everything else went according to the spec.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ofw] ib-diags: compatability issue with ibstat

[Hefty, Sean]

> 	1. Remove ibstat and use ibv_devinfo instead
> 	2. Change ibstat to obtain the fdr10 information using ibverbs
> 	3. Move the is_fdr10 functionality into OS specific files or code
> sections of ibstat
> 	4. Change ibstat to obtain fdr10 data using MADs or some other OS
> independent mechanism
> 	5. Change ibumad so that it provides the fdr10 data
> 	  Note: umad has OS dependent implementation anyway, but the windows
> umad would
> 	        depend on the data being available from ibverbs
> 	6. ???
> 
> 	Can someone explain why the information can't come from the
> umad_port::rate field?
> 
> 
> Because fdr10 is non standard, there's no rate for this; it overloads QDR
> based rates as both are 10 Gbps but have different link encodings.

Ok, then for choices 6-8:

6. Have ibstat just report the rate without specifying the link encoding
7. Use umad_port:rate to encode the rate, link encoding, inter-packet delay, etc.
   (I'm only partially kidding here.)
8. Drop ibstat from windows.  Windows users will be directed to use ibv_devinfo.

Ira, what are your thoughts on this?

Re: ib-diags: compatability issue with ibstat

[Hal Rosenstock]

On 9/23/2011 12:57 PM, Jason Gunthorpe wrote:
> On Fri, Sep 23, 2011 at 08:57:12AM -0400, Hal Rosenstock wrote:
> 
>>> Technically umad already needs to use verbs on Linux because that is
>>> the only way to get the subnet timeout value that is needed to compute
>>> the MAD RPC timeout. Currently umad ignores this part of IBA ...
>>
>> Determining the MAD timeout is more complicated than this. Subnet
>> timeout is insufficient (so you would need more than just this from
>> verbs - things that would not be part of verbs) if this mechanism were
>> to be made compliant rather than a configurable timeout as it is now.
> 
> I didn't have any problem with this in python-rdma. 

Sure; you have an algorithm that uses subnet timeout v. a configurable
timeout.

> The only wonky bit
> was that verbs was needed for subnet timeout because it isn't in
> sysfs. Everything else went according to the spec.

I don't see how your algorithm can be compliant with just subnet timeout.

-- Hal

> Jason


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Jason Gunthorpe]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1072 bytes --]

On Fri, Sep 23, 2011 at 01:31:20PM -0400, Hal Rosenstock wrote:

> > The only wonky bit
> > was that verbs was needed for subnet timeout because it isn't in
> > sysfs. Everything else went according to the spec.
> 
> I don't see how your algorithm can be compliant with just subnet timeout.

Really?

13.4.6.3  specifies the formula

                 CommTimeValueOut    CommTimeValueIn    RespTimeValue
4.096 µsec x ( 2                  +2                 +2               )

Which boils down to 

4.096 µsec x (2 x 2**(Subnet_Timeout) + 2**(RespTimeValue))

For the on subnet case.

RespTimeValue comes from the class port info of the target manager,
and C13-13.1.1 gives guidance what value to use for the initial class
port info query to get the value.

Which is exactly what I implemented.

It isn't hard.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Hal Rosenstock]

On 9/23/2011 1:55 PM, Jason Gunthorpe wrote:
> On Fri, Sep 23, 2011 at 01:31:20PM -0400, Hal Rosenstock wrote:
> 
>>> The only wonky bit
>>> was that verbs was needed for subnet timeout because it isn't in
>>> sysfs. Everything else went according to the spec.
>>
>> I don't see how your algorithm can be compliant with just subnet timeout.
> 
> Really?
> 
> 13.4.6.3  specifies the formula
> 
>                  CommTimeValueOut    CommTimeValueIn    RespTimeValue
> 4.096 µsec x ( 2                  +2                 +2               )
> 
> Which boils down to 
> 
> 4.096 µsec x (2 x 2**(Subnet_Timeout) + 2**(RespTimeValue))
> 
> For the on subnet case.
> 
> RespTimeValue comes from the class port info of the target manager,
> and C13-13.1.1 gives guidance what value to use for the initial class
> port info query to get the value.
> 
> Which is exactly what I implemented.
> 
> It isn't hard.

and for SM MADs you need PortInfo:RespTimeValue (do you do SM MADs too ?)

My point was that verbs alone is insufficient as you need a MAD to do
this and the first round trip needs an artificial timeout (before you
know the RespTimeValue from wherever you get it). Clearly, for non SM
MADs, having subnet timeout as a port attribute saves that round trip.

-- Hal

> Jason
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ib-diags: compatability issue with ibstat

[Jason Gunthorpe]

On Fri, Sep 23, 2011 at 02:01:50PM -0400, Hal Rosenstock wrote:
> On 9/23/2011 1:55 PM, Jason Gunthorpe wrote:
> > On Fri, Sep 23, 2011 at 01:31:20PM -0400, Hal Rosenstock wrote:
> > 
> >>> The only wonky bit
> >>> was that verbs was needed for subnet timeout because it isn't in
> >>> sysfs. Everything else went according to the spec.
> >>
> >> I don't see how your algorithm can be compliant with just subnet timeout.
> > 
> > Really?
> > 
> > 13.4.6.3  specifies the formula
> > 
> >                  CommTimeValueOut    CommTimeValueIn    RespTimeValue
> > 4.096 ?sec x ( 2                  +2                 +2               )
> > 
> > Which boils down to 
> > 
> > 4.096 ?sec x (2 x 2**(Subnet_Timeout) + 2**(RespTimeValue))
> > 
> > For the on subnet case.
> > 
> > RespTimeValue comes from the class port info of the target manager,
> > and C13-13.1.1 gives guidance what value to use for the initial class
> > port info query to get the value.
> > 
> > Which is exactly what I implemented.
> > 
> > It isn't hard.
> 
> and for SM MADs you need PortInfo:RespTimeValue (do you do SM MADs too ?)

You should really read what I wrote. 

To be very clear, my implementation uses SubnetTimeout and a fixed
value for RespTimeValue for initial queries directed at the SA.

It switches to SubnetTimeout plus CPI.RespTimeValue once the SA's CPI is
queried.

It uses PathRecord.PacketLifetime and a fixed value for RespTimeValue
for initial queries to non-SA managers in the network, and again
switches to CPI.RespTimeValue once the manager's CPI is queried - eg
the PMA code does all this.

This is all exactly the required behavior outlined in the spec.

> My point was that verbs alone is insufficient as you need a MAD to do
> this and the first round trip needs an artificial timeout (before you
> know the RespTimeValue from wherever you get it). Clearly, for non SM
> MADs, having subnet timeout as a port attribute saves that round trip.

And my original point is that Subnet_Timeout is not optional, IBA says
you need the value to compute the timeouts.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ofw] ib-diags: compatability issue with ibstat

[Ira Weiny]

On Fri, 23 Sep 2011 10:15:04 -0700
"Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> > 	1. Remove ibstat and use ibv_devinfo instead
> > 	2. Change ibstat to obtain the fdr10 information using ibverbs
> > 	3. Move the is_fdr10 functionality into OS specific files or code
> > sections of ibstat
> > 	4. Change ibstat to obtain fdr10 data using MADs or some other OS
> > independent mechanism
> > 	5. Change ibumad so that it provides the fdr10 data
> > 	  Note: umad has OS dependent implementation anyway, but the windows
> > umad would
> > 	        depend on the data being available from ibverbs
> > 	6. ???
> > 
> > 	Can someone explain why the information can't come from the
> > umad_port::rate field?
> > 
> > 
> > Because fdr10 is non standard, there's no rate for this; it overloads QDR
> > based rates as both are 10 Gbps but have different link encodings.
> 
> Ok, then for choices 6-8:
> 
> 6. Have ibstat just report the rate without specifying the link encoding
> 7. Use umad_port:rate to encode the rate, link encoding, inter-packet delay, etc.
>    (I'm only partially kidding here.)
> 8. Drop ibstat from windows.  Windows users will be directed to use ibv_devinfo.
> 
> Ira, what are your thoughts on this?

I don't think 6 is really an option.  Reporting FDR10 as QDR will just confuse
people.

Option 7 might be possible.  But I don't think it is really the best way to
proceed, see below.

I think adding link encoding to ibverbs would be nice.  However, this is not
standard.  So it muddy's ibverbs.

Getting the data from a vendor MAD is probably the only _compliant_ way to do
this.  However, this means ibv_devinfo can not be the conical tool for this
information without a dependency on libibumad.

So it looks like option 4 is best.  Since I don't have this hardware I have to
go by the code Hal provided.  It looks like we can see an FDR10 capable HCA is
indicated by the devid.  For the devid's which are FDR10 query the Vendor MAD
and report the special rate.

I think this is really the only solution at this time.  I really wanted to
replace ibstat with ibv_devinfo but it looks like that will not be possible
without mudding ibverbs.[*]

The code to query the "Mellanox extended PortInfo" is in libibnetdisc &
libibmad.  I just want to think a bit about how to make it available to
smpquery, libibnetdisc, and ibstat in a clean manner.  Otherwise I would have
a patch.

Ira

[*] Roland is free to chime in here.

-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html