Extend LID space in InfiniBand beyond 48k? Is this really possible?

Extend LID space in InfiniBand beyond 48k? Is this really possible?

[Vangelis Tasoulas]

[-- Attachment #1.1: Type: text/plain, Size: 976 bytes --]

Hello,

I was reading a year+ old article where Steve Scott (CRAY CTO) was
interviewed here:
https://www.nextplatform.com/2016/01/08/cray-cto-connects-the-dots-on-future-interconnects/

At a certain point Steve mentions (or at least that's what the article
claims):

/InfiniBand has another limit in terms of the number of logical IDs, or
LIDs, it can support, which is around 48,000 end points, and for the
most point, people have stayed within that limit. There is an extended
version that has some higher packet overhead that can go to higher
scalability./

Is this true? Is there any reference that one can point me to for the
part of the "extended version that has some higher packet overhead that
can go to higher scalability"?

To the best of my knowledge, there is no extended version defined in the
InfiniBand specification. What is mentioned is that /The unicast LID
range is a flat identifier space defined as 0x0001 to 0xBFFF./

Vangelis


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Extend LID space in InfiniBand beyond 48k? Is this really possible?

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

On Wed, Mar 29, 2017 at 06:07:47PM +0200, Vangelis Tasoulas wrote:
> Hello,
>
> I was reading a year+ old article where Steve Scott (CRAY CTO) was
> interviewed here:
> https://www.nextplatform.com/2016/01/08/cray-cto-connects-the-dots-on-future-interconnects/
>
> At a certain point Steve mentions (or at least that's what the article
> claims):
>
> /InfiniBand has another limit in terms of the number of logical IDs, or
> LIDs, it can support, which is around 48,000 end points, and for the
> most point, people have stayed within that limit. There is an extended
> version that has some higher packet overhead that can go to higher
> scalability./
>
> Is this true? Is there any reference that one can point me to for the
> part of the "extended version that has some higher packet overhead that
> can go to higher scalability"?
>
> To the best of my knowledge, there is no extended version defined in the
> InfiniBand specification. What is mentioned is that /The unicast LID
> range is a flat identifier space defined as 0x0001 to 0xBFFF./

Right, there is no any extended version of that spec.
As far as I know, Intel's OmniPath technology is marketed as new and
extended version of Infiniband [1], so it is probably that he referenced to it.

[1] http://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-architecture-fabric-overview.html

>
> Vangelis
>




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: Extend LID space in InfiniBand beyond 48k? Is this really possible?

[Hefty, Sean]

> > I was reading a year+ old article where Steve Scott (CRAY CTO) was
> > interviewed here:
> > https://www.nextplatform.com/2016/01/08/cray-cto-connects-the-dots-
> on-future-interconnects/
> >
> > At a certain point Steve mentions (or at least that's what the
> article
> > claims):
> >
> > /InfiniBand has another limit in terms of the number of logical IDs,
> or
> > LIDs, it can support, which is around 48,000 end points, and for the
> > most point, people have stayed within that limit. There is an
> extended
> > version that has some higher packet overhead that can go to higher
> > scalability./
> >
> > Is this true? Is there any reference that one can point me to for
> the
> > part of the "extended version that has some higher packet overhead
> that
> > can go to higher scalability"?
> >
> > To the best of my knowledge, there is no extended version defined in
> the
> > InfiniBand specification. What is mentioned is that /The unicast LID
> > range is a flat identifier space defined as 0x0001 to 0xBFFF./
> 
> Right, there is no any extended version of that spec.
> As far as I know, Intel's OmniPath technology is marketed as new and
> extended version of Infiniband [1], so it is probably that he
> referenced to it.

At one point there was talk about using the GRH to extend IB subnet address space.  I can't remember what became of that proposal, but I can't quickly find any mention of it in the spec.  But adding the GRH would result in a small increase to packet overhead.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html