Re: [opensm] RFC: new routing options (repost)

[Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>]

Hi Alex,

Thanks.  Are you still reviewing the remote_guid_sorting patch (the 2/4
patch)?  Or do you feel there is work there that needs to be done?

Al

On Mon, 2011-07-04 at 03:52 -0700, Alex Netes wrote:
> Hi Al, Hared,
> 
> Applied:
>   [PATCH 1/4] Support port shifting.
>   [PATCH 3/4] Support scatter ports.
>   [PATCH 4/4] Cleanup scatter ports patch. 
> 
> Thanks.
> 
> On 17:56 Wed 06 Apr     , Albert Chu wrote:
> > Hey Alex, Jared,
> > 
> > On Wed, 2011-04-06 at 11:14 -0700, Albert Chu wrote:
> > > Hey Alex,
> > > 
> > > On Wed, 2011-04-06 at 07:09 -0700, Alex Netes wrote:
> > > > Hi Al, Jared,
> > > > 
> > > > On 14:31 Wed 23 Mar     , Albert Chu wrote:
> > > > > > 
> > > > > > 1) Port Shifting
> > > > > > 
> > > > > > This is similar to what was done with some of the LMC > 0 code.
> > > > > > Congestion would occur due to "alignment" of routes w/ common traffic
> > > > > > patterns.  However, we found that it was also necessary for LMC=0 and
> > > > > > only for used-ports.  For example, lets say there are 4 ports (called A,
> > > > > > B, C, D) and we are routing lids 1-9 through them.  Suppose only routing
> > > > > > through A, B, and C will reach lids 1-9.
> > > > > > 
> > > > > > The LFT would normally be:
> > > > > > 
> > > > > > A: 1 4 7
> > > > > > B: 2 5 8
> > > > > > C: 3 6 9
> > > > > > D:
> > > > > > 
> > > > > > The Port Shifting option would make this:
> > > > > > 
> > > > > > A: 1 6 8
> > > > > > B: 2 4 9
> > > > > > C: 3 5 7
> > > > > > D:
> > > > > > 
> > > > > > This option by itself improved the mpiGraph average send/recv bandwidth
> > > > > > from 420 MB/s and 508 MB/s to to 991 MB/s and 1172 MB/s.
> > > > > > 
> > > > 
> > > > After thinking about this a little more and reviewing Jared Carr's - Scatter ports
> > > > patch, I think we should combine these efforts into one framework as Al
> > > > suggested.
> > 
> > As I was beginning to integrate Jared's patch with mine, it ends up that
> > algorithmically/architecturally, it isn't as easy (or similar) as I had
> > originally thought.  In particular, it has issues with LMC > 0.
> > Normally you want to route through a port that is least forwarded
> > through or goes through systems it hasn't seen yet.  This sort of
> > conflicts with the idea of selecting a port randomly.
> > 
> > I'm going to throw out the following patch series as a starting point
> > for discussion on scatter ports.  My original two patches have been
> > updated with new log messages and some minor tweaks.
> > 
> > My attempt of integration of Jared's scatter patch is included.  It has
> > a variety of cleanup (b/c of conflicts w/ my patches), 1 or 2 gotchas I
> > caught, and various tweaks for code consistency with my patches/other
> > OpenSM code.  Jared's original code algorithm is largely unchanged, but
> > I did modify it to deal with LMC > 0 better (by basically ignoring LMC).
> > 
> > Jared, LMK what you think and if it'll work for you.
> > 
> > Al
> > 
> > P.S.  Jared, I made you author on the 3rd patch naturally.
> > 
> > > Moreover, isn't "port_shifting" too much fabric oriented? Do
> > > > general OpenSM users will find this useful for them?
> > > > Moreover, how can user identify that port_shifting may improve performance for
> > > > him.
> > > 
> > > I will admit, I'm unsure of how much non-HPC users would benefit from
> > > this option, be hurt by it, or if they would even care.  I can't speak
> > > for all users, but here at LLNL and at most of the lab HPC sites, people
> > > play with the options and experiment to find the best routing algorithm
> > > + settings that support their environment.  I would imagine the
> > > port_shifting option would just be another option for people to
> > > experiment with.
> > > 
> > > I think adding Jared's Scatter Ports would be easy to merge into my line
> > > of patches.  Let me see if I can integrate his patch into my line
> > > easily.
> > > 
> > > > Is providing shift factor (more than the suggested 1) will help to make it
> > > > suitable foo a general case?
> > > 
> > > That seems like a good idea, we certainly could support an arbitrary
> > > shift, allowing users to experiment if there is a better one for their
> > > particular environment.
> > > 
> > > > > > 2) Remote Guid Sorting
> > > > > > 
> > > > > > Most core/spine switches we've seen thus far have had line boards
> > > > > > connected to spine boards in a consistent pattern.  However, we recently
> > > > > > got some Qlogic switches that connect from line/leaf boards to spine
> > > > > > boards in a (to the casual observer) random pattern.  I'm sure there was
> > > > > > a good electrical/board reason for this design, but it does hurt routing
> > > > > > b/c updn doesn't account for this.  Here's an output from iblinkinfo as
> > > > > > an example.
> > > > > > 
> > > > 
> > > > Why this problem can't be addressed by guid_routing_order_file option?
> > > 
> > > The problem we encountered in our fabric is predominantly a
> > > switch-to-switch routing issue with a spine switch.  The
> > > guid_routing_order_file wouldn't be able to solve this, since its input
> > > is just end ports.
> > > 
> > > Or another way to say it, this option directly affects the routing
> > > decisions made.  The guid_routing_order_file does not, it only affects
> > > the order in which routes are chosen (which can have consequences, but
> > > the routing algorithm itself is unchanged).
> > > 
> > > Al
> > > 
> > > > 
> > > > --Alex
> > -- 
> > Albert Chu
> > chu11-i2BcT+NCU+M@public.gmane.org
> > Computer Scientist
> > High Performance Systems Division
> > Lawrence Livermore National Laboratory
> 
> 
-- 
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html