NTB query

NTB query

[Peter Boyle]

[-- Attachment #1.1: Type: text/plain, Size: 1913 bytes --]


Hi,

sorry for cold emailing you; I’m interested in the possibility to set up an ugly home
grown, domain specific HPC approach.

We’ve built cartesian grid machines (e.g. 3d) in the past with only nearest neighbour
communication and no routing. Our QCDOC computer is an example:

https://web.archive.org/web/20081007081750/http://www.bnl.gov/lqcd/linkable_files/pdf/pap231.pdf <https://web.archive.org/web/20081007081750/http://www.bnl.gov/lqcd/linkable_files/pdf/pap231.pdf>

An earlier version QCDSP won the Gordon Bell Prize (I wasn’t part of the team at that time).

I’m curious whether NTB or similar can be used to direct connect a single CPU with PCIe Root complex
to 6 neighbours, and out of this build a  3d mesh where the neighbours can RDMA each others memory.

I’m well aware it would really be very painful, but I don’t have sufficient documentation on the root complexes of 
current CPU’s that integrate them to know whether the hardware in principle out there would
let me do this. I’m also not saying I’m planning on doing this; just presently researching if it would in principle
be possible.

My dream is that the power, price performance of lower end chips is great, but the cost of
high end interconnected is killing and driving to high end nodes.

e.g. I get 100GF/s on a quad core costing $400, but only 400 GF/s on a dual socket Xeon costin $7000.

- I’d rather expect $1600 and connect four quad-cores; but the $4000 of OPA cards, cables and switch required is crippling.

NTB is in principle THE  solution for this.

Ideally, we’d build custom boards, racks, backplanes like our QCDOC design and pull something like 8 links at x 4 lanes
out of each chip and build a “free” but restricted interconnect where the only extra parts are copper traces costing only a few 
extra dollars per node.

Thanks for any information,

Peter

[-- Attachment #1.2: Type: text/html, Size: 3014 bytes --]

[-- Attachment #2: Edinburgh University charitable status --]
[-- Type: text/plain, Size: 109 bytes --]

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.