From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ian Campbell Subject: Re: Are there source codes for Xen-IB? Date: Tue, 31 May 2011 10:04:59 +0100 Message-ID: <1306832699.775.133.camel@zakaz.uk.xensource.com> References: <1306147218.20576.79.camel@zakaz.uk.xensource.com> <1306232138.20576.147.camel@zakaz.uk.xensource.com> <1306395870.775.8.camel@zakaz.uk.xensource.com> <1306419040.775.34.camel@zakaz.uk.xensource.com> <20110526185006.GJ32595@reaktio.net> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Joseph Glanville Cc: maus , "xen-devel@lists.xensource.com" , "Fajar A. Nugraha" List-Id: xen-devel@lists.xenproject.org On Sat, 2011-05-28 at 19:15 +0100, Joseph Glanville wrote: > Hi, > > I happen to have done a huge amount of research into this as our > infrastructure is all based on converged Infiniband. > > Xen-IB and SmartIO were two seperate attempts at building a > paravirtualised Infinband driver for Xen that allowed guests to use > native Infinband protocols at close to native speeds. Being > paravirtualised they were live-migration cappable and many domU could > share a single Infiniband adapter. > > As far as I am aware both of the projects are dead, Ah, thanks for the info, sounds like there was some interesting ideas here once upon a time. I'm not 100% sure that Xen-IB and SmartIO were actually different projects, http://nowlab.cse.ohio-state.edu/projects/xen/index.html seems to suggest they are actually the same. So I guess I stand corrected: the xen-smartio repo does contain some PV IB drivers but since they are for 2.6.16 and for the Xen PV architecture (not KVM) I don't know how much use they will be to the OP. FWIW I also found http://wiki.xensource.com/xenwiki/XenSmartIO and http://www.openfabrics.org/archives/nov2006sc/xen-ib-presentation.pdf while further researching. Ian. > however SR-IOV > isn't long off for Infiniband, Mellanox is on target to push theirs > upstream into OFED 1.6 due in November, Qlogic and Oracle/Sun will be > quick to follow I would imagine. > > SR-IOV will allow pretty similar functionality but less portable in > terms of live-migrate etc. > > It is worth noting here that you can't bridge IPoIB, IPoIB is not a > layer 2 device and doesn't understand Ethernet. You can use EoIB > (Ethernet over IB) or routing to provide network access to domUs. > > Feel free to email me if you need any help with Infiniband and Xen. > > Joseph. > > On 27 May 2011 04:50, Pasi Kärkkäinen wrote: > > On Thu, May 26, 2011 at 03:10:40PM +0100, Ian Campbell wrote: > >> On Thu, 2011-05-26 at 15:04 +0100, maus wrote: > >> > Hi, > >> > > >> > I know use IP-over-IB can let KVM or Xen use IB, but what I have to do > >> > is like Xen-IB. Not just use IB. > >> > >> What is Xen-IB exactly? > >> > > > > Could "Xen-IB" mean live migration over Infiniband RDMA (Remote DMA) ? > > I can't really think of other uses.. > > > > -- Pasi > > > > > >> > So, there is no toturial for Xen-IB,right? > >> > >> That's right, there is no documentation and anyone who would have been > >> involved with this stuff is long gone. > >> > >> Ian. > >> > >> > > >> > > >> > Yi-Man > >> > > >> > > >> > > >> > 2011/5/26 Ian Campbell > >> > On Wed, 2011-05-25 at 14:40 +0100, maus wrote: > >> > > Hi, > >> > > > >> > > I have a project which is make KVM use InfiniBand. > >> > > However, since Xen can use InfiniBand through Xen-IB, I want > >> > to try to > >> > > make Xen use InfiniBand first. > >> > > Through that way, I think it may give me some idea about how > >> > to make > >> > > KVM use InfiniBand. > >> > > So, as you said, I just use "Updated defconfig to include > >> > infiniband > >> > > driver as modules for dom0/domU" to do? > >> > > And since I don't know how to I do first for Xen-IB, is > >> > there any > >> > > tutorial for Xen-IB? > >> > > Or does Xen build-in support IB currently? > >> > > >> > > >> > I'm not sure what IB support you expect Xen to have over and > >> > above the > >> > driver and utils support, which are functions of the dom0 > >> > kernel and > >> > distro and not the hypervisor. > >> > > >> > Have you got IB working on a native system? I expect that if > >> > you do that > >> > then Xen or KVM will be able to make use of it basically for > >> > free, i.e. > >> > configure a VIF on an IP-over-IB bridge or use a local > >> > IP-over-IB > >> > address in dom0 for migration etc etc. > >> > > >> > Ian. > >> > > >> > > >> > > > >> > > Yi-Man > >> > > > >> > > > >> > > > >> > > 2011/5/24 Ian Campbell > >> > > Please do not top post. > >> > > > >> > > On Tue, 2011-05-24 at 11:07 +0100, maus wrote: > >> > > > Hi, > >> > > > > >> > > > There many file in > >> > > http://xenbits.xensource.com/ext/xen-smartio/ , > >> > > > which should I use for Xen-IB? > >> > > > Use the following three? > >> > > > Updated defconfig to include infiniband driver as > >> > modules > >> > > for > >> > > > dom0/domU > >> > > > Updating drivers/infiniband to latest. Should > >> > shrink size of > >> > > > xen_ib.patch in > >> > > > Add support for Infiniband into Xen > >> > > > >> > > > >> > > Again, these changesets are >5 years old and have > >> > not been > >> > > looked at or > >> > > maintained since -- I am very sceptical that you > >> > will find > >> > > anything of > >> > > use on a modern Xen system in there. > >> > > > >> > > I expect that IB support in the kernels we currently > >> > use for > >> > > dom0 is > >> > > vastly improved over what was available in the > >> > 2.6.16 tree > >> > > referenced in > >> > > the above repository. Perhaps you don't even need > >> > anything > >> > > special > >> > > outside of what is available in the upstream dom0 > >> > kernel. > >> > > > >> > > Perhaps if you explain what you are actually trying > >> > to achieve > >> > > then > >> > > people will be able to better advise you? > >> > > > >> > > Ian. > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > > >> > >> > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > >