Re: [Xen-users] XCP with 64-bit dom0?

Re: [Xen-users] XCP with 64-bit dom0?

[Muriel]

>>> Hi all,
>>>
>>> For the last couple weeks or so, I've been pounding away at building a
>>> version of the XCP host tools to run on a 64-bit dom0 (not just a
>>> 64-bit kernel, but a full 64-bit tool stack).  I'm starting with the
>>> basic build setup of Centos 5.5 with arch x86_64.  Naturally, I'm
>>> rebuilding a large number of RPM packages from source, and I'm running
>>> into package hell as I try to track down proper 64-bit source versions
>>> of packages, fix build and compile issues, and get up to the point of
>>> being able to build the xapi-libs and xapi packages.
>>
>> I've never tried it.  Is there a particular reason WHY you want to do
>> it?  I can't think of any benefit, and the downside is your dom0 will
>> use more RAM, which means less RAM available for your domUs.
>>
>> -Dustin
 > I'm trying to integrate live-cloning (via the xen snowflock codebase)
 > into the XCP control path, which requires me to run a 64-bit PV
 > kernel, and also requires 64-bit dom0 tools to talk properly with Xen.
 >
 > Perhaps I should re-ask this question in Xen-devel.  It may be a more
 > appropriate venue.
 >
 > -kannan


Ok, i will reply directly to xen-devel to move the discussion.
I'm very interested and i'm starting to recompile. I'm trying to 
integrate into an existing 64bit cluster a dom0 based on xpc. What 
packages are fundamental to have a stable environment?

Thanks,
Muriel

Re: [Xen-users] XCP with 64-bit dom0?

[Kannan Vijayan]

On Fri, Apr 22, 2011 at 6:36 AM, Muriel <mucawhite@gmail.com> wrote:
>> I'm trying to integrate live-cloning (via the xen snowflock codebase)
>> into the XCP control path, which requires me to run a 64-bit PV
>> kernel, and also requires 64-bit dom0 tools to talk properly with Xen.
>>
>> Perhaps I should re-ask this question in Xen-devel.  It may be a more
>> appropriate venue.
>>
>> -kannan
>
>
> Ok, i will reply directly to xen-devel to move the discussion.
> I'm very interested and i'm starting to recompile. I'm trying to integrate
> into an existing 64bit cluster a dom0 based on xpc. What packages are
> fundamental to have a stable environment?
>
> Thanks,
> Muriel
>

I've been working at it from the other end, with a goal of "shortest
path to working".  I'm coming in with relative unfamiliarity with the
XCP architecture, and started working top-down.

I can boot the XCP host with a 64-bit snowflock kernel.  I
subsequently ran into problems with a 32-bit blktap userspace toolset
having problems talking to a 64-bit dom0 kernel, so I started
rebuilding that and kept moving down as needed.  I got the blktap
issue resolved, but currently I'm running into an issue with
xc_dom_boot_mem_init:

/var/log/xensource.log:[20110418T16:56:55.946Z|debug|localhost|1003
unix-RPC||cli] Xapi_cli.exception_handler: Got exception
INTERNAL_ERROR: [ XenguestHelper.Xc_dom_linux_build_failure(4, "
xc_dom_boot_mem_init: can't allocate low memory for dom\\\"") ]

Once again, I think this is due to a mismatch between 32-bit/64-bit
addresses.  The call is made through a tool called 'xenguest' which is
built alongside xapi and is used by xapi.  I had to go and dig pretty
deep in the rebuild tree to generate the dependencies needed to build
a 64-bit xapi and xenguest.

So far, these are the high-level packages I've had to rebuild to
target a 64-bit runtime (all as part of getting to a 64-bit build of
xapi and xenguest):

blktap-1.0.0-566.x86_64.rpm
blktap-devel-1.0.0-566.x86_64.rpm
ocaml-3.11.0-unknown.x86_64.rpm
ocaml-camlp4-3.11.0-unknown.x86_64.rpm
ocaml-findlib-1.1.2pl1-16.x86_64.rpm
ocaml-findlib-devel-1.1.2pl1-16.x86_64.rpm
ocaml-getopt-20040811-unknown.x86_64.rpm
ocaml-getopt-devel-20040811-unknown.x86_64.rpm
ocaml-type-conv-1.6.8-unknown.x86_64.rpm
ocaml-xmlm-1.0.2-unknown.x86_64.rpm
ocaml-xmlm-devel-1.0.2-unknown.x86_64.rpm
omake-0.9.8.5-unknown.x86_64.rpm
xapi-client-devel-0.2-unknown.x86_64.rpm
xapi-core-0.2-unknown.x86_64.rpm
xapi-datamodel-devel-0.2-unknown.x86_64.rpm
xapi-docs-0.2-unknown.x86_64.rpm
xapi-libs-devel-0-unknown.x86_64.rpm
xapi-libs-fe-0-unknown.x86_64.rpm
xapi-libs-utils-0-unknown.x86_64.rpm
xapi-squeezed-0.2-unknown.x86_64.rpm
xapi-tests-0.2-unknown.x86_64.rpm
xapi-www-0.2-unknown.x86_64.rpm
xapi-xe-0.2-unknown.x86_64.rpm
xapi-xenops-0.2-unknown.x86_64.rpm
xen-blktap-3.4.2-1.0.0.700.20051.x86_64.rpm
xen-devel-3.4.2-1.0.0.700.20051.x86_64.rpm
xen-firmware-3.4.2-1.0.0.700.20051.x86_64.rpm
xen-hypervisor-3.4.2-1.0.0.700.20051.x86_64.rpm
xen-tools-3.4.2-1.0.0.700.20051.x86_64.rpm


Current problem: Installing these directly on a normal XCP deployment
is infeasible.  It wants to pull a TON of 64-bit support packages, and
quickly descends into conflict hell.

So now I'm looking for ways to build an XCP install image, and hoping
I can replace all the base packages with 64-bit versions and go from
there.  I've looked around for instructions on building a full install
image (found the instructions for building/modifying xapi.. those were
very useful earlier in the process.. but not for this).

If I can't rebuild an XCP install disk, the other tactic I have in
mind is to edit an existing install ISO and switch out the existing
packages for 64-bit surrogates.

I haven't tried either of these approaches yet.  Does anybody know of
docs, or have thoughts, on either of these subjects?  I'm picking up
what I need to know as I go along, so my overall picture of the system
is coalescing over time, but is almost definitely not complete at this
time.

Just a note: from what I _have_ seen of the system, it's very nicely
designed.  I like the xapi object model and schema design.  Simple,
but complete, clear, and understandable.  A short analysis was enough
to figure out how best to approach the integration of live-cloning
semantics into XCP.  Kudos on the good work.

Cheers.
-kannan

Re: Re: [Xen-users] XCP with 64-bit dom0?

[Todd Deshane]

On Mon, Apr 25, 2011 at 11:39 AM, Kannan Vijayan
<kvijayan@gridcentric.ca> wrote:
> On Fri, Apr 22, 2011 at 6:36 AM, Muriel <mucawhite@gmail.com> wrote:
>>> I'm trying to integrate live-cloning (via the xen snowflock codebase)
>>> into the XCP control path, which requires me to run a 64-bit PV
>>> kernel, and also requires 64-bit dom0 tools to talk properly with Xen.
>>>

Have you considered running the 64 bit code in a VM?

The dom0 in XCP is 32 bit because of performance. It is recommended
for performance reasons to use 64bit HVM and then use the PVonHVM
drivers.

http://wiki.xensource.com/xenwiki/XenLinuxPVonHVMdrivers


>>> Perhaps I should re-ask this question in Xen-devel.  It may be a more
>>> appropriate venue.
>>>
>>> -kannan
>>
>>
>> Ok, i will reply directly to xen-devel to move the discussion.
>> I'm very interested and i'm starting to recompile. I'm trying to integrate
>> into an existing 64bit cluster a dom0 based on xpc. What packages are
>> fundamental to have a stable environment?
>>
>> Thanks,
>> Muriel
>>
>
> I've been working at it from the other end, with a goal of "shortest
> path to working".  I'm coming in with relative unfamiliarity with the
> XCP architecture, and started working top-down.
>
> I can boot the XCP host with a 64-bit snowflock kernel.  I
> subsequently ran into problems with a 32-bit blktap userspace toolset
> having problems talking to a 64-bit dom0 kernel, so I started
> rebuilding that and kept moving down as needed.  I got the blktap
> issue resolved, but currently I'm running into an issue with
> xc_dom_boot_mem_init:
>
> /var/log/xensource.log:[20110418T16:56:55.946Z|debug|localhost|1003
> unix-RPC||cli] Xapi_cli.exception_handler: Got exception
> INTERNAL_ERROR: [ XenguestHelper.Xc_dom_linux_build_failure(4, "
> xc_dom_boot_mem_init: can't allocate low memory for dom\\\"") ]
>
> Once again, I think this is due to a mismatch between 32-bit/64-bit
> addresses.  The call is made through a tool called 'xenguest' which is
> built alongside xapi and is used by xapi.  I had to go and dig pretty
> deep in the rebuild tree to generate the dependencies needed to build
> a 64-bit xapi and xenguest.
>
> So far, these are the high-level packages I've had to rebuild to
> target a 64-bit runtime (all as part of getting to a 64-bit build of
> xapi and xenguest):
>
> blktap-1.0.0-566.x86_64.rpm
> blktap-devel-1.0.0-566.x86_64.rpm
> ocaml-3.11.0-unknown.x86_64.rpm
> ocaml-camlp4-3.11.0-unknown.x86_64.rpm
> ocaml-findlib-1.1.2pl1-16.x86_64.rpm
> ocaml-findlib-devel-1.1.2pl1-16.x86_64.rpm
> ocaml-getopt-20040811-unknown.x86_64.rpm
> ocaml-getopt-devel-20040811-unknown.x86_64.rpm
> ocaml-type-conv-1.6.8-unknown.x86_64.rpm
> ocaml-xmlm-1.0.2-unknown.x86_64.rpm
> ocaml-xmlm-devel-1.0.2-unknown.x86_64.rpm
> omake-0.9.8.5-unknown.x86_64.rpm
> xapi-client-devel-0.2-unknown.x86_64.rpm
> xapi-core-0.2-unknown.x86_64.rpm
> xapi-datamodel-devel-0.2-unknown.x86_64.rpm
> xapi-docs-0.2-unknown.x86_64.rpm
> xapi-libs-devel-0-unknown.x86_64.rpm
> xapi-libs-fe-0-unknown.x86_64.rpm
> xapi-libs-utils-0-unknown.x86_64.rpm
> xapi-squeezed-0.2-unknown.x86_64.rpm
> xapi-tests-0.2-unknown.x86_64.rpm
> xapi-www-0.2-unknown.x86_64.rpm
> xapi-xe-0.2-unknown.x86_64.rpm
> xapi-xenops-0.2-unknown.x86_64.rpm
> xen-blktap-3.4.2-1.0.0.700.20051.x86_64.rpm
> xen-devel-3.4.2-1.0.0.700.20051.x86_64.rpm
> xen-firmware-3.4.2-1.0.0.700.20051.x86_64.rpm
> xen-hypervisor-3.4.2-1.0.0.700.20051.x86_64.rpm
> xen-tools-3.4.2-1.0.0.700.20051.x86_64.rpm
>
>
> Current problem: Installing these directly on a normal XCP deployment
> is infeasible.  It wants to pull a TON of 64-bit support packages, and
> quickly descends into conflict hell.
>
> So now I'm looking for ways to build an XCP install image, and hoping
> I can replace all the base packages with 64-bit versions and go from
> there.  I've looked around for instructions on building a full install
> image (found the instructions for building/modifying xapi.. those were
> very useful earlier in the process.. but not for this).
>
> If I can't rebuild an XCP install disk, the other tactic I have in
> mind is to edit an existing install ISO and switch out the existing
> packages for 64-bit surrogates.
>
> I haven't tried either of these approaches yet.  Does anybody know of
> docs, or have thoughts, on either of these subjects?  I'm picking up
> what I need to know as I go along, so my overall picture of the system
> is coalescing over time, but is almost definitely not complete at this
> time.
>
> Just a note: from what I _have_ seen of the system, it's very nicely
> designed.  I like the xapi object model and schema design.  Simple,
> but complete, clear, and understandable.  A short analysis was enough
> to figure out how best to approach the integration of live-cloning
> semantics into XCP.  Kudos on the good work.
>
> Cheers.
> -kannan
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>



-- 
Todd Deshane
http://www.linkedin.com/in/deshantm
http://www.xen.org/products/cloudxen.html