Fw: Xen on /. again

Fw: Xen on /. again

[Trent Jaeger]

[-- Attachment #1: Type: text/plain, Size: 2448 bytes --]

Hi,

I work with Reiner and was formerly the manager of the L4 SawMill project.

SawMill was a very different project where we were looking to manage the 
complexity of OS code and improve security using multiservers -- i.e., 
isolation of individual parts of the operating system from each other 
running on the hypervisor.  Few modifications to the hypervisor (in this 
case, L4) were required for this, but many mods to the OS code.

The new project aims to add security checks on hypervisor resources in the 
hypervisor.  Thus, changes to the hypervisor are necessary. 

The best analogue for what is being proposed is the addition of the Linux 
Security Modules (LSM) interface to Linux 2.6 -- this enables mandatory 
access control to be enforced on the use of Linux resources by independent 
security modules which permits flexible security choices.

Here is a brief list of the lessons that I think that we may be able to 
apply to Xen work based on both the SawMill and LSM experiences.

- minimize (manual) changes required to guest OSs (at this time SawMill 
required many changes)
- enable use of many drivers (i.e., enable driver development via driver 
model -- Xen aims to provide this)
- isolation boundaries cost more than you might think, but computers are a 
lot faster now (10X+)
- we need mandatory access control interface like LSM to have flexible 
control of resources (rather than dump resources to a control partition)
- authorize access at bind time rather than at use time (no critical path 
impact)

I am sure that there are others.

Regards,
Trent.
------------------------------------------------------------
Trent Jaeger
IBM T.J. Watson Research Center
19 Skyline Drive, Hawthorne, NY 10532
(914) 784-7225, FAX (914) 784-7225
----- Forwarded by Reiner Sailer/Watson/IBM on 01/18/2005 07:34 PM ----- 

Jacob Gorm Hansen <jacobg@diku.dk>  

01/18/2005 07:28 PM 

To
Reiner Sailer/Watson/IBM@IBMUS 
cc
xen-devel@lists.sourceforge.net 
Subject
Re: [Xen-devel] Xen on /. again







Reiner Sailer wrote:

> It is not that fast. sHype is not in any Xen source
> yet. Currently it is running on our research hypervisor.
> We want to discuss with the Xen community while we
> are porting it to Xen so we can address comments and
> adapt it where necessary.

hi,

is sHype related to the old L4 Sawmill project in any way? Are there any 
lessons learned from that project that would be relevant to the work on 
Xen?

Jacob


[-- Attachment #2: Type: text/html, Size: 4340 bytes --]

Re: Fw: Xen on /. again

[Jacob Gorm Hansen]

Trent Jaeger wrote:

> - isolation boundaries cost more than you might think, but computers are 
> a lot faster now (10X+)
> - we need mandatory access control interface like LSM to have flexible 
> control of resources (rather than dump resources to a control partition)
> - authorize access at bind time rather than at use time (no critical 
> path impact)

Is there any reason to allow communication between VMs, other than 
driver access? It seems to me you can just use network communication 
(potentially signed or encrypted) when you need to communicate, and that 
anything else would violate the fundamental 'virtual machine' abstraction?

Also, I suppose you will wish to prevent covert channels between 
domains, e.g. domains communicating using various timing attacks (I move 
the disk head to the other end of the disk if I wish to tell you 
something), or by allocating/freeing certains parts of memory.

How much will you need to dumb down the VMs view of what is going on in 
the machine to achieve this (not expose real time information, not 
expose real page tables), and how much of a VMM will there be left when 
you are done?

best regards,
Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Mark Williamson]

> Is there any reason to allow communication between VMs, other than
> driver access? It seems to me you can just use network communication
> (potentially signed or encrypted) when you need to communicate, and that
> anything else would violate the fundamental 'virtual machine' abstraction?

You might want to have two domU's shared memory to implement a "direct 
point-to-point" ethernet link, to avoid the overheads of going through dom0.  
For most people this might not matter but it's nice to have the option for 
maximal performance.

> Also, I suppose you will wish to prevent covert channels between
> domains, e.g. domains communicating using various timing attacks (I move
> the disk head to the other end of the disk if I wish to tell you
> something), or by allocating/freeing certains parts of memory.
>
> How much will you need to dumb down the VMs view of what is going on in
> the machine to achieve this (not expose real time information, not
> expose real page tables), and how much of a VMM will there be left when
> you are done?

Well domains are not aware of each other's memory usage, so I wouldn't have 
thought that allocation / exposing real page tables would matter.  (Except 
dom0 can of course see everything if it wants).

Timing related attacks are somewhat trickier to eliminate covert channels in, 
although some randomisation can limit the bandwidth.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Jacob Gorm Hansen]

Mark Williamson wrote:

> Well domains are not aware of each other's memory usage, so I wouldn't have 
> thought that allocation / exposing real page tables would matter.  (Except 
> dom0 can of course see everything if it wants).

Microkernel people like to make the argument that you could create a 
low-bandwidth covert channel by systematically allocating and freeing a 
set of pages, and because domains see real page frame numbers they can 
learn the state of the other guy by looking at what pages they get in 
return from an alloc call. I suppose you could randomize the 
page-allocator, but then you will have to leave a certain amount of 
pages unused at all times, to have enough random pages to choose from.

(For myself, I would much rather have the real page tables and find a 
way to live with the covert channels, but I am not building 
military-grade systems).

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Mark Williamson]

> Microkernel people like to make the argument that you could create a
> low-bandwidth covert channel by systematically allocating and freeing a
> set of pages, and because domains see real page frame numbers they can
> learn the state of the other guy by looking at what pages they get in
> return from an alloc call. I suppose you could randomize the
> page-allocator, but then you will have to leave a certain amount of
> pages unused at all times, to have enough random pages to choose from.

Oh, I suppose you could do this using the balloon driver to relinquish / 
reclaim physical memory - I hadn't thought of that.  You could just restrict 
domain's ability to do these operations and retain the ability to see the 
real page tables.  Ballooning probably isn't that important...

The mechanisms for the network driver donating pages to the backend would be 
the other place where restrictions might be needed in this respect.

> (For myself, I would much rather have the real page tables and find a
> way to live with the covert channels, but I am not building
> military-grade systems).

Yes, it's worth being able to limit these things as much as possible whilst 
still retaining functionality - ideally I guess there'd be a sliding scale of 
covert channel bandwidth vs. performance / functionality.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Trent Jaeger]

[-- Attachment #1: Type: text/plain, Size: 3662 bytes --]

> Is there any reason to allow communication between VMs, other than
driver access? 

In general, I agree with Mark's reply to this.  We may want to implement a 
faster mechanism for communication between 2 VMs where security 
requirements allow.  However, this implies that we have a precise 
understanding of how the mechanisms impact security requirements.

> Also, I suppose you will wish to prevent covert channels between

Preventing covert channels is not a focus of our initial work. 

Our focus initially is on providing basic mandatory access control 
mechanisms where MAC policies may aim for confinement of errors (e.g., 
type enforcement policy), protect VM integrity (e.g., via Biba or 
Clark-Wilson), and/or prevent leakage (e.g., MLS).  We have done work on 
policies of all these types for Linux.

For MLS, I expect that we will work to identify covert storage channels 
(these are generally straightforward) and identify possible solutions, so 
that these can be closed. 

However, some mechanisms may be useful in some cases, but not meet 
security requirements in others.  I am more familiar with the I/O 
descriptor ring mechanism than how the real page tables are exposed, so 
I'll use this as an example. 

The I/O descriptor rings provide high performance communication, but 
cannot be used to implement a low secrecy to high secrecy communication. 
Of course, the low secrecy VM can see how the high secrecy component 
removes data from the ring, so this is a covert (storage) channel.  Thus, 
another mechanism will have to be used. This could be setup above Xen 
(d0?), but the choice must meet the VMs security requirements.

Timing channels are much more difficult to find/solve and much more 
expensive to solve, so I don't expect much work on them for a while.  Some 
general techniques are known (e.g., fuzzy time), but these are expensive, 
so it is not clear when/if there will be a demand. 

> How much will you need to dumb down the VMs view of what is going on in

Too early to say, but hopefully we only need to dumb down VMs where 
requirements dictate.

Regards,
Trent.
------------------------------------------------------------
Trent Jaeger
IBM T.J. Watson Research Center
19 Skyline Drive, Hawthorne, NY 10532
(914) 784-7225, FAX (914) 784-7225




Jacob Gorm Hansen <jacobg@diku.dk>
01/20/2005 05:11 PM
 
        To:     Trent Jaeger/Watson/IBM@IBMUS
        cc:     Xen-devel@lists.sourceforge.net
        Subject:        Re: Fw: [Xen-devel] Xen on /. again


Trent Jaeger wrote:

> - isolation boundaries cost more than you might think, but computers are
> a lot faster now (10X+)
> - we need mandatory access control interface like LSM to have flexible
> control of resources (rather than dump resources to a control partition)
> - authorize access at bind time rather than at use time (no critical
> path impact)

Is there any reason to allow communication between VMs, other than
driver access? It seems to me you can just use network communication
(potentially signed or encrypted) when you need to communicate, and that
anything else would violate the fundamental 'virtual machine' abstraction?

Also, I suppose you will wish to prevent covert channels between
domains, e.g. domains communicating using various timing attacks (I move
the disk head to the other end of the disk if I wish to tell you
something), or by allocating/freeing certains parts of memory.

How much will you need to dumb down the VMs view of what is going on in
the machine to achieve this (not expose real time information, not
expose real page tables), and how much of a VMM will there be left when
you are done?

best regards,
Jacob


[-- Attachment #2: Type: text/html, Size: 5233 bytes --]

Re: Fw: Xen on /. again

[David Hopwood]

Mark Williamson wrote:
>>Also, I suppose you will wish to prevent covert channels between
>>domains, e.g. domains communicating using various timing attacks (I move
>>the disk head to the other end of the disk if I wish to tell you
>>something), or by allocating/freeing certains parts of memory.
>>
>>How much will you need to dumb down the VMs view of what is going on in
>>the machine to achieve this (not expose real time information, not
>>expose real page tables), and how much of a VMM will there be left when
>>you are done?
> 
> Well domains are not aware of each other's memory usage, so I wouldn't have 
> thought that allocation / exposing real page tables would matter.  (Except 
> dom0 can of course see everything if it wants).

Information about other domains' memory usage is leaked via the
hardware->physical mapping.

> Timing related attacks are somewhat trickier to eliminate covert channels in, 
> although some randomisation can limit the bandwidth.

Eliminating covert channels is completely infeasible. I don't see any
value in aiming for this. It's not a useful security property in most
circumstances.

-- 
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Mark Williamson]

> Information about other domains' memory usage is leaked via the
> hardware->physical mapping.

OK, I was forgetting about the domain memory reservation hypercalls.  It's 
probably reasonable just to throw away ballooning functionality where this 
might be a problem.

The main problem (as I see it) is going to be the network interface, whose 
performance depends on page-flipping.  You can eliminate the security problem 
without hiding machine address if you copy incoming packets but that's going 
to hurt performance :-(

> > Timing related attacks are somewhat trickier to eliminate covert channels
> > in, although some randomisation can limit the bandwidth.
>
> Eliminating covert channels is completely infeasible. I don't see any
> value in aiming for this. It's not a useful security property in most
> circumstances.

I agree it's not useful in the majority of circumstances.  If it's required it 
can be implemented at a later date but the returns for the amount of time 
invested are likely to be smaller.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Reiner Sailer]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]

xen-devel-admin@lists.sourceforge.net wrote on 01/20/2005 07:48:58 PM:

> Mark Williamson wrote:
> >>Also, I suppose you will wish to prevent covert channels between
> >>domains, e.g. domains communicating using various timing attacks (I 
move
> >>the disk head to the other end of the disk if I wish to tell you
> >>something), or by allocating/freeing certains parts of memory.
> >>
> >>How much will you need to dumb down the VMs view of what is going on 
in
> >>the machine to achieve this (not expose real time information, not
> >>expose real page tables), and how much of a VMM will there be left 
when
> >>you are done?
> > 
> > Well domains are not aware of each other's memory usage, so I wouldn't 
have 
> > thought that allocation / exposing real page tables would matter. 
(Except 
> > dom0 can of course see everything if it wants).
> 
> Information about other domains' memory usage is leaked via the
> hardware->physical mapping.

There might be a problem of object-reuse and isolation. If memory isn't 
cleared before
it is given to another partition, then this is an error. If disk space 
isn't cleared before it is
reused by another partition, then this seems serious, too.

Question:
a) Does the balloon driver clean pages before releasing it to other 
partitions? (I guess not)
b) Does the page-swap thas happens when a packet is received clear the 
pages? (I guess not)
c) If a) or b) use un-cleaned pages: did anybody try to sniff data 
(passwords, keys) from another 
     non-cooperating partitions out of re-used pages?

> > Timing related attacks are somewhat trickier to eliminate covert 
> channels in, 
> > although some randomisation can limit the bandwidth.
> 
> Eliminating covert channels is completely infeasible. I don't see any
> value in aiming for this. It's not a useful security property in most
> circumstances.
> -- 
> David Hopwood <david.nospam.hopwood@blueyonder.co.uk>

To eliminate ALL covert channels might not be in our interest, but to 
ignore all of 
them seems unnatural as well.

Certainly, covert channels are a way to spend many discussions. 

Reiner

[-- Attachment #2: Type: text/html, Size: 3008 bytes --]

Re: Fw: Xen on /. again

[David Hopwood]

Mark Williamson wrote:
>>Information about other domains' memory usage is leaked via the
>>hardware->physical mapping.
> 
> OK, I was forgetting about the domain memory reservation hypercalls.  It's 
> probably reasonable just to throw away ballooning functionality where this 
> might be a problem.
> 
> The main problem (as I see it) is going to be the network interface, whose 
> performance depends on page-flipping.  You can eliminate the security problem 
> without hiding machine address if you copy incoming packets but that's going 
> to hurt performance :-(
> 
>>>Timing related attacks are somewhat trickier to eliminate covert channels
>>>in, although some randomisation can limit the bandwidth.
>>
>>Eliminating covert channels is completely infeasible. I don't see any
>>value in aiming for this. It's not a useful security property in most
>>circumstances.
> 
> I agree it's not useful in the majority of circumstances.  If it's required it 
> can be implemented at a later date but the returns for the amount of time 
> invested are likely to be smaller.

It almost certainly can't be implemented at a later date. Even attempting
to do so (without really succeeding) would require significant incompatible
changes to the hypervisor interface.

The idea of limiting covert channels should have been abandoned when it
became clear that it isn't feasible without severely constraining the
efficiency and functionality of an operating system. Unfortunately it is
too interesting a problem, so a lot of effort has been essentially wasted
in research into this area, without ever coming up with any way to limit
the bandwidth to a useful extent. Attackers only need a very small
bandwidth to transmit many of the things that are most useful from their
point of view (cryptographic keys, passwords, compressed answers from a
program that can look at any amount of data), so claims about limiting the
bandwidth really just give a false sense of security.

-- 
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Mark Williamson]

> It almost certainly can't be implemented at a later date. Even attempting
> to do so (without really succeeding) would require significant incompatible
> changes to the hypervisor interface.

What changes are required depend on what channels you're trying to eliminate.  
You could limit the aforementioned covert channels in the network interface, 
block device head scheduling and also CPU scheduling without changing the 
hypervisor interface at all.

Whether this is worth the effort is another matter, however, as you rightly 
point out ;-)

> Attackers only need a very small 
> bandwidth to transmit many of the things that are most useful from their
> point of view (cryptographic keys, passwords, compressed answers from a
> program that can look at any amount of data), so claims about limiting the
> bandwidth really just give a false sense of security.

Yes.  You just have to hope that organisational security measures compensate 
for the covert channels that remain.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Jody Belka]

On Thu, Jan 20, 2005 at 10:41:06PM +0000, Mark Williamson wrote:
> You might want to have two domU's shared memory to implement a "direct 
> point-to-point" ethernet link, to avoid the overheads of going through dom0.
> For most people this might not matter but it's nice to have the option for 
> maximal performance.

This should be possible by using the xen front & backend network drivers,
right? The only thing right now is that i can't see how to get the backend
interface to be created anywhere other than dom0. I can set up a domain as
a network interface backend ok, and i can create a domain with a vif pointing
at that domain, but no new vif appears in the network backend dom.


NB1: I'm on a slightly old build right now (2004/12/12 17:03:10 1.1655) so
     i don't know if it's still the case, but when providing a backend domain
     for a vif only a number is accepted. It'd be much nicer if a name could
     be used.

NB2: Again, i don't know if it's still the case, but the network backend driver
     can't be built without physical device access enabled, but there isn't a
     dependency for that in the kernel config system.


J

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Steven Hand]

>Mark Williamson wrote:
>>>Also, I suppose you will wish to prevent covert channels between
>>>domains, e.g. domains communicating using various timing attacks (I move
>>>the disk head to the other end of the disk if I wish to tell you
>>>something), or by allocating/freeing certains parts of memory.
>>>
>>>How much will you need to dumb down the VMs view of what is going on in
>>>the machine to achieve this (not expose real time information, not
>>>expose real page tables), and how much of a VMM will there be left when
>>>you are done?
>> 
>> Well domains are not aware of each other's memory usage, so I wouldn't have 
>> thought that allocation / exposing real page tables would matter.  (Except 
>> dom0 can of course see everything if it wants).
>
>Information about other domains' memory usage is leaked via the
>hardware->physical mapping.

Unprivileged domains cannot see each others hardware->physical mappings. 
If we use full shadow mode (either with or without VT-x), domains cannot
even see their own hardware->physical mappings. 

Explicit page reuse (via either network page-flipping or balloon drivers) 
can be eliminated, albeit at a cost to performance. Networking can still
be provided by using either multiple NICs or hardware virtualization in 
the NIC (somewhat like the arseNIC work we did a few years back). Allowing
networking of course has other well-known risks (see below). 

>> Timing related attacks are somewhat trickier to eliminate covert channels in
>> although some randomisation can limit the bandwidth.
>
>Eliminating covert channels is completely infeasible. I don't see any
>value in aiming for this. It's not a useful security property in most
>circumstances.

Well in the current version of Xen, there are plenty of non-covert channels 
(like network connections and raw shared memory event channels) which probably 
make more sense to look at first :-) 

I think received wisdom is that it's certainly very difficult (and 
very expensive in terms of time and, usually, performance) to eliminate 
all covert channels. It may also be impossible depending on requirements
(e.g. if one allows networked connections to arbitrary third parties). 
The full virtualization support in VT may be useful though. 


cheers, 

S.



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Steven Hand]

>
>>Mark Williamson wrote:
>>>>Also, I suppose you will wish to prevent covert channels between
>>>>domains, e.g. domains communicating using various timing attacks (I move
>>>>the disk head to the other end of the disk if I wish to tell you
>>>>something), or by allocating/freeing certains parts of memory.
>>>>
>>>>How much will you need to dumb down the VMs view of what is going on in
>>>>the machine to achieve this (not expose real time information, not
>>>>expose real page tables), and how much of a VMM will there be left when
>>>>you are done?
>>> 
>>> Well domains are not aware of each other's memory usage, so I wouldn't have
 
>>> thought that allocation / exposing real page tables would matter.  (Except 
>>> dom0 can of course see everything if it wants).
>>
>>Information about other domains' memory usage is leaked via the
>>hardware->physical mapping.
>
>Unprivileged domains cannot see each others hardware->physical mappings. 
 
Oops - I read this originally as 'virtual->physical'. The MPT does indeed
currently allow domains to view each others hardware->physical mappings. 
Replacing the single MPT with per-domain viewable sparse mapping structures 
is possible though, albeit at a performance cost.  Full virt with VT-x also 
eliminates this issue.

cheers, 

S.



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Mark A. Williamson]

> > You might want to have two domU's shared memory to implement a "direct
> > point-to-point" ethernet link, to avoid the overheads of going through
> > dom0. For most people this might not matter but it's nice to have the
> > option for maximal performance.
>
> This should be possible by using the xen front & backend network drivers,
> right? The only thing right now is that i can't see how to get the backend
> interface to be created anywhere other than dom0.

Currently the backend driver will only start if it's in dom0 or in a network 
backend domain.  The main obstacles to doing this are:
1) the backend domain currently needs to be privileged in order to map the 
frontends memory, so this can't work between an arbitrary pair of domUs (this 
is beginning to change as grant tables are integrated).
2) the tools don't support this kind of configuration yet

> NB2: Again, i don't know if it's still the case, but the network backend
> driver can't be built without physical device access enabled, but there
> isn't a dependency for that in the kernel config system.

Since the current assumption is that the backend controls a physical 
interface, the backend is built iff the domain has physical device support.  
Might need to change in future...

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Mark A. Williamson]

Replied to this earlier but I think my MUA might have gobbled it :-(

> This should be possible by using the xen front & backend network drivers,
> right?

Yes but it needs a bit of extra code.

1) AFAIK, the backend currently needs to be privileged to map the frontend's 
memory - this in beginning to change as the grant tables stuff is merged
2) the tools don't support this kind of config yet

> The only thing right now is that i can't see how to get the backend 
> interface to be created anywhere other than dom0. I can set up a domain as
> a network interface backend ok, and i can create a domain with a vif
> pointing at that domain, but no new vif appears in the network backend dom.

Are you specifying what backend the frontend should be connected to in the 
frontend's config file?  Are you specifying that the backend is a backend in 
its config file?

This path through Xend hasn't been tested thoroughly, since most people aren't 
using this sort of configuration yet.

> NB2: Again, i don't know if it's still the case, but the network backend
> driver can't be built without physical device access enabled, but there
> isn't a dependency for that in the kernel config system.

Currently backends are built iff physical device access is enabled, since the 
assumption is that they will run in a privileged domain and virtualise a real 
device.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Jody Belka]

On Fri, Jan 21, 2005 at 11:22:20AM +0000, Mark A. Williamson wrote:
> > The only thing right now is that i can't see how to get the backend 
> > interface to be created anywhere other than dom0. I can set up a domain as
> > a network interface backend ok, and i can create a domain with a vif
> > pointing at that domain, but no new vif appears in the network backend dom.
> 
> Are you specifying what backend the frontend should be connected to in the 
> frontend's config file?

Yes

> Are you specifying that the backend is a backend in its config file?

Yes

> > NB2: Again, i don't know if it's still the case, but the network backend
> > driver can't be built without physical device access enabled, but there
> > isn't a dependency for that in the kernel config system.
> 
> Currently backends are built iff physical device access is enabled, since the 
> assumption is that they will run in a privileged domain and virtualise a real 
> device.

What i was meaning, was that the network backend can be chosen in the config
without physical device access enabled, but the kernel won't then build.


J

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Mark Williamson]

> > Are you specifying what backend the frontend should be connected to in
> > the frontend's config file?
> Yes
> > Are you specifying that the backend is a backend in its config file?
> Yes

Just checking ;-)  This stuff hasn't been tested much, so if you've set all 
the appropriate variables it could be a Xend bug.

Has anybody had driver domains working with the current Xend?  When?

> What i was meaning, was that the network backend can be chosen in the
> config without physical device access enabled, but the kernel won't then
> build.

Oh I see, sorry.  I guess it doesn't work because the relevant macros for 
mapping other domains memory are not defined.  I think a subsequent patch has 
fixed this - thanks for pointing it out.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Fw: Xen on /. again

[Jody Belka]

On Sat, Jan 22, 2005 at 05:20:35PM +0000, Mark Williamson wrote:
> > > Are you specifying what backend the frontend should be connected to in
> > > the frontend's config file?
> > Yes
> > > Are you specifying that the backend is a backend in its config file?
> > Yes
> 
> Just checking ;-)  This stuff hasn't been tested much, so if you've set all 
> the appropriate variables it could be a Xend bug.

That's fine. Yeah, i marked the backend domain as a netif domain, started it up,
inserted the domain ID into the config for the frontend domain (hassle, but
necessary as the current code only takes an ID, and not a name), and started up
the frontend domain. The additional interface showed up in the frontend domain,
but nothing appeared in the backend.


J

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Multiple netif device channels (was Fw: Xen on /. again)

[Mark Williamson]

> That's fine. Yeah, i marked the backend domain as a netif domain, started
> it up, inserted the domain ID into the config for the frontend domain
> (hassle, but necessary as the current code only takes an ID, and not a
> name), and started up the frontend domain. The additional interface showed
> up in the frontend domain, but nothing appeared in the backend.

Remind me: is the frontend having netifs attached to two different backends?  
What happens if you try to attach both netifs to the driver domain (not 
dom0)?

Hmmm, that's strange.  I'm surprised the additional interface showed up in one 
but not the other but I'm not familiar with the support for multiple netifs.

Mike: any idea what might cause this behaviour?  Is this known not to work?

Do you have any experience in the Xend guts?  You could try turning on tracing 
at the top of messages.py and observe exactly what control interface messages 
are being sent.  This is likely to be quite tedious but may allow you to 
infer what information about netifs is flowing between the domains.  You may 
also want to instrument the probing code in netfront.c and netback.c.

Sorry not to be more helpful,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Multiple netif device channels (was Fw: Xen on /. again)

[Jody Belka]

On Sun, Jan 23, 2005 at 05:52:23PM +0000, Mark Williamson wrote:
> > That's fine. Yeah, i marked the backend domain as a netif domain, started
> > it up, inserted the domain ID into the config for the frontend domain
> > (hassle, but necessary as the current code only takes an ID, and not a
> > name), and started up the frontend domain. The additional interface showed
> > up in the frontend domain, but nothing appeared in the backend.
> 
> Remind me: is the frontend having netifs attached to two different backends?  
> What happens if you try to attach both netifs to the driver domain (not 
> dom0)?

Attempted layout is as follows:

Dom0 - single physical network adaptor
       netif backend
       
DomU1 - no physical network adaptor
        netif backend
	netif frontend -> Dom0
	
DomU2 - no physical network adaptor
        netif frontend -> Dom0
        netif frontend -> DomU1

Interfaces for all the frontends appear inside the relevant domains, but the
only the 2 dom0 backend vifs appear; nothing shows up inside domU1.

I'll try connecting DomU2 only to DomU1 wrt networking when i next get a chance.

> Do you have any experience in the Xend guts?  You could try turning on tracing 
> at the top of messages.py and observe exactly what control interface messages 
> are being sent.  This is likely to be quite tedious but may allow you to 
> infer what information about netifs is flowing between the domains.  You may 
> also want to instrument the probing code in netfront.c and netback.c.

A little bit. i'll try and see if i can see what's happening with it. At a
minimum it'd give me a trace i can post that might help someone else figure
out what's going on.

> Sorry not to be more helpful,

np,


J

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Multiple netif device channels (was Fw: Xen on /. again)

[Mark Williamson]

> Attempted layout is as follows:
>
> Dom0 - single physical network adaptor
>        netif backend
Fine.

> DomU1 - no physical network adaptor
>         netif backend
> 	netif frontend -> Dom0

Ah, this would be a problem in any case.

Backend drivers need to run in privileged domains.  This basically means that 
they have access to a PCI device.  You could try the workaround Andy 
suggested a few days ago (grant access to a PCI device but don't compile the 
driver into the kernel so it leaves it alone).

> DomU2 - no physical network adaptor
>         netif frontend -> Dom0
>         netif frontend -> DomU1

Fine.

> A little bit. i'll try and see if i can see what's happening with it. At a
> minimum it'd give me a trace i can post that might help someone else figure
> out what's going on.

Yup.  But try getting privileges to work properly in DomU1 first.

HTH,
Mark

> > Sorry not to be more helpful,
>
> np,
>
>
> J


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Multiple netif device channels

[Jody Belka]

On Sun, Jan 23, 2005 at 05:52:23PM +0000, Mark Williamson wrote:
> Hmmm, that's strange.  I'm surprised the additional interface showed up in one 
> but not the other but I'm not familiar with the support for multiple netifs.

Ok, an update on the matter. The reason no interface appeared in the domU netif
backend was because xend wasn't actually configuring the domain as a backend.
The reason being that xend was setting the backend configuration flags /after/
constructing the domain image [1], so nothing was happening.

With that changed, a vif appears in the netif backend when the other domU
is started up. However, no actual network connection can be made between the
domains; ping just responds back with a destination host unreachable error.
Although that is better than before i upgraded to the latest -testing (which
i thought i should do before proceeding further); before the upgrade any
traffic from the netif domain to the vif caused a panic in said domain.

Seperately, although still connected to networking, the netfront driver has
code in it that prevents it from loading in a backend domain as well as the
initial domain. That should probably be reduced down to just checking for the
initial domain.

And lastly, on a completely seperate xen matter, i seem to recall some people
mentioning recently about not being able to add/remove vbd's in running
domains. I had a look at the code, and i'd guess that (like me lately) these
people are running 2.6 in the domU's. The code to update vbd devices is
currently commented out in the 2.6 sparse tree, as it hasn't been ported to the
2.6 blkfront driver yet (the code in place is a pure copy from the 2.4 driver).



J

[1] XendDomainInfo.py / XendDomainInfo.construct() - method starts @ line 427
    The configure_backends() line needs to be before the construct_image() 
    one. The configure_restart() line doesn't really matter, as that's purely
    a xend thing.

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Multiple netif device channels

[Mark Williamson]

> Ok, an update on the matter. The reason no interface appeared in the domU
> netif backend was because xend wasn't actually configuring the domain as a
> backend. The reason being that xend was setting the backend configuration
> flags /after/ constructing the domain image [1], so nothing was happening.

Ah.  Do you have a patch for this we could apply?

> With that changed, a vif appears in the netif backend when the other domU
> is started up. However, no actual network connection can be made between
> the domains; ping just responds back with a destination host unreachable
> error.

Have you tried the hack Andy suggested (giving the backend domain access to a 
PCI device it has no drivers for) so that the backend domain will be 
privileged?  If you don't elevate its privilege level some how it will 
definitely not be able to function as a backend.

> Seperately, although still connected to networking, the netfront driver has
> code in it that prevents it from loading in a backend domain as well as the
> initial domain. That should probably be reduced down to just checking for
> the initial domain.

Yes, this can probably be safely removed now.  Does removing the backend check 
work for you?

> And lastly, on a completely seperate xen matter, i seem to recall some
> people mentioning recently about not being able to add/remove vbd's in
> running domains. I had a look at the code, and i'd guess that (like me
> lately) these people are running 2.6 in the domU's. The code to update vbd
> devices is currently commented out in the 2.6 sparse tree, as it hasn't
> been ported to the 2.6 blkfront driver yet (the code in place is a pure
> copy from the 2.4 driver).

Right, it's probably a reasonably simple update if somebody takes the time to 
trawl through and figure out what needs updating.

Thanks for working with us on this.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Multiple netif device channels

[Jody Belka]

[-- Attachment #1: Type: text/plain, Size: 2699 bytes --]

On Wed, Jan 26, 2005 at 08:36:34PM +0000, Mark Williamson wrote:
> > Ok, an update on the matter. The reason no interface appeared in the domU
> > netif backend was because xend wasn't actually configuring the domain as a
> > backend. The reason being that xend was setting the backend configuration
> > flags /after/ constructing the domain image [1], so nothing was happening.
> 
> Ah.  Do you have a patch for this we could apply?

Yep, patch attached. Also includes a change to make specifying the backend
domain more user friendly.


> > With that changed, a vif appears in the netif backend when the other domU
> > is started up. However, no actual network connection can be made between
> > the domains; ping just responds back with a destination host unreachable
> > error.
> 
> Have you tried the hack Andy suggested (giving the backend domain access to a 
> PCI device it has no drivers for) so that the backend domain will be 
> privileged?  If you don't elevate its privilege level some how it will 
> definitely not be able to function as a backend.

Yay. Tried that (pointed the backend domain to the unused onboard audio
controller :) and it worked perfectly.

One thing though; all backend interfaces have the same mac address,
FE:FF:FF:FF:FF:FF. I see that Keir committed this change, along with a note
about problems with bridging with the previous code.

If i modify the code such that i can specify an explicit backend vif mac
address to xend, and then that gets passed into the backend (with
fe:ff:ff:ff:ff:ff being sent if no explicit one is given), what would be the
best way to go about avoiding the problems mentioned when doing so?

In case you're wondering why i need this change, it's because i need some way
to distingush the vif's in the backend domain, so that i can configure them
easily with the linux hotplug system (since, obviously, xend is neither running
in the domain, nor can the dom0 instance configure the interface)


> > Seperately, although still connected to networking, the netfront driver has
> > code in it that prevents it from loading in a backend domain as well as the
> > initial domain. That should probably be reduced down to just checking for
> > the initial domain.
> 
> Yes, this can probably be safely removed now.  Does removing the backend check 
> work for you?

Seems to work fine. Tried it out after i finally got the inter-domU networking
working. No problems appeared with front only, back only or front-and-back
configurations. Patch attached for this as well.


> Thanks for working with us on this.

No probs. Had a good time working through the code to figure out what was going
on.



J

-- 
Jody Belka
knew (at) pimb (dot) org

[-- Attachment #2: tools.patch --]
[-- Type: text/plain, Size: 1845 bytes --]

diff -durN tools.orig/python/xen/xend/XendDomainInfo.py tools/python/xen/xend/XendDomainInfo.py
--- tools.orig/python/xen/xend/XendDomainInfo.py	2005-01-27 01:06:28.840379840 +0100
+++ tools/python/xen/xend/XendDomainInfo.py	2005-01-25 16:20:44.000000000 +0100
@@ -449,9 +449,9 @@
 
             self.init_domain()
             self.configure_console()
+            self.configure_backends()
             self.construct_image()
             self.configure_restart()
-            self.configure_backends()
             deferred = self.configure()
             def cberr(err):
                 self.destroy()
diff -durN tools.orig/python/xen/xend/server/netif.py tools/python/xen/xend/server/netif.py
--- tools.orig/python/xen/xend/server/netif.py	2005-01-25 10:09:39.000000000 +0100
+++ tools/python/xen/xend/server/netif.py	2005-01-27 01:19:29.043770896 +0100
@@ -140,7 +140,8 @@
         self.ipaddr = self._get_config_ipaddr(config) or []
         
         try:
-            self.backendDomain = int(sxp.child_value(config, 'backend', '0'))
+            xd = get_component('xen.xend.XendDomain')
+            self.backendDomain = int(xd.domain_lookup(sxp.child_value(config, 'backend', '0')).id)
         except:
             raise XendError('invalid backend domain')
 
@@ -161,7 +162,8 @@
         bridge = sxp.child_value(config, 'bridge')
         script = sxp.child_value(config, 'script')
         ipaddr = self._get_config_ipaddr(config)
-        backendDomain = sxp.child_value(config, 'backend', '0')
+        xd = get_component('xen.xend.XendDomain')
+        backendDomain = str(xd.domain_lookup(sxp.child_value(config, 'backend', '0')).id)
         if (mac is not None) and (mac != self.mac):
             raise XendError("cannot change mac")
         if (backendDomain is not None) and (backendDomain != str(self.backendDomain)):

[-- Attachment #3: netfront.patch --]
[-- Type: text/plain, Size: 606 bytes --]

diff -durN linux-2.6.10-xen-sparse.orig/drivers/xen/netfront/netfront.c linux-2.6.10-xen-sparse/drivers/xen/netfront/netfront.c
--- linux-2.6.10-xen-sparse.orig/drivers/xen/netfront/netfront.c	2005-01-27 01:02:55.724778336 +0100
+++ linux-2.6.10-xen-sparse/drivers/xen/netfront/netfront.c	2005-01-27 01:03:07.039058304 +0100
@@ -1295,8 +1295,7 @@
 {
     int err = 0;
 
-    if ( (xen_start_info.flags & SIF_INITDOMAIN) ||
-         (xen_start_info.flags & SIF_NET_BE_DOMAIN) )
+    if ( xen_start_info.flags & SIF_INITDOMAIN )
         return 0;
 
     IPRINTK("Initialising virtual ethernet driver.\n");

Re: Multiple netif device channels

[Jody Belka]

Realised i forgot proper comments and the sign-offs.

Patch: tools.patch
Comment: Moves backend configuration to before construction of domain image, so
         that the backend drivers actually get activated in driver domains.
         Also allows domain names as well as IDs when specifying backend domains
         for vbd and disk configuration items.
Signed-off-by: knew-xen@pimb.org

Patch: netfront.patch
Comment: Remove the check preventing the netfront driver from starting up in
         network backend domains.
Signed-off-by: knew-xen@pimb.org


J

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Fw: Xen on /. again

[Ian Pratt]

> Question: 
> a) Does the balloon driver clean pages before releasing it to 
> other partitions? (I guess not) 

Yes it does.

> b) Does the page-swap thas happens when a packet is received 
> clear the pages? (I guess not) 

There is a pool of pages that are used for network IO. They're not
normally scrubbed when they get flipped, on the grounds that network
packets could be sniffed on the wire anyhow. Pages are scrubbed before
entering the pool. 

I guess we could have a config option that also zeroed the payload area
of a packet when the skb is freed, for those that want belt and braces.

[NB: one area we need to investigate is whether the kernel ever does
decrypt in-place in the skb data area, and if so, throw the page out of
the pool.]

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl