Problem Reading from XenStore in DomU

Problem Reading from XenStore in DomU

[Dagaen Golomb]

Hi All,

I'm having an interesting issue. I am working on a project that
requires me to share memory between dom0 and domUs. I have this
successfully working using the grant table and the XenStore to
communicate grefs.

My issue is this. I have one domU running Ubuntu 12.04 with a default
3.8.x kernel that has no issue reading or writing from the XenStore.
My work also requires some kernel modifications, and we have made
these changes in the 4.1.0 kernel. In particular, we've only added a
simple hypercall. This modified kernel is what dom0 is running, on top
of Xen 4.7 rc1.

The guest I mentioned before with the default kernel can still read
and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
running our kernel.

The issue I'm having is with another newly created guest (i.e., it is
not a copy of the working one, this is because I needed more space and
couldn't expand the original disk image). It is also running Ubuntu
12.04 but has been upgraded to our modified kernel. On this guest I
can write to the XenStore, and see that the writes were indeed
successful using xenstore-ls in dom0. However, when this same guest
attempts to read from the XenStore, it doesn't return an error code
but instead just blocks indefinitely. I've waiting many minutes to
make sure its not just blocking for a while, it appears like it will
block forever. The block is happening when I start the transaction.
I've also tried not using a transaction, in which case it blocks on
the read itself.

I have an inkling this may be something as simple as a configuration
issue, but I can't seem to find anything. Also, the fact that writes
work fine but reads do not is perplexing me.

Any help would be appreciated!

Regards,
Dagaen Golomb
Ph.D. Student, University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Andrew Cooper]

On 15/05/16 17:40, Dagaen Golomb wrote:
> Hi All,
>
> I'm having an interesting issue. I am working on a project that
> requires me to share memory between dom0 and domUs. I have this
> successfully working using the grant table and the XenStore to
> communicate grefs.
>
> My issue is this. I have one domU running Ubuntu 12.04 with a default
> 3.8.x kernel that has no issue reading or writing from the XenStore.
> My work also requires some kernel modifications, and we have made
> these changes in the 4.1.0 kernel. In particular, we've only added a
> simple hypercall. This modified kernel is what dom0 is running, on top
> of Xen 4.7 rc1.
>
> The guest I mentioned before with the default kernel can still read
> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
> running our kernel.
>
> The issue I'm having is with another newly created guest (i.e., it is
> not a copy of the working one, this is because I needed more space and
> couldn't expand the original disk image). It is also running Ubuntu
> 12.04 but has been upgraded to our modified kernel. On this guest I
> can write to the XenStore, and see that the writes were indeed
> successful using xenstore-ls in dom0. However, when this same guest
> attempts to read from the XenStore, it doesn't return an error code
> but instead just blocks indefinitely. I've waiting many minutes to
> make sure its not just blocking for a while, it appears like it will
> block forever. The block is happening when I start the transaction.
> I've also tried not using a transaction, in which case it blocks on
> the read itself.
>
> I have an inkling this may be something as simple as a configuration
> issue, but I can't seem to find anything. Also, the fact that writes
> work fine but reads do not is perplexing me.
>
> Any help would be appreciated!

Nothing should block like this.  Without seeing your patch, I can't
comment as to whether you have accidentally broken things.

Other avenues of investigation are to look at what the xenstored process
is doing in dom0 (is it idle? or is it spinning?), and to look in the
xenstored log file to see if anything suspicious occurs.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

>> Hi All,
>>
>> I'm having an interesting issue. I am working on a project that
>> requires me to share memory between dom0 and domUs. I have this
>> successfully working using the grant table and the XenStore to
>> communicate grefs.
>>
>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>> My work also requires some kernel modifications, and we have made
>> these changes in the 4.1.0 kernel. In particular, we've only added a
>> simple hypercall. This modified kernel is what dom0 is running, on top
>> of Xen 4.7 rc1.
>>
>> The guest I mentioned before with the default kernel can still read
>> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
>> running our kernel.
>>
>> The issue I'm having is with another newly created guest (i.e., it is
>> not a copy of the working one, this is because I needed more space and
>> couldn't expand the original disk image). It is also running Ubuntu
>> 12.04 but has been upgraded to our modified kernel. On this guest I
>> can write to the XenStore, and see that the writes were indeed
>> successful using xenstore-ls in dom0. However, when this same guest
>> attempts to read from the XenStore, it doesn't return an error code
>> but instead just blocks indefinitely. I've waiting many minutes to
>> make sure its not just blocking for a while, it appears like it will
>> block forever. The block is happening when I start the transaction.
>> I've also tried not using a transaction, in which case it blocks on
>> the read itself.
>>
>> I have an inkling this may be something as simple as a configuration
>> issue, but I can't seem to find anything. Also, the fact that writes
>> work fine but reads do not is perplexing me.
>>
>> Any help would be appreciated!
>
> Nothing should block like this.  Without seeing your patch, I can't
> comment as to whether you have accidentally broken things.

I don't see any way the patch could be causing this. It simply adds
another function and case clause to an already-existing hypercall, and
when you call the hypercall with that option it returns the current
budget of a passed-in vcpu. It doesn't even come close to touching
grant mechanics, and doesn't modify any state - it simply returns a
value that previously was hidden in the kernel.

> Other avenues of investigation are to look at what the xenstored process
> is doing in dom0 (is it idle? or is it spinning?), and to look in the
> xenstored log file to see if anything suspicious occurs.

I tried booting into older, stock kernels. They all work with the
read. However, I do not see why the kernel modification would be the
issue as described above. I also have the dom0 running this kernel and
it reads and writes the XenStore just dandy. Are there any kernel
config issues that could do this? This is very confusing. I have one
instance of it working with reads so it can be inherent to the kernel
build itself...

Xenstore (or any grep-ed "xen" process for that matter) are acting
fine. No changes in memory or CPU when blocking vs before blocking.
xenstored-access.log is this:

[20160515T19:45:42.207Z]  A992         newconn
[20160515T19:45:42.210Z]  A992         endconn
[20160515T19:45:42.223Z]  A993         newconn
[20160515T19:45:42.223Z]  A993         write     /local/domain/15/mbroe \017\001
[20160515T19:45:42.223Z]  A993         endconn
[20160515T19:45:42.232Z]  A994         newconn
[20160515T19:45:42.232Z]  A994.1       setperms  /local/domain/15/mbroe n0 b15
[20160515T19:45:42.233Z]  A994.1       setperms
/local/domain/15/mbroe/gref n0 b15
[20160515T19:45:42.233Z]  A994.1       setperms
/local/domain/15/mbroe/min_budget n0 b15
[20160515T19:45:42.234Z]  A994.1       commit
[20160515T19:45:42.234Z]  A994         endconn
[20160515T19:46:25.215Z]  A995         newconn
[20160515T19:46:25.215Z]  A995         endconn
[20160515T19:46:29.540Z]  A996         newconn
[20160515T19:46:29.541Z]  A996         watch
/local/domain/15/mbroe/max_block max_block
[20160515T19:46:29.541Z]  A996         w event
/local/domain/15/mbroe/max_block max_block
[20160515T19:46:29.542Z]  A996.1       commit
[20160515T19:46:29.543Z]  A996         watch
/local/domain/15/mbroe/num_blocks num_blocks
[20160515T19:46:29.543Z]  A996         w event
/local/domain/15/mbroe/num_blocks num_blocks
[20160515T19:46:29.544Z]  A996.2       commit
[20160515T19:46:29.545Z]  A996.3       write     /local/domain/15/mbroe/gref 8
[20160515T19:46:29.546Z]  A996.3       commit
[20160515T19:46:29.546Z]  A996.4       write
/local/domain/15/mbroe/min_budget 10
[20160515T19:46:29.547Z]  A996.4       commit
[20160515T19:46:29.548Z]  A996         endconn

** Above is dom0, running our kernel, doing both read and write just
fine, and setting permission for dom15. **

[20160515T19:47:44.562Z]  D15          watch
/local/domain/15/mbroe/gref FFFF88007B2F3390
[20160515T19:47:44.562Z]  D15          w event
/local/domain/15/mbroe/gref FFFF88007B2F3390

** Above is shown while dom15 is trying to read, but blocking **

[20160515T19:48:39.392Z]  D15          unwatch
/local/domain/15/mbroe/gref FFFF88007B2F3390

** This line is after terminating dom15 program while its still waiting.

The only issue I could see is the FFFF88007B2F3390. This is supposed
to be the key name a presume (such as gref). Maybe its an issue with
the compiled binary, and it ends up watching on a key that doesn't
exist (nor ever will). I will look into this as it looks promising!
Thanks!

I'm don't see any other logs for xenstore, if there are more I please
point them to me. xenstored.log in same directory is recognized as
binary fine, and when I open anyways all I see is "Xen Storage Daemon,
version 1.0" repeatedly.

Regards,
Dagaen Golomb
Ph.D. Student, University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

>>> I'm having an interesting issue. I am working on a project that
>>> requires me to share memory between dom0 and domUs. I have this
>>> successfully working using the grant table and the XenStore to
>>> communicate grefs.
>>>
>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>> My work also requires some kernel modifications, and we have made
>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>> of Xen 4.7 rc1.
>>>
>>> The guest I mentioned before with the default kernel can still read
>>> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
>>> running our kernel.
>>>
>>> The issue I'm having is with another newly created guest (i.e., it is
>>> not a copy of the working one, this is because I needed more space and
>>> couldn't expand the original disk image). It is also running Ubuntu
>>> 12.04 but has been upgraded to our modified kernel. On this guest I
>>> can write to the XenStore, and see that the writes were indeed
>>> successful using xenstore-ls in dom0. However, when this same guest
>>> attempts to read from the XenStore, it doesn't return an error code
>>> but instead just blocks indefinitely. I've waiting many minutes to
>>> make sure its not just blocking for a while, it appears like it will
>>> block forever. The block is happening when I start the transaction.
>>> I've also tried not using a transaction, in which case it blocks on
>>> the read itself.
>>>
>>> I have an inkling this may be something as simple as a configuration
>>> issue, but I can't seem to find anything. Also, the fact that writes
>>> work fine but reads do not is perplexing me.
>>>
>>> Any help would be appreciated!
>>
>> Nothing should block like this.  Without seeing your patch, I can't
>> comment as to whether you have accidentally broken things.
>
> I don't see any way the patch could be causing this. It simply adds
> another function and case clause to an already-existing hypercall, and
> when you call the hypercall with that option it returns the current
> budget of a passed-in vcpu. It doesn't even come close to touching
> grant mechanics, and doesn't modify any state - it simply returns a
> value that previously was hidden in the kernel.
>
>> Other avenues of investigation are to look at what the xenstored process
>> is doing in dom0 (is it idle? or is it spinning?), and to look in the
>> xenstored log file to see if anything suspicious occurs.
>
> I tried booting into older, stock kernels. They all work with the
> read. However, I do not see why the kernel modification would be the
> issue as described above. I also have the dom0 running this kernel and
> it reads and writes the XenStore just dandy. Are there any kernel
> config issues that could do this? This is very confusing. I have one
> instance of it working with reads so it can be inherent to the kernel
> build itself...
>
> Xenstore (or any grep-ed "xen" process for that matter) are acting
> fine. No changes in memory or CPU when blocking vs before blocking.
> xenstored-access.log is this:
>
> [20160515T19:45:42.207Z]  A992         newconn
> [20160515T19:45:42.210Z]  A992         endconn
> [20160515T19:45:42.223Z]  A993         newconn
> [20160515T19:45:42.223Z]  A993         write     /local/domain/15/mbroe \017\001
> [20160515T19:45:42.223Z]  A993         endconn
> [20160515T19:45:42.232Z]  A994         newconn
> [20160515T19:45:42.232Z]  A994.1       setperms  /local/domain/15/mbroe n0 b15
> [20160515T19:45:42.233Z]  A994.1       setperms
> /local/domain/15/mbroe/gref n0 b15
> [20160515T19:45:42.233Z]  A994.1       setperms
> /local/domain/15/mbroe/min_budget n0 b15
> [20160515T19:45:42.234Z]  A994.1       commit
> [20160515T19:45:42.234Z]  A994         endconn
> [20160515T19:46:25.215Z]  A995         newconn
> [20160515T19:46:25.215Z]  A995         endconn
> [20160515T19:46:29.540Z]  A996         newconn
> [20160515T19:46:29.541Z]  A996         watch
> /local/domain/15/mbroe/max_block max_block
> [20160515T19:46:29.541Z]  A996         w event
> /local/domain/15/mbroe/max_block max_block
> [20160515T19:46:29.542Z]  A996.1       commit
> [20160515T19:46:29.543Z]  A996         watch
> /local/domain/15/mbroe/num_blocks num_blocks
> [20160515T19:46:29.543Z]  A996         w event
> /local/domain/15/mbroe/num_blocks num_blocks
> [20160515T19:46:29.544Z]  A996.2       commit
> [20160515T19:46:29.545Z]  A996.3       write     /local/domain/15/mbroe/gref 8
> [20160515T19:46:29.546Z]  A996.3       commit
> [20160515T19:46:29.546Z]  A996.4       write
> /local/domain/15/mbroe/min_budget 10
> [20160515T19:46:29.547Z]  A996.4       commit
> [20160515T19:46:29.548Z]  A996         endconn
>
> ** Above is dom0, running our kernel, doing both read and write just
> fine, and setting permission for dom15. **
>
> [20160515T19:47:44.562Z]  D15          watch
> /local/domain/15/mbroe/gref FFFF88007B2F3390
> [20160515T19:47:44.562Z]  D15          w event
> /local/domain/15/mbroe/gref FFFF88007B2F3390
>
> ** Above is shown while dom15 is trying to read, but blocking **
>
> [20160515T19:48:39.392Z]  D15          unwatch
> /local/domain/15/mbroe/gref FFFF88007B2F3390
>
> ** This line is after terminating dom15 program while its still waiting.
>
> The only issue I could see is the FFFF88007B2F3390. This is supposed
> to be the key name a presume (such as gref). Maybe its an issue with
> the compiled binary, and it ends up watching on a key that doesn't
> exist (nor ever will). I will look into this as it looks promising!
> Thanks!

Update: I checked further up the logs and the other working kernels
produce values like this as well. Also, I am using the same exact
binary between kernel versions, not recompiling.

>
> I'm don't see any other logs for xenstore, if there are more I please
> point them to me. xenstored.log in same directory is recognized as
> binary fine, and when I open anyways all I see is "Xen Storage Daemon,
> version 1.0" repeatedly.

Regards,
Dagaen Golomb
Ph.D. Student, University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Meng Xu]

On Sun, May 15, 2016 at 3:54 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>> Hi All,
>>>
>>> I'm having an interesting issue. I am working on a project that
>>> requires me to share memory between dom0 and domUs. I have this
>>> successfully working using the grant table and the XenStore to
>>> communicate grefs.
>>>
>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>> My work also requires some kernel modifications, and we have made
>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>> of Xen 4.7 rc1.
>>>
>>> The guest I mentioned before with the default kernel can still read
>>> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
>>> running our kernel.
>>>
>>> The issue I'm having is with another newly created guest (i.e., it is
>>> not a copy of the working one, this is because I needed more space and
>>> couldn't expand the original disk image). It is also running Ubuntu
>>> 12.04 but has been upgraded to our modified kernel. On this guest I
>>> can write to the XenStore, and see that the writes were indeed
>>> successful using xenstore-ls in dom0. However, when this same guest
>>> attempts to read from the XenStore, it doesn't return an error code
>>> but instead just blocks indefinitely. I've waiting many minutes to
>>> make sure its not just blocking for a while, it appears like it will
>>> block forever. The block is happening when I start the transaction.
>>> I've also tried not using a transaction, in which case it blocks on
>>> the read itself.
>>>
>>> I have an inkling this may be something as simple as a configuration
>>> issue, but I can't seem to find anything. Also, the fact that writes
>>> work fine but reads do not is perplexing me.
>>>
>>> Any help would be appreciated!
>>
>> Nothing should block like this.  Without seeing your patch, I can't
>> comment as to whether you have accidentally broken things.
>
> I don't see any way the patch could be causing this. It simply adds
> another function and case clause to an already-existing hypercall, and
> when you call the hypercall with that option it returns the current
> budget of a passed-in vcpu. It doesn't even come close to touching
> grant mechanics, and doesn't modify any state - it simply returns a
> value that previously was hidden in the kernel.
>
>> Other avenues of investigation are to look at what the xenstored process
>> is doing in dom0 (is it idle? or is it spinning?), and to look in the
>> xenstored log file to see if anything suspicious occurs.
>
> I tried booting into older, stock kernels. They all work with the
> read. However, I do not see why the kernel modification would be the
> issue as described above. I also have the dom0 running this kernel and
> it reads and writes the XenStore just dandy. Are there any kernel
> config issues that could do this?

What if you use the .config of the kernel in the working domU to
compile the kernel in the not-working domU?
I assume you used the same kernel source code for both domUs.


Best Regards,

Meng
-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

>>>> Hi All,
>>>>
>>>> I'm having an interesting issue. I am working on a project that
>>>> requires me to share memory between dom0 and domUs. I have this
>>>> successfully working using the grant table and the XenStore to
>>>> communicate grefs.
>>>>
>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>> My work also requires some kernel modifications, and we have made
>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>> of Xen 4.7 rc1.
>>>>
>>>> The guest I mentioned before with the default kernel can still read
>>>> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
>>>> running our kernel.
>>>>
>>>> The issue I'm having is with another newly created guest (i.e., it is
>>>> not a copy of the working one, this is because I needed more space and
>>>> couldn't expand the original disk image). It is also running Ubuntu
>>>> 12.04 but has been upgraded to our modified kernel. On this guest I
>>>> can write to the XenStore, and see that the writes were indeed
>>>> successful using xenstore-ls in dom0. However, when this same guest
>>>> attempts to read from the XenStore, it doesn't return an error code
>>>> but instead just blocks indefinitely. I've waiting many minutes to
>>>> make sure its not just blocking for a while, it appears like it will
>>>> block forever. The block is happening when I start the transaction.
>>>> I've also tried not using a transaction, in which case it blocks on
>>>> the read itself.
>>>>
>>>> I have an inkling this may be something as simple as a configuration
>>>> issue, but I can't seem to find anything. Also, the fact that writes
>>>> work fine but reads do not is perplexing me.
>>>>
>>>> Any help would be appreciated!
>>>
>>> Nothing should block like this.  Without seeing your patch, I can't
>>> comment as to whether you have accidentally broken things.
>>
>> I don't see any way the patch could be causing this. It simply adds
>> another function and case clause to an already-existing hypercall, and
>> when you call the hypercall with that option it returns the current
>> budget of a passed-in vcpu. It doesn't even come close to touching
>> grant mechanics, and doesn't modify any state - it simply returns a
>> value that previously was hidden in the kernel.
>>
>>> Other avenues of investigation are to look at what the xenstored process
>>> is doing in dom0 (is it idle? or is it spinning?), and to look in the
>>> xenstored log file to see if anything suspicious occurs.
>>
>> I tried booting into older, stock kernels. They all work with the
>> read. However, I do not see why the kernel modification would be the
>> issue as described above. I also have the dom0 running this kernel and
>> it reads and writes the XenStore just dandy. Are there any kernel
>> config issues that could do this?
>
> What if you use the .config of the kernel in the working domU to
> compile the kernel in the not-working domU?
> I assume you used the same kernel source code for both domUs.

I could try this. Right now I did the same procedure, but not the same
.config (both times working off of oldconfig and then defaults for new
items). I'm not sure if it really makes sense for them to have the
same config, but its worth a try.

Also, I did use the same source for both.

Regards,
Dagaen Golomb
Ph.D. Student, University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

>>>>> Hi All,
>>>>>
>>>>> I'm having an interesting issue. I am working on a project that
>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>> successfully working using the grant table and the XenStore to
>>>>> communicate grefs.
>>>>>
>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>> My work also requires some kernel modifications, and we have made
>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>> of Xen 4.7 rc1.
>>>>>
>>>>> The guest I mentioned before with the default kernel can still read
>>>>> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
>>>>> running our kernel.
>>>>>
>>>>> The issue I'm having is with another newly created guest (i.e., it is
>>>>> not a copy of the working one, this is because I needed more space and
>>>>> couldn't expand the original disk image). It is also running Ubuntu
>>>>> 12.04 but has been upgraded to our modified kernel. On this guest I
>>>>> can write to the XenStore, and see that the writes were indeed
>>>>> successful using xenstore-ls in dom0. However, when this same guest
>>>>> attempts to read from the XenStore, it doesn't return an error code
>>>>> but instead just blocks indefinitely. I've waiting many minutes to
>>>>> make sure its not just blocking for a while, it appears like it will
>>>>> block forever. The block is happening when I start the transaction.
>>>>> I've also tried not using a transaction, in which case it blocks on
>>>>> the read itself.
>>>>>
>>>>> I have an inkling this may be something as simple as a configuration
>>>>> issue, but I can't seem to find anything. Also, the fact that writes
>>>>> work fine but reads do not is perplexing me.
>>>>>
>>>>> Any help would be appreciated!
>>>>
>>>> Nothing should block like this.  Without seeing your patch, I can't
>>>> comment as to whether you have accidentally broken things.
>>>
>>> I don't see any way the patch could be causing this. It simply adds
>>> another function and case clause to an already-existing hypercall, and
>>> when you call the hypercall with that option it returns the current
>>> budget of a passed-in vcpu. It doesn't even come close to touching
>>> grant mechanics, and doesn't modify any state - it simply returns a
>>> value that previously was hidden in the kernel.
>>>
>>>> Other avenues of investigation are to look at what the xenstored process
>>>> is doing in dom0 (is it idle? or is it spinning?), and to look in the
>>>> xenstored log file to see if anything suspicious occurs.
>>>
>>> I tried booting into older, stock kernels. They all work with the
>>> read. However, I do not see why the kernel modification would be the
>>> issue as described above. I also have the dom0 running this kernel and
>>> it reads and writes the XenStore just dandy. Are there any kernel
>>> config issues that could do this?
>>
>> What if you use the .config of the kernel in the working domU to
>> compile the kernel in the not-working domU?
>> I assume you used the same kernel source code for both domUs.
>
> I could try this. Right now I did the same procedure, but not the same
> .config (both times working off of oldconfig and then defaults for new
> items). I'm not sure if it really makes sense for them to have the
> same config, but its worth a try.
>
> Also, I did use the same source for both.

I just tried using the same .config file/ Due to differences in the
tool stack (notably gcc version) the only change was the stack
protector from strong to normal. Compiled as expected but issue still
persists. I'm stumped.

Regards,
Dagaen Golomb
Ph.D. Student, University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 1175 bytes --]

On 5/15/16 11:40 AM, Dagaen Golomb wrote:
> Hi All,
> 
> I'm having an interesting issue. I am working on a project that
> requires me to share memory between dom0 and domUs. I have this
> successfully working using the grant table and the XenStore to
> communicate grefs.
> 
> My issue is this. I have one domU running Ubuntu 12.04 with a default
> 3.8.x kernel that has no issue reading or writing from the XenStore.
> My work also requires some kernel modifications, and we have made
> these changes in the 4.1.0 kernel. In particular, we've only added a
> simple hypercall. This modified kernel is what dom0 is running, on top
> of Xen 4.7 rc1.

Without reading the rest of the thread but seeing the kernel versions.
Can you check how you're communicating to xenstore? Is it via
/dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
that default didn't land until Xen 4.7. Since you're on the right
versions I expect you're using /dev/xen/xenbus but you never know.

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
> > Hi All,
> >
> > I'm having an interesting issue. I am working on a project that
> > requires me to share memory between dom0 and domUs. I have this
> > successfully working using the grant table and the XenStore to
> > communicate grefs.
> >
> > My issue is this. I have one domU running Ubuntu 12.04 with a default
> > 3.8.x kernel that has no issue reading or writing from the XenStore.
> > My work also requires some kernel modifications, and we have made
> > these changes in the 4.1.0 kernel. In particular, we've only added a
> > simple hypercall. This modified kernel is what dom0 is running, on top
> > of Xen 4.7 rc1.
>
> Without reading the rest of the thread but seeing the kernel versions.
> Can you check how you're communicating to xenstore? Is it via
> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
> that default didn't land until Xen 4.7. Since you're on the right
> versions I expect you're using /dev/xen/xenbus but you never know.

How do I know which is being used? /dev/xen/xenbus is there and so is
process/xen/xenbus. Could this be a problem with header version
mismatches or something similar? I'm using the xen/xenstore.h header
file for all of my xenstore interactions. I'm running Xen 4.7 so it
should be in /dev/, and the old kernel is before 3.14 but the new one
is after, but I would presume the standard headers are updated to
account for this. Is there an easy way to check for this? Also, would
the same issue cause writes to fails? Because writes from the same
domain work fine, and appear to other domains using xenstore-ls.

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 1999 bytes --]

On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>> Hi All,
>>>
>>> I'm having an interesting issue. I am working on a project that
>>> requires me to share memory between dom0 and domUs. I have this
>>> successfully working using the grant table and the XenStore to
>>> communicate grefs.
>>>
>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>> My work also requires some kernel modifications, and we have made
>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>> of Xen 4.7 rc1.
>>
>> Without reading the rest of the thread but seeing the kernel versions.
>> Can you check how you're communicating to xenstore? Is it via
>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>> that default didn't land until Xen 4.7. Since you're on the right
>> versions I expect you're using /dev/xen/xenbus but you never know.
> 
> How do I know which is being used? /dev/xen/xenbus is there and so is
> process/xen/xenbus. Could this be a problem with header version
> mismatches or something similar? I'm using the xen/xenstore.h header
> file for all of my xenstore interactions. I'm running Xen 4.7 so it
> should be in /dev/, and the old kernel is before 3.14 but the new one
> is after, but I would presume the standard headers are updated to
> account for this. Is there an easy way to check for this? Also, would
> the same issue cause writes to fails? Because writes from the same
> domain work fine, and appear to other domains using xenstore-ls.
> 
> Regards,
> Dagaen Golomb
> 

Use strace on the process and see what gets opened.

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>> Hi All,
>>>>
>>>> I'm having an interesting issue. I am working on a project that
>>>> requires me to share memory between dom0 and domUs. I have this
>>>> successfully working using the grant table and the XenStore to
>>>> communicate grefs.
>>>>
>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>> My work also requires some kernel modifications, and we have made
>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>> of Xen 4.7 rc1.
>>>
>>> Without reading the rest of the thread but seeing the kernel versions.
>>> Can you check how you're communicating to xenstore? Is it via
>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>> that default didn't land until Xen 4.7. Since you're on the right
>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>
>> How do I know which is being used? /dev/xen/xenbus is there and so is
>> process/xen/xenbus. Could this be a problem with header version
>> mismatches or something similar? I'm using the xen/xenstore.h header
>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>> should be in /dev/, and the old kernel is before 3.14 but the new one
>> is after, but I would presume the standard headers are updated to
>> account for this. Is there an easy way to check for this? Also, would
>> the same issue cause writes to fails? Because writes from the same
>> domain work fine, and appear to other domains using xenstore-ls.
>>
>> Regards,
>> Dagaen Golomb
>>
>
> Use strace on the process and see what gets opened.

Ah, of course. It seems both the working and non-working domains are
using /proc/...

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Meng Xu]

On Sun, May 15, 2016 at 9:41 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>> Hi All,
>>>>>
>>>>> I'm having an interesting issue. I am working on a project that
>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>> successfully working using the grant table and the XenStore to
>>>>> communicate grefs.
>>>>>
>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>> My work also requires some kernel modifications, and we have made
>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>> of Xen 4.7 rc1.
>>>>
>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>> Can you check how you're communicating to xenstore? Is it via
>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>
>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>> process/xen/xenbus. Could this be a problem with header version
>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>> is after, but I would presume the standard headers are updated to
>>> account for this. Is there an easy way to check for this? Also, would
>>> the same issue cause writes to fails? Because writes from the same
>>> domain work fine, and appear to other domains using xenstore-ls.
>>>
>>> Regards,
>>> Dagaen Golomb
>>>
>>
>> Use strace on the process and see what gets opened.
>
> Ah, of course. It seems both the working and non-working domains are
> using /proc/...

Then according to Doug, "Anything after 3.14 will give you
> deadlocks if you try to use /proc/xen/xenbus.". Maybe the non-working domU uses kernel version after 3.14?

Meng
-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

[-- Attachment #1.1: Type: text/plain, Size: 2621 bytes --]

> >>>>> Hi All,
> >>>>>
> >>>>> I'm having an interesting issue. I am working on a project that
> >>>>> requires me to share memory between dom0 and domUs. I have this
> >>>>> successfully working using the grant table and the XenStore to
> >>>>> communicate grefs.
> >>>>>
> >>>>> My issue is this. I have one domU running Ubuntu 12.04 with a
default
> >>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
> >>>>> My work also requires some kernel modifications, and we have made
> >>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
> >>>>> simple hypercall. This modified kernel is what dom0 is running, on
top
> >>>>> of Xen 4.7 rc1.
> >>>>
> >>>> Without reading the rest of the thread but seeing the kernel
versions.
> >>>> Can you check how you're communicating to xenstore? Is it via
> >>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give
you
> >>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer
should
> >>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
> >>>> that default didn't land until Xen 4.7. Since you're on the right
> >>>> versions I expect you're using /dev/xen/xenbus but you never know.
> >>>
> >>> How do I know which is being used? /dev/xen/xenbus is there and so is
> >>> process/xen/xenbus. Could this be a problem with header version
> >>> mismatches or something similar? I'm using the xen/xenstore.h header
> >>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
> >>> should be in /dev/, and the old kernel is before 3.14 but the new one
> >>> is after, but I would presume the standard headers are updated to
> >>> account for this. Is there an easy way to check for this? Also, would
> >>> the same issue cause writes to fails? Because writes from the same
> >>> domain work fine, and appear to other domains using xenstore-ls.
> >>>
> >>> Regards,
> >>> Dagaen Golomb
> >>>
> >>
> >> Use strace on the process and see what gets opened.
> >
> > Ah, of course. It seems both the working and non-working domains are
> > using /proc/...
>
> Then according to Doug, "Anything after 3.14 will give you
> > deadlocks if you try to use /proc/xen/xenbus.". Maybe the non-working
domU uses kernel version after 3.14.

It does, being the custom kernel on version 4.1.0. But Dom0 uses this same
exact kernel and reads/writes just fine! The only solution if this is
indeed the problem appears to be changing the kernel source we build on or
some hacky method such as symlinking /proc/.. to /dev/.., there has to be
an elegant real solution to this...

Regards,
Dagaen Golomb

[-- Attachment #1.2: Type: text/html, Size: 3506 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Meng Xu]

Hi Doug,

Do you happen to know if Xen has an existing mechanism to make
/dev/xen/xenbus as a default device for xenstored?

On Sun, May 15, 2016 at 11:30 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>> >>>>> Hi All,
>> >>>>>
>> >>>>> I'm having an interesting issue. I am working on a project that
>> >>>>> requires me to share memory between dom0 and domUs. I have this
>> >>>>> successfully working using the grant table and the XenStore to
>> >>>>> communicate grefs.
>> >>>>>
>> >>>>> My issue is this. I have one domU running Ubuntu 12.04 with a
>> >>>>> default
>> >>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>> >>>>> My work also requires some kernel modifications, and we have made
>> >>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>> >>>>> simple hypercall. This modified kernel is what dom0 is running, on
>> >>>>> top
>> >>>>> of Xen 4.7 rc1.
>> >>>>
>> >>>> Without reading the rest of the thread but seeing the kernel
>> >>>> versions.
>> >>>> Can you check how you're communicating to xenstore? Is it via
>> >>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give
>> >>>> you
>> >>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer
>> >>>> should
>> >>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>> >>>> that default didn't land until Xen 4.7. Since you're on the right
>> >>>> versions I expect you're using /dev/xen/xenbus but you never know.
>> >>>
>> >>> How do I know which is being used? /dev/xen/xenbus is there and so is
>> >>> process/xen/xenbus. Could this be a problem with header version
>> >>> mismatches or something similar? I'm using the xen/xenstore.h header
>> >>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>> >>> should be in /dev/, and the old kernel is before 3.14 but the new one
>> >>> is after, but I would presume the standard headers are updated to
>> >>> account for this. Is there an easy way to check for this? Also, would
>> >>> the same issue cause writes to fails? Because writes from the same
>> >>> domain work fine, and appear to other domains using xenstore-ls.
>> >>>
>> >>> Regards,
>> >>> Dagaen Golomb
>> >>>
>> >>
>> >> Use strace on the process and see what gets opened.
>> >
>> > Ah, of course. It seems both the working and non-working domains are
>> > using /proc/...
>>
>> Then according to Doug, "Anything after 3.14 will give you
>> > deadlocks if you try to use /proc/xen/xenbus.". Maybe the non-working
>> > domU uses kernel version after 3.14.
>
> It does, being the custom kernel on version 4.1.0. But Dom0 uses this same
> exact kernel and reads/writes just fine! The only solution if this is indeed
> the problem appears to be changing the kernel source we build on or some
> hacky method such as symlinking /proc/.. to /dev/.., there has to be an
> elegant real solution to this...

Hi Dagaen,

Maybe we can try to create a symlink /proc/xen/xenbus to
/dev/xen/xenbus and see if works.

I'm not that sure about if you can just define a env. variable
XENSTORED_PATH to /dev/xen/xenbus to make /dev/xen/xenbus as a default
choice.. But it's no harm to try.

BTW, this is a useful link to refer to:
http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg01679.html


Best Regards,

Meng
-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Andrew Cooper]

On 16/05/16 04:57, Meng Xu wrote:
> Then according to Doug, "Anything after 3.14 will give you
>>>> deadlocks if you try to use /proc/xen/xenbus.". Maybe the non-working
>>>> domU uses kernel version after 3.14.
>> It does, being the custom kernel on version 4.1.0. But Dom0 uses this same
>> exact kernel and reads/writes just fine! The only solution if this is indeed
>> the problem appears to be changing the kernel source we build on or some
>> hacky method such as symlinking /proc/.. to /dev/.., there has to be an
>> elegant real solution to this...
> Hi Dagaen,
>
> Maybe we can try to create a symlink /proc/xen/xenbus to
> /dev/xen/xenbus and see if works.

Set XENSTORED_PATH=/dev/xen/xenbus in the environment.  This should
override the default order.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Jonathan Creekmore]

Dagaen Golomb writes:

> It does, being the custom kernel on version 4.1.0. But Dom0 uses this same
> exact kernel and reads/writes just fine! The only solution if this is
> indeed the problem appears to be changing the kernel source we build on or
> some hacky method such as symlinking /proc/.. to /dev/.., there has to be
> an elegant real solution to this...

When I was chasing down a similar problem back in August, I discovered
that Dom0 accesses XenStore through a unix domain socket and not through
the kernel interface, so hitting it through xenstore-ls always seemed to
work. That is why Dom0 reads/writes to XenStore ok with the same kernel
that fails for your DomU. When I moved XenStore to a stubdomain, I /did/
see the problem in Dom0 with new enough kernels, which is why the
XenStore access library was changed to prefer the dev device instead of
the proc file.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

[-- Attachment #1.1: Type: text/plain, Size: 1171 bytes --]

Regards,
Dagaen Golomb
On May 16, 2016 09:13, "Jonathan Creekmore" <jonathan.creekmore@gmail.com>
wrote:
>
>
> Dagaen Golomb writes:
>
> > It does, being the custom kernel on version 4.1.0. But Dom0 uses this
same
> > exact kernel and reads/writes just fine! The only solution if this is
> > indeed the problem appears to be changing the kernel source we build on
or
> > some hacky method such as symlinking /proc/.. to /dev/.., there has to
be
> > an elegant real solution to this...
>
> When I was chasing down a similar problem back in August, I discovered
> that Dom0 accesses XenStore through a unix domain socket and not through
> the kernel interface, so hitting it through xenstore-ls always seemed to
> work. That is why Dom0 reads/writes to XenStore ok with the same kernel
> that fails for your DomU. When I moved XenStore to a stubdomain, I /did/
> see the problem in Dom0 with new enough kernels, which is why the
> XenStore access library was changed to prefer the dev device instead of
> the proc file.

Great, I'll try out the XENSTORED_PATH variable when I get the chance and
see if it resolves the issue.

Regards,
Dagaen Golomb

Regards,
Dagaen Golomb

[-- Attachment #1.2: Type: text/html, Size: 1463 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 2356 bytes --]

On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>> Hi All,
>>>>>
>>>>> I'm having an interesting issue. I am working on a project that
>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>> successfully working using the grant table and the XenStore to
>>>>> communicate grefs.
>>>>>
>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>> My work also requires some kernel modifications, and we have made
>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>> of Xen 4.7 rc1.
>>>>
>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>> Can you check how you're communicating to xenstore? Is it via
>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>
>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>> process/xen/xenbus. Could this be a problem with header version
>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>> is after, but I would presume the standard headers are updated to
>>> account for this. Is there an easy way to check for this? Also, would
>>> the same issue cause writes to fails? Because writes from the same
>>> domain work fine, and appear to other domains using xenstore-ls.
>>>
>>> Regards,
>>> Dagaen Golomb
>>>
>>
>> Use strace on the process and see what gets opened.
> 
> Ah, of course. It seems both the working and non-working domains are
> using /proc/...
> 
> Regards,
> Dagaen Golomb
> 


How are you starting them? Can you confirm its attempting /dev/ first?
Xen 4.7 should prefer /dev/.

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>> successfully working using the grant table and the XenStore to
>>>>>> communicate grefs.
>>>>>>
>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>> My work also requires some kernel modifications, and we have made
>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>> of Xen 4.7 rc1.
>>>>>
>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>
>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>> process/xen/xenbus. Could this be a problem with header version
>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>> is after, but I would presume the standard headers are updated to
>>>> account for this. Is there an easy way to check for this? Also, would
>>>> the same issue cause writes to fails? Because writes from the same
>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>
>>>> Regards,
>>>> Dagaen Golomb
>>>>
>>>
>>> Use strace on the process and see what gets opened.
>>
>> Ah, of course. It seems both the working and non-working domains are
>> using /proc/...
>>
>> Regards,
>> Dagaen Golomb
>>
>
>
> How are you starting them? Can you confirm its attempting /dev/ first?
> Xen 4.7 should prefer /dev/.

The previous versions used /proc/ as well according to strace, but
worked. According to others, this may be an issue with newer kernels
(3.14+) using /proc/.

I experimented with our 4.1.0 kernel. At first it still wasn't working
because I was using sudo as the program needs superuser access... each
time this cleared out any environment.
Using sudo su, then setting the environment to use /dev/, and then
running the program successfully reads! I'll consider this the
solution. Thanks everyone.

Now to make this environment setting apply to all users/shells,
directly from boot. I guess /etc/bashrc will do the trick?

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>>> successfully working using the grant table and the XenStore to
>>>>>>> communicate grefs.
>>>>>>>
>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>>> My work also requires some kernel modifications, and we have made
>>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>>> of Xen 4.7 rc1.
>>>>>>
>>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>>
>>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>>> process/xen/xenbus. Could this be a problem with header version
>>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>>> is after, but I would presume the standard headers are updated to
>>>>> account for this. Is there an easy way to check for this? Also, would
>>>>> the same issue cause writes to fails? Because writes from the same
>>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>>
>>>>> Regards,
>>>>> Dagaen Golomb
>>>>>
>>>>
>>>> Use strace on the process and see what gets opened.
>>>
>>> Ah, of course. It seems both the working and non-working domains are
>>> using /proc/...
>>>
>>> Regards,
>>> Dagaen Golomb
>>>
>>
>>
>> How are you starting them? Can you confirm its attempting /dev/ first?
>> Xen 4.7 should prefer /dev/.
>
> The previous versions used /proc/ as well according to strace, but
> worked. According to others, this may be an issue with newer kernels
> (3.14+) using /proc/.
>
> I experimented with our 4.1.0 kernel. At first it still wasn't working
> because I was using sudo as the program needs superuser access... each
> time this cleared out any environment.
> Using sudo su, then setting the environment to use /dev/, and then
> running the program successfully reads! I'll consider this the
> solution. Thanks everyone.
>
> Now to make this environment setting apply to all users/shells,
> directly from boot. I guess /etc/bashrc will do the trick?

Also, this wasn't fixed when I recompiled on this kernel, so if the
newer headers point to /dev/ instead of /proc/, it must be using the
old headers for the 3.2/3.13 kernels even though its running 4.1.0.
I thought installing the kernel did headers as well, does it not? Or
do I need to place custom kernel headers myself?

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

On Mon, May 16, 2016 at 12:11 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
> On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>>>> successfully working using the grant table and the XenStore to
>>>>>>>> communicate grefs.
>>>>>>>>
>>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>>>> My work also requires some kernel modifications, and we have made
>>>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>>>> of Xen 4.7 rc1.
>>>>>>>
>>>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>>>
>>>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>>>> process/xen/xenbus. Could this be a problem with header version
>>>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>>>> is after, but I would presume the standard headers are updated to
>>>>>> account for this. Is there an easy way to check for this? Also, would
>>>>>> the same issue cause writes to fails? Because writes from the same
>>>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>>>
>>>>>> Regards,
>>>>>> Dagaen Golomb
>>>>>>
>>>>>
>>>>> Use strace on the process and see what gets opened.
>>>>
>>>> Ah, of course. It seems both the working and non-working domains are
>>>> using /proc/...
>>>>
>>>> Regards,
>>>> Dagaen Golomb
>>>>
>>>
>>>
>>> How are you starting them? Can you confirm its attempting /dev/ first?
>>> Xen 4.7 should prefer /dev/.

For all kernels in my domU, without setting any environment variables
they use /proc/.
For 4.1.0 this did not work, but works with /dev/ when using
environment variable.
Is this supposed to try /dev/ before /proc/? Because this doesn't
appear the case according to strace.
I think this is a bug.

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 3313 bytes --]

On 5/16/16 11:20 AM, Dagaen Golomb wrote:
> On Mon, May 16, 2016 at 12:11 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>> On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>>>>> successfully working using the grant table and the XenStore to
>>>>>>>>> communicate grefs.
>>>>>>>>>
>>>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>>>>> My work also requires some kernel modifications, and we have made
>>>>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>>>>> of Xen 4.7 rc1.
>>>>>>>>
>>>>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>>>>
>>>>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>>>>> process/xen/xenbus. Could this be a problem with header version
>>>>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>>>>> is after, but I would presume the standard headers are updated to
>>>>>>> account for this. Is there an easy way to check for this? Also, would
>>>>>>> the same issue cause writes to fails? Because writes from the same
>>>>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Dagaen Golomb
>>>>>>>
>>>>>>
>>>>>> Use strace on the process and see what gets opened.
>>>>>
>>>>> Ah, of course. It seems both the working and non-working domains are
>>>>> using /proc/...
>>>>>
>>>>> Regards,
>>>>> Dagaen Golomb
>>>>>
>>>>
>>>>
>>>> How are you starting them? Can you confirm its attempting /dev/ first?
>>>> Xen 4.7 should prefer /dev/.
> 
> For all kernels in my domU, without setting any environment variables
> they use /proc/.
> For 4.1.0 this did not work, but works with /dev/ when using
> environment variable.
> Is this supposed to try /dev/ before /proc/? Because this doesn't
> appear the case according to strace.
> I think this is a bug.
> 
> Regards,
> Dagaen Golomb
> 

Is your domU using Xen 4.7 compiled utilities? or are you using distro
provided xenstore libraries?

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

On Mon, May 16, 2016 at 3:12 PM, Doug Goldstein <cardoe@cardoe.com> wrote:
> On 5/16/16 11:20 AM, Dagaen Golomb wrote:
>> On Mon, May 16, 2016 at 12:11 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>> On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>>> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>>>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>>>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>>>>>> successfully working using the grant table and the XenStore to
>>>>>>>>>> communicate grefs.
>>>>>>>>>>
>>>>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>>>>>> My work also requires some kernel modifications, and we have made
>>>>>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>>>>>> of Xen 4.7 rc1.
>>>>>>>>>
>>>>>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>>>>>
>>>>>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>>>>>> process/xen/xenbus. Could this be a problem with header version
>>>>>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>>>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>>>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>>>>>> is after, but I would presume the standard headers are updated to
>>>>>>>> account for this. Is there an easy way to check for this? Also, would
>>>>>>>> the same issue cause writes to fails? Because writes from the same
>>>>>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Dagaen Golomb
>>>>>>>>
>>>>>>>
>>>>>>> Use strace on the process and see what gets opened.
>>>>>>
>>>>>> Ah, of course. It seems both the working and non-working domains are
>>>>>> using /proc/...
>>>>>>
>>>>>> Regards,
>>>>>> Dagaen Golomb
>>>>>>
>>>>>
>>>>>
>>>>> How are you starting them? Can you confirm its attempting /dev/ first?
>>>>> Xen 4.7 should prefer /dev/.
>>
>> For all kernels in my domU, without setting any environment variables
>> they use /proc/.
>> For 4.1.0 this did not work, but works with /dev/ when using
>> environment variable.
>> Is this supposed to try /dev/ before /proc/? Because this doesn't
>> appear the case according to strace.
>> I think this is a bug.
>>
>> Regards,
>> Dagaen Golomb
>>
>
> Is your domU using Xen 4.7 compiled utilities? or are you using distro
> provided xenstore libraries?

I am using the newest binaries available from the distro repo (Ubuntu
12.04), so version 4.4.
I'm going to guess you might say this is part of the issue. 4.4 and
4.7 aren't really *that* far off, though.
Legacy interfaces should not be breaking, so I still think this is a
bug that needs to be addressed somewhere, maybe by determining what
exactly the issue is with new kernels and /proc/ tree XenStore access,
if thisis what older tools will be using.

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

On Mon, May 16, 2016 at 3:52 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
> On Mon, May 16, 2016 at 3:12 PM, Doug Goldstein <cardoe@cardoe.com> wrote:
>> On 5/16/16 11:20 AM, Dagaen Golomb wrote:
>>> On Mon, May 16, 2016 at 12:11 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>>> On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>>>> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>>>>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>>>>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>>>>>>> successfully working using the grant table and the XenStore to
>>>>>>>>>>> communicate grefs.
>>>>>>>>>>>
>>>>>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>>>>>>> My work also requires some kernel modifications, and we have made
>>>>>>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>>>>>>> of Xen 4.7 rc1.
>>>>>>>>>>
>>>>>>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>>>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>>>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>>>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>>>>>>
>>>>>>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>>>>>>> process/xen/xenbus. Could this be a problem with header version
>>>>>>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>>>>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>>>>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>>>>>>> is after, but I would presume the standard headers are updated to
>>>>>>>>> account for this. Is there an easy way to check for this? Also, would
>>>>>>>>> the same issue cause writes to fails? Because writes from the same
>>>>>>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Dagaen Golomb
>>>>>>>>>
>>>>>>>>
>>>>>>>> Use strace on the process and see what gets opened.
>>>>>>>
>>>>>>> Ah, of course. It seems both the working and non-working domains are
>>>>>>> using /proc/...
>>>>>>>
>>>>>>> Regards,
>>>>>>> Dagaen Golomb
>>>>>>>
>>>>>>
>>>>>>
>>>>>> How are you starting them? Can you confirm its attempting /dev/ first?
>>>>>> Xen 4.7 should prefer /dev/.
>>>
>>> For all kernels in my domU, without setting any environment variables
>>> they use /proc/.
>>> For 4.1.0 this did not work, but works with /dev/ when using
>>> environment variable.
>>> Is this supposed to try /dev/ before /proc/? Because this doesn't
>>> appear the case according to strace.
>>> I think this is a bug.
>>>
>>> Regards,
>>> Dagaen Golomb
>>>
>>
>> Is your domU using Xen 4.7 compiled utilities? or are you using distro
>> provided xenstore libraries?
>
> I am using the newest binaries available from the distro repo (Ubuntu
> 12.04), so version 4.4.
> I'm going to guess you might say this is part of the issue. 4.4 and
> 4.7 aren't really *that* far off, though.
> Legacy interfaces should not be breaking, so I still think this is a
> bug that needs to be addressed somewhere, maybe by determining what
> exactly the issue is with new kernels and /proc/ tree XenStore access,
> if thisis what older tools will be using.

Should I try to use newer utilities? What would be the easiest method
of acquiring and installing them?
Should building and installing the *-tools directives in the xen
source do the trick? I'd like to see if this fixes the issue.

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

On Mon, May 16, 2016 at 3:56 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
> On Mon, May 16, 2016 at 3:52 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>> On Mon, May 16, 2016 at 3:12 PM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>> On 5/16/16 11:20 AM, Dagaen Golomb wrote:
>>>> On Mon, May 16, 2016 at 12:11 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>>>> On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>>>>> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>>>>>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>>>>>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>>>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>>>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>>>>>>>> successfully working using the grant table and the XenStore to
>>>>>>>>>>>> communicate grefs.
>>>>>>>>>>>>
>>>>>>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>>>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>>>>>>>> My work also requires some kernel modifications, and we have made
>>>>>>>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>>>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>>>>>>>> of Xen 4.7 rc1.
>>>>>>>>>>>
>>>>>>>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>>>>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>>>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>>>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>>>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>>>>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>>>>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>>>>>>>
>>>>>>>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>>>>>>>> process/xen/xenbus. Could this be a problem with header version
>>>>>>>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>>>>>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>>>>>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>>>>>>>> is after, but I would presume the standard headers are updated to
>>>>>>>>>> account for this. Is there an easy way to check for this? Also, would
>>>>>>>>>> the same issue cause writes to fails? Because writes from the same
>>>>>>>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Dagaen Golomb
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Use strace on the process and see what gets opened.
>>>>>>>>
>>>>>>>> Ah, of course. It seems both the working and non-working domains are
>>>>>>>> using /proc/...
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Dagaen Golomb
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> How are you starting them? Can you confirm its attempting /dev/ first?
>>>>>>> Xen 4.7 should prefer /dev/.
>>>>
>>>> For all kernels in my domU, without setting any environment variables
>>>> they use /proc/.
>>>> For 4.1.0 this did not work, but works with /dev/ when using
>>>> environment variable.
>>>> Is this supposed to try /dev/ before /proc/? Because this doesn't
>>>> appear the case according to strace.
>>>> I think this is a bug.
>>>>
>>>> Regards,
>>>> Dagaen Golomb
>>>>
>>>
>>> Is your domU using Xen 4.7 compiled utilities? or are you using distro
>>> provided xenstore libraries?
>>
>> I am using the newest binaries available from the distro repo (Ubuntu
>> 12.04), so version 4.4.
>> I'm going to guess you might say this is part of the issue. 4.4 and
>> 4.7 aren't really *that* far off, though.
>> Legacy interfaces should not be breaking, so I still think this is a
>> bug that needs to be addressed somewhere, maybe by determining what
>> exactly the issue is with new kernels and /proc/ tree XenStore access,
>> if thisis what older tools will be using.
>
> Should I try to use newer utilities? What would be the easiest method
> of acquiring and installing them?
> Should building and installing the *-tools directives in the xen
> source do the trick? I'd like to see if this fixes the issue.

I installed a package for xenstore-utils 4.6, and installed over the
4.4 in my guest. Issue persists.

Regards,
Dagaen Golomb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 4914 bytes --]

On 5/16/16 3:18 PM, Dagaen Golomb wrote:
> On Mon, May 16, 2016 at 3:56 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>> On Mon, May 16, 2016 at 3:52 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>> On Mon, May 16, 2016 at 3:12 PM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>>> On 5/16/16 11:20 AM, Dagaen Golomb wrote:
>>>>> On Mon, May 16, 2016 at 12:11 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>>>>> On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <dgolomb@seas.upenn.edu> wrote:
>>>>>>> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <cardoe@cardoe.com> wrote:
>>>>>>>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
>>>>>>>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
>>>>>>>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm having an interesting issue. I am working on a project that
>>>>>>>>>>>>> requires me to share memory between dom0 and domUs. I have this
>>>>>>>>>>>>> successfully working using the grant table and the XenStore to
>>>>>>>>>>>>> communicate grefs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>>>>>>>>>>>>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>>>>>>>>>>>>> My work also requires some kernel modifications, and we have made
>>>>>>>>>>>>> these changes in the 4.1.0 kernel. In particular, we've only added a
>>>>>>>>>>>>> simple hypercall. This modified kernel is what dom0 is running, on top
>>>>>>>>>>>>> of Xen 4.7 rc1.
>>>>>>>>>>>>
>>>>>>>>>>>> Without reading the rest of the thread but seeing the kernel versions.
>>>>>>>>>>>> Can you check how you're communicating to xenstore? Is it via
>>>>>>>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14 will give you
>>>>>>>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and newer should
>>>>>>>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd but making
>>>>>>>>>>>> that default didn't land until Xen 4.7. Since you're on the right
>>>>>>>>>>>> versions I expect you're using /dev/xen/xenbus but you never know.
>>>>>>>>>>>
>>>>>>>>>>> How do I know which is being used? /dev/xen/xenbus is there and so is
>>>>>>>>>>> process/xen/xenbus. Could this be a problem with header version
>>>>>>>>>>> mismatches or something similar? I'm using the xen/xenstore.h header
>>>>>>>>>>> file for all of my xenstore interactions. I'm running Xen 4.7 so it
>>>>>>>>>>> should be in /dev/, and the old kernel is before 3.14 but the new one
>>>>>>>>>>> is after, but I would presume the standard headers are updated to
>>>>>>>>>>> account for this. Is there an easy way to check for this? Also, would
>>>>>>>>>>> the same issue cause writes to fails? Because writes from the same
>>>>>>>>>>> domain work fine, and appear to other domains using xenstore-ls.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Dagaen Golomb
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Use strace on the process and see what gets opened.
>>>>>>>>>
>>>>>>>>> Ah, of course. It seems both the working and non-working domains are
>>>>>>>>> using /proc/...
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Dagaen Golomb
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> How are you starting them? Can you confirm its attempting /dev/ first?
>>>>>>>> Xen 4.7 should prefer /dev/.
>>>>>
>>>>> For all kernels in my domU, without setting any environment variables
>>>>> they use /proc/.
>>>>> For 4.1.0 this did not work, but works with /dev/ when using
>>>>> environment variable.
>>>>> Is this supposed to try /dev/ before /proc/? Because this doesn't
>>>>> appear the case according to strace.
>>>>> I think this is a bug.
>>>>>
>>>>> Regards,
>>>>> Dagaen Golomb
>>>>>
>>>>
>>>> Is your domU using Xen 4.7 compiled utilities? or are you using distro
>>>> provided xenstore libraries?
>>>
>>> I am using the newest binaries available from the distro repo (Ubuntu
>>> 12.04), so version 4.4.
>>> I'm going to guess you might say this is part of the issue. 4.4 and
>>> 4.7 aren't really *that* far off, though.
>>> Legacy interfaces should not be breaking, so I still think this is a
>>> bug that needs to be addressed somewhere, maybe by determining what
>>> exactly the issue is with new kernels and /proc/ tree XenStore access,
>>> if thisis what older tools will be using.
>>
>> Should I try to use newer utilities? What would be the easiest method
>> of acquiring and installing them?
>> Should building and installing the *-tools directives in the xen
>> source do the trick? I'd like to see if this fixes the issue.
> 
> I installed a package for xenstore-utils 4.6, and installed over the
> 4.4 in my guest. Issue persists.
> 
> Regards,
> Dagaen Golomb
> 

Changing Xen bits won't help until you use Xen 4.7. The underlying issue
is the Linux kernel interface is broken from 3.14 and on.

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Problem Reading from XenStore in DomU

[Dagaen Golomb]

[-- Attachment #1.1: Type: text/plain, Size: 5295 bytes --]

On May 16, 2016 6:25 PM, "Doug Goldstein" <cardoe@cardoe.com> wrote:
>
> On 5/16/16 3:18 PM, Dagaen Golomb wrote:
> > On Mon, May 16, 2016 at 3:56 PM, Dagaen Golomb <dgolomb@seas.upenn.edu>
wrote:
> >> On Mon, May 16, 2016 at 3:52 PM, Dagaen Golomb <dgolomb@seas.upenn.edu>
wrote:
> >>> On Mon, May 16, 2016 at 3:12 PM, Doug Goldstein <cardoe@cardoe.com>
wrote:
> >>>> On 5/16/16 11:20 AM, Dagaen Golomb wrote:
> >>>>> On Mon, May 16, 2016 at 12:11 PM, Dagaen Golomb <
dgolomb@seas.upenn.edu> wrote:
> >>>>>> On Mon, May 16, 2016 at 12:03 PM, Dagaen Golomb <
dgolomb@seas.upenn.edu> wrote:
> >>>>>>> On Mon, May 16, 2016 at 11:55 AM, Doug Goldstein <
cardoe@cardoe.com> wrote:
> >>>>>>>> On 5/15/16 8:41 PM, Dagaen Golomb wrote:
> >>>>>>>>>> On 5/15/16 8:28 PM, Dagaen Golomb wrote:
> >>>>>>>>>>>> On 5/15/16 11:40 AM, Dagaen Golomb wrote:
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm having an interesting issue. I am working on a project
that
> >>>>>>>>>>>>> requires me to share memory between dom0 and domUs. I have
this
> >>>>>>>>>>>>> successfully working using the grant table and the XenStore
to
> >>>>>>>>>>>>> communicate grefs.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> My issue is this. I have one domU running Ubuntu 12.04 with
a default
> >>>>>>>>>>>>> 3.8.x kernel that has no issue reading or writing from the
XenStore.
> >>>>>>>>>>>>> My work also requires some kernel modifications, and we
have made
> >>>>>>>>>>>>> these changes in the 4.1.0 kernel. In particular, we've
only added a
> >>>>>>>>>>>>> simple hypercall. This modified kernel is what dom0 is
running, on top
> >>>>>>>>>>>>> of Xen 4.7 rc1.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Without reading the rest of the thread but seeing the kernel
versions.
> >>>>>>>>>>>> Can you check how you're communicating to xenstore? Is it via
> >>>>>>>>>>>> /dev/xen/xenbus or /proc/xen/xenbus? Anything after 3.14
will give you
> >>>>>>>>>>>> deadlocks if you try to use /proc/xen/xenbus. Xen 4.6 and
newer should
> >>>>>>>>>>>> prefer /dev/xen/xenbus. Same thing can happen with privcmd
but making
> >>>>>>>>>>>> that default didn't land until Xen 4.7. Since you're on the
right
> >>>>>>>>>>>> versions I expect you're using /dev/xen/xenbus but you never
know.
> >>>>>>>>>>>
> >>>>>>>>>>> How do I know which is being used? /dev/xen/xenbus is there
and so is
> >>>>>>>>>>> process/xen/xenbus. Could this be a problem with header
version
> >>>>>>>>>>> mismatches or something similar? I'm using the xen/xenstore.h
header
> >>>>>>>>>>> file for all of my xenstore interactions. I'm running Xen 4.7
so it
> >>>>>>>>>>> should be in /dev/, and the old kernel is before 3.14 but the
new one
> >>>>>>>>>>> is after, but I would presume the standard headers are
updated to
> >>>>>>>>>>> account for this. Is there an easy way to check for this?
Also, would
> >>>>>>>>>>> the same issue cause writes to fails? Because writes from the
same
> >>>>>>>>>>> domain work fine, and appear to other domains using
xenstore-ls.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Dagaen Golomb
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Use strace on the process and see what gets opened.
> >>>>>>>>>
> >>>>>>>>> Ah, of course. It seems both the working and non-working
domains are
> >>>>>>>>> using /proc/...
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Dagaen Golomb
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> How are you starting them? Can you confirm its attempting /dev/
first?
> >>>>>>>> Xen 4.7 should prefer /dev/.
> >>>>>
> >>>>> For all kernels in my domU, without setting any environment
variables
> >>>>> they use /proc/.
> >>>>> For 4.1.0 this did not work, but works with /dev/ when using
> >>>>> environment variable.
> >>>>> Is this supposed to try /dev/ before /proc/? Because this doesn't
> >>>>> appear the case according to strace.
> >>>>> I think this is a bug.
> >>>>>
> >>>>> Regards,
> >>>>> Dagaen Golomb
> >>>>>
> >>>>
> >>>> Is your domU using Xen 4.7 compiled utilities? or are you using
distro
> >>>> provided xenstore libraries?
> >>>
> >>> I am using the newest binaries available from the distro repo (Ubuntu
> >>> 12.04), so version 4.4.
> >>> I'm going to guess you might say this is part of the issue. 4.4 and
> >>> 4.7 aren't really *that* far off, though.
> >>> Legacy interfaces should not be breaking, so I still think this is a
> >>> bug that needs to be addressed somewhere, maybe by determining what
> >>> exactly the issue is with new kernels and /proc/ tree XenStore access,
> >>> if thisis what older tools will be using.
> >>
> >> Should I try to use newer utilities? What would be the easiest method
> >> of acquiring and installing them?
> >> Should building and installing the *-tools directives in the xen
> >> source do the trick? I'd like to see if this fixes the issue.
> >
> > I installed a package for xenstore-utils 4.6, and installed over the
> > 4.4 in my guest. Issue persists.
> >
> > Regards,
> > Dagaen Golomb
> >
>
> Changing Xen bits won't help until you use Xen 4.7. The underlying issue
> is the Linux kernel interface is broken from 3.14 and on.

OK, thanks. So this is actually an issue in the Linux codebase and not the
Xen one? Good to know.

For now its been resolved with the environment variable and this will work
for out purposes. Thanks for the help!

Regards,
Dagaen Golomb

[-- Attachment #1.2: Type: text/html, Size: 8585 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel