One shared memory read/write issue between 2 different domains

One shared memory read/write issue between 2 different domains

[chunjie_zhu]

[-- Attachment #1.1: Type: text/plain, Size: 1952 bytes --]

Dear xen-devel team,



Recently I encountered one technical issue on Xen platform. I want to request your help.



[Problem Description]

I code one ring buffer by myself for 2 domains to transmit data, one domain is producer, and the other is consumer. I do not use the ring buffer defined in “include/xen/interface/io/ring.h”. Just like the ring buffer provided by xen, I also use 2 pointers for producer and consumer, separately. To ensure memory access order is consistent, I use memory barrier between memory store/load operation and pointer update operation.



The transmitted data contains 2 parts, one is header, and the other is payload. Each time, one domain transmits the header first, and then the payload is sent out.



Each time I can get right header at the consumer side, however, sometimes, the data I get at the consumer side contains part of invalid information.



[Develop Environment]

Xen-4.0.1 + Linux-2.6.24-29-xen



Could you please give me some suggestion on this issue?



[Other Questions]

1.      Is it reasonable to use spin lock to protect shared memory between 2 different domains?

2.      Previously I use one atomic variable to do the synchronization between producer and consumer, however, sometimes I cannot get correct header at the consumer side. Is it one known-issue?



I am looking forward to your letters. Thanks in advance.



Regards,
Chunjie

============================================



阴风催急雨，木叶舞萧萧。

闲看天色变，忧心朋宾来。




TREND MICRO EMAIL NOTICE
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.

[-- Attachment #1.2: Type: text/html, Size: 8722 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: One shared memory read/write issue between 2 different domains

[Tim Deegan]

Hi, 

At 17:39 +0800 on 25 May (1306345173), chunjie_zhu@trendmicro.com.cn wrote:
> I code one ring buffer by myself for 2 domains to transmit data, one
> domain is producer, and the other is consumer. I do not use the ring
> buffer defined in ??include/xen/interface/io/ring.h??.

Why not?  That one has had a lot of testing and debugging.

> Just like the ring buffer provided by xen, I also use 2 pointers for
> producer and consumer, separately. To ensure memory access order is
> consistent, I use memory barrier between memory store/load operation
> and pointer update operation.
> 
> The transmitted data contains 2 parts, one is header, and the other is
> payload. Each time, one domain transmits the header first, and then
> the payload is sent out.
> 
> Each time I can get right header at the consumer side, however,
> sometimes, the data I get at the consumer side contains part of
> invalid information.
> 
> Could you please give me some suggestion on this issue?

Hard to say from just this description, but probably you're missing
either a hardware memory barrier or a compiler one.   Either that or
something in the rest of your code is corrupting the rings. 

Try comparing your code very carefully to the existing, working ring
buffer code.

> [Other Questions]
> 
> 1.  Is it reasonable to use spin lock to protect shared memory between
> 2 different domains?

It's probably not a great idea.  You have to worry about what happens if
one of the domains crashes while holding the lock, for example.

> 2.  Previously I use one atomic variable to do the synchronization
> between producer and consumer, however, sometimes I cannot get correct
> header at the consumer side. Is it one known-issue?

Again, you're probably missing a compiler barrier.  AIUI, updates to
atomic variables are only guaranteed not to be reordered WRT other write
of the _same_ variable.

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

RE: One shared memory read/write issue between 2 different domains

[chunjie_zhu]

Hi Tim，

Thanks very much for your answer. I will follow your suggestion and try again.

Regards,
Chunjie
============================================
一盘棋不同色子，
一样人各样心肠。

-----Original Message-----
From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
Sent: 2011年6月6日 17:10
To: Chunjie Zhu (RD-CN)
Cc: xen-devel@lists.xensource.com; Jianxin Guo (RD-CN)
Subject: Re: [Xen-devel] One shared memory read/write issue between 2 different domains

Hi,

At 17:39 +0800 on 25 May (1306345173), chunjie_zhu@trendmicro.com.cn wrote:
> I code one ring buffer by myself for 2 domains to transmit data, one
> domain is producer, and the other is consumer. I do not use the ring
> buffer defined in ??include/xen/interface/io/ring.h??.

Why not?  That one has had a lot of testing and debugging.

> Just like the ring buffer provided by xen, I also use 2 pointers for
> producer and consumer, separately. To ensure memory access order is
> consistent, I use memory barrier between memory store/load operation
> and pointer update operation.
>
> The transmitted data contains 2 parts, one is header, and the other is
> payload. Each time, one domain transmits the header first, and then
> the payload is sent out.
>
> Each time I can get right header at the consumer side, however,
> sometimes, the data I get at the consumer side contains part of
> invalid information.
>
> Could you please give me some suggestion on this issue?

Hard to say from just this description, but probably you're missing
either a hardware memory barrier or a compiler one.   Either that or
something in the rest of your code is corrupting the rings.

Try comparing your code very carefully to the existing, working ring buffer code.

> [Other Questions]
>
> 1.  Is it reasonable to use spin lock to protect shared memory between
> 2 different domains?

It's probably not a great idea.  You have to worry about what happens if one of the domains crashes while holding the lock, for example.

> 2.  Previously I use one atomic variable to do the synchronization
> between producer and consumer, however, sometimes I cannot get correct
> header at the consumer side. Is it one known-issue?

Again, you're probably missing a compiler barrier.  AIUI, updates to atomic variables are only guaranteed not to be reordered WRT other write of the _same_ variable.

Tim.

--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)
TREND MICRO EMAIL NOTICE
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.