RE: Memory ordering question in the shutdown deferral code

["Durrant, Paul" <pdurrant@amazon.co.uk>]

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 21 September 2020 12:41
> To: Jan Beulich <jbeulich@suse.com>; Stefano Stabellini <sstabellini@kernel.org>;
> andrew.cooper3@citrix.com; George Dunlap <george.dunlap@citrix.com>; Durrant, Paul
> <pdurrant@amazon.co.uk>
> Cc: Xia, Hongyan <hongyxia@amazon.com>; xen-devel@lists.xenproject.org
> Subject: RE: [EXTERNAL] Memory ordering question in the shutdown deferral code
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open
> attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> (+ Xen-devel)
> 
> Sorry I forgot to CC xen-devel.
> 
> On 21/09/2020 12:38, Julien Grall wrote:
> > Hi all,
> >
> > I have started to look at the deferral code (see
> > vcpu_start_shutdown_deferral()) because we need it for LiveUpdate and
> > Arm will soon use it.
> >
> > The current implementation is using an smp_mb() to ensure ordering
> > between a write then a read. The code looks roughly (I have slightly
> > adapted it to make my question more obvious):
> >
> > domain_shutdown()
> >      d->is_shutting_down = 1;
> >      smp_mb();
> >      if ( !vcpu0->defer_shutdown )
> >      {
> >        vcpu_pause_nosync(v);
> >        v->paused_for_shutdown = 1;
> >      }
> >
> > vcpu_start_shutdown_deferral()
> >      vcpu0->defer_shutdown = 1;
> >      smp_mb();
> >      if ( unlikely(d->is_shutting_down) )
> >        vcpu_check_shutdown(v);
> >
> >      return vcpu0->defer_shutdown;
> >
> > smp_mb() should only guarantee ordering (this may be stronger on some
> > arch), so I think there is a race between the two functions.
> >
> > It would be possible to pause the vCPU in domain_shutdown() because
> > vcpu0->defer_shutdown wasn't yet seen.
> >
> > Equally, vcpu_start_shutdown_deferral() may not see d->is_shutting_down
> > and therefore Xen may continue to send the I/O. Yet the vCPU will be
> > paused so the I/O will never complete.
> >

The barrier enforces global order, right? So, if domain_shutdown() pauses the vcpu then is_shutting_down must necessarily be visible all cpus. Thus vcpu_start_shutdown referral will execute vcpu_check_shutdown(), so I'm having a hard time seeing the race.

> > I am not fully familiar with the IOREQ code, but it sounds to me this is
> > not the behavior that was intended. Can someone more familiar with the
> > code confirm it?
> >

No indeed. I think emulation should complete before the vcpu pauses.

  Paul

> > Cheers,
> >
> 
> --
> Julien Grall