From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Bligh Subject: Re: [PATCHv3] QEMU(upstream): Disable xen's use of O_DIRECT by default as it results in crashes. Date: Mon, 18 Mar 2013 15:40:30 +0000 Message-ID: References: <1363609123-20748-1-git-send-email-alex@alex.org.uk> <51471767.8030604@redhat.com> <7AC8953FE45335FB794B6DFE@Ximines.local> <51471F14.7030209@redhat.com> <6D0F4ACDA3B7FCF1A50F8B52@Ximines.local> <5147298C.8080900@redhat.com> Reply-To: Alex Bligh Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <5147298C.8080900@redhat.com> Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Paolo Bonzini Cc: Ian Campbell , Stefano Stabellini , George Dunlap , Ian Jackson , qemu-devel@nongnu.org, xen-devel , Anthony Liguori , Alex Bligh List-Id: xen-devel@lists.xenproject.org Paolo, --On 18 March 2013 15:49:48 +0100 Paolo Bonzini wrote: >>> The problem is that the target's page cache might host image data from a >>> previous run. If you do not use O_DIRECT, it will not see the changes >>> made on the source. >> >> I was under the impression that with cache=writeback, qemu doesn't >> use O_DIRECT, in which case why isn't there the same danger under >> kvm, i.e. that the target page cache contains data from a previous >> run? > > KVM in fact only supports migration using cache=none. This does not > apply of course if you're using cache-coherent storage, such as rbd or > gluster; or if you're using one of the userspace backends that bypass > the page cache, like NBD or libiscsi. Well right now emulated devices under xen with qemu-upstream dm do not use O_DIRECT. So that's going to hit this problem anyway. Let me understand the problem a little more. Let's say a VM has a disk backed by NFS. It runs on node A, at which point pages are introduced to the page cache. It then migrates to node B, with node A flushing its dirty pages to disk before the migration completed, but not emptying the page cache completely. It's then migrated back to node A in which case I think you are saying that node A may still hold the page cache entries from prior to the first migration. However, node A has closed and reopened the file. Doesn't NFSv4 notice that the file has changed at this point and invalidate all page cache entries? Else (re)opening a file on one NFS client that has been changed on another would never work. I know this is unpredictable if you haven't closed the file, but in this instance A has closed the file. >> Disabling migration seems a bit excessive when migration isn't disabled >> with cache=unsafe (AFAIK) > > It is not QEMU's task. There are cases where the cache= options are > unnecessary or ignored. But indeed libvirt warns (or blocks, I don't > remember) in this case. Well we are kvm user too, and I understand hat kvm (as opposed to libvirt) does not error or warn if you live migrate with cache=writeback. I've no problem if xl or libvirt or whatever error or warn. My usage is API based, rather than xl / libvirt based. > If libxl does migration without O_DIRECT, then that's a bug in libxl. > What about blkback? IIRC it uses bios, so it also bypasses the page > cache. Possibly a bug in xl rather than libxl, but as no emulated devices use O_DIRECT, that bug is already there, and isn't in QEMU. >> Stefano did ack the patch, and for a one line change it's been >> through a pretty extensive discussion on xen-devel ... > > It may be a one-line change, but it completely changes the paths that > I/O goes through. Apparently the discussion was not enough. What would you suggest? -- Alex Bligh