From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pankaj Gupta Subject: Re: [RFC v2 2/2] pmem: device flush over VIRTIO Date: Thu, 26 Apr 2018 12:40:53 -0400 (EDT) Message-ID: <58645254.23011245.1524760853269.JavaMail.zimbra@redhat.com> References: <20180425112415.12327-1-pagupta@redhat.com> <20180425112415.12327-3-pagupta@redhat.com> <20180426131517.GB30991@stefanha-x1.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20180426131517.GB30991-lxVrvc10SDRcolVlb+j0YCZi+YwRKgec@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Stefan Hajnoczi Cc: jack-AlSwsSmVLrQ@public.gmane.org, kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, david-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org, ross zwisler , qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, lcapitulino-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, niteshnarayanlal-PkbjNfxxIARBDgjK7y7TUQ@public.gmane.org, mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org, marcel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, nilal-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, riel-ebMLmSuQjDVBDgjK7y7TUQ@public.gmane.org, stefanha-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, pbonzini-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, kwolf-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, xiaoguangrong eric , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, imammedo-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org List-Id: linux-nvdimm@lists.01.org > > On Wed, Apr 25, 2018 at 04:54:14PM +0530, Pankaj Gupta wrote: > > This patch adds functionality to perform > > flush from guest to hosy over VIRTIO > > when 'ND_REGION_VIRTIO'flag is set on > > nd_negion. Flag is set by 'virtio-pmem' > > driver. > > > > Signed-off-by: Pankaj Gupta > > --- > > drivers/nvdimm/region_devs.c | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c > > index a612be6..6c6454e 100644 > > --- a/drivers/nvdimm/region_devs.c > > +++ b/drivers/nvdimm/region_devs.c > > @@ -20,6 +20,7 @@ > > #include > > #include "nd-core.h" > > #include "nd.h" > > +#include > > > > /* > > * For readq() and writeq() on 32-bit builds, the hi-lo, lo-hi order is > > @@ -1074,6 +1075,12 @@ void nvdimm_flush(struct nd_region *nd_region) > > struct nd_region_data *ndrd = dev_get_drvdata(&nd_region->dev); > > int i, idx; > > > > + /* call PV device flush */ > > + if (test_bit(ND_REGION_VIRTIO, &nd_region->flags)) { > > + virtio_pmem_flush(&nd_region->dev); > > + return; > > + } > > How does libnvdimm know when flush has completed? > > Callers expect the flush to be finished when nvdimm_flush() returns but > the virtio driver has only queued the request, it hasn't waited for > completion! I tried to implement what nvdimm does right now. It just writes to flush hint address to make sure data persists. I just did not want to block guest write requests till host side fsync completes. Operations(write/fsync) on same file would be blocking at guest side and wait time could be worse for operations on different guest files because all these operations would happen ultimately on same file at host. I think with current way, we can achieve an asynchronous queuing mechanism on cost of not 100% sure when fsync would complete but it is assured it will happen. Also, its entire block flush. I am open for suggestions here, this is my current thought and implementation. Thanks, Pankaj From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756635AbeDZQk4 (ORCPT ); Thu, 26 Apr 2018 12:40:56 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57124 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754462AbeDZQky (ORCPT ); Thu, 26 Apr 2018 12:40:54 -0400 Date: Thu, 26 Apr 2018 12:40:53 -0400 (EDT) From: Pankaj Gupta To: Stefan Hajnoczi Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, qemu-devel@nongnu.org, linux-nvdimm@ml01.01.org, linux-mm@kvack.org, jack@suse.cz, stefanha@redhat.com, dan j williams , riel@surriel.com, haozhong zhang , nilal@redhat.com, kwolf@redhat.com, pbonzini@redhat.com, ross zwisler , david@redhat.com, xiaoguangrong eric , hch@infradead.org, marcel@redhat.com, mst@redhat.com, niteshnarayanlal@hotmail.com, imammedo@redhat.com, lcapitulino@redhat.com Message-ID: <58645254.23011245.1524760853269.JavaMail.zimbra@redhat.com> In-Reply-To: <20180426131517.GB30991@stefanha-x1.localdomain> References: <20180425112415.12327-1-pagupta@redhat.com> <20180425112415.12327-3-pagupta@redhat.com> <20180426131517.GB30991@stefanha-x1.localdomain> Subject: Re: [RFC v2 2/2] pmem: device flush over VIRTIO MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.67.116.119, 10.4.195.10] Thread-Topic: pmem: device flush over VIRTIO Thread-Index: LQYKnvVqMrEPJ59FVtpvxGsK/zFRrw== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > > On Wed, Apr 25, 2018 at 04:54:14PM +0530, Pankaj Gupta wrote: > > This patch adds functionality to perform > > flush from guest to hosy over VIRTIO > > when 'ND_REGION_VIRTIO'flag is set on > > nd_negion. Flag is set by 'virtio-pmem' > > driver. > > > > Signed-off-by: Pankaj Gupta > > --- > > drivers/nvdimm/region_devs.c | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c > > index a612be6..6c6454e 100644 > > --- a/drivers/nvdimm/region_devs.c > > +++ b/drivers/nvdimm/region_devs.c > > @@ -20,6 +20,7 @@ > > #include > > #include "nd-core.h" > > #include "nd.h" > > +#include > > > > /* > > * For readq() and writeq() on 32-bit builds, the hi-lo, lo-hi order is > > @@ -1074,6 +1075,12 @@ void nvdimm_flush(struct nd_region *nd_region) > > struct nd_region_data *ndrd = dev_get_drvdata(&nd_region->dev); > > int i, idx; > > > > + /* call PV device flush */ > > + if (test_bit(ND_REGION_VIRTIO, &nd_region->flags)) { > > + virtio_pmem_flush(&nd_region->dev); > > + return; > > + } > > How does libnvdimm know when flush has completed? > > Callers expect the flush to be finished when nvdimm_flush() returns but > the virtio driver has only queued the request, it hasn't waited for > completion! I tried to implement what nvdimm does right now. It just writes to flush hint address to make sure data persists. I just did not want to block guest write requests till host side fsync completes. Operations(write/fsync) on same file would be blocking at guest side and wait time could be worse for operations on different guest files because all these operations would happen ultimately on same file at host. I think with current way, we can achieve an asynchronous queuing mechanism on cost of not 100% sure when fsync would complete but it is assured it will happen. Also, its entire block flush. I am open for suggestions here, this is my current thought and implementation. Thanks, Pankaj From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48958) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBjx4-0003qv-Sr for qemu-devel@nongnu.org; Thu, 26 Apr 2018 12:40:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBjx1-0004J7-ON for qemu-devel@nongnu.org; Thu, 26 Apr 2018 12:40:58 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51594) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBjx1-0004Ih-G3 for qemu-devel@nongnu.org; Thu, 26 Apr 2018 12:40:55 -0400 Date: Thu, 26 Apr 2018 12:40:53 -0400 (EDT) From: Pankaj Gupta Message-ID: <58645254.23011245.1524760853269.JavaMail.zimbra@redhat.com> In-Reply-To: <20180426131517.GB30991@stefanha-x1.localdomain> References: <20180425112415.12327-1-pagupta@redhat.com> <20180425112415.12327-3-pagupta@redhat.com> <20180426131517.GB30991@stefanha-x1.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC v2 2/2] pmem: device flush over VIRTIO List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, qemu-devel@nongnu.org, linux-nvdimm@ml01.01.org, linux-mm@kvack.org, jack@suse.cz, stefanha@redhat.com, dan j williams , riel@surriel.com, haozhong zhang , nilal@redhat.com, kwolf@redhat.com, pbonzini@redhat.com, ross zwisler , david@redhat.com, xiaoguangrong eric , hch@infradead.org, marcel@redhat.com, mst@redhat.com, niteshnarayanlal@hotmail.com, imammedo@redhat.com, lcapitulino@redhat.com > > On Wed, Apr 25, 2018 at 04:54:14PM +0530, Pankaj Gupta wrote: > > This patch adds functionality to perform > > flush from guest to hosy over VIRTIO > > when 'ND_REGION_VIRTIO'flag is set on > > nd_negion. Flag is set by 'virtio-pmem' > > driver. > > > > Signed-off-by: Pankaj Gupta > > --- > > drivers/nvdimm/region_devs.c | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c > > index a612be6..6c6454e 100644 > > --- a/drivers/nvdimm/region_devs.c > > +++ b/drivers/nvdimm/region_devs.c > > @@ -20,6 +20,7 @@ > > #include > > #include "nd-core.h" > > #include "nd.h" > > +#include > > > > /* > > * For readq() and writeq() on 32-bit builds, the hi-lo, lo-hi order is > > @@ -1074,6 +1075,12 @@ void nvdimm_flush(struct nd_region *nd_region) > > struct nd_region_data *ndrd = dev_get_drvdata(&nd_region->dev); > > int i, idx; > > > > + /* call PV device flush */ > > + if (test_bit(ND_REGION_VIRTIO, &nd_region->flags)) { > > + virtio_pmem_flush(&nd_region->dev); > > + return; > > + } > > How does libnvdimm know when flush has completed? > > Callers expect the flush to be finished when nvdimm_flush() returns but > the virtio driver has only queued the request, it hasn't waited for > completion! I tried to implement what nvdimm does right now. It just writes to flush hint address to make sure data persists. I just did not want to block guest write requests till host side fsync completes. Operations(write/fsync) on same file would be blocking at guest side and wait time could be worse for operations on different guest files because all these operations would happen ultimately on same file at host. I think with current way, we can achieve an asynchronous queuing mechanism on cost of not 100% sure when fsync would complete but it is assured it will happen. Also, its entire block flush. I am open for suggestions here, this is my current thought and implementation. Thanks, Pankaj