Re: [PATCH v6 6/8] powerpc/pmem: Avoid the barrier in flush routines

From: Dan Williams <dan.j.williams@intel.com>
To: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc: "Michal Suchánek" <msuchanek@suse.de>,
	linuxppc-dev <linuxppc-dev@lists.ozlabs.org>,
	"Michael Ellerman" <mpe@ellerman.id.au>,
	linux-nvdimm <linux-nvdimm@lists.01.org>,
	"Jan Kara" <jack@suse.cz>
Subject: Re: [PATCH v6 6/8] powerpc/pmem: Avoid the barrier in flush routines
Date: Tue, 30 Jun 2020 22:08:48 -0700	[thread overview]
Message-ID: <CAPcyv4jHy=jQVdfFJTN=o3=wgCeLXd3Q29e+qoMUtZDA9KWZGw@mail.gmail.com> (raw)
In-Reply-To: <4a7bf5c8-a5c7-4292-c7ad-89bcefd7b22d@linux.ibm.com>

On Tue, Jun 30, 2020 at 8:09 PM Aneesh Kumar K.V
<aneesh.kumar@linux.ibm.com> wrote:
>
> On 7/1/20 1:15 AM, Dan Williams wrote:
> > On Tue, Jun 30, 2020 at 2:21 AM Aneesh Kumar K.V
> > <aneesh.kumar@linux.ibm.com> wrote:
> > [..]
> >>>> The bio argument isn't for range based flushing, it is for flush
> >>>> operations that need to complete asynchronously.
> >>> How does the block layer determine that the pmem device needs
> >>> asynchronous fushing?
> >>>
> >>
> >>          set_bit(ND_REGION_ASYNC, &ndr_desc.flags);
> >>
> >> and dax_synchronous(dev)
> >
> > Yes, but I think it is overkill to have an indirect function call just
> > for a single instruction.
> >
> > How about something like this instead, to share a common pmem_wmb()
> > across x86 and powerpc.
> >
> > diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c
> > index 20ff30c2ab93..b14009060c83 100644
> > --- a/drivers/nvdimm/region_devs.c
> > +++ b/drivers/nvdimm/region_devs.c
> > @@ -1180,6 +1180,13 @@ int nvdimm_flush(struct nd_region *nd_region,
> > struct bio *bio)
> >   {
> >          int rc = 0;
> >
> > +       /*
> > +        * pmem_wmb() is needed to 'sfence' all previous writes such
> > +        * that they are architecturally visible for the platform buffer
> > +        * flush.
> > +        */
> > +       pmem_wmb();
> > +
> >          if (!nd_region->flush)
> >                  rc = generic_nvdimm_flush(nd_region);
> >          else {
> > @@ -1206,17 +1213,14 @@ int generic_nvdimm_flush(struct nd_region *nd_region)
> >          idx = this_cpu_add_return(flush_idx, hash_32(current->pid + idx, 8));
> >
> >          /*
> > -        * The first wmb() is needed to 'sfence' all previous writes
> > -        * such that they are architecturally visible for the platform
> > -        * buffer flush.  Note that we've already arranged for pmem
> > -        * writes to avoid the cache via memcpy_flushcache().  The final
> > -        * wmb() ensures ordering for the NVDIMM flush write.
> > +        * Note that we've already arranged for pmem writes to avoid the
> > +        * cache via memcpy_flushcache().  The final wmb() ensures
> > +        * ordering for the NVDIMM flush write.
> >           */
> > -       wmb();
>
>
> The series already convert this to pmem_wmb().
>
> >          for (i = 0; i < nd_region->ndr_mappings; i++)
> >                  if (ndrd_get_flush_wpq(ndrd, i, 0))
> >                          writeq(1, ndrd_get_flush_wpq(ndrd, i, idx));
> > -       wmb();
> > +       pmem_wmb();
>
>
> Should this be pmem_wmb()? This is ordering the above writeq() right?

Correct, this can just be wmb().

>
> >
> >          return 0;
> >   }
> >
>
> This still results in two pmem_wmb() on platforms that doesn't have
> flush_wpq. I was trying to avoid that by adding a nd_region->flush call
> back.

How about skip or exit early out of generic_nvdimm_flush if
ndrd->flush_wpq is NULL? That still saves an indirect branch at the
cost of another conditional, but that should still be worth it.
_______________________________________________
Linux-nvdimm mailing list -- linux-nvdimm@lists.01.org
To unsubscribe send an email to linux-nvdimm-leave@lists.01.org