Linux-NFS Archive on lore.kernel.org
 help / color / Atom feed
From: Olga Kornievskaia <olga.kornievskaia@gmail.com>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: trond.myklebust@hammerspace.com,
	Anna Schumaker <anna.schumaker@netapp.com>,
	"J. Bruce Fields" <bfields@redhat.com>,
	linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v7 15/19] NFSD add COPY_NOTIFY operation
Date: Wed, 2 Oct 2019 11:32:07 -0400
Message-ID: <CAN-5tyF6L1wtsxa4BbKvd+VJk0k=AdVpq11s==HmvH33PeW7ow@mail.gmail.com> (raw)
In-Reply-To: <20191002013523.GA9000@fieldses.org>

On Tue, Oct 1, 2019 at 9:35 PM J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Tue, Oct 01, 2019 at 08:14:39PM -0400, Olga Kornievskaia wrote:
> > On Tue, Oct 1, 2019 at 4:59 PM J. Bruce Fields <bfields@fieldses.org> wrote:
> > >
> > > On Mon, Sep 16, 2019 at 05:13:49PM -0400, Olga Kornievskaia wrote:
> > > > @@ -2914,7 +2983,8 @@ static bool client_has_state(struct nfs4_client *clp)
> > > >  #endif
> > > >               || !list_empty(&clp->cl_delegations)
> > > >               || !list_empty(&clp->cl_sessions)
> > > > -             || !list_empty(&clp->async_copies);
> > > > +             || !list_empty(&clp->async_copies)
> > > > +             || client_has_copy_notifies(clp);
> > >
> > > Sorry, remind me--how is the copy_notify stateid cleaned up?  Is it just
> > > timed out by the laundromat thread, or is our client destroying it when
> > > the copy is done?
> >
> > Copy_notify stateid in most cases should be removed by
> > nfs4_put_stid()/ the "parent" stateid going away (close, unlock,
> > delegreturn)
>
> Oh, right, the parent stateid going away will be enough, good.
>
> But doesn't that mean the client_has_copy_notifies() check here is
> redundant?  If it's true, then the earlier client_has_openowner() check
> was also true.

Honestly, I didn't remove it because I didn't know if I was missing
some weird case where on clientid destruction we'd still have copy
notify left and thus need to delay it. But as you see nfs4_put_stid()
calls nfs4_free_cpntf_statelist() which iterates the list of the
copy_notify states and deletes them. Sounds like I should just remove
the client_has_copy_notifies() check.

>
> --b.
>
> > (or in unlikely case, if the client sent an
> > OFFLOAD_CANCEL to the stateid, say if copy was canceled). Otherwise,
> > copy notify stateid will time out and be removed by the laundromat
> > thread. So if a client didn't send a close/unlock/delegreturn and
> > quickly tries to delete a clientid, it will get ERR_CLID_INUSE but I
> > think if there was no close, the server will have an issue of having
> > an open stateid state.
> >
> > > I'm just wondering if this can result in NFSERR_CLID_INUSE just because
> > > a copy was done recently.
> > >
> > > --b.
> > >
> > > >  }
> > > >
> > > >  static __be32 copy_impl_id(struct nfs4_client *clp,
> > > > @@ -5192,6 +5262,9 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)
> > > >       struct list_head *pos, *next, reaplist;
> > > >       time_t cutoff = get_seconds() - nn->nfsd4_lease;
> > > >       time_t t, new_timeo = nn->nfsd4_lease;
> > > > +     struct nfs4_cpntf_state *cps;
> > > > +     copy_stateid_t *cps_t;
> > > > +     int i;
> > > >
> > > >       dprintk("NFSD: laundromat service - starting\n");
> > > >
> > > > @@ -5202,6 +5275,17 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)
> > > >       dprintk("NFSD: end of grace period\n");
> > > >       nfsd4_end_grace(nn);
> > > >       INIT_LIST_HEAD(&reaplist);
> > > > +
> > > > +     spin_lock(&nn->s2s_cp_lock);
> > > > +     idr_for_each_entry(&nn->s2s_cp_stateids, cps_t, i) {
> > > > +             cps = container_of(cps_t, struct nfs4_cpntf_state, cp_stateid);
> > > > +             if (cps->cp_stateid.sc_type == NFS4_COPYNOTIFY_STID &&
> > > > +                             !time_after((unsigned long)cps->cpntf_time,
> > > > +                             (unsigned long)cutoff))
> > > > +                     _free_cpntf_state_locked(nn, cps);
> > > > +     }
> > > > +     spin_unlock(&nn->s2s_cp_lock);
> > > > +
> > > >       spin_lock(&nn->client_lock);
> > > >       list_for_each_safe(pos, next, &nn->client_lru) {
> > > >               clp = list_entry(pos, struct nfs4_client, cl_lru);
> > > > @@ -5577,6 +5661,24 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)
> > > >  out:
> > > >       return status;
> > > >  }
> > > > +static void
> > > > +_free_cpntf_state_locked(struct nfsd_net *nn, struct nfs4_cpntf_state *cps)
> > > > +{
> > > > +     WARN_ON_ONCE(cps->cp_stateid.sc_type != NFS4_COPYNOTIFY_STID);
> > > > +     if (!refcount_dec_and_test(&cps->cp_stateid.sc_count))
> > > > +             return;
> > > > +     list_del(&cps->cp_list);
> > > > +     idr_remove(&nn->s2s_cp_stateids,
> > > > +                cps->cp_stateid.stid.si_opaque.so_id);
> > > > +     kfree(cps);
> > > > +}
> > > > +
> > > > +void nfs4_put_cpntf_state(struct nfsd_net *nn, struct nfs4_cpntf_state *cps)
> > > > +{
> > > > +     spin_lock(&nn->s2s_cp_lock);
> > > > +     _free_cpntf_state_locked(nn, cps);
> > > > +     spin_unlock(&nn->s2s_cp_lock);
> > > > +}
> > > >
> > > >  /*
> > > >   * Checks for stateid operations
> > > > diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
> > > > index d9e7cbd..967b937 100644
> > > > --- a/fs/nfsd/state.h
> > > > +++ b/fs/nfsd/state.h
> > > > @@ -56,6 +56,14 @@
> > > >       stateid_opaque_t        si_opaque;
> > > >  } stateid_t;
> > > >
> > > > +typedef struct {
> > > > +     stateid_t               stid;
> > > > +#define NFS4_COPY_STID 1
> > > > +#define NFS4_COPYNOTIFY_STID 2
> > > > +     unsigned char           sc_type;
> > > > +     refcount_t              sc_count;
> > > > +} copy_stateid_t;
> > > > +
> > > >  #define STATEID_FMT  "(%08x/%08x/%08x/%08x)"
> > > >  #define STATEID_VAL(s) \
> > > >       (s)->si_opaque.so_clid.cl_boot, \
> > > > @@ -96,6 +104,7 @@ struct nfs4_stid {
> > > >  #define NFS4_REVOKED_DELEG_STID 16
> > > >  #define NFS4_CLOSED_DELEG_STID 32
> > > >  #define NFS4_LAYOUT_STID 64
> > > > +     struct list_head        sc_cp_list;
> > > >       unsigned char           sc_type;
> > > >       stateid_t               sc_stateid;
> > > >       spinlock_t              sc_lock;
> > > > @@ -104,6 +113,17 @@ struct nfs4_stid {
> > > >       void                    (*sc_free)(struct nfs4_stid *);
> > > >  };
> > > >
> > > > +/* Keep a list of stateids issued by the COPY_NOTIFY, associate it with the
> > > > + * parent OPEN/LOCK/DELEG stateid.
> > > > + */
> > > > +struct nfs4_cpntf_state {
> > > > +     copy_stateid_t          cp_stateid;
> > > > +     struct list_head        cp_list;        /* per parent nfs4_stid */
> > > > +     stateid_t               cp_p_stateid;   /* copy of parent's stateid */
> > > > +     clientid_t              cp_p_clid;      /* copy of parent's clid */
> > > > +     time_t                  cpntf_time;     /* last time stateid used */
> > > > +};
> > > > +
> > > >  /*
> > > >   * Represents a delegation stateid. The nfs4_client holds references to these
> > > >   * and they are put when it is being destroyed or when the delegation is
> > > > @@ -624,8 +644,10 @@ __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
> > > >                    struct nfs4_stid **s, struct nfsd_net *nn);
> > > >  struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *slab,
> > > >                                 void (*sc_free)(struct nfs4_stid *));
> > > > -int nfs4_init_cp_state(struct nfsd_net *nn, struct nfsd4_copy *copy);
> > > > -void nfs4_free_cp_state(struct nfsd4_copy *copy);
> > > > +int nfs4_init_copy_state(struct nfsd_net *nn, struct nfsd4_copy *copy);
> > > > +void nfs4_free_copy_state(struct nfsd4_copy *copy);
> > > > +struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state(struct nfsd_net *nn,
> > > > +                     struct nfs4_stid *p_stid);
> > > >  void nfs4_unhash_stid(struct nfs4_stid *s);
> > > >  void nfs4_put_stid(struct nfs4_stid *s);
> > > >  void nfs4_inc_and_copy_stateid(stateid_t *dst, struct nfs4_stid *stid);
> > > > @@ -655,6 +677,8 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(struct xdr_netobj name
> > > >  extern void nfs4_put_copy(struct nfsd4_copy *copy);
> > > >  extern struct nfsd4_copy *
> > > >  find_async_copy(struct nfs4_client *clp, stateid_t *staetid);
> > > > +extern void nfs4_put_cpntf_state(struct nfsd_net *nn,
> > > > +                              struct nfs4_cpntf_state *cps);
> > > >  static inline void get_nfs4_file(struct nfs4_file *fi)
> > > >  {
> > > >       refcount_inc(&fi->fi_ref);
> > > > diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
> > > > index 8231fe0..2937e06 100644
> > > > --- a/fs/nfsd/xdr4.h
> > > > +++ b/fs/nfsd/xdr4.h
> > > > @@ -542,7 +542,7 @@ struct nfsd4_copy {
> > > >       struct nfsd_file        *nf_src;
> > > >       struct nfsd_file        *nf_dst;
> > > >
> > > > -     stateid_t               cp_stateid;
> > > > +     copy_stateid_t          cp_stateid;
> > > >
> > > >       struct list_head        copies;
> > > >       struct task_struct      *copy_task;
> > > > --
> > > > 1.8.3.1

  reply index

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-16 21:13 [PATCH v7 00/19] client and server support for "inter" SSC copy Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 01/19] NFS NFSD: defining nl4_servers structure needed by both Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 02/19] NFS: add COPY_NOTIFY operation Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 03/19] NFS: add ca_source_server<> to COPY Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 04/19] NFS: also send OFFLOAD_CANCEL to source server Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 05/19] NFS: inter ssc open Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 06/19] NFS: skip recovery of copy open on dest server Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 07/19] NFS: for "inter" copy treat ESTALE as ENOTSUPP Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 08/19] NFS: COPY handle ERR_OFFLOAD_DENIED Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 09/19] NFS: handle source server reboot Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 10/19] NFS: replace cross device check in copy_file_range Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 11/19] NFSD fill-in netloc4 structure Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 12/19] NFSD add ca_source_server<> to COPY Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 13/19] NFSD return nfs4_stid in nfs4_preprocess_stateid_op Olga Kornievskaia
2019-10-02 15:52   ` bfields
2019-10-02 16:12     ` Olga Kornievskaia
2019-10-02 16:15       ` Olga Kornievskaia
2019-10-02 16:34       ` J. Bruce Fields
2019-10-02 16:52         ` Olga Kornievskaia
2019-10-02 16:57           ` J. Bruce Fields
2019-09-16 21:13 ` [PATCH v7 14/19] NFSD COPY_NOTIFY xdr Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 15/19] NFSD add COPY_NOTIFY operation Olga Kornievskaia
2019-10-01 20:59   ` bfields
2019-10-02  0:14     ` Olga Kornievskaia
2019-10-02  1:35       ` J. Bruce Fields
2019-10-02 15:32         ` Olga Kornievskaia [this message]
2019-09-16 21:13 ` [PATCH v7 16/19] NFSD check stateids against copy stateids Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 17/19] NFSD generalize nfsd4_compound_state flag names Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 18/19] NFSD: allow inter server COPY to have a STALE source server fh Olga Kornievskaia
2019-10-02 19:55   ` bfields
2019-10-07 14:31     ` Olga Kornievskaia
2019-10-07 18:20       ` J. Bruce Fields
2019-09-16 21:13 ` [PATCH v7 19/19] NFSD add nfs4 inter ssc to nfsd4_copy Olga Kornievskaia
2019-09-19 19:55   ` [PATCH 1/1] " Olga Kornievskaia
2019-09-30 19:06 ` [PATCH v7 00/19] client and server support for "inter" SSC copy Olga Kornievskaia
2019-10-01 17:13   ` bfields
2019-10-01 17:47     ` Olga Kornievskaia
2019-10-01 17:50       ` J. Bruce Fields
2019-10-01 19:03         ` Olga Kornievskaia

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAN-5tyF6L1wtsxa4BbKvd+VJk0k=AdVpq11s==HmvH33PeW7ow@mail.gmail.com' \
    --to=olga.kornievskaia@gmail.com \
    --cc=anna.schumaker@netapp.com \
    --cc=bfields@fieldses.org \
    --cc=bfields@redhat.com \
    --cc=linux-nfs@vger.kernel.org \
    --cc=trond.myklebust@hammerspace.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nfs/0 linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ https://lore.kernel.org/linux-nfs \
		linux-nfs@vger.kernel.org
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-nfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git