All of lore.kernel.org
 help / color / mirror / Atom feed
From: Olga Kornievskaia <olga.kornievskaia@gmail.com>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: trond.myklebust@hammerspace.com,
	Anna Schumaker <anna.schumaker@netapp.com>,
	"J. Bruce Fields" <bfields@redhat.com>,
	linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v7 15/19] NFSD add COPY_NOTIFY operation
Date: Wed, 2 Oct 2019 11:32:07 -0400	[thread overview]
Message-ID: <CAN-5tyF6L1wtsxa4BbKvd+VJk0k=AdVpq11s==HmvH33PeW7ow@mail.gmail.com> (raw)
In-Reply-To: <20191002013523.GA9000@fieldses.org>

On Tue, Oct 1, 2019 at 9:35 PM J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Tue, Oct 01, 2019 at 08:14:39PM -0400, Olga Kornievskaia wrote:
> > On Tue, Oct 1, 2019 at 4:59 PM J. Bruce Fields <bfields@fieldses.org> wrote:
> > >
> > > On Mon, Sep 16, 2019 at 05:13:49PM -0400, Olga Kornievskaia wrote:
> > > > @@ -2914,7 +2983,8 @@ static bool client_has_state(struct nfs4_client *clp)
> > > >  #endif
> > > >               || !list_empty(&clp->cl_delegations)
> > > >               || !list_empty(&clp->cl_sessions)
> > > > -             || !list_empty(&clp->async_copies);
> > > > +             || !list_empty(&clp->async_copies)
> > > > +             || client_has_copy_notifies(clp);
> > >
> > > Sorry, remind me--how is the copy_notify stateid cleaned up?  Is it just
> > > timed out by the laundromat thread, or is our client destroying it when
> > > the copy is done?
> >
> > Copy_notify stateid in most cases should be removed by
> > nfs4_put_stid()/ the "parent" stateid going away (close, unlock,
> > delegreturn)
>
> Oh, right, the parent stateid going away will be enough, good.
>
> But doesn't that mean the client_has_copy_notifies() check here is
> redundant?  If it's true, then the earlier client_has_openowner() check
> was also true.

Honestly, I didn't remove it because I didn't know if I was missing
some weird case where on clientid destruction we'd still have copy
notify left and thus need to delay it. But as you see nfs4_put_stid()
calls nfs4_free_cpntf_statelist() which iterates the list of the
copy_notify states and deletes them. Sounds like I should just remove
the client_has_copy_notifies() check.

>
> --b.
>
> > (or in unlikely case, if the client sent an
> > OFFLOAD_CANCEL to the stateid, say if copy was canceled). Otherwise,
> > copy notify stateid will time out and be removed by the laundromat
> > thread. So if a client didn't send a close/unlock/delegreturn and
> > quickly tries to delete a clientid, it will get ERR_CLID_INUSE but I
> > think if there was no close, the server will have an issue of having
> > an open stateid state.
> >
> > > I'm just wondering if this can result in NFSERR_CLID_INUSE just because
> > > a copy was done recently.
> > >
> > > --b.
> > >
> > > >  }
> > > >
> > > >  static __be32 copy_impl_id(struct nfs4_client *clp,
> > > > @@ -5192,6 +5262,9 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)
> > > >       struct list_head *pos, *next, reaplist;
> > > >       time_t cutoff = get_seconds() - nn->nfsd4_lease;
> > > >       time_t t, new_timeo = nn->nfsd4_lease;
> > > > +     struct nfs4_cpntf_state *cps;
> > > > +     copy_stateid_t *cps_t;
> > > > +     int i;
> > > >
> > > >       dprintk("NFSD: laundromat service - starting\n");
> > > >
> > > > @@ -5202,6 +5275,17 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)
> > > >       dprintk("NFSD: end of grace period\n");
> > > >       nfsd4_end_grace(nn);
> > > >       INIT_LIST_HEAD(&reaplist);
> > > > +
> > > > +     spin_lock(&nn->s2s_cp_lock);
> > > > +     idr_for_each_entry(&nn->s2s_cp_stateids, cps_t, i) {
> > > > +             cps = container_of(cps_t, struct nfs4_cpntf_state, cp_stateid);
> > > > +             if (cps->cp_stateid.sc_type == NFS4_COPYNOTIFY_STID &&
> > > > +                             !time_after((unsigned long)cps->cpntf_time,
> > > > +                             (unsigned long)cutoff))
> > > > +                     _free_cpntf_state_locked(nn, cps);
> > > > +     }
> > > > +     spin_unlock(&nn->s2s_cp_lock);
> > > > +
> > > >       spin_lock(&nn->client_lock);
> > > >       list_for_each_safe(pos, next, &nn->client_lru) {
> > > >               clp = list_entry(pos, struct nfs4_client, cl_lru);
> > > > @@ -5577,6 +5661,24 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)
> > > >  out:
> > > >       return status;
> > > >  }
> > > > +static void
> > > > +_free_cpntf_state_locked(struct nfsd_net *nn, struct nfs4_cpntf_state *cps)
> > > > +{
> > > > +     WARN_ON_ONCE(cps->cp_stateid.sc_type != NFS4_COPYNOTIFY_STID);
> > > > +     if (!refcount_dec_and_test(&cps->cp_stateid.sc_count))
> > > > +             return;
> > > > +     list_del(&cps->cp_list);
> > > > +     idr_remove(&nn->s2s_cp_stateids,
> > > > +                cps->cp_stateid.stid.si_opaque.so_id);
> > > > +     kfree(cps);
> > > > +}
> > > > +
> > > > +void nfs4_put_cpntf_state(struct nfsd_net *nn, struct nfs4_cpntf_state *cps)
> > > > +{
> > > > +     spin_lock(&nn->s2s_cp_lock);
> > > > +     _free_cpntf_state_locked(nn, cps);
> > > > +     spin_unlock(&nn->s2s_cp_lock);
> > > > +}
> > > >
> > > >  /*
> > > >   * Checks for stateid operations
> > > > diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
> > > > index d9e7cbd..967b937 100644
> > > > --- a/fs/nfsd/state.h
> > > > +++ b/fs/nfsd/state.h
> > > > @@ -56,6 +56,14 @@
> > > >       stateid_opaque_t        si_opaque;
> > > >  } stateid_t;
> > > >
> > > > +typedef struct {
> > > > +     stateid_t               stid;
> > > > +#define NFS4_COPY_STID 1
> > > > +#define NFS4_COPYNOTIFY_STID 2
> > > > +     unsigned char           sc_type;
> > > > +     refcount_t              sc_count;
> > > > +} copy_stateid_t;
> > > > +
> > > >  #define STATEID_FMT  "(%08x/%08x/%08x/%08x)"
> > > >  #define STATEID_VAL(s) \
> > > >       (s)->si_opaque.so_clid.cl_boot, \
> > > > @@ -96,6 +104,7 @@ struct nfs4_stid {
> > > >  #define NFS4_REVOKED_DELEG_STID 16
> > > >  #define NFS4_CLOSED_DELEG_STID 32
> > > >  #define NFS4_LAYOUT_STID 64
> > > > +     struct list_head        sc_cp_list;
> > > >       unsigned char           sc_type;
> > > >       stateid_t               sc_stateid;
> > > >       spinlock_t              sc_lock;
> > > > @@ -104,6 +113,17 @@ struct nfs4_stid {
> > > >       void                    (*sc_free)(struct nfs4_stid *);
> > > >  };
> > > >
> > > > +/* Keep a list of stateids issued by the COPY_NOTIFY, associate it with the
> > > > + * parent OPEN/LOCK/DELEG stateid.
> > > > + */
> > > > +struct nfs4_cpntf_state {
> > > > +     copy_stateid_t          cp_stateid;
> > > > +     struct list_head        cp_list;        /* per parent nfs4_stid */
> > > > +     stateid_t               cp_p_stateid;   /* copy of parent's stateid */
> > > > +     clientid_t              cp_p_clid;      /* copy of parent's clid */
> > > > +     time_t                  cpntf_time;     /* last time stateid used */
> > > > +};
> > > > +
> > > >  /*
> > > >   * Represents a delegation stateid. The nfs4_client holds references to these
> > > >   * and they are put when it is being destroyed or when the delegation is
> > > > @@ -624,8 +644,10 @@ __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
> > > >                    struct nfs4_stid **s, struct nfsd_net *nn);
> > > >  struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *slab,
> > > >                                 void (*sc_free)(struct nfs4_stid *));
> > > > -int nfs4_init_cp_state(struct nfsd_net *nn, struct nfsd4_copy *copy);
> > > > -void nfs4_free_cp_state(struct nfsd4_copy *copy);
> > > > +int nfs4_init_copy_state(struct nfsd_net *nn, struct nfsd4_copy *copy);
> > > > +void nfs4_free_copy_state(struct nfsd4_copy *copy);
> > > > +struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state(struct nfsd_net *nn,
> > > > +                     struct nfs4_stid *p_stid);
> > > >  void nfs4_unhash_stid(struct nfs4_stid *s);
> > > >  void nfs4_put_stid(struct nfs4_stid *s);
> > > >  void nfs4_inc_and_copy_stateid(stateid_t *dst, struct nfs4_stid *stid);
> > > > @@ -655,6 +677,8 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(struct xdr_netobj name
> > > >  extern void nfs4_put_copy(struct nfsd4_copy *copy);
> > > >  extern struct nfsd4_copy *
> > > >  find_async_copy(struct nfs4_client *clp, stateid_t *staetid);
> > > > +extern void nfs4_put_cpntf_state(struct nfsd_net *nn,
> > > > +                              struct nfs4_cpntf_state *cps);
> > > >  static inline void get_nfs4_file(struct nfs4_file *fi)
> > > >  {
> > > >       refcount_inc(&fi->fi_ref);
> > > > diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
> > > > index 8231fe0..2937e06 100644
> > > > --- a/fs/nfsd/xdr4.h
> > > > +++ b/fs/nfsd/xdr4.h
> > > > @@ -542,7 +542,7 @@ struct nfsd4_copy {
> > > >       struct nfsd_file        *nf_src;
> > > >       struct nfsd_file        *nf_dst;
> > > >
> > > > -     stateid_t               cp_stateid;
> > > > +     copy_stateid_t          cp_stateid;
> > > >
> > > >       struct list_head        copies;
> > > >       struct task_struct      *copy_task;
> > > > --
> > > > 1.8.3.1

  reply	other threads:[~2019-10-02 15:32 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-16 21:13 [PATCH v7 00/19] client and server support for "inter" SSC copy Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 01/19] NFS NFSD: defining nl4_servers structure needed by both Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 02/19] NFS: add COPY_NOTIFY operation Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 03/19] NFS: add ca_source_server<> to COPY Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 04/19] NFS: also send OFFLOAD_CANCEL to source server Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 05/19] NFS: inter ssc open Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 06/19] NFS: skip recovery of copy open on dest server Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 07/19] NFS: for "inter" copy treat ESTALE as ENOTSUPP Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 08/19] NFS: COPY handle ERR_OFFLOAD_DENIED Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 09/19] NFS: handle source server reboot Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 10/19] NFS: replace cross device check in copy_file_range Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 11/19] NFSD fill-in netloc4 structure Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 12/19] NFSD add ca_source_server<> to COPY Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 13/19] NFSD return nfs4_stid in nfs4_preprocess_stateid_op Olga Kornievskaia
2019-10-02 15:52   ` J. Bruce Fields
2019-10-02 16:12     ` Olga Kornievskaia
2019-10-02 16:15       ` Olga Kornievskaia
2019-10-02 16:34       ` J. Bruce Fields
2019-10-02 16:52         ` Olga Kornievskaia
2019-10-02 16:57           ` J. Bruce Fields
2019-09-16 21:13 ` [PATCH v7 14/19] NFSD COPY_NOTIFY xdr Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 15/19] NFSD add COPY_NOTIFY operation Olga Kornievskaia
2019-10-01 20:59   ` J. Bruce Fields
2019-10-02  0:14     ` Olga Kornievskaia
2019-10-02  1:35       ` J. Bruce Fields
2019-10-02 15:32         ` Olga Kornievskaia [this message]
2019-09-16 21:13 ` [PATCH v7 16/19] NFSD check stateids against copy stateids Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 17/19] NFSD generalize nfsd4_compound_state flag names Olga Kornievskaia
2019-09-16 21:13 ` [PATCH v7 18/19] NFSD: allow inter server COPY to have a STALE source server fh Olga Kornievskaia
2019-10-02 19:55   ` J. Bruce Fields
2019-10-07 14:31     ` Olga Kornievskaia
2019-10-07 18:20       ` J. Bruce Fields
2019-09-16 21:13 ` [PATCH v7 19/19] NFSD add nfs4 inter ssc to nfsd4_copy Olga Kornievskaia
2019-09-19 19:55   ` [PATCH 1/1] " Olga Kornievskaia
2019-09-30 19:06 ` [PATCH v7 00/19] client and server support for "inter" SSC copy Olga Kornievskaia
2019-10-01 17:13   ` J. Bruce Fields
2019-10-01 17:47     ` Olga Kornievskaia
2019-10-01 17:50       ` J. Bruce Fields
2019-10-01 19:03         ` Olga Kornievskaia

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAN-5tyF6L1wtsxa4BbKvd+VJk0k=AdVpq11s==HmvH33PeW7ow@mail.gmail.com' \
    --to=olga.kornievskaia@gmail.com \
    --cc=anna.schumaker@netapp.com \
    --cc=bfields@fieldses.org \
    --cc=bfields@redhat.com \
    --cc=linux-nfs@vger.kernel.org \
    --cc=trond.myklebust@hammerspace.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.