From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D2EC4727C for ; Wed, 30 Sep 2020 17:55:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 67BBF20706 for ; Wed, 30 Sep 2020 17:55:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1601488504; bh=j3sE66fyqXA3v8uvjFB/9tQ3+EU63dABwffNYqDWUF0=; h=Subject:From:To:Cc:Date:In-Reply-To:References:List-ID:From; b=Ec3srJJ4gMyucsVG/MbIKyuPvJXZ8SN61dz68w9F7UD5fFE3ivRTIgA6sv1CtDvBT mwKfrAOLlVqrW/k+GGTgdjWXh3dLFVgfWWkwAU5KTe4ObBF5XNDHsL9ROZvgzpWdiT NRBAsXND1yMH3tx8Ac8BmNYT0RcA40f7jqTC0+dc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725893AbgI3RzC (ORCPT ); Wed, 30 Sep 2020 13:55:02 -0400 Received: from mail.kernel.org ([198.145.29.99]:50940 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725799AbgI3RzC (ORCPT ); Wed, 30 Sep 2020 13:55:02 -0400 Received: from tleilax.poochiereds.net (68-20-15-154.lightspeed.rlghnc.sbcglobal.net [68.20.15.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 44D8420706; Wed, 30 Sep 2020 17:55:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1601488501; bh=j3sE66fyqXA3v8uvjFB/9tQ3+EU63dABwffNYqDWUF0=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=DvwTc4y3Uju+uA653MESJzdKWMkZfpyAN3Z7khbLlX5P0xZ+DsK0lF/47iFGglrs0 Eg6IrpPj9BQPaqrnM8hHqF8yhPvAXE9KtAJHOS+SxDsOR2KMCk1dMmgQTLFip8p+WM OOW+GkZvMWcK5j8GADZGR4Bcdb5At/Nt/E1Ydkyg= Message-ID: <3723969aaff3f3f66b3b3e9ea8b6e1c5aab5e429.camel@kernel.org> Subject: Re: [RFC PATCH 0/4] ceph: fix spurious recover_session=clean errors From: Jeff Layton To: "Yan, Zheng" Cc: Ilya Dryomov , ceph-devel , Patrick Donnelly Date: Wed, 30 Sep 2020 13:55:00 -0400 In-Reply-To: References: <20200925140851.320673-1-jlayton@kernel.org> <5ba09f6b5493457341aaa273a3d3bddb155a37b4.camel@kernel.org> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-1.fc32) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org On Wed, 2020-09-30 at 16:45 +0800, Yan, Zheng wrote: > On Wed, Sep 30, 2020 at 3:50 AM Jeff Layton wrote: > > On Tue, 2020-09-29 at 18:44 +0800, Yan, Zheng wrote: > > > On Tue, Sep 29, 2020 at 4:55 PM Ilya Dryomov wrote: > > > > On Tue, Sep 29, 2020 at 10:28 AM Yan, Zheng wrote: > > > > > On Fri, Sep 25, 2020 at 10:08 PM Jeff Layton wrote: > > > > > > Ilya noticed that he would get spurious EACCES errors on calls done just > > > > > > after blocklisting the client on mounts with recover_session=clean. The > > > > > > session would get marked as REJECTED and that caused in-flight calls to > > > > > > die with EACCES. This patchset seems to smooth over the problem, but I'm > > > > > > not fully convinced it's the right approach. > > > > > > > > > > > > > > > > the root is cause is that client does not recover session instantly > > > > > after getting rejected by mds. Before session gets recovered, client > > > > > continues to return error. > > > > > > > > Hi Zheng, > > > > > > > > I don't think it's about whether that happens instantly or not. > > > > In the example from [1], the first "ls" would fail even if issued > > > > minutes after the session reject message and the reconnect. From > > > > the user's POV it is well after the automatic recovery promised by > > > > recover_session=clean. > > > > > > > > [1] https://tracker.ceph.com/issues/47385 > > > > > > Reconnect should close all old session. It's likely because that > > > client didn't detect it's blacklisted. > > > > > > > I should have described this better -- let me explain: > > > > It did detect that it was blocklisted (almost immediately) because the > > MDS shuts down the session. I think it immediately sends a > > SESSION_REJECT message when blacklisting and indicates that it has been > > blocklisted. > > > > At that point the session is CEPH_MDS_SESSION_REJECTED. The next MDS > > calls through would see that it was in that state and would return > > -EACCES. Eventually, the delayed work runs and then the session gets > > reconnected, and further calls proceed normally. > > > > So, I think this is just a timing thing for the most part. The workqueue > > job runs on a delay of round_jiffies_relative(HZ * 5);, and that's long > > enough for the disruption to be noticeable. > > > > While this was happening during 'ls' for Ilya, it could happen in > > anything that involves sending a request to the MDS. I think we want to > > prevent new opens from erroring out during this window if we can. > > > > The real question is whether this is safe in all cases. For instance, if > > the call that we're idling is dependent on holding certain caps, then > > it's possible we will have lost them when we got REJECTED. > > > > The session in rejected state is new session. It should hold no caps. > Right. We're actually OK here wrt to async requests as they will return EJUKEBOX and the caller will redrive a synchronous request. Other MClientRequest calls don't require that the client hold any caps, AFAICT, so idling them until we can establish a new session should be OK, no? > > Hmm...so that means patch 4/4 is probably wrong. I'll comment further in > > a reply to that patch. > > > > > > Thanks, > > > > > > > > Ilya > > > > > > > > > > The potential issue I see is that the client could take cap references to > > > > > > do a call on a session that has been blocklisted. We then queue the > > > > > > message and reestablish the session, but we may not have been granted > > > > > > the same caps by the MDS at that point. > > > > > > > > > > > > If this is a problem, then we probably need to rework it so that we > > > > > > return a distinct error code in this situation and have the upper layers > > > > > > issue a completely new mds request (with new cap refs, etc.) > > > > > > > > > > > > Obviously, that's a much more invasive approach though, so it would be > > > > > > nice to avoid that if this would suffice. > > > > > > > > > > > > Jeff Layton (4): > > > > > > ceph: don't WARN when removing caps due to blocklisting > > > > > > ceph: don't mark mount as SHUTDOWN when recovering session > > > > > > ceph: remove timeout on allowing reconnect after blocklisting > > > > > > ceph: queue request when CLEANRECOVER is set > > > > > > > > > > > > fs/ceph/caps.c | 2 +- > > > > > > fs/ceph/mds_client.c | 10 ++++------ > > > > > > fs/ceph/super.c | 13 +++++++++---- > > > > > > fs/ceph/super.h | 1 - > > > > > > 4 files changed, 14 insertions(+), 12 deletions(-) > > > > > > > > > > > > -- > > > > > > 2.26.2 > > > > > > > > > > -- > > Jeff Layton > > -- Jeff Layton