From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 052EAC48BE6 for ; Wed, 16 Jun 2021 09:13:37 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CB9B861107 for ; Wed, 16 Jun 2021 09:13:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CB9B861107 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4z9C+8StC9DAx71+otqlGoKNIc8BoWgXpM65svZNidU=; b=RyHaXK2aJvPqdZ JG/R0jPqGwMMxRQ2eDmMlo2NGBvj7VNIupnQKgrL2gVgCiJFfbYhCWCfXoIr0abJLUsR+Yt253Ak9 ejzLIxHbqbG+v1PWyZkL36OZpPAoVxjMyeaQ9ZPWoBQxU2n12UkUAPeTyWSk74OZLOjYiVuUkkzxZ JTpMcc/hB+LIS/gipG++dXMg9bMXlq5ysHbPFIxcWqKYldlQX8zAD+o87IcWpczfYeh2B+9DzIMaM oYhTc2u80B1NmA/m830QABIdP6uvcHVmlLTJof3aLnl8ygHyWafoHq/GsVf5duk1T7Q+7rkZfUUoy eSXK0db/IMKmBbP4mlCQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltRak-005Wfe-BZ; Wed, 16 Jun 2021 09:12:10 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltRae-005WeV-5I for linux-arm-kernel@lists.infradead.org; Wed, 16 Jun 2021 09:12:06 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6C56431B; Wed, 16 Jun 2021 02:12:01 -0700 (PDT) Received: from e120937-lin (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C54813F70D; Wed, 16 Jun 2021 02:11:57 -0700 (PDT) Date: Wed, 16 Jun 2021 10:11:55 +0100 From: Cristian Marussi To: Jonathan Cameron Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, virtualization@lists.linux-foundation.org, virtio-dev@lists.oasis-open.org, sudeep.holla@arm.com, james.quinlan@broadcom.com, f.fainelli@gmail.com, etienne.carriere@linaro.org, vincent.guittot@linaro.org, souvik.chakravarty@arm.com, igor.skalkin@opensynergy.com, peter.hilber@opensynergy.com, alex.bennee@linaro.org, jean-philippe@linaro.org, mikhail.golubev@opensynergy.com, anton.yakovlev@opensynergy.com, Vasyl.Vavrychuk@opensynergy.com, Andriy.Tryshnivskyy@opensynergy.com Subject: Re: [PATCH v4 04/16] firmware: arm_scmi: Introduce monotonically increasing tokens Message-ID: <20210616091155.GD35368@e120937-lin> References: <20210611165937.701-1-cristian.marussi@arm.com> <20210611165937.701-5-cristian.marussi@arm.com> <20210614145301.00002cd9@Huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210614145301.00002cd9@Huawei.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210616_021204_340779_5797C6E5 X-CRM114-Status: GOOD ( 37.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, On Mon, Jun 14, 2021 at 02:53:01PM +0100, Jonathan Cameron wrote: > On Fri, 11 Jun 2021 17:59:25 +0100 > Cristian Marussi wrote: > > > Tokens are sequence numbers embedded in the each SCMI message header: they > > are used to correlate commands with responses (and delayed responses), but > > their usage and policy of selection is entirely up to the caller (usually > > the OSPM agent), while they are completely opaque to the callee (SCMI > > server platform) which merely copies them back from the command into the > > response message header. > > This also means that the platform does not, can not and should not enforce > > any kind of policy on received messages depending on the contained sequence > > number: platform can perfectly handle concurrent requests carrying the same > > identifiying token if that should happen. > > > > Moreover the platform is not required to produce in-order responses to > > agent requests, the only constraint in these regards is that in case of > > an asynchronous message the delayed response must be sent after the > > immediate response for the synchronous part of the command transaction. > > > > Currenly the SCMI stack of the OSPM agent selects a token for the egressing > > commands picking the lowest possible number which is not already in use by > > an existing in-flight transaction, which means, in other words, that we > > immediately reuse any token after its transaction has completed or it has > > timed out: this policy indeed does simplify management and lookup of tokens > > and associated xfers. > > > > Under the above assumptions and constraints, since there is really no state > > shared between the agent and the platform to let the platform know when a > > token and its associated message has timed out, the current policy of early > > reuse of tokens can easily lead to the situation in which a spurious or > > late received response (or delayed_response), related to an old stale and > > timed out transaction, can be wrongly associated to a newer valid in-flight > > xfer that just happens to have reused the same token. > > > > This misbehaviour on such ghost responses is more easily exposed on those > > transports that naturally have an higher level of parallelism in processing > > multiple concurrent in-flight messages. > > > > This commit introduces a new policy of selection of tokens for the OSPM > > agent: each new transfer now gets the next available and monotonically > > increasing token, until tokens are exhausted and the counter rolls over. > > > > Such new policy mitigates the above issues with ghost responses since the > > tokens are now reused as late as possible (when they roll back ideally) > > and so it is much easier to identify such ghost responses to stale timed > > out transactions: this also helps in simplifying the specific transports > > implementation since stale transport messages can be easily identified > > and discarded early on in the rx path without the need to cross check > > their actual state with the core transport layer. > > This mitigation is even more effective when, as is usually the case, the > > maximum number of pending messages is capped by the platform to a much > > lower number than the whole possible range of tokens values (2^10). > > > > This internal policy change in the core SCMI transport layer is fully > > transparent to the specific transports so it has not and should not have > > any impact on the transports implementation. > > > > The empirically observed cost of such new procedure of token selection > > amounts in the best case to ~10us out of an observed full transaction cost > > of 3ms for the completion of a synchronous sensor reading command on a > > platform supporting commands completion interrupts. > > Hi Cristian, > > Just curious... How badly did a cyclic IDR perform for this usecase? > Feature wise it seems suitable, but perhaps to heavy weight for this > rather constrained case where you can assume the number of IDs in > use at a time is rather small. > > Also, I've not looked closely at the code so there may be other relevant > constraint or subtlety I'm missing. > I'll give it a go to implement it with a cyclic IDR and see how much it improves the performance; I'm not really strong towards an implementation than another, the only real requirement here was to use the full set of 1024 possible tokens before reusing them, even if in reality the set of possibly pending transactions is far less than 1024 (max_msg <<< 1024) and the set of probably pending ones is even smaller. Thanks for the hint about the IDRs I'll see how it pans out in term of complexity and perf before re-posting. Cristian _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel