From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F9A2C43461 for ; Tue, 8 Sep 2020 18:28:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 78CE12087C for ; Tue, 8 Sep 2020 18:28:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731719AbgIHS23 (ORCPT ); Tue, 8 Sep 2020 14:28:29 -0400 Received: from foss.arm.com ([217.140.110.172]:56900 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731396AbgIHQKZ (ORCPT ); Tue, 8 Sep 2020 12:10:25 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 038A215DB; Tue, 8 Sep 2020 06:26:35 -0700 (PDT) Received: from bogus (unknown [10.57.10.112]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 98FA43F73C; Tue, 8 Sep 2020 06:26:32 -0700 (PDT) Date: Tue, 8 Sep 2020 14:26:26 +0100 From: Sudeep Holla To: Arnd Bergmann Cc: Viresh Kumar , Jassi Brar , Rob Herring , Frank Rowand , Bjorn Andersson , Vincent Guittot , Sudeep Holla , linux-arm-kernel , Devicetree List , Linux Kernel Mailing List Subject: Re: [RFC] dt-bindings: mailbox: add doorbell support to ARM MHU Message-ID: <20200908132602.GA27241@bogus> References: <20200605045645.GD12397@bogus> <20200605085830.GA32372@bogus> <20200610093334.yznxl2esv5ht27ns@vireshk-i7> <20200611100027.GB18781@bogus> <20200612052853.nds4iycie6ldjnnr@vireshk-i7> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Sep 08, 2020 at 11:14:50AM +0200, Arnd Bergmann wrote: > Picking up the old thread again after and getting pinged by multiple > colleagues about it (thanks!) reading through the history. > > On Fri, Jun 12, 2020 at 7:29 AM Viresh Kumar wrote: > > > > On 11-06-20, 19:34, Jassi Brar wrote: > > > In the first post in this thread, Viresh lamented that mailbox > > > introduces "a few ms" delay in the scheduler path. > > > Your own tests show that is certainly not the case -- average is the > > > same as proposed virtual channels 50-100us, the best case is 3us vs > > > 53us for virtual channels. > > > > Hmmm, I am not sure where is the confusion here Jassi. There are two > > things which are very very different from each other. > > > > - Time taken by the mailbox framework (and remote for acknowledging > > it) for completion of a single request, this can be 3us to 100s of > > us. This is clear for everyone. THIS IS NOT THE PROBLEM. > > > > - Delay introduced by few of such requests on the last one, i.e. 5 > > normal requests followed by an important one (like DVFS), the last > > one needs to wait for the first 5 to finish first. THIS IS THE > > PROBLEM. > > Earlier, Jassi also commented "Linux does not provide real-time > guarantees", which to me is what actually causes the issue here: > > Linux having timeouts when communicating to the firmware means > that it relies on the hardware and firmware having real-time behavior > even when not providing real-time guarantees to its processes. > > When comparing the two usage models, it's clear that the minimum > latency for a message delivery is always at least the time time > to process an interrupt, plus at least one expensive MMIO read > and one less expensive posted MMIO write for an Ack. If we > have a doorbell plus out-of-band message, we need an extra > DMA barrier and a read from coherent memory, both of which can > be noticeable. As soon as messages are queued in the current > model, the maximum latency increases by a potentially unbounded > number of round-trips, while in the doorbell model that problem > does not exist, so I agree that we need to handle both modes > in the kernel deal with all existing hardware as well as firmware > that requires low-latency communication. > > It also sounds like that debate is already settled because there > are platforms using both modes, and in the kernel we usually > end up supporting the platforms that our users have, whether > we think it's a good idea or not. > Thanks for the nice summary of the discussion so far. > The only questions that I see in need of being answered are: > > 1. Should the binding use just different "#mbox-cells" values or > also different "compatible" strings to tell that difference? I initially proposed latter, but Rob preferred the former which makes sense for the reasons you have mentioned below. > 2. Should one driver try to handle both modes or should there > be two drivers? > > It sounds like Jassi strongly prefers separate drivers, which > would make separate compatible strings the more practical > approach. Indeed. > While the argument can be made that a single > piece of hardware should only have one DT description, > the counter-argument would be that the behavior described > by the DT here is made up by both the hardware and the > firmware behind it, and they are in fact different. > I am too fine either way. -- Regards, Sudeep From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B66FC2BBD0 for ; Tue, 8 Sep 2020 13:27:51 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ED28923C2D for ; Tue, 8 Sep 2020 13:27:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="QpOMKz+C" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED28923C2D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Zz0v/S64adgGQdzm6viaauRf0TZdy7rROK2FYuxyZAI=; b=QpOMKz+CVHN7sHTmuPtqrxbj/ zQ8xP88m7QeDnhjifHbtPvZCf4XeFFvCAYVvYIZ+mei0IjFRlibdf3YTsGo/DlAuNtH4bDBNEcurE PORIUIUIt7TmADdnMirZfRmgvNWgW2LjFodhlbmcYr0GeOt5dMPkVyRLg0/lXxLFSMOxbGkP58RJG M1/HIt1zUHh5pS90Hzw3k1mc3oWznKITzHuXfoyMaE1T9Ri/arEiJDeMEYSchf5U+EgtkpVl8WLIl QK9+U4kWpjkEeSh5hb2B8SNfSY3pj3emOXqZP+5MGwDbChMcGkzpNY7prmXQbZLx/kpPP7bx8fyJq 7TQ1Q4EoQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kFddw-0002XQ-OM; Tue, 08 Sep 2020 13:26:40 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kFddt-0002WZ-Nz for linux-arm-kernel@lists.infradead.org; Tue, 08 Sep 2020 13:26:38 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 038A215DB; Tue, 8 Sep 2020 06:26:35 -0700 (PDT) Received: from bogus (unknown [10.57.10.112]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 98FA43F73C; Tue, 8 Sep 2020 06:26:32 -0700 (PDT) Date: Tue, 8 Sep 2020 14:26:26 +0100 From: Sudeep Holla To: Arnd Bergmann Subject: Re: [RFC] dt-bindings: mailbox: add doorbell support to ARM MHU Message-ID: <20200908132602.GA27241@bogus> References: <20200605045645.GD12397@bogus> <20200605085830.GA32372@bogus> <20200610093334.yznxl2esv5ht27ns@vireshk-i7> <20200611100027.GB18781@bogus> <20200612052853.nds4iycie6ldjnnr@vireshk-i7> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200908_092637_873558_047B1DA9 X-CRM114-Status: GOOD ( 33.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Herring , Vincent Guittot , Devicetree List , Viresh Kumar , Jassi Brar , Linux Kernel Mailing List , Bjorn Andersson , Sudeep Holla , Frank Rowand , linux-arm-kernel Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Sep 08, 2020 at 11:14:50AM +0200, Arnd Bergmann wrote: > Picking up the old thread again after and getting pinged by multiple > colleagues about it (thanks!) reading through the history. > > On Fri, Jun 12, 2020 at 7:29 AM Viresh Kumar wrote: > > > > On 11-06-20, 19:34, Jassi Brar wrote: > > > In the first post in this thread, Viresh lamented that mailbox > > > introduces "a few ms" delay in the scheduler path. > > > Your own tests show that is certainly not the case -- average is the > > > same as proposed virtual channels 50-100us, the best case is 3us vs > > > 53us for virtual channels. > > > > Hmmm, I am not sure where is the confusion here Jassi. There are two > > things which are very very different from each other. > > > > - Time taken by the mailbox framework (and remote for acknowledging > > it) for completion of a single request, this can be 3us to 100s of > > us. This is clear for everyone. THIS IS NOT THE PROBLEM. > > > > - Delay introduced by few of such requests on the last one, i.e. 5 > > normal requests followed by an important one (like DVFS), the last > > one needs to wait for the first 5 to finish first. THIS IS THE > > PROBLEM. > > Earlier, Jassi also commented "Linux does not provide real-time > guarantees", which to me is what actually causes the issue here: > > Linux having timeouts when communicating to the firmware means > that it relies on the hardware and firmware having real-time behavior > even when not providing real-time guarantees to its processes. > > When comparing the two usage models, it's clear that the minimum > latency for a message delivery is always at least the time time > to process an interrupt, plus at least one expensive MMIO read > and one less expensive posted MMIO write for an Ack. If we > have a doorbell plus out-of-band message, we need an extra > DMA barrier and a read from coherent memory, both of which can > be noticeable. As soon as messages are queued in the current > model, the maximum latency increases by a potentially unbounded > number of round-trips, while in the doorbell model that problem > does not exist, so I agree that we need to handle both modes > in the kernel deal with all existing hardware as well as firmware > that requires low-latency communication. > > It also sounds like that debate is already settled because there > are platforms using both modes, and in the kernel we usually > end up supporting the platforms that our users have, whether > we think it's a good idea or not. > Thanks for the nice summary of the discussion so far. > The only questions that I see in need of being answered are: > > 1. Should the binding use just different "#mbox-cells" values or > also different "compatible" strings to tell that difference? I initially proposed latter, but Rob preferred the former which makes sense for the reasons you have mentioned below. > 2. Should one driver try to handle both modes or should there > be two drivers? > > It sounds like Jassi strongly prefers separate drivers, which > would make separate compatible strings the more practical > approach. Indeed. > While the argument can be made that a single > piece of hardware should only have one DT description, > the counter-argument would be that the behavior described > by the DT here is made up by both the hardware and the > firmware behind it, and they are in fact different. > I am too fine either way. -- Regards, Sudeep _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel