From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B005C48BDF for ; Fri, 11 Jun 2021 00:54:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4080661376 for ; Fri, 11 Jun 2021 00:54:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231140AbhFKA42 (ORCPT ); Thu, 10 Jun 2021 20:56:28 -0400 Received: from so254-9.mailgun.net ([198.61.254.9]:32015 "EHLO so254-9.mailgun.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231192AbhFKA42 (ORCPT ); Thu, 10 Jun 2021 20:56:28 -0400 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1623372871; h=Message-ID: References: In-Reply-To: Subject: Cc: To: From: Date: Content-Transfer-Encoding: Content-Type: MIME-Version: Sender; bh=dcfJebiUK44kAjvGVha7ugPfEUhpoyAPBqW17zs14Co=; b=RUzeZI8uduZGWl1MeLgveqgSa6V/r9h48oGULm+6WUiKkYQK0WZIi01Db4zqa5Z8yMl2+cgK 4OR8TQ6wOlpt5YpTVSmP47IlxaERPqRXpMGUPusyyK+475O/l2PyktbnZorsZGX0UZdNzCBh 5fIaoc5L4bWgJ0kR//cBxkTHv5U= X-Mailgun-Sending-Ip: 198.61.254.9 X-Mailgun-Sid: WyI1MzIzYiIsICJsaW51eC1hcm0tbXNtQHZnZXIua2VybmVsLm9yZyIsICJiZTllNGEiXQ== Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n01.prod.us-east-1.postgun.com with SMTP id 60c2b42fabfd22a3dca34415 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Fri, 11 Jun 2021 00:54:07 GMT Sender: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 142B1C4360C; Fri, 11 Jun 2021 00:54:07 +0000 (UTC) Received: from mail.codeaurora.org (localhost.localdomain [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: saiprakash.ranjan) by smtp.codeaurora.org (Postfix) with ESMTPSA id 5D9D1C4338A; Fri, 11 Jun 2021 00:54:06 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 11 Jun 2021 06:24:06 +0530 From: Sai Prakash Ranjan To: Krishna Reddy Cc: Robin Murphy , linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, Will Deacon , linux-arm-kernel@lists.infradead.org, Thierry Reding Subject: Re: [PATCH] iommu/io-pgtable-arm: Optimize partial walk flush for large scatter-gather list In-Reply-To: References: <20210609145315.25750-1-saiprakash.ranjan@codeaurora.org> <35bfd245-45e2-8083-b620-330d6dbd7bd7@arm.com> <12067ffb8243b220cf03e83aaac3e823@codeaurora.org> <266f190e-99ae-9175-cf13-7a77730af389@arm.com> <61c69d23-324a-85d7-2458-dfff8df9280b@arm.com> Message-ID: <07001b4ed6c0a491eacce6e4dc13ab5e@codeaurora.org> X-Sender: saiprakash.ranjan@codeaurora.org User-Agent: Roundcube Webmail/1.3.9 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Hi Krishna, On 2021-06-11 06:07, Krishna Reddy wrote: >> > No, the unmap latency is not just in some test case written, the issue >> > is very real and we have workloads where camera is reporting frame >> > drops because of this unmap latency in the order of 100s of milliseconds. >> > And hardware team recommends using ASID based invalidations for >> > anything larger than 128 TLB entries. So yes, we have taken note of >> > impacts here before going this way and hence feel more inclined to >> > make this qcom specific if required. > > Seems like the real issue here is not the unmap API latency. > It should be the high number of back to back SMMU TLB invalidate > register writes that is resulting > in lower ISO BW to Camera and overflow. Isn't it? > Even Tegra186 SoC has similar issue and HW team recommended to rate > limit the number of > back to back SMMU tlb invalidate registers writes. The subsequent > Tegra194 SoC has a dedicated SMMU for > ISO clients to avoid the impact of TLB invalidates from NISO clients on > ISO BW. > Not exactly, this issue is not specific to camera. If you look at the numbers in the commit text, even for the test device its the same observation. It depends on the buffer size we are unmapping which affects the number of TLBIs issue. I am not aware of any such HW side bw issues for camera specifically on QCOM devices. Thanks, Sai >>> Thinking some more, I >>> wonder if the Tegra folks might have an opinion to add here, given >>> that their multiple-SMMU solution was seemingly about trying to get >>> enough TLB and pagetable walk bandwidth in the first place? > > While it is good to reduce the number of tlb register writes, Flushing > all TLB entries at context granularity arbitrarily > can have negative impact on active traffic and BW. I don't have much > data on possible impact at this point. > Can the flushing at context granularity be made a quirk than > performing it as default? > > -KR -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_RED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1867C48BDF for ; Fri, 11 Jun 2021 00:54:36 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 72A0061285 for ; Fri, 11 Jun 2021 00:54:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72A0061285 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3B62F60AB0; Fri, 11 Jun 2021 00:54:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hh3CdO71zLuv; Fri, 11 Jun 2021 00:54:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 24D9B60AA7; Fri, 11 Jun 2021 00:54:35 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DC590C000E; Fri, 11 Jun 2021 00:54:34 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3CE26C000B for ; Fri, 11 Jun 2021 00:54:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with UTF8SMTP id 3143260AAA for ; Fri, 11 Jun 2021 00:54:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with UTF8SMTP id dQu_Zoj0gh6X for ; Fri, 11 Jun 2021 00:54:32 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from m43-7.mailgun.net (m43-7.mailgun.net [69.72.43.7]) by smtp3.osuosl.org (Postfix) with UTF8SMTPS id 7307760AA7 for ; Fri, 11 Jun 2021 00:54:29 +0000 (UTC) DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1623372872; h=Message-ID: References: In-Reply-To: Subject: Cc: To: From: Date: Content-Transfer-Encoding: Content-Type: MIME-Version: Sender; bh=dcfJebiUK44kAjvGVha7ugPfEUhpoyAPBqW17zs14Co=; b=Wr1vEiqqVwpScbOYUQeoHspCHLGGbfyX/vvjJgBDj8HrdZaceu6bC5eae8JpIIBZ029MynYb 8HP/QLdFMH3rOc/e3AzqKgZEAm2tcoeYamRXiZgyGCMhcfczyhwPdJPp6zCwc3fgVItKrqgU /8Oa+Eq32uERM5XNubenBKH/x3g= X-Mailgun-Sending-Ip: 69.72.43.7 X-Mailgun-Sid: WyI3NDkwMCIsICJpb21tdUBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZyIsICJiZTllNGEiXQ== Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n01.prod.us-east-1.postgun.com with SMTP id 60c2b42fabfd22a3dca34391 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Fri, 11 Jun 2021 00:54:07 GMT Received: by smtp.codeaurora.org (Postfix, from userid 1001) id B3734C43460; Fri, 11 Jun 2021 00:54:06 +0000 (UTC) Received: from mail.codeaurora.org (localhost.localdomain [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: saiprakash.ranjan) by smtp.codeaurora.org (Postfix) with ESMTPSA id 5D9D1C4338A; Fri, 11 Jun 2021 00:54:06 +0000 (UTC) MIME-Version: 1.0 Date: Fri, 11 Jun 2021 06:24:06 +0530 From: Sai Prakash Ranjan To: Krishna Reddy Subject: Re: [PATCH] iommu/io-pgtable-arm: Optimize partial walk flush for large scatter-gather list In-Reply-To: References: <20210609145315.25750-1-saiprakash.ranjan@codeaurora.org> <35bfd245-45e2-8083-b620-330d6dbd7bd7@arm.com> <12067ffb8243b220cf03e83aaac3e823@codeaurora.org> <266f190e-99ae-9175-cf13-7a77730af389@arm.com> <61c69d23-324a-85d7-2458-dfff8df9280b@arm.com> Message-ID: <07001b4ed6c0a491eacce6e4dc13ab5e@codeaurora.org> X-Sender: saiprakash.ranjan@codeaurora.org User-Agent: Roundcube Webmail/1.3.9 Cc: Will Deacon , linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, Thierry Reding , Robin Murphy , linux-arm-kernel@lists.infradead.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" Hi Krishna, On 2021-06-11 06:07, Krishna Reddy wrote: >> > No, the unmap latency is not just in some test case written, the issue >> > is very real and we have workloads where camera is reporting frame >> > drops because of this unmap latency in the order of 100s of milliseconds. >> > And hardware team recommends using ASID based invalidations for >> > anything larger than 128 TLB entries. So yes, we have taken note of >> > impacts here before going this way and hence feel more inclined to >> > make this qcom specific if required. > > Seems like the real issue here is not the unmap API latency. > It should be the high number of back to back SMMU TLB invalidate > register writes that is resulting > in lower ISO BW to Camera and overflow. Isn't it? > Even Tegra186 SoC has similar issue and HW team recommended to rate > limit the number of > back to back SMMU tlb invalidate registers writes. The subsequent > Tegra194 SoC has a dedicated SMMU for > ISO clients to avoid the impact of TLB invalidates from NISO clients on > ISO BW. > Not exactly, this issue is not specific to camera. If you look at the numbers in the commit text, even for the test device its the same observation. It depends on the buffer size we are unmapping which affects the number of TLBIs issue. I am not aware of any such HW side bw issues for camera specifically on QCOM devices. Thanks, Sai >>> Thinking some more, I >>> wonder if the Tegra folks might have an opinion to add here, given >>> that their multiple-SMMU solution was seemingly about trying to get >>> enough TLB and pagetable walk bandwidth in the first place? > > While it is good to reduce the number of tlb register writes, Flushing > all TLB entries at context granularity arbitrarily > can have negative impact on active traffic and BW. I don't have much > data on possible impact at this point. > Can the flushing at context granularity be made a quirk than > performing it as default? > > -KR -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu