From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9ABBC6FD1D for ; Sat, 1 Apr 2023 15:55:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229870AbjDAPzc (ORCPT ); Sat, 1 Apr 2023 11:55:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229461AbjDAPza (ORCPT ); Sat, 1 Apr 2023 11:55:30 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 835B820620; Sat, 1 Apr 2023 08:55:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680364526; x=1711900526; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=Gh91C+ZNgnlHe95VEAG2ZvyvuVbTO3suX8aj+cqURKc=; b=A7Iz9/5KUtB8mfSBV71mY5Md0QkuQHVhnnycr3BTT2oBdoEfNUo8BbNM 1wpo4Qcm9g1w1x6OOsuIUy0+9WBArhHZigFllGm12OX0a04N+y2ZVA81y Kdmx47zswHFgX0t/g4jNWDcJj+DOrUxo6BsLuoFMzVF/vj8xUHY9FPZqY 5hyWHlIhWJZx4m+/C97RNhsCul4kzS+X3br4rrEqSOTQkMPuYGBafnGzh HQg31Wk3P2sei9SZKhJoLLuDTKPc+uDwyeh3P3hlYjr/EUiY23BmBBjD/ 5rsRnqtJmBrIOzQsS8Uuw4Z+wBRC9q1X0IEXlO428arbJ9vYlc4wOw0MZ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="321319307" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="321319307" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 08:55:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662716294" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662716294" Received: from yilunxu-optiplex-7050.sh.intel.com (HELO localhost) ([10.239.159.165]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 08:55:23 -0700 Date: Sat, 1 Apr 2023 23:43:48 +0800 From: Xu Yilun To: "Manne, Nava kishore" Cc: "mdf@kernel.org" , "hao.wu@intel.com" , "trix@redhat.com" , "michal.simek@xilinx.com" , "linux-fpga@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] fpga: zynqmp: Make word align the configuration data Message-ID: References: <20230314094222.66916-1-nava.kishore.manne@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2023-03-28 at 09:33:17 +0000, Manne, Nava kishore wrote: > Hi Yilun, > > Please find my response inline. > > > -----Original Message----- > > From: Xu Yilun > > Sent: Saturday, March 18, 2023 2:55 PM > > To: Manne, Nava kishore > > Cc: mdf@kernel.org; hao.wu@intel.com; trix@redhat.com; > > michal.simek@xilinx.com; linux-fpga@vger.kernel.org; linux-arm- > > kernel@lists.infradead.org; linux-kernel@vger.kernel.org > > Subject: Re: [PATCH] fpga: zynqmp: Make word align the configuration data > > > > On 2023-03-14 at 15:12:22 +0530, Nava kishore Manne wrote: > > > To avoid unwanted copies at firmware(PMUFW) this patch provides a fix > > > > The copy happens in firmware? Please help briefly describe the firmware > > operations in commit message. > > > > Yes, If the firmware receives unaligned Bitstream file from Linux to make them align > it will do one more copy at firmware and this copy takes much time as firmware code > runs on microblaze(32-bit processor and runs at lower frequency). > So, we suggested the users to handle the alignment issues at top layers(Before submitting request to the firmware). > > Will update the description in v2. > > > > to align programmable logic(PL) configuration data if the data is not > > > word-aligned. To align the configuration data this patch adds a few > > > padding bytes and these additional padding bytes will not create any > > > functional impact on the PL configuration. > > > > > > Signed-off-by: Nava kishore Manne > > > --- > > > drivers/fpga/zynqmp-fpga.c | 15 +++++++++++++-- > > > 1 file changed, 13 insertions(+), 2 deletions(-) > > > > > > diff --git a/drivers/fpga/zynqmp-fpga.c b/drivers/fpga/zynqmp-fpga.c > > > index c60f20949c47..70a12dc6e15c 100644 > > > --- a/drivers/fpga/zynqmp-fpga.c > > > +++ b/drivers/fpga/zynqmp-fpga.c > > > @@ -15,6 +15,9 @@ > > > /* Constant Definitions */ > > > #define IXR_FPGA_DONE_MASK BIT(3) > > > > > > +#define DUMMY_PAD_BYTE 0xFF > > > +#define FPGA_WORD_SIZE 4 > > > + > > > /** > > > * struct zynqmp_fpga_priv - Private data structure > > > * @dev: Device data structure > > > @@ -41,18 +44,26 @@ static int zynqmp_fpga_ops_write(struct > > fpga_manager *mgr, > > > const char *buf, size_t size) > > > { > > > struct zynqmp_fpga_priv *priv; > > > + int word_align, ret, index; > > > dma_addr_t dma_addr; > > > u32 eemi_flags = 0; > > > char *kbuf; > > > - int ret; > > > > > > priv = mgr->priv; > > > + word_align = size % FPGA_WORD_SIZE; > > > + if (word_align) > > > + word_align = FPGA_WORD_SIZE - word_align; > > > + > > > + size = size + word_align; > > > > Does the Macro ALIGN() help? > > > > Will fix in v2. > > > > > > > kbuf = dma_alloc_coherent(priv->dev, size, &dma_addr, > > GFP_KERNEL); > > > if (!kbuf) > > > return -ENOMEM; > > > > > > - memcpy(kbuf, buf, size); > > > > This is historical, but why do the realloc & copy? Any better way? > > > > Firmware internally uses the AXI DMA engine to transfer PL data from memory to the device > and it supports only continues DMA-able memory access(It will not support scatter-gather memory access). > So, this extra copy is needed to copy the data from kernel memory(allocated by the firmware subsystem using page allocators) I see, but seems the issues are generic. Same issues for versal-fpga 1. fpga_mgr_buf_load_mapped(), fpgamanager_ops::write are intended for contiguous (and maybe dma-safe) buffer but fpga_mgr_firmware_load() breaks this. This may also affect spi backend drivers. 2. Some drivers needs dma addr rather than cpu addr, this could be generally supported. But allocating a bounce buffer by dma_alloc_coherent() is not a good idea. 3. fpga_manager_ops::write() may also be called for each fragment of a sg buffer, could the firmware OK to handle this case? I think framework refactoring is needed for these issues. I may not have time to do the refactoring but will support these topics in first priority. Thanks, Yilun > to continues DMA-able memory. > > > > + for (index = 0; index < word_align; index++) > > > + kbuf[index] = DUMMY_PAD_BYTE; > > > + > > > + memcpy(&kbuf[index], buf, size - index); > > > > Generally I object to massive copy in fpga_manager_ops::write if not > > necessary. If there is an alignment requirement from HW, it should be > > noticed to the caller in some way, before the buffer is created. > > > > Agree, we should find a way to support this kind of use cases. > > Regards, > Navakishore. > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DCBE8C76196 for ; Sat, 1 Apr 2023 15:56:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IqtvZRqCg/MkOFJ8IaZ+mDMKDw7bDWYgpG9Zv7gCIbg=; b=XMcucFiE+ez01i hgqZqew5CSPCLp5p1iU00LbFDfPuyVGNQkFJReM3zQKhlsz2fOQPkxoIQ7NlDlSmgQ1pWUUrkJSXW xJBpXBlJcaKAKwZGC57RN4OPjPHrgwgjv5H91rygyXdDbCoWHaK+RMc6yqIdXqQCV0HEkmNY7/3vl lWUsDcZLA/Ktj3mX7aDO4GF1CZI5yPJL8rPJXNTgVovR7K+yPTAVzYhcPeCkuIecVmJvY7e7sjElr G12F5pSf5er2geMnKButLUiYgkeTpWmGUwDA0fdAszRzP8YMk9Iwip9VafEKGqHXXfiC4vFneVm9d u3Sx0UxJdUVQDYm/EXeg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pidZk-00ApT9-0v; Sat, 01 Apr 2023 15:55:32 +0000 Received: from mga12.intel.com ([192.55.52.136]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pidZh-00ApSJ-0X for linux-arm-kernel@lists.infradead.org; Sat, 01 Apr 2023 15:55:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680364529; x=1711900529; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=Gh91C+ZNgnlHe95VEAG2ZvyvuVbTO3suX8aj+cqURKc=; b=ThIFUN3Ezse4vOeaPpLrhW3Q7PbyHT5Jn9UpOj1sVFtkOxBBNVysiVq6 QghZ0RUZU9hxOgLf7qNLAkShW0908+GL0KKAMl+ohzfmnVydpS2LuBt0i vVHjb27RdB1rD7bHvXnrBFnct8Pr/vAl4reXV4bq/hKnYIgWP1uJSpPXR 4NxV89NytmIV2YxRXKFP8hRTPds2uJytsid2xTJ/uZMa+Bxwdqu8rXuWM O/d4nqtEVy6fOHwZDLRt6CfoWHI6F+WFZdMZBj15kjk8xxHX93nrFpbyE o0/y4ufAoZvs1BiNgqhiIsHyyMt6xCCOydDVLvPoSOnyiois1HGSMQa9Q A==; X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="321319311" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="321319311" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2023 08:55:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10667"; a="662716294" X-IronPort-AV: E=Sophos;i="5.98,310,1673942400"; d="scan'208";a="662716294" Received: from yilunxu-optiplex-7050.sh.intel.com (HELO localhost) ([10.239.159.165]) by orsmga006.jf.intel.com with ESMTP; 01 Apr 2023 08:55:23 -0700 Date: Sat, 1 Apr 2023 23:43:48 +0800 From: Xu Yilun To: "Manne, Nava kishore" Cc: "mdf@kernel.org" , "hao.wu@intel.com" , "trix@redhat.com" , "michal.simek@xilinx.com" , "linux-fpga@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] fpga: zynqmp: Make word align the configuration data Message-ID: References: <20230314094222.66916-1-nava.kishore.manne@amd.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230401_085529_253647_3E8ADC9B X-CRM114-Status: GOOD ( 40.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2023-03-28 at 09:33:17 +0000, Manne, Nava kishore wrote: > Hi Yilun, > > Please find my response inline. > > > -----Original Message----- > > From: Xu Yilun > > Sent: Saturday, March 18, 2023 2:55 PM > > To: Manne, Nava kishore > > Cc: mdf@kernel.org; hao.wu@intel.com; trix@redhat.com; > > michal.simek@xilinx.com; linux-fpga@vger.kernel.org; linux-arm- > > kernel@lists.infradead.org; linux-kernel@vger.kernel.org > > Subject: Re: [PATCH] fpga: zynqmp: Make word align the configuration data > > > > On 2023-03-14 at 15:12:22 +0530, Nava kishore Manne wrote: > > > To avoid unwanted copies at firmware(PMUFW) this patch provides a fix > > > > The copy happens in firmware? Please help briefly describe the firmware > > operations in commit message. > > > > Yes, If the firmware receives unaligned Bitstream file from Linux to make them align > it will do one more copy at firmware and this copy takes much time as firmware code > runs on microblaze(32-bit processor and runs at lower frequency). > So, we suggested the users to handle the alignment issues at top layers(Before submitting request to the firmware). > > Will update the description in v2. > > > > to align programmable logic(PL) configuration data if the data is not > > > word-aligned. To align the configuration data this patch adds a few > > > padding bytes and these additional padding bytes will not create any > > > functional impact on the PL configuration. > > > > > > Signed-off-by: Nava kishore Manne > > > --- > > > drivers/fpga/zynqmp-fpga.c | 15 +++++++++++++-- > > > 1 file changed, 13 insertions(+), 2 deletions(-) > > > > > > diff --git a/drivers/fpga/zynqmp-fpga.c b/drivers/fpga/zynqmp-fpga.c > > > index c60f20949c47..70a12dc6e15c 100644 > > > --- a/drivers/fpga/zynqmp-fpga.c > > > +++ b/drivers/fpga/zynqmp-fpga.c > > > @@ -15,6 +15,9 @@ > > > /* Constant Definitions */ > > > #define IXR_FPGA_DONE_MASK BIT(3) > > > > > > +#define DUMMY_PAD_BYTE 0xFF > > > +#define FPGA_WORD_SIZE 4 > > > + > > > /** > > > * struct zynqmp_fpga_priv - Private data structure > > > * @dev: Device data structure > > > @@ -41,18 +44,26 @@ static int zynqmp_fpga_ops_write(struct > > fpga_manager *mgr, > > > const char *buf, size_t size) > > > { > > > struct zynqmp_fpga_priv *priv; > > > + int word_align, ret, index; > > > dma_addr_t dma_addr; > > > u32 eemi_flags = 0; > > > char *kbuf; > > > - int ret; > > > > > > priv = mgr->priv; > > > + word_align = size % FPGA_WORD_SIZE; > > > + if (word_align) > > > + word_align = FPGA_WORD_SIZE - word_align; > > > + > > > + size = size + word_align; > > > > Does the Macro ALIGN() help? > > > > Will fix in v2. > > > > > > > kbuf = dma_alloc_coherent(priv->dev, size, &dma_addr, > > GFP_KERNEL); > > > if (!kbuf) > > > return -ENOMEM; > > > > > > - memcpy(kbuf, buf, size); > > > > This is historical, but why do the realloc & copy? Any better way? > > > > Firmware internally uses the AXI DMA engine to transfer PL data from memory to the device > and it supports only continues DMA-able memory access(It will not support scatter-gather memory access). > So, this extra copy is needed to copy the data from kernel memory(allocated by the firmware subsystem using page allocators) I see, but seems the issues are generic. Same issues for versal-fpga 1. fpga_mgr_buf_load_mapped(), fpgamanager_ops::write are intended for contiguous (and maybe dma-safe) buffer but fpga_mgr_firmware_load() breaks this. This may also affect spi backend drivers. 2. Some drivers needs dma addr rather than cpu addr, this could be generally supported. But allocating a bounce buffer by dma_alloc_coherent() is not a good idea. 3. fpga_manager_ops::write() may also be called for each fragment of a sg buffer, could the firmware OK to handle this case? I think framework refactoring is needed for these issues. I may not have time to do the refactoring but will support these topics in first priority. Thanks, Yilun > to continues DMA-able memory. > > > > + for (index = 0; index < word_align; index++) > > > + kbuf[index] = DUMMY_PAD_BYTE; > > > + > > > + memcpy(&kbuf[index], buf, size - index); > > > > Generally I object to massive copy in fpga_manager_ops::write if not > > necessary. If there is an alignment requirement from HW, it should be > > noticed to the caller in some way, before the buffer is created. > > > > Agree, we should find a way to support this kind of use cases. > > Regards, > Navakishore. > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel