From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51E3BC04AAA for ; Mon, 6 May 2019 09:13:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C589D2082F for ; Mon, 6 May 2019 09:13:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hx7zMRcV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726283AbfEFJNq (ORCPT ); Mon, 6 May 2019 05:13:46 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:45875 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725851AbfEFJNq (ORCPT ); Mon, 6 May 2019 05:13:46 -0400 Received: by mail-pf1-f195.google.com with SMTP id e24so6410320pfi.12; Mon, 06 May 2019 02:13:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=BEpDXfcgdebtitWCBJfk2+G6kXUwoa4FR1FqkUaFaSE=; b=hx7zMRcV7cqpbxoJbif3fYfpdB5BzjPxaisQiDx1hrTgZo2898PdxtvRnCdlJC0Oxk 0jJcSuOIjB5fL7pZ7Qd8N5tf5Yf42NP2vBY59Rd5kwm9w1ogwsKByf2Tc6QuIvQUYgxj dJ64okCgzJlzsDg64g+/p4/qDTXtVqZ3/H4yaBmIQn30Feox58SURzt7b738jFyv5+ES 6nm8bVqYWNM1J/k9KZpNU7VY0Ombta/3N4CNtXZhQPfPA+g4WTnd3agS6mb5UGS1QGF2 hPLUnPfPXK7xBv+eRNfoADQm8hU7O4H5vYcqh1Di7W4ALlaKGW3rpMiDsDrMQoSjpDnv eChA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=BEpDXfcgdebtitWCBJfk2+G6kXUwoa4FR1FqkUaFaSE=; b=cHfLHQusoygZe35DnvM/ALsySrZvGJMfLzYQs61SjgLY5+xobnfFk1dLZynxoVJFcD ni2OkPVvwYFj1oTBwohAFQ8B22I68F69r2xWjZg/MD2wfkWlh5uTuFfgAhVbwQkgu5MG ZbO32BOkZxa4BKH5ag6C93o7bbnfO4cqkeUI5ymj9MyqTfB374ppDuOXW9bWfUon4siy +oySKLfPdcO0HOozRzsDkTTSA4m5C3BVCMvX3sj0gvDC27gKUTmx/6aWk5by4u5JRleF HDklf1JaOg9S9io+OvXVugFKL2KOLJCmEgk1cfy8Z4UbIeKtDty7wGJEld5odf7evFB9 aHdg== X-Gm-Message-State: APjAAAVWrncgSWLjI/V77JV/T56xucQ6SKQkHHvH6Bgc9ioQZxdgta9c 1GiR2hm8mqSqmc7LINK4ye0xjJZDmwQ4DADy6xc= X-Google-Smtp-Source: APXvYqwvdoGOlEFMX+koFix/vFPmjVLpzS1pDoZOlntwmG6VOPVoU6gi1m+a58eQdXJqKTBGJbrf2jnMKan/MD4ouK0= X-Received: by 2002:a63:dd58:: with SMTP id g24mr30463674pgj.161.1557134024213; Mon, 06 May 2019 02:13:44 -0700 (PDT) MIME-Version: 1.0 References: <1556891348-74209-1-git-send-email-lsun@mellanox.com> In-Reply-To: <1556891348-74209-1-git-send-email-lsun@mellanox.com> From: Andy Shevchenko Date: Mon, 6 May 2019 12:13:32 +0300 Message-ID: Subject: Re: [PATCH v16] platform/mellanox: Add TmFifo driver for Mellanox BlueField Soc To: Liming Sun Cc: David Woods , Andy Shevchenko , Darren Hart , Vadim Pasternak , Linux Kernel Mailing List , Platform Driver Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, May 3, 2019 at 4:49 PM Liming Sun wrote: > > This commit adds the TmFifo platform driver for Mellanox BlueField > Soc. TmFifo is a shared FIFO which enables external host machine > to exchange data with the SoC via USB or PCIe. The driver is based > on virtio framework and has console and network access enabled. > Pushed to my review and testing queue, thanks! > Reviewed-by: Vadim Pasternak > Signed-off-by: Liming Sun > --- > v15->v16: > Rebase and resubmit (no new changes). > v14->v15: > Fixes for comments from Andy: > - Remove the 'union' definition of mlxbf_tmfifo_msg_hdr and use > on-the-fly conversion when sending the 8-byte message header > into the FIFO; > - Update comment of mlxbf_tmfifo_msg_hdr explaining why '__be16' > is needed for the 'len' field. The SoC sends data stream into > the FIFO and the other side reads it. The byte order of the data > stream (byte-stream) stays the same. The 'len' field is encoded > into network byte order so upper-level applications in external > host machine with different endianness could decode it. This > implementation was verified over USB with an external PPC host > machine running in big-endian mode. > - Move the 'dev_err()' line to the end of the block in function > mlxbf_tmfifo_alloc_vrings(); > - Remove the 'irq_info->index < MLXBF_TM_MAX_IRQ' check in > mlxbf_tmfifo_irq_handler() since it's unnecessary; > - Remove the 'if (desc_head)' check in > mlxbf_tmfifo_release_pending_pkt() since function > mlxbf_tmfifo_get_pkt_len() is already NULL-aware; > - Adjust the testing order of 'if (!(vring->index & BIT(0)))' > in bool mlxbf_tmfifo_virtio_notify() to test the positive case > 'if (vring->index & BIT(0))' first; > - Add '(u64)offset' conversion in mlxbf_tmfifo_virtio_get() to > avoid 32-bit length addition overflow; > - Update the 'efi.get_variable' statement into single line in > mlxbf_tmfifo_get_cfg_mac(); > - Use new helper devm_platform_ioremap_resource() to replace > 'platform_get_resource() + devm_ioremap_resource()' in > mlxbf_tmfifo_probe(); > v13->v14: > Fixes for comments from Andy: > - Add a blank line to separate the virtio header files; > - Update the comment for 'union mlxbf_tmfifo_msg_hdr' to be > more clear how this union is used; > - Update the 'mlxbf_tmfifo_net_default_mac[ETH_ALEN]' definition > to be two lines; > - Reformat macro MLXBF_TMFIFO_NET_FEATURES to put the definition > in a seperate line; > - Update all 'fifo' to 'FIFO' in the comments; > - Update mlxbf_tmfifo_alloc_vrings() to specifically release the > allocated entries in case of failures, so the logic looks more > clear. In the caller function the mlxbf_tmfifo_free_vrings() > might be called again in case of other failures, which is ok > since the 'va' pointer will be set to NULL once released; > - Update mlxbf_tmfifo_timer() to change the first statement to > one line; > - Update one memcpy() to ether_addr_copy() in > mlxbf_tmfifo_get_cfg_mac(); > - Remove 'fifo->pdev' since it is really not needed; > - Define temporary variable to update the mlxbf_tmfifo_create_vdev() > statement into single line. > New changes by Liming: > - Reorder the logic a little bit in mlxbf_tmfifo_timer(). Previously > it has logic like "!a || !b" while the '!b' will not be evaluated > if '!a' is true. It was changed to this way during review, but is > actually not the desired behavior since both bits need to be > tested/set in fifo->pend_events. This issue was found during > verification which caused extra delays for Tx packets. > v12->v13: > Rebase and resubmit (no new changes). > v11->v12: > Fixed the two unsolved comments from v11. > - "Change macro mlxbf_vdev_to_tmfifo() to one line" > Done. Seems not hard. > - "Is it appropriate use of devm_* for 'tm_vdev =3D devm_kzalloc'" > Yes, understand the comment now. The tmfifo is fixed, but the > vdev is dynamic. Use kzalloc() instead, and free the device > in the release callback which is the right place for it. > v10->v11: > Fixes for comments from Andy: > - Use GENMASK_ULL() instead of GENMASK() in mlxbf-tmfifo-regs.h > - Removed the cpu_to_le64()/le64_to_cpu() conversion since > readq()/writeq() already takes care of it. > - Remove the "if (irq)" check in mlxbf_tmfifo_disable_irqs(). > - Add "u32 count" temp variable in mlxbf_tmfifo_get_tx_avail(). > - Clean up mlxbf_tmfifo_get_cfg_mac(), use ETH_ALEN instead of > value 6. > - Change the tx_buf to use Linux existing 'struct circ_buf'. > Comment not applied: > - "Change macro mlxbf_vdev_to_tmfifo() to one line" > Couldn't fit in one line with 80 chracters > - "Is it appropriate use of devm_* for 'tm_vdev =3D devm_kzalloc'" > This is SoC, the device won't be closed or detached. > The only case is when the driver is unloaded. So it appears > ok to use devm_kzalloc() since it's allocated during probe() > and released during module unload. > Comments from Vadim: OK > v9->v10: > Fixes for comments from Andy: > - Use devm_ioremap_resource() instead of devm_ioremap(). > - Use kernel-doc comments. > - Keep Makefile contents sorted. > - Use same fixed format for offsets. > - Use SZ_1K/SZ_32K instead of 1024/23*1024. > - Remove unnecessary comments. > - Use one style for max numbers. > - More comments for mlxbf_tmfifo_vdev and mlxbf_tmfifo_data_64bit. > - Use globally defined MTU instead of new definition. > - Remove forward declaration of mlxbf_tmfifo_remove(). > - Remove PAGE_ALIGN() for dma_alloc_coherent)(). > - Remove the cast of "struct vring *". > - Check return result of test_and_set_bit(). > - Add a macro mlxbt_vdev_to_tmfifo(). > - Several other minor coding style comments. > Comment not applied: > - "Shouldn't be rather helper in EFI lib in kernel" > Looks like efi.get_variable() is the way I found in the kernel > tree. > - "this one is not protected anyhow? Potential race condition" > In mlxbf_tmfifo_console_tx(), the spin-lock is used to protect the > 'tx_buf' only, not the FIFO writes. So there is no race condition. > - "Is __packed needed in mlxbf_tmfifo_msg_hdr". > Yes, it is needed to make sure the structure is 8 bytes. > Fixes for comments from Vadim: > - Use tab in mlxbf-tmfifo-regs.h > - Use kernel-doc comments for struct mlxbf_tmfifo_msg_hdr and > mlxbf_tmfifo_irq_info as well. > - Use _MAX instead of _CNT in the macro definition to be consistent. > - Fix the MODULE_LICENSE. > - Use BIT_ULL() instead of BIT(). > - Remove argument of 'avail' for mlxbf_tmfifo_rxtx_header() and > mlxbf_tmfifo_rxtx_word() > - Revise logic in mlxbf_tmfifo_rxtx_one_desc() to remove the > WARN_ON(). > - Change "union mlxbf_tmfifo_u64 u" to "union mlxbf_tmfifo_u64 buf" > in mlxbf_tmfifo_rxtx_word(). > - Change date type of vring_change from 'int' to 'bool'. > - Remove the blank lines after Signed-off. > - Don=E2=80=99t use declaration in the middle. > - Make the network header initialization in some more elegant way. > - Change label done to mlxbf_tmfifo_desc_done. > - Remove some unnecessary comments, and several other misc coding > style comments. > - Simplify code logic in mlxbf_tmfifo_virtio_notify() > New changes by Liming: > - Simplify the Rx/Tx function arguments to make it more readable. > v8->v9: > Fixes for comments from Andy: > - Use modern devm_xxx() API instead. > Fixes for comments from Vadim: > - Split the Rx/Tx function into smaller funcitons. > - File name, copyright information. > - Function and variable name conversion. > - Local variable and indent coding styles. > - Remove unnecessary 'inline' declarations. > - Use devm_xxx() APIs. > - Move the efi_char16_t MAC address definition to global. > - Fix warnings reported by 'checkpatch --strict'. > - Fix warnings reported by 'make CF=3D"-D__CHECK_ENDIAN__"'. > - Change select VIRTIO_xxx to depends on VIRTIO_ in Kconfig. > - Merge mlxbf_tmfifo_vdev_tx_buf_push() and > mlxbf_tmfifo_vdev_tx_buf_pop(). > - Add union to avoid casting between __le64 and u64. > - Several other misc coding style comments. > New changes by Liming: > - Removed the DT binding documentation since only ACPI is > supported for now by UEFI on the SoC. > v8: Re-submit under drivers/platform/mellanox for the target-side > platform driver only. > v7: Added host side drivers into the same patch set. > v5~v6: Coding style fix. > v1~v4: Initial version for directory drivers/soc/mellanox. > --- > drivers/platform/mellanox/Kconfig | 12 +- > drivers/platform/mellanox/Makefile | 1 + > drivers/platform/mellanox/mlxbf-tmfifo-regs.h | 63 ++ > drivers/platform/mellanox/mlxbf-tmfifo.c | 1281 +++++++++++++++++++= ++++++ > 4 files changed, 1356 insertions(+), 1 deletion(-) > create mode 100644 drivers/platform/mellanox/mlxbf-tmfifo-regs.h > create mode 100644 drivers/platform/mellanox/mlxbf-tmfifo.c > > diff --git a/drivers/platform/mellanox/Kconfig b/drivers/platform/mellano= x/Kconfig > index cd8a908..530fe7e 100644 > --- a/drivers/platform/mellanox/Kconfig > +++ b/drivers/platform/mellanox/Kconfig > @@ -5,7 +5,7 @@ > > menuconfig MELLANOX_PLATFORM > bool "Platform support for Mellanox hardware" > - depends on X86 || ARM || COMPILE_TEST > + depends on X86 || ARM || ARM64 || COMPILE_TEST > ---help--- > Say Y here to get to see options for platform support for > Mellanox systems. This option alone does not add any kernel cod= e. > @@ -34,4 +34,14 @@ config MLXREG_IO > to system resets operation, system reset causes monitoring and = some > kinds of mux selection. > > +config MLXBF_TMFIFO > + tristate "Mellanox BlueField SoC TmFifo platform driver" > + depends on ARM64 > + depends on ACPI > + depends on VIRTIO_CONSOLE && VIRTIO_NET > + help > + Say y here to enable TmFifo support. The TmFifo driver provides > + platform driver support for the TmFifo which supports console > + and networking based on the virtio framework. > + > endif # MELLANOX_PLATFORM > diff --git a/drivers/platform/mellanox/Makefile b/drivers/platform/mellan= ox/Makefile > index 57074d9c..a229bda1 100644 > --- a/drivers/platform/mellanox/Makefile > +++ b/drivers/platform/mellanox/Makefile > @@ -3,5 +3,6 @@ > # Makefile for linux/drivers/platform/mellanox > # Mellanox Platform-Specific Drivers > # > +obj-$(CONFIG_MLXBF_TMFIFO) +=3D mlxbf-tmfifo.o > obj-$(CONFIG_MLXREG_HOTPLUG) +=3D mlxreg-hotplug.o > obj-$(CONFIG_MLXREG_IO) +=3D mlxreg-io.o > diff --git a/drivers/platform/mellanox/mlxbf-tmfifo-regs.h b/drivers/plat= form/mellanox/mlxbf-tmfifo-regs.h > new file mode 100644 > index 0000000..e4f0d2e > --- /dev/null > +++ b/drivers/platform/mellanox/mlxbf-tmfifo-regs.h > @@ -0,0 +1,63 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright (c) 2019, Mellanox Technologies. All rights reserved. > + */ > + > +#ifndef __MLXBF_TMFIFO_REGS_H__ > +#define __MLXBF_TMFIFO_REGS_H__ > + > +#include > +#include > + > +#define MLXBF_TMFIFO_TX_DATA 0x00 > +#define MLXBF_TMFIFO_TX_STS 0x08 > +#define MLXBF_TMFIFO_TX_STS__LENGTH 0x0001 > +#define MLXBF_TMFIFO_TX_STS__COUNT_SHIFT 0 > +#define MLXBF_TMFIFO_TX_STS__COUNT_WIDTH 9 > +#define MLXBF_TMFIFO_TX_STS__COUNT_RESET_VAL 0 > +#define MLXBF_TMFIFO_TX_STS__COUNT_RMASK GENMASK_ULL(8, 0) > +#define MLXBF_TMFIFO_TX_STS__COUNT_MASK GENMASK_U= LL(8, 0) > +#define MLXBF_TMFIFO_TX_CTL 0x10 > +#define MLXBF_TMFIFO_TX_CTL__LENGTH 0x0001 > +#define MLXBF_TMFIFO_TX_CTL__LWM_SHIFT 0 > +#define MLXBF_TMFIFO_TX_CTL__LWM_WIDTH 8 > +#define MLXBF_TMFIFO_TX_CTL__LWM_RESET_VAL 128 > +#define MLXBF_TMFIFO_TX_CTL__LWM_RMASK GENMASK_ULL(7, 0) > +#define MLXBF_TMFIFO_TX_CTL__LWM_MASK GENMASK_ULL(7, 0) > +#define MLXBF_TMFIFO_TX_CTL__HWM_SHIFT 8 > +#define MLXBF_TMFIFO_TX_CTL__HWM_WIDTH 8 > +#define MLXBF_TMFIFO_TX_CTL__HWM_RESET_VAL 128 > +#define MLXBF_TMFIFO_TX_CTL__HWM_RMASK GENMASK_ULL(7, 0) > +#define MLXBF_TMFIFO_TX_CTL__HWM_MASK GENMASK_ULL(15, 8= ) > +#define MLXBF_TMFIFO_TX_CTL__MAX_ENTRIES_SHIFT 32 > +#define MLXBF_TMFIFO_TX_CTL__MAX_ENTRIES_WIDTH 9 > +#define MLXBF_TMFIFO_TX_CTL__MAX_ENTRIES_RESET_VAL 256 > +#define MLXBF_TMFIFO_TX_CTL__MAX_ENTRIES_RMASK GENMASK_ULL(8, 0) > +#define MLXBF_TMFIFO_TX_CTL__MAX_ENTRIES_MASK GENMASK_ULL(40, 3= 2) > +#define MLXBF_TMFIFO_RX_DATA 0x00 > +#define MLXBF_TMFIFO_RX_STS 0x08 > +#define MLXBF_TMFIFO_RX_STS__LENGTH 0x0001 > +#define MLXBF_TMFIFO_RX_STS__COUNT_SHIFT 0 > +#define MLXBF_TMFIFO_RX_STS__COUNT_WIDTH 9 > +#define MLXBF_TMFIFO_RX_STS__COUNT_RESET_VAL 0 > +#define MLXBF_TMFIFO_RX_STS__COUNT_RMASK GENMASK_ULL(8, 0) > +#define MLXBF_TMFIFO_RX_STS__COUNT_MASK GENMASK_U= LL(8, 0) > +#define MLXBF_TMFIFO_RX_CTL 0x10 > +#define MLXBF_TMFIFO_RX_CTL__LENGTH 0x0001 > +#define MLXBF_TMFIFO_RX_CTL__LWM_SHIFT 0 > +#define MLXBF_TMFIFO_RX_CTL__LWM_WIDTH 8 > +#define MLXBF_TMFIFO_RX_CTL__LWM_RESET_VAL 128 > +#define MLXBF_TMFIFO_RX_CTL__LWM_RMASK GENMASK_ULL(7, 0) > +#define MLXBF_TMFIFO_RX_CTL__LWM_MASK GENMASK_ULL(7, 0) > +#define MLXBF_TMFIFO_RX_CTL__HWM_SHIFT 8 > +#define MLXBF_TMFIFO_RX_CTL__HWM_WIDTH 8 > +#define MLXBF_TMFIFO_RX_CTL__HWM_RESET_VAL 128 > +#define MLXBF_TMFIFO_RX_CTL__HWM_RMASK GENMASK_ULL(7, 0) > +#define MLXBF_TMFIFO_RX_CTL__HWM_MASK GENMASK_ULL(15, 8= ) > +#define MLXBF_TMFIFO_RX_CTL__MAX_ENTRIES_SHIFT 32 > +#define MLXBF_TMFIFO_RX_CTL__MAX_ENTRIES_WIDTH 9 > +#define MLXBF_TMFIFO_RX_CTL__MAX_ENTRIES_RESET_VAL 256 > +#define MLXBF_TMFIFO_RX_CTL__MAX_ENTRIES_RMASK GENMASK_ULL(8, 0) > +#define MLXBF_TMFIFO_RX_CTL__MAX_ENTRIES_MASK GENMASK_ULL(40, 3= 2) > + > +#endif /* !defined(__MLXBF_TMFIFO_REGS_H__) */ > diff --git a/drivers/platform/mellanox/mlxbf-tmfifo.c b/drivers/platform/= mellanox/mlxbf-tmfifo.c > new file mode 100644 > index 0000000..9a5c9fd > --- /dev/null > +++ b/drivers/platform/mellanox/mlxbf-tmfifo.c > @@ -0,0 +1,1281 @@ > +// SPDX-License-Identifier: GPL-2.0+ > +/* > + * Mellanox BlueField SoC TmFifo driver > + * > + * Copyright (C) 2019 Mellanox Technologies > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include > +#include > +#include > +#include > +#include > + > +#include "mlxbf-tmfifo-regs.h" > + > +/* Vring size. */ > +#define MLXBF_TMFIFO_VRING_SIZE SZ_1K > + > +/* Console Tx buffer size. */ > +#define MLXBF_TMFIFO_CON_TX_BUF_SIZE SZ_32K > + > +/* Console Tx buffer reserved space. */ > +#define MLXBF_TMFIFO_CON_TX_BUF_RSV_SIZE 8 > + > +/* House-keeping timer interval. */ > +#define MLXBF_TMFIFO_TIMER_INTERVAL (HZ / 10) > + > +/* Virtual devices sharing the TM FIFO. */ > +#define MLXBF_TMFIFO_VDEV_MAX (VIRTIO_ID_CONSOLE + 1) > + > +/* > + * Reserve 1/16 of TmFifo space, so console messages are not starved by > + * the networking traffic. > + */ > +#define MLXBF_TMFIFO_RESERVE_RATIO 16 > + > +/* Message with data needs at least two words (for header & data). */ > +#define MLXBF_TMFIFO_DATA_MIN_WORDS 2 > + > +struct mlxbf_tmfifo; > + > +/** > + * mlxbf_tmfifo_vring - Structure of the TmFifo virtual ring > + * @va: virtual address of the ring > + * @dma: dma address of the ring > + * @vq: pointer to the virtio virtqueue > + * @desc: current descriptor of the pending packet > + * @desc_head: head descriptor of the pending packet > + * @cur_len: processed length of the current descriptor > + * @rem_len: remaining length of the pending packet > + * @pkt_len: total length of the pending packet > + * @next_avail: next avail descriptor id > + * @num: vring size (number of descriptors) > + * @align: vring alignment size > + * @index: vring index > + * @vdev_id: vring virtio id (VIRTIO_ID_xxx) > + * @fifo: pointer to the tmfifo structure > + */ > +struct mlxbf_tmfifo_vring { > + void *va; > + dma_addr_t dma; > + struct virtqueue *vq; > + struct vring_desc *desc; > + struct vring_desc *desc_head; > + int cur_len; > + int rem_len; > + u32 pkt_len; > + u16 next_avail; > + int num; > + int align; > + int index; > + int vdev_id; > + struct mlxbf_tmfifo *fifo; > +}; > + > +/* Interrupt types. */ > +enum { > + MLXBF_TM_RX_LWM_IRQ, > + MLXBF_TM_RX_HWM_IRQ, > + MLXBF_TM_TX_LWM_IRQ, > + MLXBF_TM_TX_HWM_IRQ, > + MLXBF_TM_MAX_IRQ > +}; > + > +/* Ring types (Rx & Tx). */ > +enum { > + MLXBF_TMFIFO_VRING_RX, > + MLXBF_TMFIFO_VRING_TX, > + MLXBF_TMFIFO_VRING_MAX > +}; > + > +/** > + * mlxbf_tmfifo_vdev - Structure of the TmFifo virtual device > + * @vdev: virtio device, in which the vdev.id.device field has the > + * VIRTIO_ID_xxx id to distinguish the virtual device. > + * @status: status of the device > + * @features: supported features of the device > + * @vrings: array of tmfifo vrings of this device > + * @config.cons: virtual console config - > + * select if vdev.id.device is VIRTIO_ID_CONSOLE > + * @config.net: virtual network config - > + * select if vdev.id.device is VIRTIO_ID_NET > + * @tx_buf: tx buffer used to buffer data before writing into the FIFO > + */ > +struct mlxbf_tmfifo_vdev { > + struct virtio_device vdev; > + u8 status; > + u64 features; > + struct mlxbf_tmfifo_vring vrings[MLXBF_TMFIFO_VRING_MAX]; > + union { > + struct virtio_console_config cons; > + struct virtio_net_config net; > + } config; > + struct circ_buf tx_buf; > +}; > + > +/** > + * mlxbf_tmfifo_irq_info - Structure of the interrupt information > + * @fifo: pointer to the tmfifo structure > + * @irq: interrupt number > + * @index: index into the interrupt array > + */ > +struct mlxbf_tmfifo_irq_info { > + struct mlxbf_tmfifo *fifo; > + int irq; > + int index; > +}; > + > +/** > + * mlxbf_tmfifo - Structure of the TmFifo > + * @vdev: array of the virtual devices running over the TmFifo > + * @lock: lock to protect the TmFifo access > + * @rx_base: mapped register base address for the Rx FIFO > + * @tx_base: mapped register base address for the Tx FIFO > + * @rx_fifo_size: number of entries of the Rx FIFO > + * @tx_fifo_size: number of entries of the Tx FIFO > + * @pend_events: pending bits for deferred events > + * @irq_info: interrupt information > + * @work: work struct for deferred process > + * @timer: background timer > + * @vring: Tx/Rx ring > + * @spin_lock: spin lock > + * @is_ready: ready flag > + */ > +struct mlxbf_tmfifo { > + struct mlxbf_tmfifo_vdev *vdev[MLXBF_TMFIFO_VDEV_MAX]; > + struct mutex lock; /* TmFifo lock */ > + void __iomem *rx_base; > + void __iomem *tx_base; > + int rx_fifo_size; > + int tx_fifo_size; > + unsigned long pend_events; > + struct mlxbf_tmfifo_irq_info irq_info[MLXBF_TM_MAX_IRQ]; > + struct work_struct work; > + struct timer_list timer; > + struct mlxbf_tmfifo_vring *vring[2]; > + spinlock_t spin_lock; /* spin lock */ > + bool is_ready; > +}; > + > +/** > + * mlxbf_tmfifo_msg_hdr - Structure of the TmFifo message header > + * @type: message type > + * @len: payload length in network byte order. Messages sent into the FI= FO > + * will be read by the other side as data stream in the same byte = order. > + * The length needs to be encoded into network order so both sides > + * could understand it. > + */ > +struct mlxbf_tmfifo_msg_hdr { > + u8 type; > + __be16 len; > + u8 unused[5]; > +} __packed __aligned(sizeof(u64)); > + > +/* > + * Default MAC. > + * This MAC address will be read from EFI persistent variable if configu= red. > + * It can also be reconfigured with standard Linux tools. > + */ > +static u8 mlxbf_tmfifo_net_default_mac[ETH_ALEN] =3D { > + 0x00, 0x1A, 0xCA, 0xFF, 0xFF, 0x01 > +}; > + > +/* EFI variable name of the MAC address. */ > +static efi_char16_t mlxbf_tmfifo_efi_name[] =3D L"RshimMacAddr"; > + > +/* Maximum L2 header length. */ > +#define MLXBF_TMFIFO_NET_L2_OVERHEAD 36 > + > +/* Supported virtio-net features. */ > +#define MLXBF_TMFIFO_NET_FEATURES \ > + (BIT_ULL(VIRTIO_NET_F_MTU) | BIT_ULL(VIRTIO_NET_F_STATUS) | \ > + BIT_ULL(VIRTIO_NET_F_MAC)) > + > +#define mlxbf_vdev_to_tmfifo(d) container_of(d, struct mlxbf_tmfifo_vdev= , vdev) > + > +/* Free vrings of the FIFO device. */ > +static void mlxbf_tmfifo_free_vrings(struct mlxbf_tmfifo *fifo, > + struct mlxbf_tmfifo_vdev *tm_vdev) > +{ > + struct mlxbf_tmfifo_vring *vring; > + int i, size; > + > + for (i =3D 0; i < ARRAY_SIZE(tm_vdev->vrings); i++) { > + vring =3D &tm_vdev->vrings[i]; > + if (vring->va) { > + size =3D vring_size(vring->num, vring->align); > + dma_free_coherent(tm_vdev->vdev.dev.parent, size, > + vring->va, vring->dma); > + vring->va =3D NULL; > + if (vring->vq) { > + vring_del_virtqueue(vring->vq); > + vring->vq =3D NULL; > + } > + } > + } > +} > + > +/* Allocate vrings for the FIFO. */ > +static int mlxbf_tmfifo_alloc_vrings(struct mlxbf_tmfifo *fifo, > + struct mlxbf_tmfifo_vdev *tm_vdev) > +{ > + struct mlxbf_tmfifo_vring *vring; > + struct device *dev; > + dma_addr_t dma; > + int i, size; > + void *va; > + > + for (i =3D 0; i < ARRAY_SIZE(tm_vdev->vrings); i++) { > + vring =3D &tm_vdev->vrings[i]; > + vring->fifo =3D fifo; > + vring->num =3D MLXBF_TMFIFO_VRING_SIZE; > + vring->align =3D SMP_CACHE_BYTES; > + vring->index =3D i; > + vring->vdev_id =3D tm_vdev->vdev.id.device; > + dev =3D &tm_vdev->vdev.dev; > + > + size =3D vring_size(vring->num, vring->align); > + va =3D dma_alloc_coherent(dev->parent, size, &dma, GFP_KE= RNEL); > + if (!va) { > + mlxbf_tmfifo_free_vrings(fifo, tm_vdev); > + dev_err(dev->parent, "dma_alloc_coherent failed\n= "); > + return -ENOMEM; > + } > + > + vring->va =3D va; > + vring->dma =3D dma; > + } > + > + return 0; > +} > + > +/* Disable interrupts of the FIFO device. */ > +static void mlxbf_tmfifo_disable_irqs(struct mlxbf_tmfifo *fifo) > +{ > + int i, irq; > + > + for (i =3D 0; i < MLXBF_TM_MAX_IRQ; i++) { > + irq =3D fifo->irq_info[i].irq; > + fifo->irq_info[i].irq =3D 0; > + disable_irq(irq); > + } > +} > + > +/* Interrupt handler. */ > +static irqreturn_t mlxbf_tmfifo_irq_handler(int irq, void *arg) > +{ > + struct mlxbf_tmfifo_irq_info *irq_info =3D arg; > + > + if (!test_and_set_bit(irq_info->index, &irq_info->fifo->pend_even= ts)) > + schedule_work(&irq_info->fifo->work); > + > + return IRQ_HANDLED; > +} > + > +/* Get the next packet descriptor from the vring. */ > +static struct vring_desc * > +mlxbf_tmfifo_get_next_desc(struct mlxbf_tmfifo_vring *vring) > +{ > + const struct vring *vr =3D virtqueue_get_vring(vring->vq); > + struct virtio_device *vdev =3D vring->vq->vdev; > + unsigned int idx, head; > + > + if (vring->next_avail =3D=3D virtio16_to_cpu(vdev, vr->avail->idx= )) > + return NULL; > + > + idx =3D vring->next_avail % vr->num; > + head =3D virtio16_to_cpu(vdev, vr->avail->ring[idx]); > + if (WARN_ON(head >=3D vr->num)) > + return NULL; > + > + vring->next_avail++; > + > + return &vr->desc[head]; > +} > + > +/* Release virtio descriptor. */ > +static void mlxbf_tmfifo_release_desc(struct mlxbf_tmfifo_vring *vring, > + struct vring_desc *desc, u32 len) > +{ > + const struct vring *vr =3D virtqueue_get_vring(vring->vq); > + struct virtio_device *vdev =3D vring->vq->vdev; > + u16 idx, vr_idx; > + > + vr_idx =3D virtio16_to_cpu(vdev, vr->used->idx); > + idx =3D vr_idx % vr->num; > + vr->used->ring[idx].id =3D cpu_to_virtio32(vdev, desc - vr->desc)= ; > + vr->used->ring[idx].len =3D cpu_to_virtio32(vdev, len); > + > + /* > + * Virtio could poll and check the 'idx' to decide whether the de= sc is > + * done or not. Add a memory barrier here to make sure the update= above > + * completes before updating the idx. > + */ > + mb(); > + vr->used->idx =3D cpu_to_virtio16(vdev, vr_idx + 1); > +} > + > +/* Get the total length of the descriptor chain. */ > +static u32 mlxbf_tmfifo_get_pkt_len(struct mlxbf_tmfifo_vring *vring, > + struct vring_desc *desc) > +{ > + const struct vring *vr =3D virtqueue_get_vring(vring->vq); > + struct virtio_device *vdev =3D vring->vq->vdev; > + u32 len =3D 0, idx; > + > + while (desc) { > + len +=3D virtio32_to_cpu(vdev, desc->len); > + if (!(virtio16_to_cpu(vdev, desc->flags) & VRING_DESC_F_N= EXT)) > + break; > + idx =3D virtio16_to_cpu(vdev, desc->next); > + desc =3D &vr->desc[idx]; > + } > + > + return len; > +} > + > +static void mlxbf_tmfifo_release_pending_pkt(struct mlxbf_tmfifo_vring *= vring) > +{ > + struct vring_desc *desc_head; > + u32 len =3D 0; > + > + if (vring->desc_head) { > + desc_head =3D vring->desc_head; > + len =3D vring->pkt_len; > + } else { > + desc_head =3D mlxbf_tmfifo_get_next_desc(vring); > + len =3D mlxbf_tmfifo_get_pkt_len(vring, desc_head); > + } > + > + if (desc_head) > + mlxbf_tmfifo_release_desc(vring, desc_head, len); > + > + vring->pkt_len =3D 0; > + vring->desc =3D NULL; > + vring->desc_head =3D NULL; > +} > + > +static void mlxbf_tmfifo_init_net_desc(struct mlxbf_tmfifo_vring *vring, > + struct vring_desc *desc, bool is_r= x) > +{ > + struct virtio_device *vdev =3D vring->vq->vdev; > + struct virtio_net_hdr *net_hdr; > + > + net_hdr =3D phys_to_virt(virtio64_to_cpu(vdev, desc->addr)); > + memset(net_hdr, 0, sizeof(*net_hdr)); > +} > + > +/* Get and initialize the next packet. */ > +static struct vring_desc * > +mlxbf_tmfifo_get_next_pkt(struct mlxbf_tmfifo_vring *vring, bool is_rx) > +{ > + struct vring_desc *desc; > + > + desc =3D mlxbf_tmfifo_get_next_desc(vring); > + if (desc && is_rx && vring->vdev_id =3D=3D VIRTIO_ID_NET) > + mlxbf_tmfifo_init_net_desc(vring, desc, is_rx); > + > + vring->desc_head =3D desc; > + vring->desc =3D desc; > + > + return desc; > +} > + > +/* House-keeping timer. */ > +static void mlxbf_tmfifo_timer(struct timer_list *t) > +{ > + struct mlxbf_tmfifo *fifo =3D container_of(t, struct mlxbf_tmfifo= , timer); > + int rx, tx; > + > + rx =3D !test_and_set_bit(MLXBF_TM_RX_HWM_IRQ, &fifo->pend_events)= ; > + tx =3D !test_and_set_bit(MLXBF_TM_TX_LWM_IRQ, &fifo->pend_events)= ; > + > + if (rx || tx) > + schedule_work(&fifo->work); > + > + mod_timer(&fifo->timer, jiffies + MLXBF_TMFIFO_TIMER_INTERVAL); > +} > + > +/* Copy one console packet into the output buffer. */ > +static void mlxbf_tmfifo_console_output_one(struct mlxbf_tmfifo_vdev *co= ns, > + struct mlxbf_tmfifo_vring *vr= ing, > + struct vring_desc *desc) > +{ > + const struct vring *vr =3D virtqueue_get_vring(vring->vq); > + struct virtio_device *vdev =3D &cons->vdev; > + u32 len, idx, seg; > + void *addr; > + > + while (desc) { > + addr =3D phys_to_virt(virtio64_to_cpu(vdev, desc->addr)); > + len =3D virtio32_to_cpu(vdev, desc->len); > + > + seg =3D CIRC_SPACE_TO_END(cons->tx_buf.head, cons->tx_buf= .tail, > + MLXBF_TMFIFO_CON_TX_BUF_SIZE); > + if (len <=3D seg) { > + memcpy(cons->tx_buf.buf + cons->tx_buf.head, addr= , len); > + } else { > + memcpy(cons->tx_buf.buf + cons->tx_buf.head, addr= , seg); > + addr +=3D seg; > + memcpy(cons->tx_buf.buf, addr, len - seg); > + } > + cons->tx_buf.head =3D (cons->tx_buf.head + len) % > + MLXBF_TMFIFO_CON_TX_BUF_SIZE; > + > + if (!(virtio16_to_cpu(vdev, desc->flags) & VRING_DESC_F_N= EXT)) > + break; > + idx =3D virtio16_to_cpu(vdev, desc->next); > + desc =3D &vr->desc[idx]; > + } > +} > + > +/* Copy console data into the output buffer. */ > +static void mlxbf_tmfifo_console_output(struct mlxbf_tmfifo_vdev *cons, > + struct mlxbf_tmfifo_vring *vring) > +{ > + struct vring_desc *desc; > + u32 len, avail; > + > + desc =3D mlxbf_tmfifo_get_next_desc(vring); > + while (desc) { > + /* Release the packet if not enough space. */ > + len =3D mlxbf_tmfifo_get_pkt_len(vring, desc); > + avail =3D CIRC_SPACE(cons->tx_buf.head, cons->tx_buf.tail= , > + MLXBF_TMFIFO_CON_TX_BUF_SIZE); > + if (len + MLXBF_TMFIFO_CON_TX_BUF_RSV_SIZE > avail) { > + mlxbf_tmfifo_release_desc(vring, desc, len); > + break; > + } > + > + mlxbf_tmfifo_console_output_one(cons, vring, desc); > + mlxbf_tmfifo_release_desc(vring, desc, len); > + desc =3D mlxbf_tmfifo_get_next_desc(vring); > + } > +} > + > +/* Get the number of available words in Rx FIFO for receiving. */ > +static int mlxbf_tmfifo_get_rx_avail(struct mlxbf_tmfifo *fifo) > +{ > + u64 sts; > + > + sts =3D readq(fifo->rx_base + MLXBF_TMFIFO_RX_STS); > + return FIELD_GET(MLXBF_TMFIFO_RX_STS__COUNT_MASK, sts); > +} > + > +/* Get the number of available words in the TmFifo for sending. */ > +static int mlxbf_tmfifo_get_tx_avail(struct mlxbf_tmfifo *fifo, int vdev= _id) > +{ > + int tx_reserve; > + u32 count; > + u64 sts; > + > + /* Reserve some room in FIFO for console messages. */ > + if (vdev_id =3D=3D VIRTIO_ID_NET) > + tx_reserve =3D fifo->tx_fifo_size / MLXBF_TMFIFO_RESERVE_= RATIO; > + else > + tx_reserve =3D 1; > + > + sts =3D readq(fifo->tx_base + MLXBF_TMFIFO_TX_STS); > + count =3D FIELD_GET(MLXBF_TMFIFO_TX_STS__COUNT_MASK, sts); > + return fifo->tx_fifo_size - tx_reserve - count; > +} > + > +/* Console Tx (move data from the output buffer into the TmFifo). */ > +static void mlxbf_tmfifo_console_tx(struct mlxbf_tmfifo *fifo, int avail= ) > +{ > + struct mlxbf_tmfifo_msg_hdr hdr; > + struct mlxbf_tmfifo_vdev *cons; > + unsigned long flags; > + int size, seg; > + void *addr; > + u64 data; > + > + /* Return if not enough space available. */ > + if (avail < MLXBF_TMFIFO_DATA_MIN_WORDS) > + return; > + > + cons =3D fifo->vdev[VIRTIO_ID_CONSOLE]; > + if (!cons || !cons->tx_buf.buf) > + return; > + > + /* Return if no data to send. */ > + size =3D CIRC_CNT(cons->tx_buf.head, cons->tx_buf.tail, > + MLXBF_TMFIFO_CON_TX_BUF_SIZE); > + if (size =3D=3D 0) > + return; > + > + /* Adjust the size to available space. */ > + if (size + sizeof(hdr) > avail * sizeof(u64)) > + size =3D avail * sizeof(u64) - sizeof(hdr); > + > + /* Write header. */ > + hdr.type =3D VIRTIO_ID_CONSOLE; > + hdr.len =3D htons(size); > + writeq(*(u64 *)&hdr, fifo->tx_base + MLXBF_TMFIFO_TX_DATA); > + > + /* Use spin-lock to protect the 'cons->tx_buf'. */ > + spin_lock_irqsave(&fifo->spin_lock, flags); > + > + while (size > 0) { > + addr =3D cons->tx_buf.buf + cons->tx_buf.tail; > + > + seg =3D CIRC_CNT_TO_END(cons->tx_buf.head, cons->tx_buf.t= ail, > + MLXBF_TMFIFO_CON_TX_BUF_SIZE); > + if (seg >=3D sizeof(u64)) { > + memcpy(&data, addr, sizeof(u64)); > + } else { > + memcpy(&data, addr, seg); > + memcpy((u8 *)&data + seg, cons->tx_buf.buf, > + sizeof(u64) - seg); > + } > + writeq(data, fifo->tx_base + MLXBF_TMFIFO_TX_DATA); > + > + if (size >=3D sizeof(u64)) { > + cons->tx_buf.tail =3D (cons->tx_buf.tail + sizeof= (u64)) % > + MLXBF_TMFIFO_CON_TX_BUF_SIZE; > + size -=3D sizeof(u64); > + } else { > + cons->tx_buf.tail =3D (cons->tx_buf.tail + size) = % > + MLXBF_TMFIFO_CON_TX_BUF_SIZE; > + size =3D 0; > + } > + } > + > + spin_unlock_irqrestore(&fifo->spin_lock, flags); > +} > + > +/* Rx/Tx one word in the descriptor buffer. */ > +static void mlxbf_tmfifo_rxtx_word(struct mlxbf_tmfifo_vring *vring, > + struct vring_desc *desc, > + bool is_rx, int len) > +{ > + struct virtio_device *vdev =3D vring->vq->vdev; > + struct mlxbf_tmfifo *fifo =3D vring->fifo; > + void *addr; > + u64 data; > + > + /* Get the buffer address of this desc. */ > + addr =3D phys_to_virt(virtio64_to_cpu(vdev, desc->addr)); > + > + /* Read a word from FIFO for Rx. */ > + if (is_rx) > + data =3D readq(fifo->rx_base + MLXBF_TMFIFO_RX_DATA); > + > + if (vring->cur_len + sizeof(u64) <=3D len) { > + /* The whole word. */ > + if (is_rx) > + memcpy(addr + vring->cur_len, &data, sizeof(u64))= ; > + else > + memcpy(&data, addr + vring->cur_len, sizeof(u64))= ; > + vring->cur_len +=3D sizeof(u64); > + } else { > + /* Leftover bytes. */ > + if (is_rx) > + memcpy(addr + vring->cur_len, &data, > + len - vring->cur_len); > + else > + memcpy(&data, addr + vring->cur_len, > + len - vring->cur_len); > + vring->cur_len =3D len; > + } > + > + /* Write the word into FIFO for Tx. */ > + if (!is_rx) > + writeq(data, fifo->tx_base + MLXBF_TMFIFO_TX_DATA); > +} > + > +/* > + * Rx/Tx packet header. > + * > + * In Rx case, the packet might be found to belong to a different vring = since > + * the TmFifo is shared by different services. In such case, the 'vring_= change' > + * flag is set. > + */ > +static void mlxbf_tmfifo_rxtx_header(struct mlxbf_tmfifo_vring *vring, > + struct vring_desc *desc, > + bool is_rx, bool *vring_change) > +{ > + struct mlxbf_tmfifo *fifo =3D vring->fifo; > + struct virtio_net_config *config; > + struct mlxbf_tmfifo_msg_hdr hdr; > + int vdev_id, hdr_len; > + > + /* Read/Write packet header. */ > + if (is_rx) { > + /* Drain one word from the FIFO. */ > + *(u64 *)&hdr =3D readq(fifo->rx_base + MLXBF_TMFIFO_RX_DA= TA); > + > + /* Skip the length 0 packets (keepalive). */ > + if (hdr.len =3D=3D 0) > + return; > + > + /* Check packet type. */ > + if (hdr.type =3D=3D VIRTIO_ID_NET) { > + vdev_id =3D VIRTIO_ID_NET; > + hdr_len =3D sizeof(struct virtio_net_hdr); > + config =3D &fifo->vdev[vdev_id]->config.net; > + if (ntohs(hdr.len) > config->mtu + > + MLXBF_TMFIFO_NET_L2_OVERHEAD) > + return; > + } else { > + vdev_id =3D VIRTIO_ID_CONSOLE; > + hdr_len =3D 0; > + } > + > + /* > + * Check whether the new packet still belongs to this vri= ng. > + * If not, update the pkt_len of the new vring. > + */ > + if (vdev_id !=3D vring->vdev_id) { > + struct mlxbf_tmfifo_vdev *tm_dev2 =3D fifo->vdev[= vdev_id]; > + > + if (!tm_dev2) > + return; > + vring->desc =3D desc; > + vring =3D &tm_dev2->vrings[MLXBF_TMFIFO_VRING_RX]= ; > + *vring_change =3D true; > + } > + vring->pkt_len =3D ntohs(hdr.len) + hdr_len; > + } else { > + /* Network virtio has an extra header. */ > + hdr_len =3D (vring->vdev_id =3D=3D VIRTIO_ID_NET) ? > + sizeof(struct virtio_net_hdr) : 0; > + vring->pkt_len =3D mlxbf_tmfifo_get_pkt_len(vring, desc); > + hdr.type =3D (vring->vdev_id =3D=3D VIRTIO_ID_NET) ? > + VIRTIO_ID_NET : VIRTIO_ID_CONSOLE; > + hdr.len =3D htons(vring->pkt_len - hdr_len); > + writeq(*(u64 *)&hdr, fifo->tx_base + MLXBF_TMFIFO_TX_DATA= ); > + } > + > + vring->cur_len =3D hdr_len; > + vring->rem_len =3D vring->pkt_len; > + fifo->vring[is_rx] =3D vring; > +} > + > +/* > + * Rx/Tx one descriptor. > + * > + * Return true to indicate more data available. > + */ > +static bool mlxbf_tmfifo_rxtx_one_desc(struct mlxbf_tmfifo_vring *vring, > + bool is_rx, int *avail) > +{ > + const struct vring *vr =3D virtqueue_get_vring(vring->vq); > + struct mlxbf_tmfifo *fifo =3D vring->fifo; > + struct virtio_device *vdev; > + bool vring_change =3D false; > + struct vring_desc *desc; > + unsigned long flags; > + u32 len, idx; > + > + vdev =3D &fifo->vdev[vring->vdev_id]->vdev; > + > + /* Get the descriptor of the next packet. */ > + if (!vring->desc) { > + desc =3D mlxbf_tmfifo_get_next_pkt(vring, is_rx); > + if (!desc) > + return false; > + } else { > + desc =3D vring->desc; > + } > + > + /* Beginning of a packet. Start to Rx/Tx packet header. */ > + if (vring->pkt_len =3D=3D 0) { > + mlxbf_tmfifo_rxtx_header(vring, desc, is_rx, &vring_chang= e); > + (*avail)--; > + > + /* Return if new packet is for another ring. */ > + if (vring_change) > + return false; > + goto mlxbf_tmfifo_desc_done; > + } > + > + /* Get the length of this desc. */ > + len =3D virtio32_to_cpu(vdev, desc->len); > + if (len > vring->rem_len) > + len =3D vring->rem_len; > + > + /* Rx/Tx one word (8 bytes) if not done. */ > + if (vring->cur_len < len) { > + mlxbf_tmfifo_rxtx_word(vring, desc, is_rx, len); > + (*avail)--; > + } > + > + /* Check again whether it's done. */ > + if (vring->cur_len =3D=3D len) { > + vring->cur_len =3D 0; > + vring->rem_len -=3D len; > + > + /* Get the next desc on the chain. */ > + if (vring->rem_len > 0 && > + (virtio16_to_cpu(vdev, desc->flags) & VRING_DESC_F_NE= XT)) { > + idx =3D virtio16_to_cpu(vdev, desc->next); > + desc =3D &vr->desc[idx]; > + goto mlxbf_tmfifo_desc_done; > + } > + > + /* Done and release the pending packet. */ > + mlxbf_tmfifo_release_pending_pkt(vring); > + desc =3D NULL; > + fifo->vring[is_rx] =3D NULL; > + > + /* Notify upper layer that packet is done. */ > + spin_lock_irqsave(&fifo->spin_lock, flags); > + vring_interrupt(0, vring->vq); > + spin_unlock_irqrestore(&fifo->spin_lock, flags); > + } > + > +mlxbf_tmfifo_desc_done: > + /* Save the current desc. */ > + vring->desc =3D desc; > + > + return true; > +} > + > +/* Rx & Tx processing of a queue. */ > +static void mlxbf_tmfifo_rxtx(struct mlxbf_tmfifo_vring *vring, bool is_= rx) > +{ > + int avail =3D 0, devid =3D vring->vdev_id; > + struct mlxbf_tmfifo *fifo; > + bool more; > + > + fifo =3D vring->fifo; > + > + /* Return if vdev is not ready. */ > + if (!fifo->vdev[devid]) > + return; > + > + /* Return if another vring is running. */ > + if (fifo->vring[is_rx] && fifo->vring[is_rx] !=3D vring) > + return; > + > + /* Only handle console and network for now. */ > + if (WARN_ON(devid !=3D VIRTIO_ID_NET && devid !=3D VIRTIO_ID_CONS= OLE)) > + return; > + > + do { > + /* Get available FIFO space. */ > + if (avail =3D=3D 0) { > + if (is_rx) > + avail =3D mlxbf_tmfifo_get_rx_avail(fifo)= ; > + else > + avail =3D mlxbf_tmfifo_get_tx_avail(fifo,= devid); > + if (avail <=3D 0) > + break; > + } > + > + /* Console output always comes from the Tx buffer. */ > + if (!is_rx && devid =3D=3D VIRTIO_ID_CONSOLE) { > + mlxbf_tmfifo_console_tx(fifo, avail); > + break; > + } > + > + /* Handle one descriptor. */ > + more =3D mlxbf_tmfifo_rxtx_one_desc(vring, is_rx, &avail)= ; > + } while (more); > +} > + > +/* Handle Rx or Tx queues. */ > +static void mlxbf_tmfifo_work_rxtx(struct mlxbf_tmfifo *fifo, int queue_= id, > + int irq_id, bool is_rx) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev; > + struct mlxbf_tmfifo_vring *vring; > + int i; > + > + if (!test_and_clear_bit(irq_id, &fifo->pend_events) || > + !fifo->irq_info[irq_id].irq) > + return; > + > + for (i =3D 0; i < MLXBF_TMFIFO_VDEV_MAX; i++) { > + tm_vdev =3D fifo->vdev[i]; > + if (tm_vdev) { > + vring =3D &tm_vdev->vrings[queue_id]; > + if (vring->vq) > + mlxbf_tmfifo_rxtx(vring, is_rx); > + } > + } > +} > + > +/* Work handler for Rx and Tx case. */ > +static void mlxbf_tmfifo_work_handler(struct work_struct *work) > +{ > + struct mlxbf_tmfifo *fifo; > + > + fifo =3D container_of(work, struct mlxbf_tmfifo, work); > + if (!fifo->is_ready) > + return; > + > + mutex_lock(&fifo->lock); > + > + /* Tx (Send data to the TmFifo). */ > + mlxbf_tmfifo_work_rxtx(fifo, MLXBF_TMFIFO_VRING_TX, > + MLXBF_TM_TX_LWM_IRQ, false); > + > + /* Rx (Receive data from the TmFifo). */ > + mlxbf_tmfifo_work_rxtx(fifo, MLXBF_TMFIFO_VRING_RX, > + MLXBF_TM_RX_HWM_IRQ, true); > + > + mutex_unlock(&fifo->lock); > +} > + > +/* The notify function is called when new buffers are posted. */ > +static bool mlxbf_tmfifo_virtio_notify(struct virtqueue *vq) > +{ > + struct mlxbf_tmfifo_vring *vring =3D vq->priv; > + struct mlxbf_tmfifo_vdev *tm_vdev; > + struct mlxbf_tmfifo *fifo; > + unsigned long flags; > + > + fifo =3D vring->fifo; > + > + /* > + * Virtio maintains vrings in pairs, even number ring for Rx > + * and odd number ring for Tx. > + */ > + if (vring->index & BIT(0)) { > + /* > + * Console could make blocking call with interrupts disab= led. > + * In such case, the vring needs to be served right away.= For > + * other cases, just set the TX LWM bit to start Tx in th= e > + * worker handler. > + */ > + if (vring->vdev_id =3D=3D VIRTIO_ID_CONSOLE) { > + spin_lock_irqsave(&fifo->spin_lock, flags); > + tm_vdev =3D fifo->vdev[VIRTIO_ID_CONSOLE]; > + mlxbf_tmfifo_console_output(tm_vdev, vring); > + spin_unlock_irqrestore(&fifo->spin_lock, flags); > + } else if (test_and_set_bit(MLXBF_TM_TX_LWM_IRQ, > + &fifo->pend_events)) { > + return true; > + } > + } else { > + if (test_and_set_bit(MLXBF_TM_RX_HWM_IRQ, &fifo->pend_eve= nts)) > + return true; > + } > + > + schedule_work(&fifo->work); > + > + return true; > +} > + > +/* Get the array of feature bits for this device. */ > +static u64 mlxbf_tmfifo_virtio_get_features(struct virtio_device *vdev) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + return tm_vdev->features; > +} > + > +/* Confirm device features to use. */ > +static int mlxbf_tmfifo_virtio_finalize_features(struct virtio_device *v= dev) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + tm_vdev->features =3D vdev->features; > + > + return 0; > +} > + > +/* Free virtqueues found by find_vqs(). */ > +static void mlxbf_tmfifo_virtio_del_vqs(struct virtio_device *vdev) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + struct mlxbf_tmfifo_vring *vring; > + struct virtqueue *vq; > + int i; > + > + for (i =3D 0; i < ARRAY_SIZE(tm_vdev->vrings); i++) { > + vring =3D &tm_vdev->vrings[i]; > + > + /* Release the pending packet. */ > + if (vring->desc) > + mlxbf_tmfifo_release_pending_pkt(vring); > + vq =3D vring->vq; > + if (vq) { > + vring->vq =3D NULL; > + vring_del_virtqueue(vq); > + } > + } > +} > + > +/* Create and initialize the virtual queues. */ > +static int mlxbf_tmfifo_virtio_find_vqs(struct virtio_device *vdev, > + unsigned int nvqs, > + struct virtqueue *vqs[], > + vq_callback_t *callbacks[], > + const char * const names[], > + const bool *ctx, > + struct irq_affinity *desc) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + struct mlxbf_tmfifo_vring *vring; > + struct virtqueue *vq; > + int i, ret, size; > + > + if (nvqs > ARRAY_SIZE(tm_vdev->vrings)) > + return -EINVAL; > + > + for (i =3D 0; i < nvqs; ++i) { > + if (!names[i]) { > + ret =3D -EINVAL; > + goto error; > + } > + vring =3D &tm_vdev->vrings[i]; > + > + /* zero vring */ > + size =3D vring_size(vring->num, vring->align); > + memset(vring->va, 0, size); > + vq =3D vring_new_virtqueue(i, vring->num, vring->align, v= dev, > + false, false, vring->va, > + mlxbf_tmfifo_virtio_notify, > + callbacks[i], names[i]); > + if (!vq) { > + dev_err(&vdev->dev, "vring_new_virtqueue failed\n= "); > + ret =3D -ENOMEM; > + goto error; > + } > + > + vqs[i] =3D vq; > + vring->vq =3D vq; > + vq->priv =3D vring; > + } > + > + return 0; > + > +error: > + mlxbf_tmfifo_virtio_del_vqs(vdev); > + return ret; > +} > + > +/* Read the status byte. */ > +static u8 mlxbf_tmfifo_virtio_get_status(struct virtio_device *vdev) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + return tm_vdev->status; > +} > + > +/* Write the status byte. */ > +static void mlxbf_tmfifo_virtio_set_status(struct virtio_device *vdev, > + u8 status) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + tm_vdev->status =3D status; > +} > + > +/* Reset the device. Not much here for now. */ > +static void mlxbf_tmfifo_virtio_reset(struct virtio_device *vdev) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + tm_vdev->status =3D 0; > +} > + > +/* Read the value of a configuration field. */ > +static void mlxbf_tmfifo_virtio_get(struct virtio_device *vdev, > + unsigned int offset, > + void *buf, > + unsigned int len) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + if ((u64)offset + len > sizeof(tm_vdev->config)) > + return; > + > + memcpy(buf, (u8 *)&tm_vdev->config + offset, len); > +} > + > +/* Write the value of a configuration field. */ > +static void mlxbf_tmfifo_virtio_set(struct virtio_device *vdev, > + unsigned int offset, > + const void *buf, > + unsigned int len) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + if ((u64)offset + len > sizeof(tm_vdev->config)) > + return; > + > + memcpy((u8 *)&tm_vdev->config + offset, buf, len); > +} > + > +static void tmfifo_virtio_dev_release(struct device *device) > +{ > + struct virtio_device *vdev =3D > + container_of(device, struct virtio_device, dev); > + struct mlxbf_tmfifo_vdev *tm_vdev =3D mlxbf_vdev_to_tmfifo(vdev); > + > + kfree(tm_vdev); > +} > + > +/* Virtio config operations. */ > +static const struct virtio_config_ops mlxbf_tmfifo_virtio_config_ops =3D= { > + .get_features =3D mlxbf_tmfifo_virtio_get_features, > + .finalize_features =3D mlxbf_tmfifo_virtio_finalize_features, > + .find_vqs =3D mlxbf_tmfifo_virtio_find_vqs, > + .del_vqs =3D mlxbf_tmfifo_virtio_del_vqs, > + .reset =3D mlxbf_tmfifo_virtio_reset, > + .set_status =3D mlxbf_tmfifo_virtio_set_status, > + .get_status =3D mlxbf_tmfifo_virtio_get_status, > + .get =3D mlxbf_tmfifo_virtio_get, > + .set =3D mlxbf_tmfifo_virtio_set, > +}; > + > +/* Create vdev for the FIFO. */ > +static int mlxbf_tmfifo_create_vdev(struct device *dev, > + struct mlxbf_tmfifo *fifo, > + int vdev_id, u64 features, > + void *config, u32 size) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev, *reg_dev =3D NULL; > + int ret; > + > + mutex_lock(&fifo->lock); > + > + tm_vdev =3D fifo->vdev[vdev_id]; > + if (tm_vdev) { > + dev_err(dev, "vdev %d already exists\n", vdev_id); > + ret =3D -EEXIST; > + goto fail; > + } > + > + tm_vdev =3D kzalloc(sizeof(*tm_vdev), GFP_KERNEL); > + if (!tm_vdev) { > + ret =3D -ENOMEM; > + goto fail; > + } > + > + tm_vdev->vdev.id.device =3D vdev_id; > + tm_vdev->vdev.config =3D &mlxbf_tmfifo_virtio_config_ops; > + tm_vdev->vdev.dev.parent =3D dev; > + tm_vdev->vdev.dev.release =3D tmfifo_virtio_dev_release; > + tm_vdev->features =3D features; > + if (config) > + memcpy(&tm_vdev->config, config, size); > + > + if (mlxbf_tmfifo_alloc_vrings(fifo, tm_vdev)) { > + dev_err(dev, "unable to allocate vring\n"); > + ret =3D -ENOMEM; > + goto vdev_fail; > + } > + > + /* Allocate an output buffer for the console device. */ > + if (vdev_id =3D=3D VIRTIO_ID_CONSOLE) > + tm_vdev->tx_buf.buf =3D devm_kmalloc(dev, > + MLXBF_TMFIFO_CON_TX_BU= F_SIZE, > + GFP_KERNEL); > + fifo->vdev[vdev_id] =3D tm_vdev; > + > + /* Register the virtio device. */ > + ret =3D register_virtio_device(&tm_vdev->vdev); > + reg_dev =3D tm_vdev; > + if (ret) { > + dev_err(dev, "register_virtio_device failed\n"); > + goto vdev_fail; > + } > + > + mutex_unlock(&fifo->lock); > + return 0; > + > +vdev_fail: > + mlxbf_tmfifo_free_vrings(fifo, tm_vdev); > + fifo->vdev[vdev_id] =3D NULL; > + if (reg_dev) > + put_device(&tm_vdev->vdev.dev); > + else > + kfree(tm_vdev); > +fail: > + mutex_unlock(&fifo->lock); > + return ret; > +} > + > +/* Delete vdev for the FIFO. */ > +static int mlxbf_tmfifo_delete_vdev(struct mlxbf_tmfifo *fifo, int vdev_= id) > +{ > + struct mlxbf_tmfifo_vdev *tm_vdev; > + > + mutex_lock(&fifo->lock); > + > + /* Unregister vdev. */ > + tm_vdev =3D fifo->vdev[vdev_id]; > + if (tm_vdev) { > + unregister_virtio_device(&tm_vdev->vdev); > + mlxbf_tmfifo_free_vrings(fifo, tm_vdev); > + fifo->vdev[vdev_id] =3D NULL; > + } > + > + mutex_unlock(&fifo->lock); > + > + return 0; > +} > + > +/* Read the configured network MAC address from efi variable. */ > +static void mlxbf_tmfifo_get_cfg_mac(u8 *mac) > +{ > + efi_guid_t guid =3D EFI_GLOBAL_VARIABLE_GUID; > + unsigned long size =3D ETH_ALEN; > + u8 buf[ETH_ALEN]; > + efi_status_t rc; > + > + rc =3D efi.get_variable(mlxbf_tmfifo_efi_name, &guid, NULL, &size= , buf); > + if (rc =3D=3D EFI_SUCCESS && size =3D=3D ETH_ALEN) > + ether_addr_copy(mac, buf); > + else > + ether_addr_copy(mac, mlxbf_tmfifo_net_default_mac); > +} > + > +/* Set TmFifo thresolds which is used to trigger interrupts. */ > +static void mlxbf_tmfifo_set_threshold(struct mlxbf_tmfifo *fifo) > +{ > + u64 ctl; > + > + /* Get Tx FIFO size and set the low/high watermark. */ > + ctl =3D readq(fifo->tx_base + MLXBF_TMFIFO_TX_CTL); > + fifo->tx_fifo_size =3D > + FIELD_GET(MLXBF_TMFIFO_TX_CTL__MAX_ENTRIES_MASK, ctl); > + ctl =3D (ctl & ~MLXBF_TMFIFO_TX_CTL__LWM_MASK) | > + FIELD_PREP(MLXBF_TMFIFO_TX_CTL__LWM_MASK, > + fifo->tx_fifo_size / 2); > + ctl =3D (ctl & ~MLXBF_TMFIFO_TX_CTL__HWM_MASK) | > + FIELD_PREP(MLXBF_TMFIFO_TX_CTL__HWM_MASK, > + fifo->tx_fifo_size - 1); > + writeq(ctl, fifo->tx_base + MLXBF_TMFIFO_TX_CTL); > + > + /* Get Rx FIFO size and set the low/high watermark. */ > + ctl =3D readq(fifo->rx_base + MLXBF_TMFIFO_RX_CTL); > + fifo->rx_fifo_size =3D > + FIELD_GET(MLXBF_TMFIFO_RX_CTL__MAX_ENTRIES_MASK, ctl); > + ctl =3D (ctl & ~MLXBF_TMFIFO_RX_CTL__LWM_MASK) | > + FIELD_PREP(MLXBF_TMFIFO_RX_CTL__LWM_MASK, 0); > + ctl =3D (ctl & ~MLXBF_TMFIFO_RX_CTL__HWM_MASK) | > + FIELD_PREP(MLXBF_TMFIFO_RX_CTL__HWM_MASK, 1); > + writeq(ctl, fifo->rx_base + MLXBF_TMFIFO_RX_CTL); > +} > + > +static void mlxbf_tmfifo_cleanup(struct mlxbf_tmfifo *fifo) > +{ > + int i; > + > + fifo->is_ready =3D false; > + del_timer_sync(&fifo->timer); > + mlxbf_tmfifo_disable_irqs(fifo); > + cancel_work_sync(&fifo->work); > + for (i =3D 0; i < MLXBF_TMFIFO_VDEV_MAX; i++) > + mlxbf_tmfifo_delete_vdev(fifo, i); > +} > + > +/* Probe the TMFIFO. */ > +static int mlxbf_tmfifo_probe(struct platform_device *pdev) > +{ > + struct virtio_net_config net_config; > + struct device *dev =3D &pdev->dev; > + struct mlxbf_tmfifo *fifo; > + int i, rc; > + > + fifo =3D devm_kzalloc(dev, sizeof(*fifo), GFP_KERNEL); > + if (!fifo) > + return -ENOMEM; > + > + spin_lock_init(&fifo->spin_lock); > + INIT_WORK(&fifo->work, mlxbf_tmfifo_work_handler); > + mutex_init(&fifo->lock); > + > + /* Get the resource of the Rx FIFO. */ > + fifo->rx_base =3D devm_platform_ioremap_resource(pdev, 0); > + if (IS_ERR(fifo->rx_base)) > + return PTR_ERR(fifo->rx_base); > + > + /* Get the resource of the Tx FIFO. */ > + fifo->tx_base =3D devm_platform_ioremap_resource(pdev, 1); > + if (IS_ERR(fifo->tx_base)) > + return PTR_ERR(fifo->tx_base); > + > + platform_set_drvdata(pdev, fifo); > + > + timer_setup(&fifo->timer, mlxbf_tmfifo_timer, 0); > + > + for (i =3D 0; i < MLXBF_TM_MAX_IRQ; i++) { > + fifo->irq_info[i].index =3D i; > + fifo->irq_info[i].fifo =3D fifo; > + fifo->irq_info[i].irq =3D platform_get_irq(pdev, i); > + rc =3D devm_request_irq(dev, fifo->irq_info[i].irq, > + mlxbf_tmfifo_irq_handler, 0, > + "tmfifo", &fifo->irq_info[i]); > + if (rc) { > + dev_err(dev, "devm_request_irq failed\n"); > + fifo->irq_info[i].irq =3D 0; > + return rc; > + } > + } > + > + mlxbf_tmfifo_set_threshold(fifo); > + > + /* Create the console vdev. */ > + rc =3D mlxbf_tmfifo_create_vdev(dev, fifo, VIRTIO_ID_CONSOLE, 0, = NULL, 0); > + if (rc) > + goto fail; > + > + /* Create the network vdev. */ > + memset(&net_config, 0, sizeof(net_config)); > + net_config.mtu =3D ETH_DATA_LEN; > + net_config.status =3D VIRTIO_NET_S_LINK_UP; > + mlxbf_tmfifo_get_cfg_mac(net_config.mac); > + rc =3D mlxbf_tmfifo_create_vdev(dev, fifo, VIRTIO_ID_NET, > + MLXBF_TMFIFO_NET_FEATURES, &net_con= fig, > + sizeof(net_config)); > + if (rc) > + goto fail; > + > + mod_timer(&fifo->timer, jiffies + MLXBF_TMFIFO_TIMER_INTERVAL); > + > + fifo->is_ready =3D true; > + return 0; > + > +fail: > + mlxbf_tmfifo_cleanup(fifo); > + return rc; > +} > + > +/* Device remove function. */ > +static int mlxbf_tmfifo_remove(struct platform_device *pdev) > +{ > + struct mlxbf_tmfifo *fifo =3D platform_get_drvdata(pdev); > + > + mlxbf_tmfifo_cleanup(fifo); > + > + return 0; > +} > + > +static const struct acpi_device_id mlxbf_tmfifo_acpi_match[] =3D { > + { "MLNXBF01", 0 }, > + {} > +}; > +MODULE_DEVICE_TABLE(acpi, mlxbf_tmfifo_acpi_match); > + > +static struct platform_driver mlxbf_tmfifo_driver =3D { > + .probe =3D mlxbf_tmfifo_probe, > + .remove =3D mlxbf_tmfifo_remove, > + .driver =3D { > + .name =3D "bf-tmfifo", > + .acpi_match_table =3D mlxbf_tmfifo_acpi_match, > + }, > +}; > + > +module_platform_driver(mlxbf_tmfifo_driver); > + > +MODULE_DESCRIPTION("Mellanox BlueField SoC TmFifo Driver"); > +MODULE_LICENSE("GPL v2"); > +MODULE_AUTHOR("Mellanox Technologies"); > -- > 1.8.3.1 > --=20 With Best Regards, Andy Shevchenko