* [PATCH v9 net-next 00/14] pds_core driver
@ 2023-04-06 23:41 Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver Shannon Nelson
` (15 more replies)
0 siblings, 16 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Summary:
--------
This patchset implements a new driver for use with the AMD/Pensando
Distributed Services Card (DSC), intended to provide core configuration
services through the auxiliary_bus and through a couple of EXPORTed
functions for use initially in VFio and vDPA feature specific drivers.
To keep this patchset to a manageable size, the pds_vdpa and pds_vfio
drivers have been split out into their own patchsets to be reviewed
separately.
Detail:
-------
AMD/Pensando is making available a new set of devices for supporting vDPA,
VFio, and potentially other features in the Distributed Services Card
(DSC). These features are implemented through a PF that serves as a Core
device for controlling and configuring its VF devices. These VF devices
have separate drivers that use the auxiliary_bus to work through the Core
device as the control path.
Currently, the DSC supports standard ethernet operations using the
ionic driver. This is not replaced by the Core-based devices - these
new devices are in addition to the existing Ethernet device. Typical DSC
configurations will include both PDS devices and Ionic Eth devices.
However, there is a potential future path for ethernet services to come
through this device as well.
The Core device is a new PCI PF/VF device managed by a new driver
'pds_core'. The PF device has access to an admin queue for configuring
the services used by the VFs, and sets up auxiliary_bus devices for each
vDPA VF for communicating with the drivers for the vDPA devices. The VFs
may be for VFio or vDPA, and other services in the future; these VF types
are selected as part of the DSC internal FW configurations, which is out
of the scope of this patchset.
When the vDPA support set is enabled in the core PF through its devlink
param, auxiliary_bus devices are created for each VF that supports the
feature. The vDPA driver then connects to and uses this auxiliary_device
to do control path configuration through the PF device. This can then be
used with the vdpa kernel module to provide devices for virtio_vdpa kernel
module for host interfaces, or vhost_vdpa kernel module for interfaces
exported into your favorite VM.
A cheap ASCII diagram of a vDPA instance looks something like this:
,----------.
| vdpa |
'----------'
| ||
ctl data
| ||
.----------. ||
| pds_vdpa | ||
'----------' ||
| ||
pds_core.vDPA.1 ||
| ||
.---------------. ||
| pds_core | ||
'---------------' ||
|| || ||
09:00.0 09:00.1
== PCI ============================================
|| ||
.----------. .----------.
,------| PF |---| VF |-------,
| '----------' '----------' |
| DSC |
| |
------------------------------------------
Changes:
v9:
- change pdsc field name id to uid to clarify the unique id used for aux device
- remove unnecessary pf->state and other checks in aux device creation
- hardcode fw slotnames for devlink info, don't use strings from FW
- handle errors from PDS_CORE_CMD_INIT devcmd call
- tighten up health thread use of config_lock
- remove pdsc_queue_health_check() layer over queuing health check
- start pds_core.rst file in first patch, add to it incrementally
- give more user interaction info in commit messages
- removed a few more extraneous includes
v8:
Link: https://lore.kernel.org/netdev/20230330234628.14627-1-shannon.nelson@amd.com/
- fixed deadlock problem, use devl_health_reporter_destroy() when devlink is locked
- don't clear client_id until after auxiliary_device_uninit()
v7:
Link: https://lore.kernel.org/netdev/20230330192313.62018-1-shannon.nelson@amd.com/
- use explicit devlink locking and devl_* APIs
- move some of devlink setup logic into probe and remove
- use debugfs_create_u{type}() for state and queue head and tail
- add include for linux/vmalloc.h
Reported-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/oe-kbuild-all/202303260420.Tgq0qobF-lkp@intel.com/
v6:
Link: https://lore.kernel.org/netdev/20230324190243.27722-1-shannon.nelson@amd.com/
- removed version.h include noticed by kernel test robot's version check
Reported-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/oe-kbuild-all/202303230742.pX3ply0t-lkp@intel.com/
- fixed up the more egregious checkpatch line length complaints
- make sure pdsc_auxbus_dev_register() checks padev pointer errcode
v5:
Link: https://lore.kernel.org/netdev/20230322185626.38758-1-shannon.nelson@amd.com/
- added devlink health reporter for FW issues
- removed asic_type, asic_rev, serial_num, fw_version from debugfs as
they are available through other means
- trimed OS info in pdsc_identify(), we don't need to send that much info to the FW
- removed reg/unreg from auxbus client API, they are now in the core when VF
is started
- removed need for pdsc definition in client by simplifying the padev to only carry
struct pci_dev pointers rather than full struct pdsc to the pf and vf
- removed the unused pdsc argument in pdsc_notify()
- moved include/linux/pds/pds_core.h to driver/../pds_core/core.h
- restored a few pds_core_if.h interface values and structs that are shared
with FW source
- moved final config_lock unlock to before tear down of timer and workqueue
to be sure there are no deadlocks while waiting for any stragglers
- changed use of PAGE_SIZE to local PDS_PAGE_SIZE to keep with FW layout needs
without regard to kernel PAGE_SIZE configuration
- removed the redundant *adminqcq argument from pdsc_adminq_post()
v4:
Link: https://lore.kernel.org/netdev/20230308051310.12544-1-shannon.nelson@amd.com/
- reworked to attach to both Core PF and vDPA VF PCI devices
- now creates auxiliary_device as part of each VF PCI probe, removes them on PCI remove
- auxiliary devices now use simple unique id rather than PCI address for identifier
- replaced home-grown event publishing with kernel-based notifier service
- dropped live_migration parameter, not needed when not creating aux device for it
- replaced devm_* functions with traditional interfaces
- added MAINTAINERS entry
- removed lingering traces of set/get_vf attribute adminq commands
- trimmed some include lists
- cleaned a kernel test robot complaint about a stray unused variable
Link: https://lore.kernel.org/oe-kbuild-all/202302181049.yeUQMeWY-lkp@intel.com/
v3:
Link: https://lore.kernel.org/netdev/20230217225558.19837-1-shannon.nelson@amd.com/
- changed names from "pensando" to "amd" and updated copyright strings
- dropped the DEVLINK_PARAM_GENERIC_ID_FW_BANK for future development
- changed the auxiliary device creation to be triggered by the
PCI bus event BOUND_DRIVER, and torn down at UNBIND_DRIVER in order
to properly handle users using the sysfs bind/unbind functions
- dropped some noisy log messages
- rebased to current net-next
RFC to v2:
Link: https://lore.kernel.org/netdev/20221207004443.33779-1-shannon.nelson@amd.com/
- added separate devlink param patches for DEVLINK_PARAM_GENERIC_ID_ENABLE_MIGRATION
and DEVLINK_PARAM_GENERIC_ID_FW_BANK, and dropped the driver specific implementations
- updated descriptions for the new devlink parameters
- dropped netdev support
- dropped vDPA patches, will followup later
- separated fw update and fw bank select into their own patches
RFC:
Link: https://lore.kernel.org/netdev/20221118225656.48309-1-snelson@pensando.io/
Shannon Nelson (14):
pds_core: initial framework for pds_core PF driver
pds_core: add devcmd device interfaces
pds_core: health timer and workqueue
pds_core: add devlink health facilities
pds_core: set up device and adminq
pds_core: Add adminq processing and commands
pds_core: add FW update feature to devlink
pds_core: set up the VIF definitions and defaults
pds_core: add initial VF device handling
pds_core: add auxiliary_bus devices
pds_core: devlink params for enabling VIF support
pds_core: add the aux client API
pds_core: publish events to the clients
pds_core: Kconfig and pds_core.rst
.../device_drivers/ethernet/amd/pds_core.rst | 144 ++++
.../device_drivers/ethernet/index.rst | 1 +
MAINTAINERS | 9 +
drivers/net/ethernet/amd/Kconfig | 12 +
drivers/net/ethernet/amd/Makefile | 1 +
drivers/net/ethernet/amd/pds_core/Makefile | 14 +
drivers/net/ethernet/amd/pds_core/adminq.c | 291 ++++++++
drivers/net/ethernet/amd/pds_core/auxbus.c | 241 +++++++
drivers/net/ethernet/amd/pds_core/core.c | 616 +++++++++++++++++
drivers/net/ethernet/amd/pds_core/core.h | 325 +++++++++
drivers/net/ethernet/amd/pds_core/debugfs.c | 202 ++++++
drivers/net/ethernet/amd/pds_core/dev.c | 352 ++++++++++
drivers/net/ethernet/amd/pds_core/devlink.c | 180 +++++
drivers/net/ethernet/amd/pds_core/fw.c | 194 ++++++
drivers/net/ethernet/amd/pds_core/main.c | 489 +++++++++++++
include/linux/pds/pds_adminq.h | 645 ++++++++++++++++++
include/linux/pds/pds_auxbus.h | 44 ++
include/linux/pds/pds_common.h | 97 +++
include/linux/pds/pds_core_if.h | 540 +++++++++++++++
include/linux/pds/pds_intr.h | 163 +++++
20 files changed, 4560 insertions(+)
create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile
create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c
create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c
create mode 100644 drivers/net/ethernet/amd/pds_core/core.c
create mode 100644 drivers/net/ethernet/amd/pds_core/core.h
create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c
create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c
create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c
create mode 100644 drivers/net/ethernet/amd/pds_core/main.c
create mode 100644 include/linux/pds/pds_adminq.h
create mode 100644 include/linux/pds/pds_auxbus.h
create mode 100644 include/linux/pds/pds_common.h
create mode 100644 include/linux/pds/pds_core_if.h
create mode 100644 include/linux/pds/pds_intr.h
--
2.17.1
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 11:26 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces Shannon Nelson
` (14 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
This is the initial PCI driver framework for the new pds_core device
driver and its family of devices. This does the very basics of
registering for the new PF PCI device 1dd8:100c, setting up debugfs
entries, and registering with devlink.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
.../device_drivers/ethernet/amd/pds_core.rst | 40 ++
.../device_drivers/ethernet/index.rst | 1 +
drivers/net/ethernet/amd/pds_core/Makefile | 8 +
drivers/net/ethernet/amd/pds_core/core.h | 63 ++
drivers/net/ethernet/amd/pds_core/debugfs.c | 34 ++
drivers/net/ethernet/amd/pds_core/main.c | 285 +++++++++
include/linux/pds/pds_common.h | 14 +
include/linux/pds/pds_core_if.h | 540 ++++++++++++++++++
8 files changed, 985 insertions(+)
create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile
create mode 100644 drivers/net/ethernet/amd/pds_core/core.h
create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c
create mode 100644 drivers/net/ethernet/amd/pds_core/main.c
create mode 100644 include/linux/pds/pds_common.h
create mode 100644 include/linux/pds/pds_core_if.h
diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
new file mode 100644
index 000000000000..58a28b255d37
--- /dev/null
+++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
@@ -0,0 +1,40 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+========================================================
+Linux Driver for the AMD/Pensando(R) DSC adapter family
+========================================================
+
+Copyright(c) 2023 Advanced Micro Devices, Inc
+
+Identifying the Adapter
+=======================
+
+To find if one or more AMD/Pensando PCI Core devices are installed on the
+host, check for the PCI devices::
+
+ # lspci -d 1dd8:100c
+ b5:00.0 Processing accelerators: Pensando Systems Device 100c
+ b6:00.0 Processing accelerators: Pensando Systems Device 100c
+
+If such devices are listed as above, then the pds_core.ko driver should find
+and configure them for use. There should be log entries in the kernel
+messages such as these::
+
+ $ dmesg | grep pds_core
+ pds_core 0000:b5:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
+ pds_core 0000:b5:00.0: FW: 1.60.0-73
+ pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
+ pds_core 0000:b6:00.0: FW: 1.60.0-73
+
+Support
+=======
+
+For general Linux networking support, please use the netdev mailing
+list, which is monitored by AMD/Pensando personnel::
+
+ netdev@vger.kernel.org
+
+For more specific support needs, please use the AMD/Pensando driver support
+email::
+
+ drivers@pensando.io
diff --git a/Documentation/networking/device_drivers/ethernet/index.rst b/Documentation/networking/device_drivers/ethernet/index.rst
index 6e9e7012d000..417ca514a4d0 100644
--- a/Documentation/networking/device_drivers/ethernet/index.rst
+++ b/Documentation/networking/device_drivers/ethernet/index.rst
@@ -14,6 +14,7 @@ Contents:
3com/vortex
amazon/ena
altera/altera_tse
+ amd/pds_core
aquantia/atlantic
chelsio/cxgb
cirrus/cs89x0
diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
new file mode 100644
index 000000000000..de3bf1d1886c
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Advanced Micro Devices, Inc.
+
+obj-$(CONFIG_PDS_CORE) := pds_core.o
+
+pds_core-y := main.o
+
+pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
new file mode 100644
index 000000000000..51c77f145bab
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _PDSC_H_
+#define _PDSC_H_
+
+#include <linux/debugfs.h>
+#include <net/devlink.h>
+
+#include <linux/pds/pds_common.h>
+#include <linux/pds/pds_core_if.h>
+
+#define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver"
+
+struct pdsc_dev_bar {
+ void __iomem *vaddr;
+ phys_addr_t bus_addr;
+ unsigned long len;
+ int res_index;
+};
+
+/* No state flags set means we are in a steady running state */
+enum pdsc_state_flags {
+ PDSC_S_FW_DEAD, /* stopped, wait on startup or recovery */
+ PDSC_S_INITING_DRIVER, /* initial startup from probe */
+ PDSC_S_STOPPING_DRIVER, /* driver remove */
+
+ /* leave this as last */
+ PDSC_S_STATE_SIZE
+};
+
+struct pdsc {
+ struct pci_dev *pdev;
+ struct dentry *dentry;
+ struct device *dev;
+ struct pdsc_dev_bar bars[PDS_CORE_BARS_MAX];
+ int hw_index;
+ int uid;
+
+ unsigned long state;
+
+ struct pds_core_dev_info_regs __iomem *info_regs;
+ struct pds_core_dev_cmd_regs __iomem *cmd_regs;
+ struct pds_core_intr __iomem *intr_ctrl;
+ u64 __iomem *intr_status;
+ u64 __iomem *db_pages;
+ dma_addr_t phy_db_pages;
+ u64 __iomem *kern_dbpage;
+};
+
+#ifdef CONFIG_DEBUG_FS
+void pdsc_debugfs_create(void);
+void pdsc_debugfs_destroy(void);
+void pdsc_debugfs_add_dev(struct pdsc *pdsc);
+void pdsc_debugfs_del_dev(struct pdsc *pdsc);
+#else
+static inline void pdsc_debugfs_create(void) { }
+static inline void pdsc_debugfs_destroy(void) { }
+static inline void pdsc_debugfs_add_dev(struct pdsc *pdsc) { }
+static inline void pdsc_debugfs_del_dev(struct pdsc *pdsc) { }
+#endif
+
+#endif /* _PDSC_H_ */
diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c
new file mode 100644
index 000000000000..9b2385c19c41
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/debugfs.c
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifdef CONFIG_DEBUG_FS
+
+#include <linux/pci.h>
+
+#include "core.h"
+
+static struct dentry *pdsc_dir;
+
+void pdsc_debugfs_create(void)
+{
+ pdsc_dir = debugfs_create_dir(PDS_CORE_DRV_NAME, NULL);
+}
+
+void pdsc_debugfs_destroy(void)
+{
+ debugfs_remove_recursive(pdsc_dir);
+}
+
+void pdsc_debugfs_add_dev(struct pdsc *pdsc)
+{
+ pdsc->dentry = debugfs_create_dir(pci_name(pdsc->pdev), pdsc_dir);
+
+ debugfs_create_ulong("state", 0400, pdsc->dentry, &pdsc->state);
+}
+
+void pdsc_debugfs_del_dev(struct pdsc *pdsc)
+{
+ debugfs_remove_recursive(pdsc->dentry);
+ pdsc->dentry = NULL;
+}
+#endif /* CONFIG_DEBUG_FS */
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
new file mode 100644
index 000000000000..1c2f3fbaa27c
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -0,0 +1,285 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/pci.h>
+
+#include <linux/pds/pds_common.h>
+
+#include "core.h"
+
+MODULE_DESCRIPTION(PDSC_DRV_DESCRIPTION);
+MODULE_AUTHOR("Advanced Micro Devices, Inc");
+MODULE_LICENSE("GPL");
+
+/* Supported devices */
+static const struct pci_device_id pdsc_id_table[] = {
+ { PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF) },
+ { 0, } /* end of table */
+};
+MODULE_DEVICE_TABLE(pci, pdsc_id_table);
+
+static void pdsc_unmap_bars(struct pdsc *pdsc)
+{
+ struct pdsc_dev_bar *bars = pdsc->bars;
+ unsigned int i;
+
+ for (i = 0; i < PDS_CORE_BARS_MAX; i++) {
+ if (bars[i].vaddr) {
+ pci_iounmap(pdsc->pdev, bars[i].vaddr);
+ bars[i].vaddr = NULL;
+ }
+
+ bars[i].len = 0;
+ bars[i].bus_addr = 0;
+ bars[i].res_index = 0;
+ }
+}
+
+static int pdsc_map_bars(struct pdsc *pdsc)
+{
+ struct pdsc_dev_bar *bar = pdsc->bars;
+ struct pci_dev *pdev = pdsc->pdev;
+ struct device *dev = pdsc->dev;
+ struct pdsc_dev_bar *bars;
+ unsigned int i, j;
+ int num_bars = 0;
+ int err;
+ u32 sig;
+
+ bars = pdsc->bars;
+ num_bars = 0;
+
+ /* Since the PCI interface in the hardware is configurable,
+ * we need to poke into all the bars to find the set we're
+ * expecting.
+ */
+ for (i = 0, j = 0; i < PDS_CORE_BARS_MAX; i++) {
+ if (!(pci_resource_flags(pdev, i) & IORESOURCE_MEM))
+ continue;
+
+ bars[j].len = pci_resource_len(pdev, i);
+ bars[j].bus_addr = pci_resource_start(pdev, i);
+ bars[j].res_index = i;
+
+ /* only map the whole bar 0 */
+ if (j > 0) {
+ bars[j].vaddr = NULL;
+ } else {
+ bars[j].vaddr = pci_iomap(pdev, i, bars[j].len);
+ if (!bars[j].vaddr) {
+ dev_err(dev, "Cannot map BAR %d, aborting\n", i);
+ return -ENODEV;
+ }
+ }
+
+ j++;
+ }
+ num_bars = j;
+
+ /* BAR0: dev_cmd and interrupts */
+ if (num_bars < 1) {
+ dev_err(dev, "No bars found\n");
+ err = -EFAULT;
+ goto err_out;
+ }
+
+ if (bar->len < PDS_CORE_BAR0_SIZE) {
+ dev_err(dev, "Resource bar size %lu too small\n", bar->len);
+ err = -EFAULT;
+ goto err_out;
+ }
+
+ pdsc->info_regs = bar->vaddr + PDS_CORE_BAR0_DEV_INFO_REGS_OFFSET;
+ pdsc->cmd_regs = bar->vaddr + PDS_CORE_BAR0_DEV_CMD_REGS_OFFSET;
+ pdsc->intr_status = bar->vaddr + PDS_CORE_BAR0_INTR_STATUS_OFFSET;
+ pdsc->intr_ctrl = bar->vaddr + PDS_CORE_BAR0_INTR_CTRL_OFFSET;
+
+ sig = ioread32(&pdsc->info_regs->signature);
+ if (sig != PDS_CORE_DEV_INFO_SIGNATURE) {
+ dev_err(dev, "Incompatible firmware signature %x", sig);
+ err = -EFAULT;
+ goto err_out;
+ }
+
+ /* BAR1: doorbells */
+ bar++;
+ if (num_bars < 2) {
+ dev_err(dev, "Doorbell bar missing\n");
+ err = -EFAULT;
+ goto err_out;
+ }
+
+ pdsc->db_pages = bar->vaddr;
+ pdsc->phy_db_pages = bar->bus_addr;
+
+ return 0;
+
+err_out:
+ pdsc_unmap_bars(pdsc);
+ pdsc->info_regs = NULL;
+ pdsc->cmd_regs = NULL;
+ pdsc->intr_status = NULL;
+ pdsc->intr_ctrl = NULL;
+ return err;
+}
+
+static int pdsc_init_vf(struct pdsc *vf)
+{
+ return -1;
+}
+
+static int pdsc_init_pf(struct pdsc *pdsc)
+{
+ struct devlink *dl;
+ int err;
+
+ pcie_print_link_status(pdsc->pdev);
+
+ err = pci_request_regions(pdsc->pdev, PDS_CORE_DRV_NAME);
+ if (err) {
+ dev_err(pdsc->dev, "Cannot request PCI regions: %pe\n",
+ ERR_PTR(err));
+ return err;
+ }
+
+ err = pdsc_map_bars(pdsc);
+ if (err)
+ goto err_out_release_regions;
+
+ dl = priv_to_devlink(pdsc);
+ devl_lock(dl);
+ devl_register(dl);
+ devl_unlock(dl);
+
+ return 0;
+
+err_out_release_regions:
+ pci_release_regions(pdsc->pdev);
+
+ return err;
+}
+
+static const struct devlink_ops pdsc_dl_ops = {
+};
+
+static const struct devlink_ops pdsc_dl_vf_ops = {
+};
+
+static DEFINE_IDA(pdsc_ida);
+
+static int pdsc_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+ struct device *dev = &pdev->dev;
+ const struct devlink_ops *ops;
+ struct devlink *dl;
+ struct pdsc *pdsc;
+ bool is_pf;
+ int err;
+
+ is_pf = !pdev->is_virtfn;
+ ops = is_pf ? &pdsc_dl_ops : &pdsc_dl_vf_ops;
+ dl = devlink_alloc(ops, sizeof(struct pdsc), dev);
+ if (!dl)
+ return -ENOMEM;
+ pdsc = devlink_priv(dl);
+
+ pdsc->pdev = pdev;
+ pdsc->dev = &pdev->dev;
+ set_bit(PDSC_S_INITING_DRIVER, &pdsc->state);
+ pci_set_drvdata(pdev, pdsc);
+ pdsc_debugfs_add_dev(pdsc);
+
+ err = ida_alloc(&pdsc_ida, GFP_KERNEL);
+ if (err < 0) {
+ dev_err(pdsc->dev, "%s: id alloc failed: %pe\n",
+ __func__, ERR_PTR(err));
+ goto err_out_free_devlink;
+ }
+ pdsc->uid = err;
+
+ /* Query system for DMA addressing limitation for the device. */
+ err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(PDS_CORE_ADDR_LEN));
+ if (err) {
+ dev_err(dev, "Unable to obtain 64-bit DMA for consistent allocations, aborting: %pe\n",
+ ERR_PTR(err));
+ goto err_out_free_ida;
+ }
+
+ err = pci_enable_device(pdev);
+ if (err) {
+ dev_err(dev, "Cannot enable PCI device: %pe\n", ERR_PTR(err));
+ goto err_out_free_ida;
+ }
+ pci_set_master(pdev);
+
+ if (is_pf)
+ err = pdsc_init_pf(pdsc);
+ else
+ err = pdsc_init_vf(pdsc);
+ if (err) {
+ dev_err(dev, "Cannot init device: %pe\n", ERR_PTR(err));
+ goto err_out_clear_master;
+ }
+
+ clear_bit(PDSC_S_INITING_DRIVER, &pdsc->state);
+ return 0;
+
+err_out_clear_master:
+ pci_clear_master(pdev);
+ pci_disable_device(pdev);
+err_out_free_ida:
+ ida_free(&pdsc_ida, pdsc->uid);
+err_out_free_devlink:
+ pdsc_debugfs_del_dev(pdsc);
+ devlink_free(dl);
+
+ return err;
+}
+
+static void pdsc_remove(struct pci_dev *pdev)
+{
+ struct pdsc *pdsc = pci_get_drvdata(pdev);
+ struct devlink *dl;
+
+ /* Unhook the registrations first to be sure there
+ * are no requests while we're stopping.
+ */
+ dl = priv_to_devlink(pdsc);
+ devl_lock(dl);
+ devl_unregister(dl);
+ devl_unlock(dl);
+
+ pdsc_unmap_bars(pdsc);
+ pci_release_regions(pdev);
+
+ pci_clear_master(pdev);
+ pci_disable_device(pdev);
+
+ ida_free(&pdsc_ida, pdsc->uid);
+ pdsc_debugfs_del_dev(pdsc);
+ devlink_free(dl);
+}
+
+static struct pci_driver pdsc_driver = {
+ .name = PDS_CORE_DRV_NAME,
+ .id_table = pdsc_id_table,
+ .probe = pdsc_probe,
+ .remove = pdsc_remove,
+};
+
+static int __init pdsc_init_module(void)
+{
+ pdsc_debugfs_create();
+ return pci_register_driver(&pdsc_driver);
+}
+
+static void __exit pdsc_cleanup_module(void)
+{
+ pci_unregister_driver(&pdsc_driver);
+ pdsc_debugfs_destroy();
+}
+
+module_init(pdsc_init_module);
+module_exit(pdsc_cleanup_module);
diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
new file mode 100644
index 000000000000..bd041a5170a6
--- /dev/null
+++ b/include/linux/pds/pds_common.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc. */
+
+#ifndef _PDS_COMMON_H_
+#define _PDS_COMMON_H_
+
+#define PDS_CORE_DRV_NAME "pds_core"
+
+/* the device's internal addressing uses up to 52 bits */
+#define PDS_CORE_ADDR_LEN 52
+#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1)
+#define PDS_PAGE_SIZE 4096
+
+#endif /* _PDS_COMMON_H_ */
diff --git a/include/linux/pds/pds_core_if.h b/include/linux/pds/pds_core_if.h
new file mode 100644
index 000000000000..eb9f1662c7c7
--- /dev/null
+++ b/include/linux/pds/pds_core_if.h
@@ -0,0 +1,540 @@
+/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc. */
+
+#ifndef _PDS_CORE_IF_H_
+#define _PDS_CORE_IF_H_
+
+#define PCI_VENDOR_ID_PENSANDO 0x1dd8
+#define PCI_DEVICE_ID_PENSANDO_CORE_PF 0x100c
+#define PCI_DEVICE_ID_VIRTIO_NET_TRANS 0x1000
+#define PCI_DEVICE_ID_PENSANDO_IONIC_ETH_VF 0x1003
+#define PCI_DEVICE_ID_PENSANDO_VDPA_VF 0x100b
+#define PDS_CORE_BARS_MAX 4
+#define PDS_CORE_PCI_BAR_DBELL 1
+
+/* Bar0 */
+#define PDS_CORE_DEV_INFO_SIGNATURE 0x44455649 /* 'DEVI' */
+#define PDS_CORE_BAR0_SIZE 0x8000
+#define PDS_CORE_BAR0_DEV_INFO_REGS_OFFSET 0x0000
+#define PDS_CORE_BAR0_DEV_CMD_REGS_OFFSET 0x0800
+#define PDS_CORE_BAR0_DEV_CMD_DATA_REGS_OFFSET 0x0c00
+#define PDS_CORE_BAR0_INTR_STATUS_OFFSET 0x1000
+#define PDS_CORE_BAR0_INTR_CTRL_OFFSET 0x2000
+#define PDS_CORE_DEV_CMD_DONE 0x00000001
+
+#define PDS_CORE_DEVCMD_TIMEOUT 5
+
+#define PDS_CORE_CLIENT_ID 0
+#define PDS_CORE_ASIC_TYPE_CAPRI 0
+
+/*
+ * enum pds_core_cmd_opcode - Device commands
+ */
+enum pds_core_cmd_opcode {
+ /* Core init */
+ PDS_CORE_CMD_NOP = 0,
+ PDS_CORE_CMD_IDENTIFY = 1,
+ PDS_CORE_CMD_RESET = 2,
+ PDS_CORE_CMD_INIT = 3,
+
+ PDS_CORE_CMD_FW_DOWNLOAD = 4,
+ PDS_CORE_CMD_FW_CONTROL = 5,
+
+ /* SR/IOV commands */
+ PDS_CORE_CMD_VF_GETATTR = 60,
+ PDS_CORE_CMD_VF_SETATTR = 61,
+ PDS_CORE_CMD_VF_CTRL = 62,
+
+ /* Add commands before this line */
+ PDS_CORE_CMD_MAX,
+ PDS_CORE_CMD_COUNT
+};
+
+/**
+ * struct pds_core_drv_identity - Driver identity information
+ * @drv_type: Driver type (enum pds_core_driver_type)
+ * @os_dist: OS distribution, numeric format
+ * @os_dist_str: OS distribution, string format
+ * @kernel_ver: Kernel version, numeric format
+ * @kernel_ver_str: Kernel version, string format
+ * @driver_ver_str: Driver version, string format
+ */
+struct pds_core_drv_identity {
+ __le32 drv_type;
+ __le32 os_dist;
+ char os_dist_str[128];
+ __le32 kernel_ver;
+ char kernel_ver_str[32];
+ char driver_ver_str[32];
+};
+
+#define PDS_DEV_TYPE_MAX 16
+/**
+ * struct pds_core_dev_identity - Device identity information
+ * @version: Version of device identify
+ * @type: Identify type (0 for now)
+ * @state: Device state
+ * @rsvd: Word boundary padding
+ * @nlifs: Number of LIFs provisioned
+ * @nintrs: Number of interrupts provisioned
+ * @ndbpgs_per_lif: Number of doorbell pages per LIF
+ * @intr_coal_mult: Interrupt coalescing multiplication factor
+ * Scale user-supplied interrupt coalescing
+ * value in usecs to device units using:
+ * device units = usecs * mult / div
+ * @intr_coal_div: Interrupt coalescing division factor
+ * Scale user-supplied interrupt coalescing
+ * value in usecs to device units using:
+ * device units = usecs * mult / div
+ * @vif_types: How many of each VIF device type is supported
+ */
+struct pds_core_dev_identity {
+ u8 version;
+ u8 type;
+ u8 state;
+ u8 rsvd;
+ __le32 nlifs;
+ __le32 nintrs;
+ __le32 ndbpgs_per_lif;
+ __le32 intr_coal_mult;
+ __le32 intr_coal_div;
+ __le16 vif_types[PDS_DEV_TYPE_MAX];
+};
+
+#define PDS_CORE_IDENTITY_VERSION_1 1
+
+/**
+ * struct pds_core_dev_identify_cmd - Driver/device identify command
+ * @opcode: Opcode PDS_CORE_CMD_IDENTIFY
+ * @ver: Highest version of identify supported by driver
+ *
+ * Expects to find driver identification info (struct pds_core_drv_identity)
+ * in cmd_regs->data. Driver should keep the devcmd interface locked
+ * while preparing the driver info.
+ */
+struct pds_core_dev_identify_cmd {
+ u8 opcode;
+ u8 ver;
+};
+
+/**
+ * struct pds_core_dev_identify_comp - Device identify command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @ver: Version of identify returned by device
+ *
+ * Device identification info (struct pds_core_dev_identity) can be found
+ * in cmd_regs->data. Driver should keep the devcmd interface locked
+ * while reading the results.
+ */
+struct pds_core_dev_identify_comp {
+ u8 status;
+ u8 ver;
+};
+
+/**
+ * struct pds_core_dev_reset_cmd - Device reset command
+ * @opcode: Opcode PDS_CORE_CMD_RESET
+ *
+ * Resets and clears all LIFs, VDevs, and VIFs on the device.
+ */
+struct pds_core_dev_reset_cmd {
+ u8 opcode;
+};
+
+/**
+ * struct pds_core_dev_reset_comp - Reset command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ */
+struct pds_core_dev_reset_comp {
+ u8 status;
+};
+
+/*
+ * struct pds_core_dev_init_data - Pointers and info needed for the Core
+ * initialization PDS_CORE_CMD_INIT command. The in and out structs are
+ * overlays on the pds_core_dev_cmd_regs.data space for passing data down
+ * to the firmware on init, and then returning initialization results.
+ */
+struct pds_core_dev_init_data_in {
+ __le64 adminq_q_base;
+ __le64 adminq_cq_base;
+ __le64 notifyq_cq_base;
+ __le32 flags;
+ __le16 intr_index;
+ u8 adminq_ring_size;
+ u8 notifyq_ring_size;
+};
+
+struct pds_core_dev_init_data_out {
+ __le32 core_hw_index;
+ __le32 adminq_hw_index;
+ __le32 notifyq_hw_index;
+ u8 adminq_hw_type;
+ u8 notifyq_hw_type;
+};
+
+/**
+ * struct pds_core_dev_init_cmd - Core device initialize
+ * @opcode: opcode PDS_CORE_CMD_INIT
+ *
+ * Initializes the core device and sets up the AdminQ and NotifyQ.
+ * Expects to find initialization data (struct pds_core_dev_init_data_in)
+ * in cmd_regs->data. Driver should keep the devcmd interface locked
+ * while preparing the driver info.
+ */
+struct pds_core_dev_init_cmd {
+ u8 opcode;
+};
+
+/**
+ * struct pds_core_dev_init_comp - Core init completion
+ * @status: Status of the command (enum pds_core_status_code)
+ *
+ * Initialization result data (struct pds_core_dev_init_data_in)
+ * is found in cmd_regs->data.
+ */
+struct pds_core_dev_init_comp {
+ u8 status;
+};
+
+/**
+ * struct pds_core_fw_download_cmd - Firmware download command
+ * @opcode: opcode
+ * @rsvd: Word boundary padding
+ * @addr: DMA address of the firmware buffer
+ * @offset: offset of the firmware buffer within the full image
+ * @length: number of valid bytes in the firmware buffer
+ */
+struct pds_core_fw_download_cmd {
+ u8 opcode;
+ u8 rsvd[3];
+ __le32 offset;
+ __le64 addr;
+ __le32 length;
+};
+
+/**
+ * struct pds_core_fw_download_comp - Firmware download completion
+ * @status: Status of the command (enum pds_core_status_code)
+ */
+struct pds_core_fw_download_comp {
+ u8 status;
+};
+
+/**
+ * enum pds_core_fw_control_oper - FW control operations
+ * @PDS_CORE_FW_INSTALL_ASYNC: Install firmware asynchronously
+ * @PDS_CORE_FW_INSTALL_STATUS: Firmware installation status
+ * @PDS_CORE_FW_ACTIVATE_ASYNC: Activate firmware asynchronously
+ * @PDS_CORE_FW_ACTIVATE_STATUS: Firmware activate status
+ * @PDS_CORE_FW_UPDATE_CLEANUP: Cleanup any firmware update leftovers
+ * @PDS_CORE_FW_GET_BOOT: Return current active firmware slot
+ * @PDS_CORE_FW_SET_BOOT: Set active firmware slot for next boot
+ * @PDS_CORE_FW_GET_LIST: Return list of installed firmware images
+ */
+enum pds_core_fw_control_oper {
+ PDS_CORE_FW_INSTALL_ASYNC = 0,
+ PDS_CORE_FW_INSTALL_STATUS = 1,
+ PDS_CORE_FW_ACTIVATE_ASYNC = 2,
+ PDS_CORE_FW_ACTIVATE_STATUS = 3,
+ PDS_CORE_FW_UPDATE_CLEANUP = 4,
+ PDS_CORE_FW_GET_BOOT = 5,
+ PDS_CORE_FW_SET_BOOT = 6,
+ PDS_CORE_FW_GET_LIST = 7,
+};
+
+enum pds_core_fw_slot {
+ PDS_CORE_FW_SLOT_INVALID = 0,
+ PDS_CORE_FW_SLOT_A = 1,
+ PDS_CORE_FW_SLOT_B = 2,
+ PDS_CORE_FW_SLOT_GOLD = 3,
+};
+
+/**
+ * struct pds_core_fw_control_cmd - Firmware control command
+ * @opcode: opcode
+ * @rsvd: Word boundary padding
+ * @oper: firmware control operation (enum pds_core_fw_control_oper)
+ * @slot: slot to operate on (enum pds_core_fw_slot)
+ */
+struct pds_core_fw_control_cmd {
+ u8 opcode;
+ u8 rsvd[3];
+ u8 oper;
+ u8 slot;
+};
+
+/**
+ * struct pds_core_fw_control_comp - Firmware control copletion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @rsvd: Word alignment space
+ * @slot: Slot number (enum pds_core_fw_slot)
+ * @rsvd1: Struct padding
+ * @color: Color bit
+ */
+struct pds_core_fw_control_comp {
+ u8 status;
+ u8 rsvd[3];
+ u8 slot;
+ u8 rsvd1[10];
+ u8 color;
+};
+
+struct pds_core_fw_name_info {
+#define PDS_CORE_FWSLOT_BUFLEN 8
+#define PDS_CORE_FWVERS_BUFLEN 32
+ char slotname[PDS_CORE_FWSLOT_BUFLEN];
+ char fw_version[PDS_CORE_FWVERS_BUFLEN];
+};
+
+struct pds_core_fw_list_info {
+#define PDS_CORE_FWVERS_LIST_LEN 16
+ u8 num_fw_slots;
+ struct pds_core_fw_name_info fw_names[PDS_CORE_FWVERS_LIST_LEN];
+} __packed;
+
+enum pds_core_vf_attr {
+ PDS_CORE_VF_ATTR_SPOOFCHK = 1,
+ PDS_CORE_VF_ATTR_TRUST = 2,
+ PDS_CORE_VF_ATTR_MAC = 3,
+ PDS_CORE_VF_ATTR_LINKSTATE = 4,
+ PDS_CORE_VF_ATTR_VLAN = 5,
+ PDS_CORE_VF_ATTR_RATE = 6,
+ PDS_CORE_VF_ATTR_STATSADDR = 7,
+};
+
+/**
+ * enum pds_core_vf_link_status - Virtual Function link status
+ * @PDS_CORE_VF_LINK_STATUS_AUTO: Use link state of the uplink
+ * @PDS_CORE_VF_LINK_STATUS_UP: Link always up
+ * @PDS_CORE_VF_LINK_STATUS_DOWN: Link always down
+ */
+enum pds_core_vf_link_status {
+ PDS_CORE_VF_LINK_STATUS_AUTO = 0,
+ PDS_CORE_VF_LINK_STATUS_UP = 1,
+ PDS_CORE_VF_LINK_STATUS_DOWN = 2,
+};
+
+/**
+ * struct pds_core_vf_setattr_cmd - Set VF attributes on the NIC
+ * @opcode: Opcode
+ * @attr: Attribute type (enum pds_core_vf_attr)
+ * @vf_index: VF index
+ * @macaddr: mac address
+ * @vlanid: vlan ID
+ * @maxrate: max Tx rate in Mbps
+ * @spoofchk: enable address spoof checking
+ * @trust: enable VF trust
+ * @linkstate: set link up or down
+ * @stats: stats addr struct
+ * @stats.pa: set DMA address for VF stats
+ * @stats.len: length of VF stats space
+ * @pad: force union to specific size
+ */
+struct pds_core_vf_setattr_cmd {
+ u8 opcode;
+ u8 attr;
+ __le16 vf_index;
+ union {
+ u8 macaddr[6];
+ __le16 vlanid;
+ __le32 maxrate;
+ u8 spoofchk;
+ u8 trust;
+ u8 linkstate;
+ struct {
+ __le64 pa;
+ __le32 len;
+ } stats;
+ u8 pad[60];
+ } __packed;
+};
+
+struct pds_core_vf_setattr_comp {
+ u8 status;
+ u8 attr;
+ __le16 vf_index;
+ __le16 comp_index;
+ u8 rsvd[9];
+ u8 color;
+};
+
+/**
+ * struct pds_core_vf_getattr_cmd - Get VF attributes from the NIC
+ * @opcode: Opcode
+ * @attr: Attribute type (enum pds_core_vf_attr)
+ * @vf_index: VF index
+ */
+struct pds_core_vf_getattr_cmd {
+ u8 opcode;
+ u8 attr;
+ __le16 vf_index;
+};
+
+struct pds_core_vf_getattr_comp {
+ u8 status;
+ u8 attr;
+ __le16 vf_index;
+ union {
+ u8 macaddr[6];
+ __le16 vlanid;
+ __le32 maxrate;
+ u8 spoofchk;
+ u8 trust;
+ u8 linkstate;
+ __le64 stats_pa;
+ u8 pad[11];
+ } __packed;
+ u8 color;
+};
+
+enum pds_core_vf_ctrl_opcode {
+ PDS_CORE_VF_CTRL_START_ALL = 0,
+ PDS_CORE_VF_CTRL_START = 1,
+};
+
+/**
+ * struct pds_core_vf_ctrl_cmd - VF control command
+ * @opcode: Opcode for the command
+ * @ctrl_opcode: VF control operation type
+ * @vf_index: VF Index. It is unused if op START_ALL is used.
+ */
+
+struct pds_core_vf_ctrl_cmd {
+ u8 opcode;
+ u8 ctrl_opcode;
+ __le16 vf_index;
+};
+
+/**
+ * struct pds_core_vf_ctrl_comp - VF_CTRL command completion.
+ * @status: Status of the command (enum pds_core_status_code)
+ */
+struct pds_core_vf_ctrl_comp {
+ u8 status;
+};
+
+/*
+ * union pds_core_dev_cmd - Overlay of core device command structures
+ */
+union pds_core_dev_cmd {
+ u8 opcode;
+ u32 words[16];
+
+ struct pds_core_dev_identify_cmd identify;
+ struct pds_core_dev_init_cmd init;
+ struct pds_core_dev_reset_cmd reset;
+ struct pds_core_fw_download_cmd fw_download;
+ struct pds_core_fw_control_cmd fw_control;
+
+ struct pds_core_vf_setattr_cmd vf_setattr;
+ struct pds_core_vf_getattr_cmd vf_getattr;
+ struct pds_core_vf_ctrl_cmd vf_ctrl;
+};
+
+/*
+ * union pds_core_dev_comp - Overlay of core device completion structures
+ */
+union pds_core_dev_comp {
+ u8 status;
+ u8 bytes[16];
+
+ struct pds_core_dev_identify_comp identify;
+ struct pds_core_dev_reset_comp reset;
+ struct pds_core_dev_init_comp init;
+ struct pds_core_fw_download_comp fw_download;
+ struct pds_core_fw_control_comp fw_control;
+
+ struct pds_core_vf_setattr_comp vf_setattr;
+ struct pds_core_vf_getattr_comp vf_getattr;
+ struct pds_core_vf_ctrl_comp vf_ctrl;
+};
+
+/**
+ * struct pds_core_dev_hwstamp_regs - Hardware current timestamp registers
+ * @tick_low: Low 32 bits of hardware timestamp
+ * @tick_high: High 32 bits of hardware timestamp
+ */
+struct pds_core_dev_hwstamp_regs {
+ u32 tick_low;
+ u32 tick_high;
+};
+
+/**
+ * struct pds_core_dev_info_regs - Device info register format (read-only)
+ * @signature: Signature value of 0x44455649 ('DEVI')
+ * @version: Current version of info
+ * @asic_type: Asic type
+ * @asic_rev: Asic revision
+ * @fw_status: Firmware status
+ * bit 0 - 1 = fw running
+ * bit 4-7 - 4 bit generation number, changes on fw restart
+ * @fw_heartbeat: Firmware heartbeat counter
+ * @serial_num: Serial number
+ * @fw_version: Firmware version
+ * @oprom_regs: oprom_regs to store oprom debug enable/disable and bmp
+ * @rsvd_pad1024: Struct padding
+ * @hwstamp: Hardware current timestamp registers
+ * @rsvd_pad2048: Struct padding
+ */
+struct pds_core_dev_info_regs {
+#define PDS_CORE_DEVINFO_FWVERS_BUFLEN 32
+#define PDS_CORE_DEVINFO_SERIAL_BUFLEN 32
+ u32 signature;
+ u8 version;
+ u8 asic_type;
+ u8 asic_rev;
+#define PDS_CORE_FW_STS_F_STOPPED 0x00
+#define PDS_CORE_FW_STS_F_RUNNING 0x01
+#define PDS_CORE_FW_STS_F_GENERATION 0xF0
+ u8 fw_status;
+ __le32 fw_heartbeat;
+ char fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN];
+ char serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN];
+ u8 oprom_regs[32]; /* reserved */
+ u8 rsvd_pad1024[916];
+ struct pds_core_dev_hwstamp_regs hwstamp; /* on 1k boundary */
+ u8 rsvd_pad2048[1016];
+} __packed;
+
+/**
+ * struct pds_core_dev_cmd_regs - Device command register format (read-write)
+ * @doorbell: Device Cmd Doorbell, write-only
+ * Write a 1 to signal device to process cmd
+ * @done: Command completed indicator, poll for completion
+ * bit 0 == 1 when command is complete
+ * @cmd: Opcode-specific command bytes
+ * @comp: Opcode-specific response bytes
+ * @rsvd: Struct padding
+ * @data: Opcode-specific side-data
+ */
+struct pds_core_dev_cmd_regs {
+ u32 doorbell;
+ u32 done;
+ union pds_core_dev_cmd cmd;
+ union pds_core_dev_comp comp;
+ u8 rsvd[48];
+ u32 data[478];
+} __packed;
+
+/**
+ * struct pds_core_dev_regs - Device register format for bar 0 page 0
+ * @info: Device info registers
+ * @devcmd: Device command registers
+ */
+struct pds_core_dev_regs {
+ struct pds_core_dev_info_regs info;
+ struct pds_core_dev_cmd_regs devcmd;
+} __packed;
+
+#ifndef __CHECKER__
+static_assert(sizeof(struct pds_core_drv_identity) <= 1912);
+static_assert(sizeof(struct pds_core_dev_identity) <= 1912);
+static_assert(sizeof(union pds_core_dev_cmd) == 64);
+static_assert(sizeof(union pds_core_dev_comp) == 16);
+static_assert(sizeof(struct pds_core_dev_info_regs) == 2048);
+static_assert(sizeof(struct pds_core_dev_cmd_regs) == 2048);
+static_assert(sizeof(struct pds_core_dev_regs) == 4096);
+#endif /* __CHECKER__ */
+
+#endif /* _PDS_CORE_IF_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 11:46 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 03/14] pds_core: health timer and workqueue Shannon Nelson
` (13 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
The devcmd interface is the basic connection to the device through the
PCI BAR for low level identification and command services. This does
the early device initialization and finds the identity data, and adds
devcmd routines to be used by later driver bits.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/Makefile | 4 +-
drivers/net/ethernet/amd/pds_core/core.c | 36 ++
drivers/net/ethernet/amd/pds_core/core.h | 52 +++
drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++
drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++
drivers/net/ethernet/amd/pds_core/main.c | 33 +-
include/linux/pds/pds_common.h | 61 ++++
include/linux/pds/pds_intr.h | 163 +++++++++
8 files changed, 763 insertions(+), 3 deletions(-)
create mode 100644 drivers/net/ethernet/amd/pds_core/core.c
create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c
create mode 100644 include/linux/pds/pds_intr.h
diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
index de3bf1d1886c..95a6c31e92d2 100644
--- a/drivers/net/ethernet/amd/pds_core/Makefile
+++ b/drivers/net/ethernet/amd/pds_core/Makefile
@@ -3,6 +3,8 @@
obj-$(CONFIG_PDS_CORE) := pds_core.o
-pds_core-y := main.o
+pds_core-y := main.o \
+ dev.o \
+ core.o
pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
new file mode 100644
index 000000000000..80d2ecb045df
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/core.c
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#include "core.h"
+
+int pdsc_setup(struct pdsc *pdsc, bool init)
+{
+ int err = 0;
+
+ if (init)
+ err = pdsc_dev_init(pdsc);
+ else
+ err = pdsc_dev_reinit(pdsc);
+ if (err)
+ return err;
+
+ clear_bit(PDSC_S_FW_DEAD, &pdsc->state);
+ return 0;
+}
+
+void pdsc_teardown(struct pdsc *pdsc, bool removing)
+{
+ pdsc_devcmd_reset(pdsc);
+
+ if (removing) {
+ kfree(pdsc->intr_info);
+ pdsc->intr_info = NULL;
+ }
+
+ if (pdsc->kern_dbpage) {
+ iounmap(pdsc->kern_dbpage);
+ pdsc->kern_dbpage = NULL;
+ }
+
+ set_bit(PDSC_S_FW_DEAD, &pdsc->state);
+}
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index 51c77f145bab..fdf171281c87 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -9,8 +9,13 @@
#include <linux/pds/pds_common.h>
#include <linux/pds/pds_core_if.h>
+#include <linux/pds/pds_intr.h>
#define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver"
+#define PDSC_TEARDOWN_RECOVERY false
+#define PDSC_TEARDOWN_REMOVING true
+#define PDSC_SETUP_RECOVERY false
+#define PDSC_SETUP_INIT true
struct pdsc_dev_bar {
void __iomem *vaddr;
@@ -19,6 +24,22 @@ struct pdsc_dev_bar {
int res_index;
};
+struct pdsc_devinfo {
+ u8 asic_type;
+ u8 asic_rev;
+ char fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN + 1];
+ char serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN + 1];
+};
+
+#define PDSC_INTR_NAME_MAX_SZ 32
+
+struct pdsc_intr_info {
+ char name[PDSC_INTR_NAME_MAX_SZ];
+ unsigned int index;
+ unsigned int vector;
+ void *data;
+};
+
/* No state flags set means we are in a steady running state */
enum pdsc_state_flags {
PDSC_S_FW_DEAD, /* stopped, wait on startup or recovery */
@@ -38,7 +59,19 @@ struct pdsc {
int uid;
unsigned long state;
+ u8 fw_status;
+ u8 fw_generation;
+ unsigned long last_fw_time;
+ u32 last_hb;
+
+ struct pdsc_devinfo dev_info;
+ struct pds_core_dev_identity dev_ident;
+ unsigned int nintrs;
+ struct pdsc_intr_info *intr_info; /* array of nintrs elements */
+ unsigned int devcmd_timeout;
+ struct mutex devcmd_lock; /* lock for dev_cmd operations */
+ struct mutex config_lock; /* lock for configuration operations */
struct pds_core_dev_info_regs __iomem *info_regs;
struct pds_core_dev_cmd_regs __iomem *cmd_regs;
struct pds_core_intr __iomem *intr_ctrl;
@@ -53,11 +86,30 @@ void pdsc_debugfs_create(void);
void pdsc_debugfs_destroy(void);
void pdsc_debugfs_add_dev(struct pdsc *pdsc);
void pdsc_debugfs_del_dev(struct pdsc *pdsc);
+void pdsc_debugfs_add_ident(struct pdsc *pdsc);
+void pdsc_debugfs_add_irqs(struct pdsc *pdsc);
#else
static inline void pdsc_debugfs_create(void) { }
static inline void pdsc_debugfs_destroy(void) { }
static inline void pdsc_debugfs_add_dev(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_del_dev(struct pdsc *pdsc) { }
+static inline void pdsc_debugfs_add_ident(struct pdsc *pdsc) { }
+static inline void pdsc_debugfs_add_irqs(struct pdsc *pdsc) { }
#endif
+int pdsc_err_to_errno(enum pds_core_status_code code);
+bool pdsc_is_fw_running(struct pdsc *pdsc);
+bool pdsc_is_fw_good(struct pdsc *pdsc);
+int pdsc_devcmd(struct pdsc *pdsc, union pds_core_dev_cmd *cmd,
+ union pds_core_dev_comp *comp, int max_seconds);
+int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd,
+ union pds_core_dev_comp *comp, int max_seconds);
+int pdsc_devcmd_init(struct pdsc *pdsc);
+int pdsc_devcmd_reset(struct pdsc *pdsc);
+int pdsc_dev_reinit(struct pdsc *pdsc);
+int pdsc_dev_init(struct pdsc *pdsc);
+
+int pdsc_setup(struct pdsc *pdsc, bool init);
+void pdsc_teardown(struct pdsc *pdsc, bool removing);
+
#endif /* _PDSC_H_ */
diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c
index 9b2385c19c41..2869a2f60678 100644
--- a/drivers/net/ethernet/amd/pds_core/debugfs.c
+++ b/drivers/net/ethernet/amd/pds_core/debugfs.c
@@ -31,4 +31,72 @@ void pdsc_debugfs_del_dev(struct pdsc *pdsc)
debugfs_remove_recursive(pdsc->dentry);
pdsc->dentry = NULL;
}
+
+static int identity_show(struct seq_file *seq, void *v)
+{
+ struct pdsc *pdsc = seq->private;
+ struct pds_core_dev_identity *ident;
+ int vt;
+
+ ident = &pdsc->dev_ident;
+
+ seq_printf(seq, "fw_heartbeat: 0x%x\n",
+ ioread32(&pdsc->info_regs->fw_heartbeat));
+
+ seq_printf(seq, "nlifs: %d\n",
+ le32_to_cpu(ident->nlifs));
+ seq_printf(seq, "nintrs: %d\n",
+ le32_to_cpu(ident->nintrs));
+ seq_printf(seq, "ndbpgs_per_lif: %d\n",
+ le32_to_cpu(ident->ndbpgs_per_lif));
+ seq_printf(seq, "intr_coal_mult: %d\n",
+ le32_to_cpu(ident->intr_coal_mult));
+ seq_printf(seq, "intr_coal_div: %d\n",
+ le32_to_cpu(ident->intr_coal_div));
+
+ seq_puts(seq, "vif_types: ");
+ for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++)
+ seq_printf(seq, "%d ",
+ le16_to_cpu(pdsc->dev_ident.vif_types[vt]));
+ seq_puts(seq, "\n");
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(identity);
+
+void pdsc_debugfs_add_ident(struct pdsc *pdsc)
+{
+ debugfs_create_file("identity", 0400, pdsc->dentry,
+ pdsc, &identity_fops);
+}
+
+static int irqs_show(struct seq_file *seq, void *v)
+{
+ struct pdsc *pdsc = seq->private;
+ struct pdsc_intr_info *intr_info;
+ int i;
+
+ seq_printf(seq, "index vector name (nintrs %d)\n", pdsc->nintrs);
+
+ if (!pdsc->intr_info)
+ return 0;
+
+ for (i = 0; i < pdsc->nintrs; i++) {
+ intr_info = &pdsc->intr_info[i];
+ if (!intr_info->vector)
+ continue;
+
+ seq_printf(seq, "% 3d % 3d %s\n",
+ i, intr_info->vector, intr_info->name);
+ }
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(irqs);
+
+void pdsc_debugfs_add_irqs(struct pdsc *pdsc)
+{
+ debugfs_create_file("irqs", 0400, pdsc->dentry, pdsc, &irqs_fops);
+}
+
#endif /* CONFIG_DEBUG_FS */
diff --git a/drivers/net/ethernet/amd/pds_core/dev.c b/drivers/net/ethernet/amd/pds_core/dev.c
new file mode 100644
index 000000000000..52385a72246d
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/dev.c
@@ -0,0 +1,349 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#include <linux/errno.h>
+#include <linux/pci.h>
+#include <linux/utsname.h>
+
+#include "core.h"
+
+int pdsc_err_to_errno(enum pds_core_status_code code)
+{
+ switch (code) {
+ case PDS_RC_SUCCESS:
+ return 0;
+ case PDS_RC_EVERSION:
+ case PDS_RC_EQTYPE:
+ case PDS_RC_EQID:
+ case PDS_RC_EINVAL:
+ case PDS_RC_ENOSUPP:
+ return -EINVAL;
+ case PDS_RC_EPERM:
+ return -EPERM;
+ case PDS_RC_ENOENT:
+ return -ENOENT;
+ case PDS_RC_EAGAIN:
+ return -EAGAIN;
+ case PDS_RC_ENOMEM:
+ return -ENOMEM;
+ case PDS_RC_EFAULT:
+ return -EFAULT;
+ case PDS_RC_EBUSY:
+ return -EBUSY;
+ case PDS_RC_EEXIST:
+ return -EEXIST;
+ case PDS_RC_EVFID:
+ return -ENODEV;
+ case PDS_RC_ECLIENT:
+ return -ECHILD;
+ case PDS_RC_ENOSPC:
+ return -ENOSPC;
+ case PDS_RC_ERANGE:
+ return -ERANGE;
+ case PDS_RC_BAD_ADDR:
+ return -EFAULT;
+ case PDS_RC_EOPCODE:
+ case PDS_RC_EINTR:
+ case PDS_RC_DEV_CMD:
+ case PDS_RC_ERROR:
+ case PDS_RC_ERDMA:
+ case PDS_RC_EIO:
+ default:
+ return -EIO;
+ }
+}
+
+bool pdsc_is_fw_running(struct pdsc *pdsc)
+{
+ pdsc->fw_status = ioread8(&pdsc->info_regs->fw_status);
+ pdsc->last_fw_time = jiffies;
+ pdsc->last_hb = ioread32(&pdsc->info_regs->fw_heartbeat);
+
+ /* Firmware is useful only if the running bit is set and
+ * fw_status != 0xff (bad PCI read)
+ */
+ return (pdsc->fw_status != 0xff) &&
+ (pdsc->fw_status & PDS_CORE_FW_STS_F_RUNNING);
+}
+
+bool pdsc_is_fw_good(struct pdsc *pdsc)
+{
+ u8 gen = pdsc->fw_status & PDS_CORE_FW_STS_F_GENERATION;
+
+ return pdsc_is_fw_running(pdsc) && gen == pdsc->fw_generation;
+}
+
+static u8 pdsc_devcmd_status(struct pdsc *pdsc)
+{
+ return ioread8(&pdsc->cmd_regs->comp.status);
+}
+
+static bool pdsc_devcmd_done(struct pdsc *pdsc)
+{
+ return ioread32(&pdsc->cmd_regs->done) & PDS_CORE_DEV_CMD_DONE;
+}
+
+static void pdsc_devcmd_dbell(struct pdsc *pdsc)
+{
+ iowrite32(0, &pdsc->cmd_regs->done);
+ iowrite32(1, &pdsc->cmd_regs->doorbell);
+}
+
+static void pdsc_devcmd_clean(struct pdsc *pdsc)
+{
+ iowrite32(0, &pdsc->cmd_regs->doorbell);
+ memset_io(&pdsc->cmd_regs->cmd, 0, sizeof(pdsc->cmd_regs->cmd));
+}
+
+static const char *pdsc_devcmd_str(int opcode)
+{
+ switch (opcode) {
+ case PDS_CORE_CMD_NOP:
+ return "PDS_CORE_CMD_NOP";
+ case PDS_CORE_CMD_IDENTIFY:
+ return "PDS_CORE_CMD_IDENTIFY";
+ case PDS_CORE_CMD_RESET:
+ return "PDS_CORE_CMD_RESET";
+ case PDS_CORE_CMD_INIT:
+ return "PDS_CORE_CMD_INIT";
+ case PDS_CORE_CMD_FW_DOWNLOAD:
+ return "PDS_CORE_CMD_FW_DOWNLOAD";
+ case PDS_CORE_CMD_FW_CONTROL:
+ return "PDS_CORE_CMD_FW_CONTROL";
+ default:
+ return "PDS_CORE_CMD_UNKNOWN";
+ }
+}
+
+static int pdsc_devcmd_wait(struct pdsc *pdsc, int max_seconds)
+{
+ struct device *dev = pdsc->dev;
+ unsigned long start_time;
+ unsigned long max_wait;
+ unsigned long duration;
+ int timeout = 0;
+ int status = 0;
+ int done = 0;
+ int err = 0;
+ int opcode;
+
+ opcode = ioread8(&pdsc->cmd_regs->cmd.opcode);
+
+ start_time = jiffies;
+ max_wait = start_time + (max_seconds * HZ);
+
+ while (!done && !timeout) {
+ done = pdsc_devcmd_done(pdsc);
+ if (done)
+ break;
+
+ timeout = time_after(jiffies, max_wait);
+ if (timeout)
+ break;
+
+ usleep_range(100, 200);
+ }
+ duration = jiffies - start_time;
+
+ if (done && duration > HZ)
+ dev_dbg(dev, "DEVCMD %d %s after %ld secs\n",
+ opcode, pdsc_devcmd_str(opcode), duration / HZ);
+
+ if (!done || timeout) {
+ dev_err(dev, "DEVCMD %d %s timeout, done %d timeout %d max_seconds=%d\n",
+ opcode, pdsc_devcmd_str(opcode), done, timeout,
+ max_seconds);
+ err = -ETIMEDOUT;
+ pdsc_devcmd_clean(pdsc);
+ }
+
+ status = pdsc_devcmd_status(pdsc);
+ err = pdsc_err_to_errno(status);
+ if (status != PDS_RC_SUCCESS && status != PDS_RC_EAGAIN)
+ dev_err(dev, "DEVCMD %d %s failed, status=%d err %d %pe\n",
+ opcode, pdsc_devcmd_str(opcode), status, err,
+ ERR_PTR(err));
+
+ return err;
+}
+
+int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd,
+ union pds_core_dev_comp *comp, int max_seconds)
+{
+ int err;
+
+ memcpy_toio(&pdsc->cmd_regs->cmd, cmd, sizeof(*cmd));
+ pdsc_devcmd_dbell(pdsc);
+ err = pdsc_devcmd_wait(pdsc, max_seconds);
+ memcpy_fromio(comp, &pdsc->cmd_regs->comp, sizeof(*comp));
+
+ return err;
+}
+
+int pdsc_devcmd(struct pdsc *pdsc, union pds_core_dev_cmd *cmd,
+ union pds_core_dev_comp *comp, int max_seconds)
+{
+ int err;
+
+ mutex_lock(&pdsc->devcmd_lock);
+ err = pdsc_devcmd_locked(pdsc, cmd, comp, max_seconds);
+ mutex_unlock(&pdsc->devcmd_lock);
+
+ return err;
+}
+
+int pdsc_devcmd_init(struct pdsc *pdsc)
+{
+ union pds_core_dev_comp comp = { 0 };
+ union pds_core_dev_cmd cmd = {
+ .opcode = PDS_CORE_CMD_INIT,
+ };
+
+ return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+}
+
+int pdsc_devcmd_reset(struct pdsc *pdsc)
+{
+ union pds_core_dev_comp comp = { 0 };
+ union pds_core_dev_cmd cmd = {
+ .reset.opcode = PDS_CORE_CMD_RESET,
+ };
+
+ return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+}
+
+static int pdsc_devcmd_identify_locked(struct pdsc *pdsc)
+{
+ union pds_core_dev_comp comp = { 0 };
+ union pds_core_dev_cmd cmd = {
+ .identify.opcode = PDS_CORE_CMD_IDENTIFY,
+ .identify.ver = PDS_CORE_IDENTITY_VERSION_1,
+ };
+
+ return pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+}
+
+static void pdsc_init_devinfo(struct pdsc *pdsc)
+{
+ pdsc->dev_info.asic_type = ioread8(&pdsc->info_regs->asic_type);
+ pdsc->dev_info.asic_rev = ioread8(&pdsc->info_regs->asic_rev);
+ pdsc->fw_generation = PDS_CORE_FW_STS_F_GENERATION &
+ ioread8(&pdsc->info_regs->fw_status);
+
+ memcpy_fromio(pdsc->dev_info.fw_version,
+ pdsc->info_regs->fw_version,
+ PDS_CORE_DEVINFO_FWVERS_BUFLEN);
+ pdsc->dev_info.fw_version[PDS_CORE_DEVINFO_FWVERS_BUFLEN] = 0;
+
+ memcpy_fromio(pdsc->dev_info.serial_num,
+ pdsc->info_regs->serial_num,
+ PDS_CORE_DEVINFO_SERIAL_BUFLEN);
+ pdsc->dev_info.serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN] = 0;
+
+ dev_dbg(pdsc->dev, "fw_version %s\n", pdsc->dev_info.fw_version);
+}
+
+static int pdsc_identify(struct pdsc *pdsc)
+{
+ struct pds_core_drv_identity drv = { 0 };
+ size_t sz;
+ int err;
+
+ drv.drv_type = cpu_to_le32(PDS_DRIVER_LINUX);
+ snprintf(drv.driver_ver_str, sizeof(drv.driver_ver_str),
+ "%s %s", PDS_CORE_DRV_NAME, utsname()->release);
+
+ /* Next let's get some info about the device
+ * We use the devcmd_lock at this level in order to
+ * get safe access to the cmd_regs->data before anyone
+ * else can mess it up
+ */
+ mutex_lock(&pdsc->devcmd_lock);
+
+ sz = min_t(size_t, sizeof(drv), sizeof(pdsc->cmd_regs->data));
+ memcpy_toio(&pdsc->cmd_regs->data, &drv, sz);
+
+ err = pdsc_devcmd_identify_locked(pdsc);
+ if (!err) {
+ sz = min_t(size_t, sizeof(pdsc->dev_ident),
+ sizeof(pdsc->cmd_regs->data));
+ memcpy_fromio(&pdsc->dev_ident, &pdsc->cmd_regs->data, sz);
+ }
+ mutex_unlock(&pdsc->devcmd_lock);
+
+ if (err) {
+ dev_err(pdsc->dev, "Cannot identify device: %pe\n",
+ ERR_PTR(err));
+ return err;
+ }
+
+ if (isprint(pdsc->dev_info.fw_version[0]) &&
+ isascii(pdsc->dev_info.fw_version[0]))
+ dev_info(pdsc->dev, "FW: %.*s\n",
+ (int)(sizeof(pdsc->dev_info.fw_version) - 1),
+ pdsc->dev_info.fw_version);
+ else
+ dev_info(pdsc->dev, "FW: (invalid string) 0x%02x 0x%02x 0x%02x 0x%02x ...\n",
+ (u8)pdsc->dev_info.fw_version[0],
+ (u8)pdsc->dev_info.fw_version[1],
+ (u8)pdsc->dev_info.fw_version[2],
+ (u8)pdsc->dev_info.fw_version[3]);
+
+ return 0;
+}
+
+int pdsc_dev_reinit(struct pdsc *pdsc)
+{
+ pdsc_init_devinfo(pdsc);
+
+ return pdsc_identify(pdsc);
+}
+
+int pdsc_dev_init(struct pdsc *pdsc)
+{
+ unsigned int nintrs;
+ int err;
+
+ /* Initial init and reset of device */
+ pdsc_init_devinfo(pdsc);
+ pdsc->devcmd_timeout = PDS_CORE_DEVCMD_TIMEOUT;
+
+ err = pdsc_devcmd_reset(pdsc);
+ if (err)
+ return err;
+
+ err = pdsc_identify(pdsc);
+ if (err)
+ return err;
+
+ pdsc_debugfs_add_ident(pdsc);
+
+ /* Now we can reserve interrupts */
+ nintrs = le32_to_cpu(pdsc->dev_ident.nintrs);
+ nintrs = min_t(unsigned int, num_online_cpus(), nintrs);
+
+ /* Get intr_info struct array for tracking */
+ pdsc->intr_info = kcalloc(nintrs, sizeof(*pdsc->intr_info), GFP_KERNEL);
+ if (!pdsc->intr_info) {
+ err = -ENOSPC;
+ goto err_out;
+ }
+
+ err = pci_alloc_irq_vectors(pdsc->pdev, nintrs, nintrs, PCI_IRQ_MSIX);
+ if (err != nintrs) {
+ dev_err(pdsc->dev, "Can't get %d intrs from OS: %pe\n",
+ nintrs, ERR_PTR(err));
+ err = -ENOSPC;
+ goto err_out;
+ }
+ pdsc->nintrs = nintrs;
+ pdsc_debugfs_add_irqs(pdsc);
+
+ return 0;
+
+err_out:
+ kfree(pdsc->intr_info);
+ pdsc->intr_info = NULL;
+
+ return err;
+}
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index 1c2f3fbaa27c..a762e9a27850 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -148,6 +148,18 @@ static int pdsc_init_pf(struct pdsc *pdsc)
if (err)
goto err_out_release_regions;
+ mutex_init(&pdsc->devcmd_lock);
+ mutex_init(&pdsc->config_lock);
+
+ mutex_lock(&pdsc->config_lock);
+ set_bit(PDSC_S_FW_DEAD, &pdsc->state);
+
+ err = pdsc_setup(pdsc, PDSC_SETUP_INIT);
+ if (err)
+ goto err_out_unmap_bars;
+
+ mutex_unlock(&pdsc->config_lock);
+
dl = priv_to_devlink(pdsc);
devl_lock(dl);
devl_register(dl);
@@ -155,6 +167,12 @@ static int pdsc_init_pf(struct pdsc *pdsc)
return 0;
+err_out_unmap_bars:
+ mutex_unlock(&pdsc->config_lock);
+ mutex_destroy(&pdsc->config_lock);
+ mutex_destroy(&pdsc->devcmd_lock);
+ pci_free_irq_vectors(pdsc->pdev);
+ pdsc_unmap_bars(pdsc);
err_out_release_regions:
pci_release_regions(pdsc->pdev);
@@ -251,8 +269,19 @@ static void pdsc_remove(struct pci_dev *pdev)
devl_unregister(dl);
devl_unlock(dl);
- pdsc_unmap_bars(pdsc);
- pci_release_regions(pdev);
+ if (!pdev->is_virtfn) {
+ mutex_lock(&pdsc->config_lock);
+ set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
+
+ pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
+ mutex_unlock(&pdsc->config_lock);
+ mutex_destroy(&pdsc->config_lock);
+ mutex_destroy(&pdsc->devcmd_lock);
+
+ pci_free_irq_vectors(pdev);
+ pdsc_unmap_bars(pdsc);
+ pci_release_regions(pdev);
+ }
pci_clear_master(pdev);
pci_disable_device(pdev);
diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
index bd041a5170a6..e45a69178f74 100644
--- a/include/linux/pds/pds_common.h
+++ b/include/linux/pds/pds_common.h
@@ -11,4 +11,65 @@
#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1)
#define PDS_PAGE_SIZE 4096
+/*
+ * enum pds_core_status_code - Device command return codes
+ */
+enum pds_core_status_code {
+ PDS_RC_SUCCESS = 0, /* Success */
+ PDS_RC_EVERSION = 1, /* Incorrect version for request */
+ PDS_RC_EOPCODE = 2, /* Invalid cmd opcode */
+ PDS_RC_EIO = 3, /* I/O error */
+ PDS_RC_EPERM = 4, /* Permission denied */
+ PDS_RC_EQID = 5, /* Bad qid */
+ PDS_RC_EQTYPE = 6, /* Bad qtype */
+ PDS_RC_ENOENT = 7, /* No such element */
+ PDS_RC_EINTR = 8, /* operation interrupted */
+ PDS_RC_EAGAIN = 9, /* Try again */
+ PDS_RC_ENOMEM = 10, /* Out of memory */
+ PDS_RC_EFAULT = 11, /* Bad address */
+ PDS_RC_EBUSY = 12, /* Device or resource busy */
+ PDS_RC_EEXIST = 13, /* object already exists */
+ PDS_RC_EINVAL = 14, /* Invalid argument */
+ PDS_RC_ENOSPC = 15, /* No space left or alloc failure */
+ PDS_RC_ERANGE = 16, /* Parameter out of range */
+ PDS_RC_BAD_ADDR = 17, /* Descriptor contains a bad ptr */
+ PDS_RC_DEV_CMD = 18, /* Device cmd attempted on AdminQ */
+ PDS_RC_ENOSUPP = 19, /* Operation not supported */
+ PDS_RC_ERROR = 29, /* Generic error */
+ PDS_RC_ERDMA = 30, /* Generic RDMA error */
+ PDS_RC_EVFID = 31, /* VF ID does not exist */
+ PDS_RC_BAD_FW = 32, /* FW file is invalid or corrupted */
+ PDS_RC_ECLIENT = 33, /* No such client id */
+};
+
+enum pds_core_driver_type {
+ PDS_DRIVER_LINUX = 1,
+ PDS_DRIVER_WIN = 2,
+ PDS_DRIVER_DPDK = 3,
+ PDS_DRIVER_FREEBSD = 4,
+ PDS_DRIVER_IPXE = 5,
+ PDS_DRIVER_ESXI = 6,
+};
+
+#define PDS_CORE_IFNAMSIZ 16
+
+/**
+ * enum pds_core_logical_qtype - Logical Queue Types
+ * @PDS_CORE_QTYPE_ADMINQ: Administrative Queue
+ * @PDS_CORE_QTYPE_NOTIFYQ: Notify Queue
+ * @PDS_CORE_QTYPE_RXQ: Receive Queue
+ * @PDS_CORE_QTYPE_TXQ: Transmit Queue
+ * @PDS_CORE_QTYPE_EQ: Event Queue
+ * @PDS_CORE_QTYPE_MAX: Max queue type supported
+ */
+enum pds_core_logical_qtype {
+ PDS_CORE_QTYPE_ADMINQ = 0,
+ PDS_CORE_QTYPE_NOTIFYQ = 1,
+ PDS_CORE_QTYPE_RXQ = 2,
+ PDS_CORE_QTYPE_TXQ = 3,
+ PDS_CORE_QTYPE_EQ = 4,
+
+ PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */
+};
+
#endif /* _PDS_COMMON_H_ */
diff --git a/include/linux/pds/pds_intr.h b/include/linux/pds/pds_intr.h
new file mode 100644
index 000000000000..56277c37248c
--- /dev/null
+++ b/include/linux/pds/pds_intr.h
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc. */
+
+#ifndef _PDS_INTR_H_
+#define _PDS_INTR_H_
+
+/*
+ * Interrupt control register
+ * @coal_init: Coalescing timer initial value, in
+ * device units. Use @identity->intr_coal_mult
+ * and @identity->intr_coal_div to convert from
+ * usecs to device units:
+ *
+ * coal_init = coal_usecs * coal_mutl / coal_div
+ *
+ * When an interrupt is sent the interrupt
+ * coalescing timer current value
+ * (@coalescing_curr) is initialized with this
+ * value and begins counting down. No more
+ * interrupts are sent until the coalescing
+ * timer reaches 0. When @coalescing_init=0
+ * interrupt coalescing is effectively disabled
+ * and every interrupt assert results in an
+ * interrupt. Reset value: 0
+ * @mask: Interrupt mask. When @mask=1 the interrupt
+ * resource will not send an interrupt. When
+ * @mask=0 the interrupt resource will send an
+ * interrupt if an interrupt event is pending
+ * or on the next interrupt assertion event.
+ * Reset value: 1
+ * @credits: Interrupt credits. This register indicates
+ * how many interrupt events the hardware has
+ * sent. When written by software this
+ * register atomically decrements @int_credits
+ * by the value written. When @int_credits
+ * becomes 0 then the "pending interrupt" bit
+ * in the Interrupt Status register is cleared
+ * by the hardware and any pending but unsent
+ * interrupts are cleared.
+ * !!!IMPORTANT!!! This is a signed register.
+ * @flags: Interrupt control flags
+ * @unmask -- When this bit is written with a 1
+ * the interrupt resource will set mask=0.
+ * @coal_timer_reset -- When this
+ * bit is written with a 1 the
+ * @coalescing_curr will be reloaded with
+ * @coalescing_init to reset the coalescing
+ * timer.
+ * @mask_on_assert: Automatically mask on assertion. When
+ * @mask_on_assert=1 the interrupt resource
+ * will set @mask=1 whenever an interrupt is
+ * sent. When using interrupts in Legacy
+ * Interrupt mode the driver must select
+ * @mask_on_assert=0 for proper interrupt
+ * operation.
+ * @coalescing_curr: Coalescing timer current value, in
+ * microseconds. When this value reaches 0
+ * the interrupt resource is again eligible to
+ * send an interrupt. If an interrupt event
+ * is already pending when @coalescing_curr
+ * reaches 0 the pending interrupt will be
+ * sent, otherwise an interrupt will be sent
+ * on the next interrupt assertion event.
+ */
+struct pds_core_intr {
+ u32 coal_init;
+ u32 mask;
+ u16 credits;
+ u16 flags;
+#define PDS_CORE_INTR_F_UNMASK 0x0001
+#define PDS_CORE_INTR_F_TIMER_RESET 0x0002
+ u32 mask_on_assert;
+ u32 coalescing_curr;
+ u32 rsvd6[3];
+};
+
+#ifndef __CHECKER__
+static_assert(sizeof(struct pds_core_intr) == 32);
+#endif /* __CHECKER__ */
+
+#define PDS_CORE_INTR_CTRL_REGS_MAX 2048
+#define PDS_CORE_INTR_CTRL_COAL_MAX 0x3F
+#define PDS_CORE_INTR_INDEX_NOT_ASSIGNED -1
+
+struct pds_core_intr_status {
+ u32 status[2];
+};
+
+/**
+ * enum pds_core_intr_mask_vals - valid values for mask and mask_assert.
+ * @PDS_CORE_INTR_MASK_CLEAR: unmask interrupt.
+ * @PDS_CORE_INTR_MASK_SET: mask interrupt.
+ */
+enum pds_core_intr_mask_vals {
+ PDS_CORE_INTR_MASK_CLEAR = 0,
+ PDS_CORE_INTR_MASK_SET = 1,
+};
+
+/**
+ * enum pds_core_intr_credits_bits - Bitwise composition of credits values.
+ * @PDS_CORE_INTR_CRED_COUNT: bit mask of credit count, no shift needed.
+ * @PDS_CORE_INTR_CRED_COUNT_SIGNED: bit mask of credit count, including sign bit.
+ * @PDS_CORE_INTR_CRED_UNMASK: unmask the interrupt.
+ * @PDS_CORE_INTR_CRED_RESET_COALESCE: reset the coalesce timer.
+ * @PDS_CORE_INTR_CRED_REARM: unmask the and reset the timer.
+ */
+enum pds_core_intr_credits_bits {
+ PDS_CORE_INTR_CRED_COUNT = 0x7fffu,
+ PDS_CORE_INTR_CRED_COUNT_SIGNED = 0xffffu,
+ PDS_CORE_INTR_CRED_UNMASK = 0x10000u,
+ PDS_CORE_INTR_CRED_RESET_COALESCE = 0x20000u,
+ PDS_CORE_INTR_CRED_REARM = (PDS_CORE_INTR_CRED_UNMASK |
+ PDS_CORE_INTR_CRED_RESET_COALESCE),
+};
+
+static inline void
+pds_core_intr_coal_init(struct pds_core_intr __iomem *intr_ctrl, u32 coal)
+{
+ iowrite32(coal, &intr_ctrl->coal_init);
+}
+
+static inline void
+pds_core_intr_mask(struct pds_core_intr __iomem *intr_ctrl, u32 mask)
+{
+ iowrite32(mask, &intr_ctrl->mask);
+}
+
+static inline void
+pds_core_intr_credits(struct pds_core_intr __iomem *intr_ctrl,
+ u32 cred, u32 flags)
+{
+ if (WARN_ON_ONCE(cred > PDS_CORE_INTR_CRED_COUNT)) {
+ cred = ioread32(&intr_ctrl->credits);
+ cred &= PDS_CORE_INTR_CRED_COUNT_SIGNED;
+ }
+
+ iowrite32(cred | flags, &intr_ctrl->credits);
+}
+
+static inline void
+pds_core_intr_clean_flags(struct pds_core_intr __iomem *intr_ctrl, u32 flags)
+{
+ u32 cred;
+
+ cred = ioread32(&intr_ctrl->credits);
+ cred &= PDS_CORE_INTR_CRED_COUNT_SIGNED;
+ cred |= flags;
+ iowrite32(cred, &intr_ctrl->credits);
+}
+
+static inline void
+pds_core_intr_clean(struct pds_core_intr __iomem *intr_ctrl)
+{
+ pds_core_intr_clean_flags(intr_ctrl, PDS_CORE_INTR_CRED_RESET_COALESCE);
+}
+
+static inline void
+pds_core_intr_mask_assert(struct pds_core_intr __iomem *intr_ctrl, u32 mask)
+{
+ iowrite32(mask, &intr_ctrl->mask_on_assert);
+}
+
+#endif /* _PDS_INTR_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 03/14] pds_core: health timer and workqueue
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 11:51 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 04/14] pds_core: add devlink health facilities Shannon Nelson
` (12 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Add in the periodic health check and the related workqueue,
as well as the handlers for when a FW reset is seen.
The firmware is polled every 5 seconds to be sure that it is
still alive and that the FW generation didn't change.
The alive check looks to see that the PCI bus is still readable
and the fw_status still has the RUNNING bit on. If not alive,
the driver stops activity and tears things down. When the FW
recovers and the alive check again succeeds, the driver sets
back up for activity.
The generation check looks at the fw_generation to see if it
has changed, which can happen if the FW crashed and recovered
or was updated in between health checks. If changed, the
driver counts that as though the alive test failed and forces
the fw_down/fw_up cycle.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/core.c | 61 ++++++++++++++++++++++++
drivers/net/ethernet/amd/pds_core/core.h | 8 ++++
drivers/net/ethernet/amd/pds_core/dev.c | 3 ++
drivers/net/ethernet/amd/pds_core/main.c | 37 ++++++++++++++
4 files changed, 109 insertions(+)
diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
index 80d2ecb045df..701d27471858 100644
--- a/drivers/net/ethernet/amd/pds_core/core.c
+++ b/drivers/net/ethernet/amd/pds_core/core.c
@@ -34,3 +34,64 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)
set_bit(PDSC_S_FW_DEAD, &pdsc->state);
}
+
+static void pdsc_fw_down(struct pdsc *pdsc)
+{
+ if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
+ dev_err(pdsc->dev, "%s: already happening\n", __func__);
+ return;
+ }
+
+ pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
+}
+
+static void pdsc_fw_up(struct pdsc *pdsc)
+{
+ int err;
+
+ if (!test_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
+ dev_err(pdsc->dev, "%s: fw not dead\n", __func__);
+ return;
+ }
+
+ err = pdsc_setup(pdsc, PDSC_SETUP_RECOVERY);
+ if (err)
+ goto err_out;
+
+ return;
+
+err_out:
+ pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
+}
+
+void pdsc_health_thread(struct work_struct *work)
+{
+ struct pdsc *pdsc = container_of(work, struct pdsc, health_work);
+ unsigned long mask;
+ bool healthy;
+
+ mutex_lock(&pdsc->config_lock);
+
+ /* Don't do a check when in a transition state */
+ mask = BIT_ULL(PDSC_S_INITING_DRIVER) |
+ BIT_ULL(PDSC_S_STOPPING_DRIVER);
+ if (pdsc->state & mask)
+ goto out_unlock;
+
+ healthy = pdsc_is_fw_good(pdsc);
+ dev_dbg(pdsc->dev, "%s: health %d fw_status %#02x fw_heartbeat %d\n",
+ __func__, healthy, pdsc->fw_status, pdsc->last_hb);
+
+ if (test_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
+ if (healthy)
+ pdsc_fw_up(pdsc);
+ } else {
+ if (!healthy)
+ pdsc_fw_down(pdsc);
+ }
+
+ pdsc->fw_generation = pdsc->fw_status & PDS_CORE_FW_STS_F_GENERATION;
+
+out_unlock:
+ mutex_unlock(&pdsc->config_lock);
+}
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index fdf171281c87..ffc9e01dec31 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -12,6 +12,8 @@
#include <linux/pds/pds_intr.h>
#define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver"
+
+#define PDSC_WATCHDOG_SECS 5
#define PDSC_TEARDOWN_RECOVERY false
#define PDSC_TEARDOWN_REMOVING true
#define PDSC_SETUP_RECOVERY false
@@ -63,12 +65,17 @@ struct pdsc {
u8 fw_generation;
unsigned long last_fw_time;
u32 last_hb;
+ struct timer_list wdtimer;
+ unsigned int wdtimer_period;
+ struct work_struct health_work;
struct pdsc_devinfo dev_info;
struct pds_core_dev_identity dev_ident;
unsigned int nintrs;
struct pdsc_intr_info *intr_info; /* array of nintrs elements */
+ struct workqueue_struct *wq;
+
unsigned int devcmd_timeout;
struct mutex devcmd_lock; /* lock for dev_cmd operations */
struct mutex config_lock; /* lock for configuration operations */
@@ -111,5 +118,6 @@ int pdsc_dev_init(struct pdsc *pdsc);
int pdsc_setup(struct pdsc *pdsc, bool init);
void pdsc_teardown(struct pdsc *pdsc, bool removing);
+void pdsc_health_thread(struct work_struct *work);
#endif /* _PDSC_H_ */
diff --git a/drivers/net/ethernet/amd/pds_core/dev.c b/drivers/net/ethernet/amd/pds_core/dev.c
index 52385a72246d..9fdec8adab2b 100644
--- a/drivers/net/ethernet/amd/pds_core/dev.c
+++ b/drivers/net/ethernet/amd/pds_core/dev.c
@@ -177,6 +177,9 @@ int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd,
err = pdsc_devcmd_wait(pdsc, max_seconds);
memcpy_fromio(comp, &pdsc->cmd_regs->comp, sizeof(*comp));
+ if (err == -ENXIO || err == -ETIMEDOUT)
+ queue_work(pdsc->wq, &pdsc->health_work);
+
return err;
}
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index a762e9a27850..5032fc199603 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -20,6 +20,17 @@ static const struct pci_device_id pdsc_id_table[] = {
};
MODULE_DEVICE_TABLE(pci, pdsc_id_table);
+static void pdsc_wdtimer_cb(struct timer_list *t)
+{
+ struct pdsc *pdsc = from_timer(pdsc, t, wdtimer);
+
+ dev_dbg(pdsc->dev, "%s: jiffies %ld\n", __func__, jiffies);
+ mod_timer(&pdsc->wdtimer,
+ round_jiffies(jiffies + pdsc->wdtimer_period));
+
+ queue_work(pdsc->wq, &pdsc->health_work);
+}
+
static void pdsc_unmap_bars(struct pdsc *pdsc)
{
struct pdsc_dev_bar *bars = pdsc->bars;
@@ -130,8 +141,11 @@ static int pdsc_init_vf(struct pdsc *vf)
return -1;
}
+#define PDSC_WQ_NAME_LEN 24
+
static int pdsc_init_pf(struct pdsc *pdsc)
{
+ char wq_name[PDSC_WQ_NAME_LEN];
struct devlink *dl;
int err;
@@ -148,6 +162,13 @@ static int pdsc_init_pf(struct pdsc *pdsc)
if (err)
goto err_out_release_regions;
+ /* General workqueue and timer, but don't start timer yet */
+ snprintf(wq_name, sizeof(wq_name), "%s.%d", PDS_CORE_DRV_NAME, pdsc->uid);
+ pdsc->wq = create_singlethread_workqueue(wq_name);
+ INIT_WORK(&pdsc->health_work, pdsc_health_thread);
+ timer_setup(&pdsc->wdtimer, pdsc_wdtimer_cb, 0);
+ pdsc->wdtimer_period = PDSC_WATCHDOG_SECS * HZ;
+
mutex_init(&pdsc->devcmd_lock);
mutex_init(&pdsc->config_lock);
@@ -165,10 +186,19 @@ static int pdsc_init_pf(struct pdsc *pdsc)
devl_register(dl);
devl_unlock(dl);
+ /* Lastly, start the health check timer */
+ mod_timer(&pdsc->wdtimer, round_jiffies(jiffies + pdsc->wdtimer_period));
+
return 0;
err_out_unmap_bars:
mutex_unlock(&pdsc->config_lock);
+ del_timer_sync(&pdsc->wdtimer);
+ if (pdsc->wq) {
+ flush_workqueue(pdsc->wq);
+ destroy_workqueue(pdsc->wq);
+ pdsc->wq = NULL;
+ }
mutex_destroy(&pdsc->config_lock);
mutex_destroy(&pdsc->devcmd_lock);
pci_free_irq_vectors(pdsc->pdev);
@@ -270,6 +300,13 @@ static void pdsc_remove(struct pci_dev *pdev)
devl_unlock(dl);
if (!pdev->is_virtfn) {
+ del_timer_sync(&pdsc->wdtimer);
+ if (pdsc->wq) {
+ flush_workqueue(pdsc->wq);
+ destroy_workqueue(pdsc->wq);
+ pdsc->wq = NULL;
+ }
+
mutex_lock(&pdsc->config_lock);
set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 04/14] pds_core: add devlink health facilities
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (2 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 03/14] pds_core: health timer and workqueue Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 11:54 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 05/14] pds_core: set up device and adminq Shannon Nelson
` (11 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Add devlink health reporting on top of our fw watchdog.
Example:
# devlink health show pci/0000:2b:00.0 reporter fw
pci/0000:2b:00.0:
reporter fw
state healthy error 0 recover 0
# devlink health diagnose pci/0000:2b:00.0 reporter fw
Status: healthy State: 1 Generation: 0 Recoveries: 0
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
.../device_drivers/ethernet/amd/pds_core.rst | 12 ++++++
drivers/net/ethernet/amd/pds_core/Makefile | 1 +
drivers/net/ethernet/amd/pds_core/core.c | 6 +++
drivers/net/ethernet/amd/pds_core/core.h | 6 +++
drivers/net/ethernet/amd/pds_core/devlink.c | 37 +++++++++++++++++++
drivers/net/ethernet/amd/pds_core/main.c | 22 +++++++++++
6 files changed, 84 insertions(+)
create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
index 58a28b255d37..90b473559bac 100644
--- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
+++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
@@ -26,6 +26,18 @@ messages such as these::
pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
pds_core 0000:b6:00.0: FW: 1.60.0-73
+Health Reporters
+================
+
+The driver supports a devlink health reporter for FW status::
+
+ # devlink health show pci/0000:2b:00.0 reporter fw
+ pci/0000:2b:00.0:
+ reporter fw
+ state healthy error 0 recover 0
+ # devlink health diagnose pci/0000:2b:00.0 reporter fw
+ Status: healthy State: 1 Generation: 0 Recoveries: 0
+
Support
=======
diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
index 95a6c31e92d2..eaca8557ba66 100644
--- a/drivers/net/ethernet/amd/pds_core/Makefile
+++ b/drivers/net/ethernet/amd/pds_core/Makefile
@@ -4,6 +4,7 @@
obj-$(CONFIG_PDS_CORE) := pds_core.o
pds_core-y := main.o \
+ devlink.o \
dev.o \
core.o
diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
index 701d27471858..52236af6b0e0 100644
--- a/drivers/net/ethernet/amd/pds_core/core.c
+++ b/drivers/net/ethernet/amd/pds_core/core.c
@@ -42,6 +42,8 @@ static void pdsc_fw_down(struct pdsc *pdsc)
return;
}
+ devlink_health_report(pdsc->fw_reporter, "FW down reported", pdsc);
+
pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
}
@@ -58,6 +60,10 @@ static void pdsc_fw_up(struct pdsc *pdsc)
if (err)
goto err_out;
+ pdsc->fw_recoveries++;
+ devlink_health_reporter_state_update(pdsc->fw_reporter,
+ DEVLINK_HEALTH_REPORTER_STATE_HEALTHY);
+
return;
err_out:
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index ffc9e01dec31..3758071c94da 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -68,6 +68,8 @@ struct pdsc {
struct timer_list wdtimer;
unsigned int wdtimer_period;
struct work_struct health_work;
+ struct devlink_health_reporter *fw_reporter;
+ u32 fw_recoveries;
struct pdsc_devinfo dev_info;
struct pds_core_dev_identity dev_ident;
@@ -88,6 +90,10 @@ struct pdsc {
u64 __iomem *kern_dbpage;
};
+int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
+ struct devlink_fmsg *fmsg,
+ struct netlink_ext_ack *extack);
+
#ifdef CONFIG_DEBUG_FS
void pdsc_debugfs_create(void);
void pdsc_debugfs_destroy(void);
diff --git a/drivers/net/ethernet/amd/pds_core/devlink.c b/drivers/net/ethernet/amd/pds_core/devlink.c
new file mode 100644
index 000000000000..717fcbf91aee
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/devlink.c
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#include "core.h"
+
+int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
+ struct devlink_fmsg *fmsg,
+ struct netlink_ext_ack *extack)
+{
+ struct pdsc *pdsc = devlink_health_reporter_priv(reporter);
+ int err = 0;
+
+ if (test_bit(PDSC_S_FW_DEAD, &pdsc->state))
+ err = devlink_fmsg_string_pair_put(fmsg, "Status", "dead");
+ else if (!pdsc_is_fw_good(pdsc))
+ err = devlink_fmsg_string_pair_put(fmsg, "Status", "unhealthy");
+ else
+ err = devlink_fmsg_string_pair_put(fmsg, "Status", "healthy");
+ if (err)
+ return err;
+
+ err = devlink_fmsg_u32_pair_put(fmsg, "State",
+ pdsc->fw_status &
+ ~PDS_CORE_FW_STS_F_GENERATION);
+ if (err)
+ return err;
+ err = devlink_fmsg_u32_pair_put(fmsg, "Generation",
+ pdsc->fw_generation >> 4);
+ if (err)
+ return err;
+ err = devlink_fmsg_u32_pair_put(fmsg, "Recoveries",
+ pdsc->fw_recoveries);
+ if (err)
+ return err;
+
+ return 0;
+}
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index 5032fc199603..82ce180d7b48 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -141,10 +141,16 @@ static int pdsc_init_vf(struct pdsc *vf)
return -1;
}
+static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {
+ .name = "fw",
+ .diagnose = pdsc_fw_reporter_diagnose,
+};
+
#define PDSC_WQ_NAME_LEN 24
static int pdsc_init_pf(struct pdsc *pdsc)
{
+ struct devlink_health_reporter *hr;
char wq_name[PDSC_WQ_NAME_LEN];
struct devlink *dl;
int err;
@@ -183,6 +189,16 @@ static int pdsc_init_pf(struct pdsc *pdsc)
dl = priv_to_devlink(pdsc);
devl_lock(dl);
+
+ hr = devl_health_reporter_create(dl, &pdsc_fw_reporter_ops, 0, pdsc);
+ if (IS_ERR(hr)) {
+ dev_warn(pdsc->dev, "Failed to create fw reporter: %pe\n", hr);
+ err = PTR_ERR(hr);
+ devl_unlock(dl);
+ goto err_out_teardown;
+ }
+ pdsc->fw_reporter = hr;
+
devl_register(dl);
devl_unlock(dl);
@@ -191,6 +207,8 @@ static int pdsc_init_pf(struct pdsc *pdsc)
return 0;
+err_out_teardown:
+ pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
err_out_unmap_bars:
mutex_unlock(&pdsc->config_lock);
del_timer_sync(&pdsc->wdtimer);
@@ -297,6 +315,10 @@ static void pdsc_remove(struct pci_dev *pdev)
dl = priv_to_devlink(pdsc);
devl_lock(dl);
devl_unregister(dl);
+ if (pdsc->fw_reporter) {
+ devl_health_reporter_destroy(pdsc->fw_reporter);
+ pdsc->fw_reporter = NULL;
+ }
devl_unlock(dl);
if (!pdev->is_virtfn) {
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 05/14] pds_core: set up device and adminq
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (3 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 04/14] pds_core: add devlink health facilities Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 12:03 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 06/14] pds_core: Add adminq processing and commands Shannon Nelson
` (10 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Set up the basic adminq and notifyq queue structures. These are
used mostly by the client drivers for feature configuration.
These are essentially the same adminq and notifyq as in the
ionic driver.
Part of this includes querying for device identity and FW
information, so we can make that available to devlink dev info.
$ devlink dev info pci/0000:b5:00.0
pci/0000:b5:00.0:
driver pds_core
serial_number FLM18420073
versions:
fixed:
asic.id 0x0
asic.rev 0x0
running:
fw 1.51.0-73
stored:
fw.goldfw 1.15.9-C-22
fw.mainfwa 1.60.0-73
fw.mainfwb 1.60.0-57
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
.../device_drivers/ethernet/amd/pds_core.rst | 47 ++
drivers/net/ethernet/amd/pds_core/core.c | 450 ++++++++++++-
drivers/net/ethernet/amd/pds_core/core.h | 154 +++++
drivers/net/ethernet/amd/pds_core/debugfs.c | 76 +++
drivers/net/ethernet/amd/pds_core/devlink.c | 61 ++
drivers/net/ethernet/amd/pds_core/main.c | 17 +-
include/linux/pds/pds_adminq.h | 637 ++++++++++++++++++
7 files changed, 1438 insertions(+), 4 deletions(-)
create mode 100644 include/linux/pds/pds_adminq.h
diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
index 90b473559bac..265d948a8c02 100644
--- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
+++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
@@ -26,6 +26,53 @@ messages such as these::
pds_core 0000:b6:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
pds_core 0000:b6:00.0: FW: 1.60.0-73
+Driver and firmware version information can be gathered with devlink::
+
+ $ devlink dev info pci/0000:b5:00.0
+ pci/0000:b5:00.0:
+ driver pds_core
+ serial_number FLM18420073
+ versions:
+ fixed:
+ asic.id 0x0
+ asic.rev 0x0
+ running:
+ fw 1.51.0-73
+ stored:
+ fw.goldfw 1.15.9-C-22
+ fw.mainfwa 1.60.0-73
+ fw.mainfwb 1.60.0-57
+
+Info versions
+=============
+
+The ``pds_core`` driver reports the following versions
+
+.. list-table:: devlink info versions implemented
+ :widths: 5 5 90
+
+ * - Name
+ - Type
+ - Description
+ * - ``fw``
+ - running
+ - Version of firmware running on the device
+ * - ``fw.goldfw``
+ - stored
+ - Version of firmware stored in the goldfw slot
+ * - ``fw.mainfwa``
+ - stored
+ - Version of firmware stored in the mainfwa slot
+ * - ``fw.mainfwb``
+ - stored
+ - Version of firmware stored in the mainfwb slot
+ * - ``asic.id``
+ - fixed
+ - The ASIC type for this device
+ * - ``asic.rev``
+ - fixed
+ - The revision of the ASIC for this device
+
Health Reporters
================
diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
index 52236af6b0e0..ec78510fcce7 100644
--- a/drivers/net/ethernet/amd/pds_core/core.c
+++ b/drivers/net/ethernet/amd/pds_core/core.c
@@ -1,10 +1,376 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+#include <linux/pci.h>
+#include <linux/vmalloc.h>
+
#include "core.h"
+void pdsc_work_thread(struct work_struct *work)
+{
+ /* stub */
+}
+
+irqreturn_t pdsc_adminq_isr(int irq, void *data)
+{
+ /* stub */
+ return IRQ_HANDLED;
+}
+
+void pdsc_intr_free(struct pdsc *pdsc, int index)
+{
+ struct pdsc_intr_info *intr_info;
+
+ if (index >= pdsc->nintrs || index < 0) {
+ WARN(true, "bad intr index %d\n", index);
+ return;
+ }
+
+ intr_info = &pdsc->intr_info[index];
+ if (!intr_info->vector)
+ return;
+ dev_dbg(pdsc->dev, "%s: idx %d vec %d name %s\n",
+ __func__, index, intr_info->vector, intr_info->name);
+
+ pds_core_intr_mask(&pdsc->intr_ctrl[index], PDS_CORE_INTR_MASK_SET);
+ pds_core_intr_clean(&pdsc->intr_ctrl[index]);
+
+ free_irq(intr_info->vector, intr_info->data);
+
+ memset(intr_info, 0, sizeof(*intr_info));
+}
+
+int pdsc_intr_alloc(struct pdsc *pdsc, char *name,
+ irq_handler_t handler, void *data)
+{
+ struct pdsc_intr_info *intr_info;
+ unsigned int index;
+ int err;
+
+ /* Find the first available interrupt */
+ for (index = 0; index < pdsc->nintrs; index++)
+ if (!pdsc->intr_info[index].vector)
+ break;
+ if (index >= pdsc->nintrs) {
+ dev_warn(pdsc->dev, "%s: no intr, index=%d nintrs=%d\n",
+ __func__, index, pdsc->nintrs);
+ return -ENOSPC;
+ }
+
+ pds_core_intr_clean_flags(&pdsc->intr_ctrl[index],
+ PDS_CORE_INTR_CRED_RESET_COALESCE);
+
+ intr_info = &pdsc->intr_info[index];
+
+ intr_info->index = index;
+ intr_info->data = data;
+ strscpy(intr_info->name, name, sizeof(intr_info->name));
+
+ /* Get the OS vector number for the interrupt */
+ err = pci_irq_vector(pdsc->pdev, index);
+ if (err < 0) {
+ dev_err(pdsc->dev, "failed to get intr vector index %d: %pe\n",
+ index, ERR_PTR(err));
+ goto err_out_free_intr;
+ }
+ intr_info->vector = err;
+
+ /* Init the device's intr mask */
+ pds_core_intr_clean(&pdsc->intr_ctrl[index]);
+ pds_core_intr_mask_assert(&pdsc->intr_ctrl[index], 1);
+ pds_core_intr_mask(&pdsc->intr_ctrl[index], PDS_CORE_INTR_MASK_SET);
+
+ /* Register the isr with a name */
+ err = request_irq(intr_info->vector, handler, 0, intr_info->name, data);
+ if (err) {
+ dev_err(pdsc->dev, "failed to get intr irq vector %d: %pe\n",
+ intr_info->vector, ERR_PTR(err));
+ goto err_out_free_intr;
+ }
+
+ return index;
+
+err_out_free_intr:
+ pdsc_intr_free(pdsc, index);
+ return err;
+}
+
+static void pdsc_qcq_intr_free(struct pdsc *pdsc, struct pdsc_qcq *qcq)
+{
+ if (!(qcq->flags & PDS_CORE_QCQ_F_INTR) ||
+ qcq->intx == PDS_CORE_INTR_INDEX_NOT_ASSIGNED)
+ return;
+
+ pdsc_intr_free(pdsc, qcq->intx);
+ qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED;
+}
+
+static int pdsc_qcq_intr_alloc(struct pdsc *pdsc, struct pdsc_qcq *qcq)
+{
+ char name[PDSC_INTR_NAME_MAX_SZ];
+ int index;
+
+ if (!(qcq->flags & PDS_CORE_QCQ_F_INTR)) {
+ qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED;
+ return 0;
+ }
+
+ snprintf(name, sizeof(name), "%s-%d-%s",
+ PDS_CORE_DRV_NAME, pdsc->pdev->bus->number, qcq->q.name);
+ index = pdsc_intr_alloc(pdsc, name, pdsc_adminq_isr, qcq);
+ if (index < 0)
+ return index;
+ qcq->intx = index;
+
+ return 0;
+}
+
+void pdsc_qcq_free(struct pdsc *pdsc, struct pdsc_qcq *qcq)
+{
+ struct device *dev = pdsc->dev;
+
+ if (!(qcq && qcq->pdsc))
+ return;
+
+ pdsc_debugfs_del_qcq(qcq);
+
+ pdsc_qcq_intr_free(pdsc, qcq);
+
+ if (qcq->q_base) {
+ dma_free_coherent(dev, qcq->q_size,
+ qcq->q_base, qcq->q_base_pa);
+ qcq->q_base = NULL;
+ qcq->q_base_pa = 0;
+ }
+
+ if (qcq->cq_base) {
+ dma_free_coherent(dev, qcq->cq_size,
+ qcq->cq_base, qcq->cq_base_pa);
+ qcq->cq_base = NULL;
+ qcq->cq_base_pa = 0;
+ }
+
+ if (qcq->cq.info) {
+ vfree(qcq->cq.info);
+ qcq->cq.info = NULL;
+ }
+ if (qcq->q.info) {
+ vfree(qcq->q.info);
+ qcq->q.info = NULL;
+ }
+
+ qcq->pdsc = NULL;
+ memset(&qcq->q, 0, sizeof(qcq->q));
+ memset(&qcq->cq, 0, sizeof(qcq->cq));
+}
+
+static void pdsc_q_map(struct pdsc_queue *q, void *base, dma_addr_t base_pa)
+{
+ struct pdsc_q_info *cur;
+ unsigned int i;
+
+ q->base = base;
+ q->base_pa = base_pa;
+
+ for (i = 0, cur = q->info; i < q->num_descs; i++, cur++)
+ cur->desc = base + (i * q->desc_size);
+}
+
+static void pdsc_cq_map(struct pdsc_cq *cq, void *base, dma_addr_t base_pa)
+{
+ struct pdsc_cq_info *cur;
+ unsigned int i;
+
+ cq->base = base;
+ cq->base_pa = base_pa;
+
+ for (i = 0, cur = cq->info; i < cq->num_descs; i++, cur++)
+ cur->comp = base + (i * cq->desc_size);
+}
+
+int pdsc_qcq_alloc(struct pdsc *pdsc, unsigned int type, unsigned int index,
+ const char *name, unsigned int flags, unsigned int num_descs,
+ unsigned int desc_size, unsigned int cq_desc_size,
+ unsigned int pid, struct pdsc_qcq *qcq)
+{
+ struct device *dev = pdsc->dev;
+ dma_addr_t cq_base_pa = 0;
+ dma_addr_t q_base_pa = 0;
+ void *q_base, *cq_base;
+ int err;
+
+ qcq->q.info = vzalloc(num_descs * sizeof(*qcq->q.info));
+ if (!qcq->q.info) {
+ err = -ENOMEM;
+ goto err_out;
+ }
+
+ qcq->pdsc = pdsc;
+ qcq->flags = flags;
+ INIT_WORK(&qcq->work, pdsc_work_thread);
+
+ qcq->q.type = type;
+ qcq->q.index = index;
+ qcq->q.num_descs = num_descs;
+ qcq->q.desc_size = desc_size;
+ qcq->q.tail_idx = 0;
+ qcq->q.head_idx = 0;
+ qcq->q.pid = pid;
+ snprintf(qcq->q.name, sizeof(qcq->q.name), "%s%u", name, index);
+
+ err = pdsc_qcq_intr_alloc(pdsc, qcq);
+ if (err)
+ goto err_out_free_q_info;
+
+ qcq->cq.info = vzalloc(num_descs * sizeof(*qcq->cq.info));
+ if (!qcq->cq.info) {
+ err = -ENOMEM;
+ goto err_out_free_irq;
+ }
+
+ qcq->cq.bound_intr = &pdsc->intr_info[qcq->intx];
+ qcq->cq.num_descs = num_descs;
+ qcq->cq.desc_size = cq_desc_size;
+ qcq->cq.tail_idx = 0;
+ qcq->cq.done_color = 1;
+
+ if (flags & PDS_CORE_QCQ_F_NOTIFYQ) {
+ /* q & cq need to be contiguous in case of notifyq */
+ qcq->q_size = PDS_PAGE_SIZE +
+ ALIGN(num_descs * desc_size, PDS_PAGE_SIZE) +
+ ALIGN(num_descs * cq_desc_size, PDS_PAGE_SIZE);
+ qcq->q_base = dma_alloc_coherent(dev,
+ qcq->q_size + qcq->cq_size,
+ &qcq->q_base_pa,
+ GFP_KERNEL);
+ if (!qcq->q_base) {
+ err = -ENOMEM;
+ goto err_out_free_cq_info;
+ }
+ q_base = PTR_ALIGN(qcq->q_base, PDS_PAGE_SIZE);
+ q_base_pa = ALIGN(qcq->q_base_pa, PDS_PAGE_SIZE);
+ pdsc_q_map(&qcq->q, q_base, q_base_pa);
+
+ cq_base = PTR_ALIGN(q_base +
+ ALIGN(num_descs * desc_size, PDS_PAGE_SIZE),
+ PDS_PAGE_SIZE);
+ cq_base_pa = ALIGN(qcq->q_base_pa +
+ ALIGN(num_descs * desc_size, PDS_PAGE_SIZE),
+ PDS_PAGE_SIZE);
+
+ } else {
+ /* q DMA descriptors */
+ qcq->q_size = PDS_PAGE_SIZE + (num_descs * desc_size);
+ qcq->q_base = dma_alloc_coherent(dev, qcq->q_size,
+ &qcq->q_base_pa,
+ GFP_KERNEL);
+ if (!qcq->q_base) {
+ err = -ENOMEM;
+ goto err_out_free_cq_info;
+ }
+ q_base = PTR_ALIGN(qcq->q_base, PDS_PAGE_SIZE);
+ q_base_pa = ALIGN(qcq->q_base_pa, PDS_PAGE_SIZE);
+ pdsc_q_map(&qcq->q, q_base, q_base_pa);
+
+ /* cq DMA descriptors */
+ qcq->cq_size = PDS_PAGE_SIZE + (num_descs * cq_desc_size);
+ qcq->cq_base = dma_alloc_coherent(dev, qcq->cq_size,
+ &qcq->cq_base_pa,
+ GFP_KERNEL);
+ if (!qcq->cq_base) {
+ err = -ENOMEM;
+ goto err_out_free_q;
+ }
+ cq_base = PTR_ALIGN(qcq->cq_base, PDS_PAGE_SIZE);
+ cq_base_pa = ALIGN(qcq->cq_base_pa, PDS_PAGE_SIZE);
+ }
+
+ pdsc_cq_map(&qcq->cq, cq_base, cq_base_pa);
+ qcq->cq.bound_q = &qcq->q;
+
+ pdsc_debugfs_add_qcq(pdsc, qcq);
+
+ return 0;
+
+err_out_free_q:
+ dma_free_coherent(dev, qcq->q_size, qcq->q_base, qcq->q_base_pa);
+err_out_free_cq_info:
+ vfree(qcq->cq.info);
+err_out_free_irq:
+ pdsc_qcq_intr_free(pdsc, qcq);
+err_out_free_q_info:
+ vfree(qcq->q.info);
+ memset(qcq, 0, sizeof(*qcq));
+err_out:
+ dev_err(dev, "qcq alloc of %s%d failed %d\n", name, index, err);
+ return err;
+}
+
+static int pdsc_core_init(struct pdsc *pdsc)
+{
+ union pds_core_dev_comp comp = { 0 };
+ union pds_core_dev_cmd cmd = {
+ .init.opcode = PDS_CORE_CMD_INIT,
+ };
+ struct pds_core_dev_init_data_out cido;
+ struct pds_core_dev_init_data_in cidi;
+ u32 dbid_count;
+ u32 dbpage_num;
+ size_t sz;
+ int err;
+
+ cidi.adminq_q_base = cpu_to_le64(pdsc->adminqcq.q_base_pa);
+ cidi.adminq_cq_base = cpu_to_le64(pdsc->adminqcq.cq_base_pa);
+ cidi.notifyq_cq_base = cpu_to_le64(pdsc->notifyqcq.cq.base_pa);
+ cidi.flags = cpu_to_le32(PDS_CORE_QINIT_F_IRQ | PDS_CORE_QINIT_F_ENA);
+ cidi.intr_index = cpu_to_le16(pdsc->adminqcq.intx);
+ cidi.adminq_ring_size = ilog2(pdsc->adminqcq.q.num_descs);
+ cidi.notifyq_ring_size = ilog2(pdsc->notifyqcq.q.num_descs);
+
+ mutex_lock(&pdsc->devcmd_lock);
+
+ sz = min_t(size_t, sizeof(cidi), sizeof(pdsc->cmd_regs->data));
+ memcpy_toio(&pdsc->cmd_regs->data, &cidi, sz);
+
+ err = pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+ if (!err) {
+ sz = min_t(size_t, sizeof(cido), sizeof(pdsc->cmd_regs->data));
+ memcpy_fromio(&cido, &pdsc->cmd_regs->data, sz);
+ }
+
+ mutex_unlock(&pdsc->devcmd_lock);
+ if (err) {
+ dev_err(pdsc->dev, "Device init command failed: %pe\n",
+ ERR_PTR(err));
+ return err;
+ }
+
+ pdsc->hw_index = le32_to_cpu(cido.core_hw_index);
+
+ dbid_count = le32_to_cpu(pdsc->dev_ident.ndbpgs_per_lif);
+ dbpage_num = pdsc->hw_index * dbid_count;
+ pdsc->kern_dbpage = pdsc_map_dbpage(pdsc, dbpage_num);
+ if (!pdsc->kern_dbpage) {
+ dev_err(pdsc->dev, "Cannot map dbpage, aborting\n");
+ return -ENOMEM;
+ }
+
+ pdsc->adminqcq.q.hw_type = cido.adminq_hw_type;
+ pdsc->adminqcq.q.hw_index = le32_to_cpu(cido.adminq_hw_index);
+ pdsc->adminqcq.q.dbval = PDS_CORE_DBELL_QID(pdsc->adminqcq.q.hw_index);
+
+ pdsc->notifyqcq.q.hw_type = cido.notifyq_hw_type;
+ pdsc->notifyqcq.q.hw_index = le32_to_cpu(cido.notifyq_hw_index);
+ pdsc->notifyqcq.q.dbval = PDS_CORE_DBELL_QID(pdsc->notifyqcq.q.hw_index);
+
+ pdsc->last_eid = 0;
+
+ return err;
+}
+
int pdsc_setup(struct pdsc *pdsc, bool init)
{
+ int numdescs;
int err = 0;
if (init)
@@ -14,17 +380,60 @@ int pdsc_setup(struct pdsc *pdsc, bool init)
if (err)
return err;
+ /* Scale the descriptor ring length based on number of CPUs and VFs */
+ numdescs = max_t(int, PDSC_ADMINQ_MIN_LENGTH, num_online_cpus());
+ numdescs += 2 * pci_sriov_get_totalvfs(pdsc->pdev);
+ numdescs = roundup_pow_of_two(numdescs);
+ err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_ADMINQ, 0, "adminq",
+ PDS_CORE_QCQ_F_CORE | PDS_CORE_QCQ_F_INTR,
+ numdescs,
+ sizeof(union pds_core_adminq_cmd),
+ sizeof(union pds_core_adminq_comp),
+ 0, &pdsc->adminqcq);
+ if (err)
+ goto err_out_teardown;
+
+ err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_NOTIFYQ, 0, "notifyq",
+ PDS_CORE_QCQ_F_NOTIFYQ,
+ PDSC_NOTIFYQ_LENGTH,
+ sizeof(struct pds_core_notifyq_cmd),
+ sizeof(union pds_core_notifyq_comp),
+ 0, &pdsc->notifyqcq);
+ if (err)
+ goto err_out_teardown;
+
+ /* NotifyQ rides on the AdminQ interrupt */
+ pdsc->notifyqcq.intx = pdsc->adminqcq.intx;
+
+ /* Set up the Core with the AdminQ and NotifyQ info */
+ err = pdsc_core_init(pdsc);
+ if (err)
+ goto err_out_teardown;
+
clear_bit(PDSC_S_FW_DEAD, &pdsc->state);
return 0;
+
+err_out_teardown:
+ pdsc_teardown(pdsc, init);
+ return err;
}
void pdsc_teardown(struct pdsc *pdsc, bool removing)
{
+ int i;
+
pdsc_devcmd_reset(pdsc);
+ pdsc_qcq_free(pdsc, &pdsc->notifyqcq);
+ pdsc_qcq_free(pdsc, &pdsc->adminqcq);
+
+ if (pdsc->intr_info) {
+ for (i = 0; i < pdsc->nintrs; i++)
+ pdsc_intr_free(pdsc, i);
- if (removing) {
- kfree(pdsc->intr_info);
- pdsc->intr_info = NULL;
+ if (removing) {
+ kfree(pdsc->intr_info);
+ pdsc->intr_info = NULL;
+ }
}
if (pdsc->kern_dbpage) {
@@ -35,6 +444,36 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)
set_bit(PDSC_S_FW_DEAD, &pdsc->state);
}
+int pdsc_start(struct pdsc *pdsc)
+{
+ pds_core_intr_mask(&pdsc->intr_ctrl[pdsc->adminqcq.intx],
+ PDS_CORE_INTR_MASK_CLEAR);
+
+ return 0;
+}
+
+static void pdsc_mask_interrupts(struct pdsc *pdsc)
+{
+ int i;
+
+ if (!pdsc->intr_info)
+ return;
+
+ /* Mask interrupts that are in use */
+ for (i = 0; i < pdsc->nintrs; i++)
+ if (pdsc->intr_info[i].vector)
+ pds_core_intr_mask(&pdsc->intr_ctrl[i],
+ PDS_CORE_INTR_MASK_SET);
+}
+
+void pdsc_stop(struct pdsc *pdsc)
+{
+ if (pdsc->wq)
+ flush_workqueue(pdsc->wq);
+
+ pdsc_mask_interrupts(pdsc);
+}
+
static void pdsc_fw_down(struct pdsc *pdsc)
{
if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
@@ -44,6 +483,7 @@ static void pdsc_fw_down(struct pdsc *pdsc)
devlink_health_report(pdsc->fw_reporter, "FW down reported", pdsc);
+ pdsc_mask_interrupts(pdsc);
pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
}
@@ -60,6 +500,10 @@ static void pdsc_fw_up(struct pdsc *pdsc)
if (err)
goto err_out;
+ err = pdsc_start(pdsc);
+ if (err)
+ goto err_out;
+
pdsc->fw_recoveries++;
devlink_health_reporter_state_update(pdsc->fw_reporter,
DEVLINK_HEALTH_REPORTER_STATE_HEALTHY);
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index 3758071c94da..ee24833ed7ee 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -9,11 +9,15 @@
#include <linux/pds/pds_common.h>
#include <linux/pds/pds_core_if.h>
+#include <linux/pds/pds_adminq.h>
#include <linux/pds/pds_intr.h>
#define PDSC_DRV_DESCRIPTION "AMD/Pensando Core Driver"
#define PDSC_WATCHDOG_SECS 5
+#define PDSC_QUEUE_NAME_MAX_SZ 32
+#define PDSC_ADMINQ_MIN_LENGTH 16 /* must be a power of two */
+#define PDSC_NOTIFYQ_LENGTH 64 /* must be a power of two */
#define PDSC_TEARDOWN_RECOVERY false
#define PDSC_TEARDOWN_REMOVING true
#define PDSC_SETUP_RECOVERY false
@@ -33,6 +37,28 @@ struct pdsc_devinfo {
char serial_num[PDS_CORE_DEVINFO_SERIAL_BUFLEN + 1];
};
+struct pdsc_queue {
+ struct pdsc_q_info *info;
+ u64 dbval;
+ u16 head_idx;
+ u16 tail_idx;
+ u8 hw_type;
+ unsigned int index;
+ unsigned int num_descs;
+ u64 dbell_count;
+ u64 features;
+ unsigned int type;
+ unsigned int hw_index;
+ union {
+ void *base;
+ struct pds_core_admin_cmd *adminq;
+ };
+ dma_addr_t base_pa; /* must be page aligned */
+ unsigned int desc_size;
+ unsigned int pid;
+ char name[PDSC_QUEUE_NAME_MAX_SZ];
+};
+
#define PDSC_INTR_NAME_MAX_SZ 32
struct pdsc_intr_info {
@@ -42,6 +68,61 @@ struct pdsc_intr_info {
void *data;
};
+struct pdsc_cq_info {
+ void *comp;
+};
+
+struct pdsc_buf_info {
+ struct page *page;
+ dma_addr_t dma_addr;
+ u32 page_offset;
+ u32 len;
+};
+
+struct pdsc_q_info {
+ union {
+ void *desc;
+ struct pdsc_admin_cmd *adminq_desc;
+ };
+ unsigned int bytes;
+ unsigned int nbufs;
+ struct pdsc_buf_info bufs[PDS_CORE_MAX_FRAGS];
+ struct pdsc_wait_context *wc;
+ void *dest;
+};
+
+struct pdsc_cq {
+ struct pdsc_cq_info *info;
+ struct pdsc_queue *bound_q;
+ struct pdsc_intr_info *bound_intr;
+ u16 tail_idx;
+ bool done_color;
+ unsigned int num_descs;
+ unsigned int desc_size;
+ void *base;
+ dma_addr_t base_pa; /* must be page aligned */
+} ____cacheline_aligned_in_smp;
+
+struct pdsc_qcq {
+ struct pdsc *pdsc;
+ void *q_base;
+ dma_addr_t q_base_pa; /* might not be page aligned */
+ void *cq_base;
+ dma_addr_t cq_base_pa; /* might not be page aligned */
+ u32 q_size;
+ u32 cq_size;
+ bool armed;
+ unsigned int flags;
+
+ struct work_struct work;
+ struct pdsc_queue q;
+ struct pdsc_cq cq;
+ int intx;
+
+ u32 accum_work;
+ struct dentry *dentry;
+};
+
/* No state flags set means we are in a steady running state */
enum pdsc_state_flags {
PDSC_S_FW_DEAD, /* stopped, wait on startup or recovery */
@@ -81,6 +162,7 @@ struct pdsc {
unsigned int devcmd_timeout;
struct mutex devcmd_lock; /* lock for dev_cmd operations */
struct mutex config_lock; /* lock for configuration operations */
+ spinlock_t adminq_lock; /* lock for adminq operations */
struct pds_core_dev_info_regs __iomem *info_regs;
struct pds_core_dev_cmd_regs __iomem *cmd_regs;
struct pds_core_intr __iomem *intr_ctrl;
@@ -88,11 +170,64 @@ struct pdsc {
u64 __iomem *db_pages;
dma_addr_t phy_db_pages;
u64 __iomem *kern_dbpage;
+
+ struct pdsc_qcq adminqcq;
+ struct pdsc_qcq notifyqcq;
+ u64 last_eid;
};
+/** enum pds_core_dbell_bits - bitwise composition of dbell values.
+ *
+ * @PDS_CORE_DBELL_QID_MASK: unshifted mask of valid queue id bits.
+ * @PDS_CORE_DBELL_QID_SHIFT: queue id shift amount in dbell value.
+ * @PDS_CORE_DBELL_QID: macro to build QID component of dbell value.
+ *
+ * @PDS_CORE_DBELL_RING_MASK: unshifted mask of valid ring bits.
+ * @PDS_CORE_DBELL_RING_SHIFT: ring shift amount in dbell value.
+ * @PDS_CORE_DBELL_RING: macro to build ring component of dbell value.
+ *
+ * @PDS_CORE_DBELL_RING_0: ring zero dbell component value.
+ * @PDS_CORE_DBELL_RING_1: ring one dbell component value.
+ * @PDS_CORE_DBELL_RING_2: ring two dbell component value.
+ * @PDS_CORE_DBELL_RING_3: ring three dbell component value.
+ *
+ * @PDS_CORE_DBELL_INDEX_MASK: bit mask of valid index bits, no shift needed.
+ */
+enum pds_core_dbell_bits {
+ PDS_CORE_DBELL_QID_MASK = 0xffffff,
+ PDS_CORE_DBELL_QID_SHIFT = 24,
+
+#define PDS_CORE_DBELL_QID(n) \
+ (((u64)(n) & PDS_CORE_DBELL_QID_MASK) << PDS_CORE_DBELL_QID_SHIFT)
+
+ PDS_CORE_DBELL_RING_MASK = 0x7,
+ PDS_CORE_DBELL_RING_SHIFT = 16,
+
+#define PDS_CORE_DBELL_RING(n) \
+ (((u64)(n) & PDS_CORE_DBELL_RING_MASK) << PDS_CORE_DBELL_RING_SHIFT)
+
+ PDS_CORE_DBELL_RING_0 = 0,
+ PDS_CORE_DBELL_RING_1 = PDS_CORE_DBELL_RING(1),
+ PDS_CORE_DBELL_RING_2 = PDS_CORE_DBELL_RING(2),
+ PDS_CORE_DBELL_RING_3 = PDS_CORE_DBELL_RING(3),
+
+ PDS_CORE_DBELL_INDEX_MASK = 0xffff,
+};
+
+static inline void pds_core_dbell_ring(u64 __iomem *db_page,
+ enum pds_core_logical_qtype qtype,
+ u64 val)
+{
+ writeq(val, &db_page[qtype]);
+}
+
int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
struct devlink_fmsg *fmsg,
struct netlink_ext_ack *extack);
+int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,
+ struct netlink_ext_ack *extack);
+
+void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num);
#ifdef CONFIG_DEBUG_FS
void pdsc_debugfs_create(void);
@@ -101,6 +236,8 @@ void pdsc_debugfs_add_dev(struct pdsc *pdsc);
void pdsc_debugfs_del_dev(struct pdsc *pdsc);
void pdsc_debugfs_add_ident(struct pdsc *pdsc);
void pdsc_debugfs_add_irqs(struct pdsc *pdsc);
+void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq);
+void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq);
#else
static inline void pdsc_debugfs_create(void) { }
static inline void pdsc_debugfs_destroy(void) { }
@@ -108,6 +245,9 @@ static inline void pdsc_debugfs_add_dev(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_del_dev(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_add_ident(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_add_irqs(struct pdsc *pdsc) { }
+static inline void pdsc_debugfs_add_qcq(struct pdsc *pdsc,
+ struct pdsc_qcq *qcq) { }
+static inline void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq) { }
#endif
int pdsc_err_to_errno(enum pds_core_status_code code);
@@ -122,8 +262,22 @@ int pdsc_devcmd_reset(struct pdsc *pdsc);
int pdsc_dev_reinit(struct pdsc *pdsc);
int pdsc_dev_init(struct pdsc *pdsc);
+int pdsc_intr_alloc(struct pdsc *pdsc, char *name,
+ irq_handler_t handler, void *data);
+void pdsc_intr_free(struct pdsc *pdsc, int index);
+void pdsc_qcq_free(struct pdsc *pdsc, struct pdsc_qcq *qcq);
+int pdsc_qcq_alloc(struct pdsc *pdsc, unsigned int type, unsigned int index,
+ const char *name, unsigned int flags, unsigned int num_descs,
+ unsigned int desc_size, unsigned int cq_desc_size,
+ unsigned int pid, struct pdsc_qcq *qcq);
int pdsc_setup(struct pdsc *pdsc, bool init);
void pdsc_teardown(struct pdsc *pdsc, bool removing);
+int pdsc_start(struct pdsc *pdsc);
+void pdsc_stop(struct pdsc *pdsc);
void pdsc_health_thread(struct work_struct *work);
+void pdsc_process_adminq(struct pdsc_qcq *qcq);
+void pdsc_work_thread(struct work_struct *work);
+irqreturn_t pdsc_adminq_isr(int irq, void *data);
+
#endif /* _PDSC_H_ */
diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c
index 2869a2f60678..7439dc62fe6f 100644
--- a/drivers/net/ethernet/amd/pds_core/debugfs.c
+++ b/drivers/net/ethernet/amd/pds_core/debugfs.c
@@ -99,4 +99,80 @@ void pdsc_debugfs_add_irqs(struct pdsc *pdsc)
debugfs_create_file("irqs", 0400, pdsc->dentry, pdsc, &irqs_fops);
}
+static const struct debugfs_reg32 intr_ctrl_regs[] = {
+ { .name = "coal_init", .offset = 0, },
+ { .name = "mask", .offset = 4, },
+ { .name = "credits", .offset = 8, },
+ { .name = "mask_on_assert", .offset = 12, },
+ { .name = "coal_timer", .offset = 16, },
+};
+
+void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq)
+{
+ struct dentry *qcq_dentry, *q_dentry, *cq_dentry;
+ struct dentry *intr_dentry;
+ struct debugfs_regset32 *intr_ctrl_regset;
+ struct pdsc_intr_info *intr = &pdsc->intr_info[qcq->intx];
+ struct pdsc_queue *q = &qcq->q;
+ struct pdsc_cq *cq = &qcq->cq;
+
+ qcq_dentry = debugfs_create_dir(q->name, pdsc->dentry);
+ if (IS_ERR_OR_NULL(qcq_dentry))
+ return;
+ qcq->dentry = qcq_dentry;
+
+ debugfs_create_x64("q_base_pa", 0400, qcq_dentry, &qcq->q_base_pa);
+ debugfs_create_x32("q_size", 0400, qcq_dentry, &qcq->q_size);
+ debugfs_create_x64("cq_base_pa", 0400, qcq_dentry, &qcq->cq_base_pa);
+ debugfs_create_x32("cq_size", 0400, qcq_dentry, &qcq->cq_size);
+ debugfs_create_x32("accum_work", 0400, qcq_dentry, &qcq->accum_work);
+
+ q_dentry = debugfs_create_dir("q", qcq->dentry);
+ if (IS_ERR_OR_NULL(q_dentry))
+ return;
+
+ debugfs_create_u32("index", 0400, q_dentry, &q->index);
+ debugfs_create_u32("num_descs", 0400, q_dentry, &q->num_descs);
+ debugfs_create_u32("desc_size", 0400, q_dentry, &q->desc_size);
+ debugfs_create_u32("pid", 0400, q_dentry, &q->pid);
+
+ debugfs_create_u16("tail", 0400, q_dentry, &q->tail_idx);
+ debugfs_create_u16("head", 0400, q_dentry, &q->head_idx);
+
+ cq_dentry = debugfs_create_dir("cq", qcq->dentry);
+ if (IS_ERR_OR_NULL(cq_dentry))
+ return;
+
+ debugfs_create_x64("base_pa", 0400, cq_dentry, &cq->base_pa);
+ debugfs_create_u32("num_descs", 0400, cq_dentry, &cq->num_descs);
+ debugfs_create_u32("desc_size", 0400, cq_dentry, &cq->desc_size);
+ debugfs_create_bool("done_color", 0400, cq_dentry, &cq->done_color);
+ debugfs_create_u16("tail", 0400, cq_dentry, &cq->tail_idx);
+
+ if (qcq->flags & PDS_CORE_QCQ_F_INTR) {
+ intr_dentry = debugfs_create_dir("intr", qcq->dentry);
+ if (IS_ERR_OR_NULL(intr_dentry))
+ return;
+
+ debugfs_create_u32("index", 0400, intr_dentry, &intr->index);
+ debugfs_create_u32("vector", 0400, intr_dentry, &intr->vector);
+
+ intr_ctrl_regset = kzalloc(sizeof(*intr_ctrl_regset),
+ GFP_KERNEL);
+ if (!intr_ctrl_regset)
+ return;
+ intr_ctrl_regset->regs = intr_ctrl_regs;
+ intr_ctrl_regset->nregs = ARRAY_SIZE(intr_ctrl_regs);
+ intr_ctrl_regset->base = &pdsc->intr_ctrl[intr->index];
+
+ debugfs_create_regset32("intr_ctrl", 0400, intr_dentry,
+ intr_ctrl_regset);
+ }
+};
+
+void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq)
+{
+ debugfs_remove_recursive(qcq->dentry);
+ qcq->dentry = NULL;
+}
#endif /* CONFIG_DEBUG_FS */
diff --git a/drivers/net/ethernet/amd/pds_core/devlink.c b/drivers/net/ethernet/amd/pds_core/devlink.c
index 717fcbf91aee..beea606c3b8d 100644
--- a/drivers/net/ethernet/amd/pds_core/devlink.c
+++ b/drivers/net/ethernet/amd/pds_core/devlink.c
@@ -3,6 +3,67 @@
#include "core.h"
+static char *fw_slotnames[] = {
+ "fw.goldfw",
+ "fw.mainfwa",
+ "fw.mainfwb",
+};
+
+int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,
+ struct netlink_ext_ack *extack)
+{
+ union pds_core_dev_cmd cmd = {
+ .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL,
+ .fw_control.oper = PDS_CORE_FW_GET_LIST,
+ };
+ struct pds_core_fw_list_info fw_list;
+ struct pdsc *pdsc = devlink_priv(dl);
+ union pds_core_dev_comp comp;
+ char buf[16];
+ int listlen;
+ int err;
+ int i;
+
+ mutex_lock(&pdsc->devcmd_lock);
+ err = pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout * 2);
+ memcpy_fromio(&fw_list, pdsc->cmd_regs->data, sizeof(fw_list));
+ mutex_unlock(&pdsc->devcmd_lock);
+ if (err && err != -EIO)
+ return err;
+
+ listlen = fw_list.num_fw_slots;
+ for (i = 0; i < listlen; i++) {
+ if (i < ARRAY_SIZE(fw_slotnames))
+ strscpy(buf, fw_slotnames[i], sizeof(buf));
+ else
+ snprintf(buf, sizeof(buf), "fw.slot_%d", i);
+ err = devlink_info_version_stored_put(req, buf,
+ fw_list.fw_names[i].fw_version);
+ }
+
+ err = devlink_info_version_running_put(req,
+ DEVLINK_INFO_VERSION_GENERIC_FW,
+ pdsc->dev_info.fw_version);
+ if (err)
+ return err;
+
+ snprintf(buf, sizeof(buf), "0x%x", pdsc->dev_info.asic_type);
+ err = devlink_info_version_fixed_put(req,
+ DEVLINK_INFO_VERSION_GENERIC_ASIC_ID,
+ buf);
+ if (err)
+ return err;
+
+ snprintf(buf, sizeof(buf), "0x%x", pdsc->dev_info.asic_rev);
+ err = devlink_info_version_fixed_put(req,
+ DEVLINK_INFO_VERSION_GENERIC_ASIC_REV,
+ buf);
+ if (err)
+ return err;
+
+ return devlink_info_serial_number_put(req, pdsc->dev_info.serial_num);
+}
+
int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
struct devlink_fmsg *fmsg,
struct netlink_ext_ack *extack)
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index 82ce180d7b48..c349f44b01c1 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -136,6 +136,13 @@ static int pdsc_map_bars(struct pdsc *pdsc)
return err;
}
+void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num)
+{
+ return pci_iomap_range(pdsc->pdev,
+ pdsc->bars[PDS_CORE_PCI_BAR_DBELL].res_index,
+ (u64)page_num << PAGE_SHIFT, PAGE_SIZE);
+}
+
static int pdsc_init_vf(struct pdsc *vf)
{
return -1;
@@ -177,6 +184,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)
mutex_init(&pdsc->devcmd_lock);
mutex_init(&pdsc->config_lock);
+ spin_lock_init(&pdsc->adminq_lock);
mutex_lock(&pdsc->config_lock);
set_bit(PDSC_S_FW_DEAD, &pdsc->state);
@@ -184,6 +192,9 @@ static int pdsc_init_pf(struct pdsc *pdsc)
err = pdsc_setup(pdsc, PDSC_SETUP_INIT);
if (err)
goto err_out_unmap_bars;
+ err = pdsc_start(pdsc);
+ if (err)
+ goto err_out_teardown;
mutex_unlock(&pdsc->config_lock);
@@ -195,7 +206,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)
dev_warn(pdsc->dev, "Failed to create fw reporter: %pe\n", hr);
err = PTR_ERR(hr);
devl_unlock(dl);
- goto err_out_teardown;
+ goto err_out_stop;
}
pdsc->fw_reporter = hr;
@@ -207,6 +218,8 @@ static int pdsc_init_pf(struct pdsc *pdsc)
return 0;
+err_out_stop:
+ pdsc_stop(pdsc);
err_out_teardown:
pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
err_out_unmap_bars:
@@ -228,6 +241,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)
}
static const struct devlink_ops pdsc_dl_ops = {
+ .info_get = pdsc_dl_info_get,
};
static const struct devlink_ops pdsc_dl_vf_ops = {
@@ -332,6 +346,7 @@ static void pdsc_remove(struct pci_dev *pdev)
mutex_lock(&pdsc->config_lock);
set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
+ pdsc_stop(pdsc);
pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
mutex_unlock(&pdsc->config_lock);
mutex_destroy(&pdsc->config_lock);
diff --git a/include/linux/pds/pds_adminq.h b/include/linux/pds/pds_adminq.h
new file mode 100644
index 000000000000..9cd58b7f5fb2
--- /dev/null
+++ b/include/linux/pds/pds_adminq.h
@@ -0,0 +1,637 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _PDS_CORE_ADMINQ_H_
+#define _PDS_CORE_ADMINQ_H_
+
+enum pds_core_adminq_flags {
+ PDS_AQ_FLAG_FASTPOLL = BIT(1), /* completion poll at 1ms */
+};
+
+/*
+ * enum pds_core_adminq_opcode - AdminQ command opcodes
+ * These commands are only processed on AdminQ, not available in devcmd
+ */
+enum pds_core_adminq_opcode {
+ PDS_AQ_CMD_NOP = 0,
+
+ /* Client control */
+ PDS_AQ_CMD_CLIENT_REG = 6,
+ PDS_AQ_CMD_CLIENT_UNREG = 7,
+ PDS_AQ_CMD_CLIENT_CMD = 8,
+
+ /* LIF commands */
+ PDS_AQ_CMD_LIF_IDENTIFY = 20,
+ PDS_AQ_CMD_LIF_INIT = 21,
+ PDS_AQ_CMD_LIF_RESET = 22,
+ PDS_AQ_CMD_LIF_GETATTR = 23,
+ PDS_AQ_CMD_LIF_SETATTR = 24,
+ PDS_AQ_CMD_LIF_SETPHC = 25,
+
+ PDS_AQ_CMD_RX_MODE_SET = 30,
+ PDS_AQ_CMD_RX_FILTER_ADD = 31,
+ PDS_AQ_CMD_RX_FILTER_DEL = 32,
+
+ /* Queue commands */
+ PDS_AQ_CMD_Q_IDENTIFY = 39,
+ PDS_AQ_CMD_Q_INIT = 40,
+ PDS_AQ_CMD_Q_CONTROL = 41,
+
+ /* SR/IOV commands */
+ PDS_AQ_CMD_VF_GETATTR = 60,
+ PDS_AQ_CMD_VF_SETATTR = 61,
+};
+
+/*
+ * enum pds_core_notifyq_opcode - NotifyQ event codes
+ */
+enum pds_core_notifyq_opcode {
+ PDS_EVENT_LINK_CHANGE = 1,
+ PDS_EVENT_RESET = 2,
+ PDS_EVENT_XCVR = 5,
+ PDS_EVENT_CLIENT = 6,
+};
+
+#define PDS_COMP_COLOR_MASK 0x80
+
+/**
+ * struct pds_core_notifyq_event - Generic event reporting structure
+ * @eid: event number
+ * @ecode: event code
+ *
+ * This is the generic event report struct from which the other
+ * actual events will be formed.
+ */
+struct pds_core_notifyq_event {
+ __le64 eid;
+ __le16 ecode;
+};
+
+/**
+ * struct pds_core_link_change_event - Link change event notification
+ * @eid: event number
+ * @ecode: event code = PDS_EVENT_LINK_CHANGE
+ * @link_status: link up/down, with error bits
+ * @link_speed: speed of the network link
+ *
+ * Sent when the network link state changes between UP and DOWN
+ */
+struct pds_core_link_change_event {
+ __le64 eid;
+ __le16 ecode;
+ __le16 link_status;
+ __le32 link_speed; /* units of 1Mbps: e.g. 10000 = 10Gbps */
+};
+
+/**
+ * struct pds_core_reset_event - Reset event notification
+ * @eid: event number
+ * @ecode: event code = PDS_EVENT_RESET
+ * @reset_code: reset type
+ * @state: 0=pending, 1=complete, 2=error
+ *
+ * Sent when the NIC or some subsystem is going to be or
+ * has been reset.
+ */
+struct pds_core_reset_event {
+ __le64 eid;
+ __le16 ecode;
+ u8 reset_code;
+ u8 state;
+};
+
+/**
+ * struct pds_core_client_event - Client event notification
+ * @eid: event number
+ * @ecode: event code = PDS_EVENT_CLIENT
+ * @client_id: client to sent event to
+ * @client_event: wrapped event struct for the client
+ *
+ * Sent when an event needs to be passed on to a client
+ */
+struct pds_core_client_event {
+ __le64 eid;
+ __le16 ecode;
+ __le16 client_id;
+ u8 client_event[54];
+};
+
+/**
+ * struct pds_core_notifyq_cmd - Placeholder for building qcq
+ * @data: anonymous field for building the qcq
+ */
+struct pds_core_notifyq_cmd {
+ __le32 data; /* Not used but needed for qcq structure */
+};
+
+/*
+ * union pds_core_notifyq_comp - Overlay of notifyq event structures
+ */
+union pds_core_notifyq_comp {
+ struct {
+ __le64 eid;
+ __le16 ecode;
+ };
+ struct pds_core_notifyq_event event;
+ struct pds_core_link_change_event link_change;
+ struct pds_core_reset_event reset;
+ u8 data[64];
+};
+
+/**
+ * struct pds_core_client_reg_cmd - Register a new client with DSC
+ * @opcode: opcode PDS_AQ_CMD_CLIENT_REG
+ * @rsvd: word boundary padding
+ * @devname: text name of client device
+ * @vif_type: what type of device (enum pds_core_vif_types)
+ *
+ * Tell the DSC of the new client, and receive a client_id from DSC.
+ */
+struct pds_core_client_reg_cmd {
+ u8 opcode;
+ u8 rsvd[3];
+ char devname[32];
+ u8 vif_type;
+};
+
+/**
+ * struct pds_core_client_reg_comp - Client registration completion
+ * @status: Status of the command (enum pdc_core_status_code)
+ * @rsvd: Word boundary padding
+ * @comp_index: Index in the descriptor ring for which this is the completion
+ * @client_id: New id assigned by DSC
+ * @rsvd1: Word boundary padding
+ * @color: Color bit
+ */
+struct pds_core_client_reg_comp {
+ u8 status;
+ u8 rsvd;
+ __le16 comp_index;
+ __le16 client_id;
+ u8 rsvd1[9];
+ u8 color;
+};
+
+/**
+ * struct pds_core_client_unreg_cmd - Unregister a client from DSC
+ * @opcode: opcode PDS_AQ_CMD_CLIENT_UNREG
+ * @rsvd: word boundary padding
+ * @client_id: id of client being removed
+ *
+ * Tell the DSC this client is going away and remove its context
+ * This uses the generic completion.
+ */
+struct pds_core_client_unreg_cmd {
+ u8 opcode;
+ u8 rsvd;
+ __le16 client_id;
+};
+
+/**
+ * struct pds_core_client_request_cmd - Pass along a wrapped client AdminQ cmd
+ * @opcode: opcode PDS_AQ_CMD_CLIENT_CMD
+ * @rsvd: word boundary padding
+ * @client_id: id of client being removed
+ * @client_cmd: the wrapped client command
+ *
+ * Proxy post an adminq command for the client.
+ * This uses the generic completion.
+ */
+struct pds_core_client_request_cmd {
+ u8 opcode;
+ u8 rsvd;
+ __le16 client_id;
+ u8 client_cmd[60];
+};
+
+#define PDS_CORE_MAX_FRAGS 16
+
+#define PDS_CORE_QCQ_F_INITED BIT(0)
+#define PDS_CORE_QCQ_F_SG BIT(1)
+#define PDS_CORE_QCQ_F_INTR BIT(2)
+#define PDS_CORE_QCQ_F_TX_STATS BIT(3)
+#define PDS_CORE_QCQ_F_RX_STATS BIT(4)
+#define PDS_CORE_QCQ_F_NOTIFYQ BIT(5)
+#define PDS_CORE_QCQ_F_CMB_RINGS BIT(6)
+#define PDS_CORE_QCQ_F_CORE BIT(7)
+
+enum pds_core_lif_type {
+ PDS_CORE_LIF_TYPE_DEFAULT = 0,
+};
+
+/**
+ * union pds_core_lif_config - LIF configuration
+ * @state: LIF state (enum pds_core_lif_state)
+ * @rsvd: Word boundary padding
+ * @name: LIF name
+ * @rsvd2: Word boundary padding
+ * @features: LIF features active (enum pds_core_hw_features)
+ * @queue_count: Queue counts per queue-type
+ * @words: Full union buffer size
+ */
+union pds_core_lif_config {
+ struct {
+ u8 state;
+ u8 rsvd[3];
+ char name[PDS_CORE_IFNAMSIZ];
+ u8 rsvd2[12];
+ __le64 features;
+ __le32 queue_count[PDS_CORE_QTYPE_MAX];
+ } __packed;
+ __le32 words[64];
+};
+
+/**
+ * struct pds_core_lif_status - LIF status register
+ * @eid: most recent NotifyQ event id
+ * @rsvd: full struct size
+ */
+struct pds_core_lif_status {
+ __le64 eid;
+ u8 rsvd[56];
+};
+
+/**
+ * struct pds_core_lif_info - LIF info structure
+ * @config: LIF configuration structure
+ * @status: LIF status structure
+ */
+struct pds_core_lif_info {
+ union pds_core_lif_config config;
+ struct pds_core_lif_status status;
+};
+
+/**
+ * struct pds_core_lif_identity - LIF identity information (type-specific)
+ * @features: LIF features (see enum pds_core_hw_features)
+ * @version: Identify structure version
+ * @hw_index: LIF hardware index
+ * @rsvd: Word boundary padding
+ * @max_nb_sessions: Maximum number of sessions supported
+ * @rsvd2: buffer padding
+ * @config: LIF config struct with features, q counts
+ */
+struct pds_core_lif_identity {
+ __le64 features;
+ u8 version;
+ u8 hw_index;
+ u8 rsvd[2];
+ __le32 max_nb_sessions;
+ u8 rsvd2[120];
+ union pds_core_lif_config config;
+};
+
+/**
+ * struct pds_core_lif_identify_cmd - Get LIF identity info command
+ * @opcode: Opcode PDS_AQ_CMD_LIF_IDENTIFY
+ * @type: LIF type (enum pds_core_lif_type)
+ * @client_id: Client identifier
+ * @ver: Version of identify returned by device
+ * @rsvd: Word boundary padding
+ * @ident_pa: DMA address to receive identity info
+ *
+ * Firmware will copy LIF identity data (struct pds_core_lif_identity)
+ * into the buffer address given.
+ */
+struct pds_core_lif_identify_cmd {
+ u8 opcode;
+ u8 type;
+ __le16 client_id;
+ u8 ver;
+ u8 rsvd[3];
+ __le64 ident_pa;
+};
+
+/**
+ * struct pds_core_lif_identify_comp - LIF identify command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @ver: Version of identify returned by device
+ * @bytes: Bytes copied into the buffer
+ * @rsvd: Word boundary padding
+ * @color: Color bit
+ */
+struct pds_core_lif_identify_comp {
+ u8 status;
+ u8 ver;
+ __le16 bytes;
+ u8 rsvd[11];
+ u8 color;
+};
+
+/**
+ * struct pds_core_lif_init_cmd - LIF init command
+ * @opcode: Opcode PDS_AQ_CMD_LIF_INIT
+ * @type: LIF type (enum pds_core_lif_type)
+ * @client_id: Client identifier
+ * @rsvd: Word boundary padding
+ * @info_pa: Destination address for LIF info (struct pds_core_lif_info)
+ */
+struct pds_core_lif_init_cmd {
+ u8 opcode;
+ u8 type;
+ __le16 client_id;
+ __le32 rsvd;
+ __le64 info_pa;
+};
+
+/**
+ * struct pds_core_lif_init_comp - LIF init command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @rsvd: Word boundary padding
+ * @hw_index: Hardware index of the initialized LIF
+ * @rsvd1: Word boundary padding
+ * @color: Color bit
+ */
+struct pds_core_lif_init_comp {
+ u8 status;
+ u8 rsvd;
+ __le16 hw_index;
+ u8 rsvd1[11];
+ u8 color;
+};
+
+/**
+ * struct pds_core_lif_reset_cmd - LIF reset command
+ * Will reset only the specified LIF.
+ * @opcode: Opcode PDS_AQ_CMD_LIF_RESET
+ * @rsvd: Word boundary padding
+ * @client_id: Client identifier
+ */
+struct pds_core_lif_reset_cmd {
+ u8 opcode;
+ u8 rsvd;
+ __le16 client_id;
+};
+
+/**
+ * enum pds_core_lif_attr - List of LIF attributes
+ * @PDS_CORE_LIF_ATTR_STATE: LIF state attribute
+ * @PDS_CORE_LIF_ATTR_NAME: LIF name attribute
+ * @PDS_CORE_LIF_ATTR_FEATURES: LIF features attribute
+ * @PDS_CORE_LIF_ATTR_STATS_CTRL: LIF statistics control attribute
+ */
+enum pds_core_lif_attr {
+ PDS_CORE_LIF_ATTR_STATE = 0,
+ PDS_CORE_LIF_ATTR_NAME = 1,
+ PDS_CORE_LIF_ATTR_FEATURES = 4,
+ PDS_CORE_LIF_ATTR_STATS_CTRL = 6,
+};
+
+/**
+ * struct pds_core_lif_setattr_cmd - Set LIF attributes on the NIC
+ * @opcode: Opcode PDS_AQ_CMD_LIF_SETATTR
+ * @attr: Attribute type (enum pds_core_lif_attr)
+ * @client_id: Client identifier
+ * @state: LIF state (enum pds_core_lif_state)
+ * @name: The name string, 0 terminated
+ * @features: Features (enum pds_core_hw_features)
+ * @stats_ctl: Stats control commands (enum pds_core_stats_ctl_cmd)
+ * @rsvd: Command Buffer padding
+ */
+struct pds_core_lif_setattr_cmd {
+ u8 opcode;
+ u8 attr;
+ __le16 client_id;
+ union {
+ u8 state;
+ char name[PDS_CORE_IFNAMSIZ];
+ __le64 features;
+ u8 stats_ctl;
+ u8 rsvd[60];
+ } __packed;
+};
+
+/**
+ * struct pds_core_lif_setattr_comp - LIF set attr command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @rsvd: Word boundary padding
+ * @comp_index: Index in the descriptor ring for which this is the completion
+ * @features: Features (enum pds_core_hw_features)
+ * @rsvd2: Word boundary padding
+ * @color: Color bit
+ */
+struct pds_core_lif_setattr_comp {
+ u8 status;
+ u8 rsvd;
+ __le16 comp_index;
+ union {
+ __le64 features;
+ u8 rsvd2[11];
+ } __packed;
+ u8 color;
+};
+
+/**
+ * struct pds_core_lif_getattr_cmd - Get LIF attributes from the NIC
+ * @opcode: Opcode PDS_AQ_CMD_LIF_GETATTR
+ * @attr: Attribute type (enum pds_core_lif_attr)
+ * @client_id: Client identifier
+ */
+struct pds_core_lif_getattr_cmd {
+ u8 opcode;
+ u8 attr;
+ __le16 client_id;
+};
+
+/**
+ * struct pds_core_lif_getattr_comp - LIF get attr command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @rsvd: Word boundary padding
+ * @comp_index: Index in the descriptor ring for which this is the completion
+ * @state: LIF state (enum pds_core_lif_state)
+ * @name: LIF name string, 0 terminated
+ * @features: Features (enum pds_core_hw_features)
+ * @rsvd2: Word boundary padding
+ * @color: Color bit
+ */
+struct pds_core_lif_getattr_comp {
+ u8 status;
+ u8 rsvd;
+ __le16 comp_index;
+ union {
+ u8 state;
+ __le64 features;
+ u8 rsvd2[11];
+ } __packed;
+ u8 color;
+};
+
+/**
+ * union pds_core_q_identity - Queue identity information
+ * @version: Queue type version that can be used with FW
+ * @supported: Bitfield of queue versions, first bit = ver 0
+ * @rsvd: Word boundary padding
+ * @features: Queue features
+ * @desc_sz: Descriptor size
+ * @comp_sz: Completion descriptor size
+ * @rsvd2: Word boundary padding
+ */
+struct pds_core_q_identity {
+ u8 version;
+ u8 supported;
+ u8 rsvd[6];
+#define PDS_CORE_QIDENT_F_CQ 0x01 /* queue has completion ring */
+ __le64 features;
+ __le16 desc_sz;
+ __le16 comp_sz;
+ u8 rsvd2[6];
+};
+
+/**
+ * struct pds_core_q_identify_cmd - queue identify command
+ * @opcode: Opcode PDS_AQ_CMD_Q_IDENTIFY
+ * @type: Logical queue type (enum pds_core_logical_qtype)
+ * @client_id: Client identifier
+ * @ver: Highest queue type version that the driver supports
+ * @rsvd: Word boundary padding
+ * @ident_pa: DMA address to receive the data (struct pds_core_q_identity)
+ */
+struct pds_core_q_identify_cmd {
+ u8 opcode;
+ u8 type;
+ __le16 client_id;
+ u8 ver;
+ u8 rsvd[3];
+ __le64 ident_pa;
+};
+
+/**
+ * struct pds_core_q_identify_comp - queue identify command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @rsvd: Word boundary padding
+ * @comp_index: Index in the descriptor ring for which this is the completion
+ * @ver: Queue type version that can be used with FW
+ * @rsvd1: Word boundary padding
+ * @color: Color bit
+ */
+struct pds_core_q_identify_comp {
+ u8 status;
+ u8 rsvd;
+ __le16 comp_index;
+ u8 ver;
+ u8 rsvd1[10];
+ u8 color;
+};
+
+/**
+ * struct pds_core_q_init_cmd - Queue init command
+ * @opcode: Opcode PDS_AQ_CMD_Q_INIT
+ * @type: Logical queue type
+ * @client_id: Client identifier
+ * @ver: Queue type version
+ * @rsvd: Word boundary padding
+ * @index: (LIF, qtype) relative admin queue index
+ * @intr_index: Interrupt control register index, or Event queue index
+ * @pid: Process ID
+ * @flags:
+ * IRQ: Interrupt requested on completion
+ * ENA: Enable the queue. If ENA=0 the queue is initialized
+ * but remains disabled, to be later enabled with the
+ * Queue Enable command. If ENA=1, then queue is
+ * initialized and then enabled.
+ * @cos: Class of service for this queue
+ * @ring_size: Queue ring size, encoded as a log2(size), in
+ * number of descriptors. The actual ring size is
+ * (1 << ring_size). For example, to select a ring size
+ * of 64 descriptors write ring_size = 6. The minimum
+ * ring_size value is 2 for a ring of 4 descriptors.
+ * The maximum ring_size value is 12 for a ring of 4k
+ * descriptors. Values of ring_size <2 and >12 are
+ * reserved.
+ * @ring_base: Queue ring base address
+ * @cq_ring_base: Completion queue ring base address
+ */
+struct pds_core_q_init_cmd {
+ u8 opcode;
+ u8 type;
+ __le16 client_id;
+ u8 ver;
+ u8 rsvd[3];
+ __le32 index;
+ __le16 pid;
+ __le16 intr_index;
+ __le16 flags;
+#define PDS_CORE_QINIT_F_IRQ 0x01 /* Request interrupt on completion */
+#define PDS_CORE_QINIT_F_ENA 0x02 /* Enable the queue */
+ u8 cos;
+#define PDS_CORE_QSIZE_MIN_LG2 2
+#define PDS_CORE_QSIZE_MAX_LG2 12
+ u8 ring_size;
+ __le64 ring_base;
+ __le64 cq_ring_base;
+} __packed;
+
+/**
+ * struct pds_core_q_init_comp - Queue init command completion
+ * @status: Status of the command (enum pds_core_status_code)
+ * @rsvd: Word boundary padding
+ * @comp_index: Index in the descriptor ring for which this is the completion
+ * @hw_index: Hardware Queue ID
+ * @hw_type: Hardware Queue type
+ * @rsvd2: Word boundary padding
+ * @color: Color
+ */
+struct pds_core_q_init_comp {
+ u8 status;
+ u8 rsvd;
+ __le16 comp_index;
+ __le32 hw_index;
+ u8 hw_type;
+ u8 rsvd2[6];
+ u8 color;
+};
+
+union pds_core_adminq_cmd {
+ u8 opcode;
+ u8 bytes[64];
+
+ struct pds_core_client_reg_cmd client_reg;
+ struct pds_core_client_unreg_cmd client_unreg;
+ struct pds_core_client_request_cmd client_request;
+
+ struct pds_core_lif_identify_cmd lif_ident;
+ struct pds_core_lif_init_cmd lif_init;
+ struct pds_core_lif_reset_cmd lif_reset;
+ struct pds_core_lif_setattr_cmd lif_setattr;
+ struct pds_core_lif_getattr_cmd lif_getattr;
+
+ struct pds_core_q_identify_cmd q_ident;
+ struct pds_core_q_init_cmd q_init;
+};
+
+union pds_core_adminq_comp {
+ struct {
+ u8 status;
+ u8 rsvd;
+ __le16 comp_index;
+ u8 rsvd2[11];
+ u8 color;
+ };
+ u32 words[4];
+
+ struct pds_core_client_reg_comp client_reg;
+
+ struct pds_core_lif_identify_comp lif_ident;
+ struct pds_core_lif_init_comp lif_init;
+ struct pds_core_lif_setattr_comp lif_setattr;
+ struct pds_core_lif_getattr_comp lif_getattr;
+
+ struct pds_core_q_identify_comp q_ident;
+ struct pds_core_q_init_comp q_init;
+};
+
+#ifndef __CHECKER__
+static_assert(sizeof(union pds_core_adminq_cmd) == 64);
+static_assert(sizeof(union pds_core_adminq_comp) == 16);
+static_assert(sizeof(union pds_core_notifyq_comp) == 64);
+#endif /* __CHECKER__ */
+
+/* The color bit is a 'done' bit for the completion descriptors
+ * where the meaning alternates between '1' and '0' for alternating
+ * passes through the completion descriptor ring.
+ */
+static inline u8 pdsc_color_match(u8 color, u8 done_color)
+{
+ return (!!(color & PDS_COMP_COLOR_MASK)) == done_color;
+}
+#endif /* _PDS_CORE_ADMINQ_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 06/14] pds_core: Add adminq processing and commands
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (4 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 05/14] pds_core: set up device and adminq Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink Shannon Nelson
` (9 subsequent siblings)
15 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Add the service routines for submitting and processing
the adminq messages and for handling notifyq events.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/Makefile | 1 +
drivers/net/ethernet/amd/pds_core/adminq.c | 289 +++++++++++++++++++++
drivers/net/ethernet/amd/pds_core/core.c | 11 -
include/linux/pds/pds_adminq.h | 8 +
4 files changed, 298 insertions(+), 11 deletions(-)
create mode 100644 drivers/net/ethernet/amd/pds_core/adminq.c
diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
index eaca8557ba66..ef76dcd7fccd 100644
--- a/drivers/net/ethernet/amd/pds_core/Makefile
+++ b/drivers/net/ethernet/amd/pds_core/Makefile
@@ -6,6 +6,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o
pds_core-y := main.o \
devlink.o \
dev.o \
+ adminq.o \
core.o
pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c
new file mode 100644
index 000000000000..25c7dd0d37e5
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/adminq.c
@@ -0,0 +1,289 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#include "core.h"
+
+struct pdsc_wait_context {
+ struct pdsc_qcq *qcq;
+ struct completion wait_completion;
+};
+
+static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
+{
+ union pds_core_notifyq_comp *comp;
+ struct pdsc *pdsc = qcq->pdsc;
+ struct pdsc_cq *cq = &qcq->cq;
+ struct pdsc_cq_info *cq_info;
+ int nq_work = 0;
+ u64 eid;
+
+ cq_info = &cq->info[cq->tail_idx];
+ comp = cq_info->comp;
+ eid = le64_to_cpu(comp->event.eid);
+ while (eid > pdsc->last_eid) {
+ u16 ecode = le16_to_cpu(comp->event.ecode);
+
+ switch (ecode) {
+ case PDS_EVENT_LINK_CHANGE:
+ dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
+ ecode, eid);
+ break;
+
+ case PDS_EVENT_RESET:
+ dev_info(pdsc->dev, "NotifyQ RESET ecode %d eid %lld\n",
+ ecode, eid);
+ break;
+
+ case PDS_EVENT_XCVR:
+ dev_info(pdsc->dev, "NotifyQ XCVR ecode %d eid %lld\n",
+ ecode, eid);
+ break;
+
+ default:
+ dev_info(pdsc->dev, "NotifyQ ecode %d eid %lld\n",
+ ecode, eid);
+ break;
+ }
+
+ pdsc->last_eid = eid;
+ cq->tail_idx = (cq->tail_idx + 1) & (cq->num_descs - 1);
+ cq_info = &cq->info[cq->tail_idx];
+ comp = cq_info->comp;
+ eid = le64_to_cpu(comp->event.eid);
+
+ nq_work++;
+ }
+
+ qcq->accum_work += nq_work;
+
+ return nq_work;
+}
+
+void pdsc_process_adminq(struct pdsc_qcq *qcq)
+{
+ union pds_core_adminq_comp *comp;
+ struct pdsc_queue *q = &qcq->q;
+ struct pdsc *pdsc = qcq->pdsc;
+ struct pdsc_cq *cq = &qcq->cq;
+ struct pdsc_q_info *q_info;
+ unsigned long irqflags;
+ int nq_work = 0;
+ int aq_work = 0;
+ int credits;
+
+ /* Don't process AdminQ when shutting down */
+ if (pdsc->state & BIT_ULL(PDSC_S_STOPPING_DRIVER)) {
+ dev_err(pdsc->dev, "%s: called while PDSC_S_STOPPING_DRIVER\n",
+ __func__);
+ return;
+ }
+
+ /* Check for NotifyQ event */
+ nq_work = pdsc_process_notifyq(&pdsc->notifyqcq);
+
+ /* Check for empty queue, which can happen if the interrupt was
+ * for a NotifyQ event and there are no new AdminQ completions.
+ */
+ if (q->tail_idx == q->head_idx)
+ goto credits;
+
+ /* Find the first completion to clean,
+ * run the callback in the related q_info,
+ * and continue while we still match done color
+ */
+ spin_lock_irqsave(&pdsc->adminq_lock, irqflags);
+ comp = cq->info[cq->tail_idx].comp;
+ while (pdsc_color_match(comp->color, cq->done_color)) {
+ q_info = &q->info[q->tail_idx];
+ q->tail_idx = (q->tail_idx + 1) & (q->num_descs - 1);
+
+ /* Copy out the completion data */
+ memcpy(q_info->dest, comp, sizeof(*comp));
+
+ complete_all(&q_info->wc->wait_completion);
+
+ if (cq->tail_idx == cq->num_descs - 1)
+ cq->done_color = !cq->done_color;
+ cq->tail_idx = (cq->tail_idx + 1) & (cq->num_descs - 1);
+ comp = cq->info[cq->tail_idx].comp;
+
+ aq_work++;
+ }
+ spin_unlock_irqrestore(&pdsc->adminq_lock, irqflags);
+
+ qcq->accum_work += aq_work;
+
+credits:
+ /* Return the interrupt credits, one for each completion */
+ credits = nq_work + aq_work;
+ if (credits)
+ pds_core_intr_credits(&pdsc->intr_ctrl[qcq->intx],
+ credits,
+ PDS_CORE_INTR_CRED_REARM);
+}
+
+void pdsc_work_thread(struct work_struct *work)
+{
+ struct pdsc_qcq *qcq = container_of(work, struct pdsc_qcq, work);
+
+ pdsc_process_adminq(qcq);
+}
+
+irqreturn_t pdsc_adminq_isr(int irq, void *data)
+{
+ struct pdsc_qcq *qcq = data;
+ struct pdsc *pdsc = qcq->pdsc;
+
+ /* Don't process AdminQ when shutting down */
+ if (pdsc->state & BIT_ULL(PDSC_S_STOPPING_DRIVER)) {
+ dev_err(pdsc->dev, "%s: called while PDSC_S_STOPPING_DRIVER\n",
+ __func__);
+ return IRQ_HANDLED;
+ }
+
+ queue_work(pdsc->wq, &qcq->work);
+ pds_core_intr_mask(&pdsc->intr_ctrl[irq], PDS_CORE_INTR_MASK_CLEAR);
+
+ return IRQ_HANDLED;
+}
+
+static int __pdsc_adminq_post(struct pdsc *pdsc,
+ struct pdsc_qcq *qcq,
+ union pds_core_adminq_cmd *cmd,
+ union pds_core_adminq_comp *comp,
+ struct pdsc_wait_context *wc)
+{
+ struct pdsc_queue *q = &qcq->q;
+ struct pdsc_q_info *q_info;
+ unsigned long irqflags;
+ unsigned int avail;
+ int ret = 0;
+ int index;
+
+ spin_lock_irqsave(&pdsc->adminq_lock, irqflags);
+
+ /* Check for space in the queue */
+ avail = q->tail_idx;
+ if (q->head_idx >= avail)
+ avail += q->num_descs - q->head_idx - 1;
+ else
+ avail -= q->head_idx + 1;
+ if (!avail) {
+ ret = -ENOSPC;
+ goto err_out_unlock;
+ }
+
+ /* Check that the FW is running */
+ if (!pdsc_is_fw_running(pdsc)) {
+ u8 fw_status = ioread8(&pdsc->info_regs->fw_status);
+
+ dev_info(pdsc->dev, "%s: post failed - fw not running %#02x:\n",
+ __func__, fw_status);
+ ret = -ENXIO;
+
+ goto err_out_unlock;
+ }
+
+ /* Post the request */
+ index = q->head_idx;
+ q_info = &q->info[index];
+ q_info->wc = wc;
+ q_info->dest = comp;
+ memcpy(q_info->desc, cmd, sizeof(*cmd));
+
+ dev_dbg(pdsc->dev, "head_idx %d tail_idx %d\n",
+ q->head_idx, q->tail_idx);
+ dev_dbg(pdsc->dev, "post admin queue command:\n");
+ dynamic_hex_dump("cmd ", DUMP_PREFIX_OFFSET, 16, 1,
+ cmd, sizeof(*cmd), true);
+
+ q->head_idx = (q->head_idx + 1) & (q->num_descs - 1);
+
+ pds_core_dbell_ring(pdsc->kern_dbpage,
+ q->hw_type, q->dbval | q->head_idx);
+ ret = index;
+
+err_out_unlock:
+ spin_unlock_irqrestore(&pdsc->adminq_lock, irqflags);
+ return ret;
+}
+
+int pdsc_adminq_post(void *pdsc_void,
+ union pds_core_adminq_cmd *cmd,
+ union pds_core_adminq_comp *comp,
+ bool fast_poll)
+{
+ struct pdsc_wait_context wc = {
+ .wait_completion =
+ COMPLETION_INITIALIZER_ONSTACK(wc.wait_completion),
+ };
+ unsigned long poll_interval = 1;
+ unsigned long poll_jiffies;
+ unsigned long time_limit;
+ unsigned long time_start;
+ unsigned long time_done;
+ unsigned long remaining;
+ struct pdsc *pdsc;
+ int err = 0;
+ int index;
+
+ pdsc = (struct pdsc *)pdsc_void;
+
+ wc.qcq = &pdsc->adminqcq;
+ index = __pdsc_adminq_post(pdsc, &pdsc->adminqcq, cmd, comp, &wc);
+ if (index < 0) {
+ err = index;
+ goto err_out;
+ }
+
+ time_start = jiffies;
+ time_limit = time_start + HZ * pdsc->devcmd_timeout;
+ do {
+ /* Timeslice the actual wait to catch IO errors etc early */
+ poll_jiffies = msecs_to_jiffies(poll_interval);
+ remaining = wait_for_completion_timeout(&wc.wait_completion,
+ poll_jiffies);
+ if (remaining)
+ break;
+
+ if (!pdsc_is_fw_running(pdsc)) {
+ u8 fw_status = ioread8(&pdsc->info_regs->fw_status);
+
+ dev_dbg(pdsc->dev, "%s: post wait failed - fw not running %#02x:\n",
+ __func__, fw_status);
+ err = -ENXIO;
+ break;
+ }
+
+ /* When fast_poll is not requested, prevent aggressive polling
+ * on failures due to timeouts by doing exponential back off.
+ */
+ if (!fast_poll && poll_interval < PDSC_ADMINQ_MAX_POLL_INTERVAL)
+ poll_interval <<= 1;
+ } while (time_before(jiffies, time_limit));
+ time_done = jiffies;
+ dev_dbg(pdsc->dev, "%s: elapsed %d msecs\n",
+ __func__, jiffies_to_msecs(time_done - time_start));
+
+ /* Check the results */
+ if (time_after_eq(time_done, time_limit))
+ err = -ETIMEDOUT;
+
+ dev_dbg(pdsc->dev, "read admin queue completion idx %d:\n", index);
+ dynamic_hex_dump("comp ", DUMP_PREFIX_OFFSET, 16, 1,
+ comp, sizeof(*comp), true);
+
+ if (remaining && comp->status)
+ err = pdsc_err_to_errno(comp->status);
+
+err_out:
+ if (err) {
+ dev_dbg(pdsc->dev, "%s: opcode %d status %d err %pe\n",
+ __func__, cmd->opcode, comp->status, ERR_PTR(err));
+ if (err == -ENXIO || err == -ETIMEDOUT)
+ queue_work(pdsc->wq, &pdsc->health_work);
+ }
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(pdsc_adminq_post);
diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
index ec78510fcce7..7c5ccff4005c 100644
--- a/drivers/net/ethernet/amd/pds_core/core.c
+++ b/drivers/net/ethernet/amd/pds_core/core.c
@@ -6,17 +6,6 @@
#include "core.h"
-void pdsc_work_thread(struct work_struct *work)
-{
- /* stub */
-}
-
-irqreturn_t pdsc_adminq_isr(int irq, void *data)
-{
- /* stub */
- return IRQ_HANDLED;
-}
-
void pdsc_intr_free(struct pdsc *pdsc, int index)
{
struct pdsc_intr_info *intr_info;
diff --git a/include/linux/pds/pds_adminq.h b/include/linux/pds/pds_adminq.h
index 9cd58b7f5fb2..797d93fd0473 100644
--- a/include/linux/pds/pds_adminq.h
+++ b/include/linux/pds/pds_adminq.h
@@ -4,6 +4,8 @@
#ifndef _PDS_CORE_ADMINQ_H_
#define _PDS_CORE_ADMINQ_H_
+#define PDSC_ADMINQ_MAX_POLL_INTERVAL 256
+
enum pds_core_adminq_flags {
PDS_AQ_FLAG_FASTPOLL = BIT(1), /* completion poll at 1ms */
};
@@ -634,4 +636,10 @@ static inline u8 pdsc_color_match(u8 color, u8 done_color)
{
return (!!(color & PDS_COMP_COLOR_MASK)) == done_color;
}
+
+int pdsc_adminq_post(void *pdsc,
+ union pds_core_adminq_cmd *cmd,
+ union pds_core_adminq_comp *comp,
+ bool fast_poll);
+
#endif /* _PDS_CORE_ADMINQ_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (5 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 06/14] pds_core: Add adminq processing and commands Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-10 15:44 ` Simon Horman
2023-04-06 23:41 ` [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults Shannon Nelson
` (8 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Add in the support for doing firmware updates. Of the two
main banks available, a and b, this updates the one not in
use and then selects it for the next boot.
Example:
devlink dev flash pci/0000:b2:00.0 \
file pensando/dsc_fw_1.63.0-22.tar
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
.../device_drivers/ethernet/amd/pds_core.rst | 10 +
drivers/net/ethernet/amd/pds_core/Makefile | 3 +-
drivers/net/ethernet/amd/pds_core/core.h | 5 +
drivers/net/ethernet/amd/pds_core/devlink.c | 9 +
drivers/net/ethernet/amd/pds_core/fw.c | 194 ++++++++++++++++++
drivers/net/ethernet/amd/pds_core/main.c | 1 +
6 files changed, 221 insertions(+), 1 deletion(-)
create mode 100644 drivers/net/ethernet/amd/pds_core/fw.c
diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
index 265d948a8c02..6faf46390f5f 100644
--- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
+++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
@@ -73,6 +73,16 @@ The ``pds_core`` driver reports the following versions
- fixed
- The revision of the ASIC for this device
+Firmware Management
+===================
+
+The ``flash`` command can update a the DSC firmware. The downloaded firmware
+will be saved into either of firmware bank 1 or bank 2, whichever is not
+currrently in use, and that bank will used for the next boot::
+
+ # devlink dev flash pci/0000:b5:00.0 \
+ file pensando/dsc_fw_1.63.0-22.tar
+
Health Reporters
================
diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
index ef76dcd7fccd..6d1d6c58a1fa 100644
--- a/drivers/net/ethernet/amd/pds_core/Makefile
+++ b/drivers/net/ethernet/amd/pds_core/Makefile
@@ -7,6 +7,7 @@ pds_core-y := main.o \
devlink.o \
dev.o \
adminq.o \
- core.o
+ core.o \
+ fw.o
pds_core-$(CONFIG_DEBUG_FS) += debugfs.o
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index ee24833ed7ee..b0818b55e516 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -226,6 +226,9 @@ int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
struct netlink_ext_ack *extack);
int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,
struct netlink_ext_ack *extack);
+int pdsc_dl_flash_update(struct devlink *dl,
+ struct devlink_flash_update_params *params,
+ struct netlink_ext_ack *extack);
void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num);
@@ -280,4 +283,6 @@ void pdsc_process_adminq(struct pdsc_qcq *qcq);
void pdsc_work_thread(struct work_struct *work);
irqreturn_t pdsc_adminq_isr(int irq, void *data);
+int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw,
+ struct netlink_ext_ack *extack);
#endif /* _PDSC_H_ */
diff --git a/drivers/net/ethernet/amd/pds_core/devlink.c b/drivers/net/ethernet/amd/pds_core/devlink.c
index beea606c3b8d..0e98137cf69b 100644
--- a/drivers/net/ethernet/amd/pds_core/devlink.c
+++ b/drivers/net/ethernet/amd/pds_core/devlink.c
@@ -3,6 +3,15 @@
#include "core.h"
+int pdsc_dl_flash_update(struct devlink *dl,
+ struct devlink_flash_update_params *params,
+ struct netlink_ext_ack *extack)
+{
+ struct pdsc *pdsc = devlink_priv(dl);
+
+ return pdsc_firmware_update(pdsc, params->fw, extack);
+}
+
static char *fw_slotnames[] = {
"fw.goldfw",
"fw.mainfwa",
diff --git a/drivers/net/ethernet/amd/pds_core/fw.c b/drivers/net/ethernet/amd/pds_core/fw.c
new file mode 100644
index 000000000000..518e48c4ef20
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/fw.c
@@ -0,0 +1,194 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#include "core.h"
+
+/* The worst case wait for the install activity is about 25 minutes when
+ * installing a new CPLD, which is very seldom. Normal is about 30-35
+ * seconds. Since the driver can't tell if a CPLD update will happen we
+ * set the timeout for the ugly case.
+ */
+#define PDSC_FW_INSTALL_TIMEOUT (25 * 60)
+#define PDSC_FW_SELECT_TIMEOUT 30
+
+/* Number of periodic log updates during fw file download */
+#define PDSC_FW_INTERVAL_FRACTION 32
+
+static int pdsc_devcmd_fw_download_locked(struct pdsc *pdsc, u64 addr,
+ u32 offset, u32 length)
+{
+ union pds_core_dev_cmd cmd = {
+ .fw_download.opcode = PDS_CORE_CMD_FW_DOWNLOAD,
+ .fw_download.offset = cpu_to_le32(offset),
+ .fw_download.addr = cpu_to_le64(addr),
+ .fw_download.length = cpu_to_le32(length),
+ };
+ union pds_core_dev_comp comp;
+
+ return pdsc_devcmd_locked(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+}
+
+static int pdsc_devcmd_fw_install(struct pdsc *pdsc)
+{
+ union pds_core_dev_cmd cmd = {
+ .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL,
+ .fw_control.oper = PDS_CORE_FW_INSTALL_ASYNC
+ };
+ union pds_core_dev_comp comp;
+ int err;
+
+ err = pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+ if (err < 0)
+ return err;
+
+ return comp.fw_control.slot;
+}
+
+static int pdsc_devcmd_fw_activate(struct pdsc *pdsc,
+ enum pds_core_fw_slot slot)
+{
+ union pds_core_dev_cmd cmd = {
+ .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL,
+ .fw_control.oper = PDS_CORE_FW_ACTIVATE_ASYNC,
+ .fw_control.slot = slot
+ };
+ union pds_core_dev_comp comp;
+
+ return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+}
+
+static int pdsc_fw_status_long_wait(struct pdsc *pdsc,
+ const char *label,
+ unsigned long timeout,
+ u8 fw_cmd,
+ struct netlink_ext_ack *extack)
+{
+ union pds_core_dev_cmd cmd = {
+ .fw_control.opcode = PDS_CORE_CMD_FW_CONTROL,
+ .fw_control.oper = fw_cmd,
+ };
+ union pds_core_dev_comp comp;
+ unsigned long start_time;
+ unsigned long end_time;
+ int err;
+
+ /* Ping on the status of the long running async install
+ * command. We get EAGAIN while the command is still
+ * running, else we get the final command status.
+ */
+ start_time = jiffies;
+ end_time = start_time + (timeout * HZ);
+ do {
+ err = pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
+ msleep(20);
+ } while (time_before(jiffies, end_time) &&
+ (err == -EAGAIN || err == -ETIMEDOUT));
+
+ if (err == -EAGAIN || err == -ETIMEDOUT) {
+ NL_SET_ERR_MSG_MOD(extack, "Firmware wait timed out");
+ dev_err(pdsc->dev, "DEV_CMD firmware wait %s timed out\n",
+ label);
+ } else if (err) {
+ NL_SET_ERR_MSG_MOD(extack, "Firmware wait failed");
+ }
+
+ return err;
+}
+
+int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw,
+ struct netlink_ext_ack *extack)
+{
+ u32 buf_sz, copy_sz, offset;
+ struct devlink *dl;
+ int next_interval;
+ u64 data_addr;
+ int err = 0;
+ u8 fw_slot;
+
+ dev_info(pdsc->dev, "Installing firmware\n");
+
+ dl = priv_to_devlink(pdsc);
+ devlink_flash_update_status_notify(dl, "Preparing to flash",
+ NULL, 0, 0);
+
+ buf_sz = sizeof(pdsc->cmd_regs->data);
+
+ dev_dbg(pdsc->dev,
+ "downloading firmware - size %d part_sz %d nparts %lu\n",
+ (int)fw->size, buf_sz, DIV_ROUND_UP(fw->size, buf_sz));
+
+ offset = 0;
+ next_interval = 0;
+ data_addr = offsetof(struct pds_core_dev_cmd_regs, data);
+ while (offset < fw->size) {
+ if (offset >= next_interval) {
+ devlink_flash_update_status_notify(dl, "Downloading",
+ NULL, offset,
+ fw->size);
+ next_interval = offset +
+ (fw->size / PDSC_FW_INTERVAL_FRACTION);
+ }
+
+ copy_sz = min_t(unsigned int, buf_sz, fw->size - offset);
+ mutex_lock(&pdsc->devcmd_lock);
+ memcpy_toio(&pdsc->cmd_regs->data, fw->data + offset, copy_sz);
+ err = pdsc_devcmd_fw_download_locked(pdsc, data_addr,
+ offset, copy_sz);
+ mutex_unlock(&pdsc->devcmd_lock);
+ if (err) {
+ dev_err(pdsc->dev,
+ "download failed offset 0x%x addr 0x%llx len 0x%x: %pe\n",
+ offset, data_addr, copy_sz, ERR_PTR(err));
+ NL_SET_ERR_MSG_MOD(extack, "Segment download failed");
+ goto err_out;
+ }
+ offset += copy_sz;
+ }
+ devlink_flash_update_status_notify(dl, "Downloading", NULL,
+ fw->size, fw->size);
+
+ devlink_flash_update_timeout_notify(dl, "Installing", NULL,
+ PDSC_FW_INSTALL_TIMEOUT);
+
+ fw_slot = pdsc_devcmd_fw_install(pdsc);
+ if (fw_slot < 0) {
+ err = fw_slot;
+ dev_err(pdsc->dev, "install failed: %pe\n", ERR_PTR(err));
+ NL_SET_ERR_MSG_MOD(extack, "Failed to start firmware install");
+ goto err_out;
+ }
+
+ err = pdsc_fw_status_long_wait(pdsc, "Installing",
+ PDSC_FW_INSTALL_TIMEOUT,
+ PDS_CORE_FW_INSTALL_STATUS,
+ extack);
+ if (err)
+ goto err_out;
+
+ devlink_flash_update_timeout_notify(dl, "Selecting", NULL,
+ PDSC_FW_SELECT_TIMEOUT);
+
+ err = pdsc_devcmd_fw_activate(pdsc, fw_slot);
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack, "Failed to start firmware select");
+ goto err_out;
+ }
+
+ err = pdsc_fw_status_long_wait(pdsc, "Selecting",
+ PDSC_FW_SELECT_TIMEOUT,
+ PDS_CORE_FW_ACTIVATE_STATUS,
+ extack);
+ if (err)
+ goto err_out;
+
+ dev_info(pdsc->dev, "Firmware update completed, slot %d\n", fw_slot);
+
+err_out:
+ if (err)
+ devlink_flash_update_status_notify(dl, "Flash failed",
+ NULL, 0, 0);
+ else
+ devlink_flash_update_status_notify(dl, "Flash done",
+ NULL, 0, 0);
+ return err;
+}
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index c349f44b01c1..d5edce8480d7 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -242,6 +242,7 @@ static int pdsc_init_pf(struct pdsc *pdsc)
static const struct devlink_ops pdsc_dl_ops = {
.info_get = pdsc_dl_info_get,
+ .flash_update = pdsc_dl_flash_update,
};
static const struct devlink_ops pdsc_dl_vf_ops = {
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (6 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 12:08 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 09/14] pds_core: add initial VF device handling Shannon Nelson
` (7 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
The Virtual Interfaces (VIFs) supported by the DSC's
configuration (vDPA, Eth, RDMA, etc) are reported in the
dev_ident struct and made visible in debugfs. At this point
only vDPA is supported in this driver - the other interfaces
are defined for future use but not yet realized.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/core.c | 48 +++++++++++++++++++++
drivers/net/ethernet/amd/pds_core/core.h | 12 ++++++
drivers/net/ethernet/amd/pds_core/debugfs.c | 24 +++++++++++
include/linux/pds/pds_common.h | 19 ++++++++
4 files changed, 103 insertions(+)
diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
index 7c5ccff4005c..ec088d490d34 100644
--- a/drivers/net/ethernet/amd/pds_core/core.c
+++ b/drivers/net/ethernet/amd/pds_core/core.c
@@ -357,6 +357,43 @@ static int pdsc_core_init(struct pdsc *pdsc)
return err;
}
+static struct pdsc_viftype pdsc_viftype_defaults[] = {
+ [PDS_DEV_TYPE_VDPA] = { .name = PDS_DEV_TYPE_VDPA_STR,
+ .vif_id = PDS_DEV_TYPE_VDPA,
+ .dl_id = DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET },
+ [PDS_DEV_TYPE_MAX] = { 0 }
+};
+
+static int pdsc_viftypes_init(struct pdsc *pdsc)
+{
+ enum pds_core_vif_types vt;
+
+ pdsc->viftype_status = kzalloc(sizeof(pdsc_viftype_defaults),
+ GFP_KERNEL);
+ if (!pdsc->viftype_status)
+ return -ENOMEM;
+
+ for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
+ bool vt_support;
+
+ if (!pdsc_viftype_defaults[vt].name)
+ continue;
+
+ /* Grab the defaults */
+ pdsc->viftype_status[vt] = pdsc_viftype_defaults[vt];
+
+ /* See what the Core device has for support */
+ vt_support = !!le16_to_cpu(pdsc->dev_ident.vif_types[vt]);
+ dev_dbg(pdsc->dev, "VIF %s is %ssupported\n",
+ pdsc->viftype_status[vt].name,
+ vt_support ? "" : "not ");
+
+ pdsc->viftype_status[vt].supported = vt_support;
+ }
+
+ return 0;
+}
+
int pdsc_setup(struct pdsc *pdsc, bool init)
{
int numdescs;
@@ -399,6 +436,14 @@ int pdsc_setup(struct pdsc *pdsc, bool init)
if (err)
goto err_out_teardown;
+ /* Set up the VIFs */
+ err = pdsc_viftypes_init(pdsc);
+ if (err)
+ goto err_out_teardown;
+
+ if (init)
+ pdsc_debugfs_add_viftype(pdsc);
+
clear_bit(PDSC_S_FW_DEAD, &pdsc->state);
return 0;
@@ -415,6 +460,9 @@ void pdsc_teardown(struct pdsc *pdsc, bool removing)
pdsc_qcq_free(pdsc, &pdsc->notifyqcq);
pdsc_qcq_free(pdsc, &pdsc->adminqcq);
+ kfree(pdsc->viftype_status);
+ pdsc->viftype_status = NULL;
+
if (pdsc->intr_info) {
for (i = 0; i < pdsc->nintrs; i++)
pdsc_intr_free(pdsc, i);
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index b0818b55e516..0133bc45913c 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -123,6 +123,15 @@ struct pdsc_qcq {
struct dentry *dentry;
};
+struct pdsc_viftype {
+ char *name;
+ bool supported;
+ bool enabled;
+ int dl_id;
+ int vif_id;
+ struct pds_auxiliary_dev *padev;
+};
+
/* No state flags set means we are in a steady running state */
enum pdsc_state_flags {
PDSC_S_FW_DEAD, /* stopped, wait on startup or recovery */
@@ -174,6 +183,7 @@ struct pdsc {
struct pdsc_qcq adminqcq;
struct pdsc_qcq notifyqcq;
u64 last_eid;
+ struct pdsc_viftype *viftype_status;
};
/** enum pds_core_dbell_bits - bitwise composition of dbell values.
@@ -238,6 +248,7 @@ void pdsc_debugfs_destroy(void);
void pdsc_debugfs_add_dev(struct pdsc *pdsc);
void pdsc_debugfs_del_dev(struct pdsc *pdsc);
void pdsc_debugfs_add_ident(struct pdsc *pdsc);
+void pdsc_debugfs_add_viftype(struct pdsc *pdsc);
void pdsc_debugfs_add_irqs(struct pdsc *pdsc);
void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq);
void pdsc_debugfs_del_qcq(struct pdsc_qcq *qcq);
@@ -247,6 +258,7 @@ static inline void pdsc_debugfs_destroy(void) { }
static inline void pdsc_debugfs_add_dev(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_del_dev(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_add_ident(struct pdsc *pdsc) { }
+static inline void pdsc_debugfs_add_viftype(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_add_irqs(struct pdsc *pdsc) { }
static inline void pdsc_debugfs_add_qcq(struct pdsc *pdsc,
struct pdsc_qcq *qcq) { }
diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c
index 7439dc62fe6f..3482b8efd751 100644
--- a/drivers/net/ethernet/amd/pds_core/debugfs.c
+++ b/drivers/net/ethernet/amd/pds_core/debugfs.c
@@ -70,6 +70,30 @@ void pdsc_debugfs_add_ident(struct pdsc *pdsc)
pdsc, &identity_fops);
}
+static int viftype_show(struct seq_file *seq, void *v)
+{
+ struct pdsc *pdsc = seq->private;
+ int vt;
+
+ for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
+ if (!pdsc->viftype_status[vt].name)
+ continue;
+
+ seq_printf(seq, "%s\t%d supported %d enabled\n",
+ pdsc->viftype_status[vt].name,
+ pdsc->viftype_status[vt].supported,
+ pdsc->viftype_status[vt].enabled);
+ }
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(viftype);
+
+void pdsc_debugfs_add_viftype(struct pdsc *pdsc)
+{
+ debugfs_create_file("viftypes", 0400, pdsc->dentry,
+ pdsc, &viftype_fops);
+}
+
static int irqs_show(struct seq_file *seq, void *v)
{
struct pdsc *pdsc = seq->private;
diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
index e45a69178f74..350295091d9d 100644
--- a/include/linux/pds/pds_common.h
+++ b/include/linux/pds/pds_common.h
@@ -51,6 +51,25 @@ enum pds_core_driver_type {
PDS_DRIVER_ESXI = 6,
};
+enum pds_core_vif_types {
+ PDS_DEV_TYPE_CORE = 0,
+ PDS_DEV_TYPE_VDPA = 1,
+ PDS_DEV_TYPE_VFIO = 2,
+ PDS_DEV_TYPE_ETH = 3,
+ PDS_DEV_TYPE_RDMA = 4,
+ PDS_DEV_TYPE_LM = 5,
+
+ /* new ones added before this line */
+ PDS_DEV_TYPE_MAX = 16 /* don't change - used in struct size */
+};
+
+#define PDS_DEV_TYPE_CORE_STR "Core"
+#define PDS_DEV_TYPE_VDPA_STR "vDPA"
+#define PDS_DEV_TYPE_VFIO_STR "VFio"
+#define PDS_DEV_TYPE_ETH_STR "Eth"
+#define PDS_DEV_TYPE_RDMA_STR "RDMA"
+#define PDS_DEV_TYPE_LM_STR "LM"
+
#define PDS_CORE_IFNAMSIZ 16
/**
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 09/14] pds_core: add initial VF device handling
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (7 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices Shannon Nelson
` (6 subsequent siblings)
15 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
This is the initial VF PCI driver framework for the new
pds_vdpa VF device, which will work in conjunction with an
auxiliary_bus client of the pds_core driver. This does the
very basics of registering for the new VF device, setting
up debugfs entries, and registering with devlink.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/core.h | 9 +++++
drivers/net/ethernet/amd/pds_core/main.c | 48 +++++++++++++++++++++++-
2 files changed, 56 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index 0133bc45913c..5be2b986c4d9 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -30,6 +30,12 @@ struct pdsc_dev_bar {
int res_index;
};
+struct pdsc_vf {
+ struct pds_auxiliary_dev *padev;
+ u16 index;
+ __le16 vif_types[PDS_DEV_TYPE_MAX];
+};
+
struct pdsc_devinfo {
u8 asic_type;
u8 asic_rev;
@@ -147,6 +153,9 @@ struct pdsc {
struct dentry *dentry;
struct device *dev;
struct pdsc_dev_bar bars[PDS_CORE_BARS_MAX];
+ struct pdsc_vf *vfs;
+ int num_vfs;
+ int vf_id;
int hw_index;
int uid;
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index d5edce8480d7..5bda66d2a0df 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -16,6 +16,7 @@ MODULE_LICENSE("GPL");
/* Supported devices */
static const struct pci_device_id pdsc_id_table[] = {
{ PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF) },
+ { PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_VDPA_VF) },
{ 0, } /* end of table */
};
MODULE_DEVICE_TABLE(pci, pdsc_id_table);
@@ -143,9 +144,51 @@ void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num)
(u64)page_num << PAGE_SHIFT, PAGE_SIZE);
}
+static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)
+{
+ struct pdsc *pdsc = pci_get_drvdata(pdev);
+ struct device *dev = pdsc->dev;
+ int ret = 0;
+
+ if (num_vfs > 0) {
+ pdsc->vfs = kcalloc(num_vfs, sizeof(struct pdsc_vf),
+ GFP_KERNEL);
+ if (!pdsc->vfs)
+ return -ENOMEM;
+ pdsc->num_vfs = num_vfs;
+
+ ret = pci_enable_sriov(pdev, num_vfs);
+ if (ret) {
+ dev_err(dev, "Cannot enable SRIOV: %pe\n",
+ ERR_PTR(ret));
+ goto no_vfs;
+ }
+
+ return num_vfs;
+ }
+
+no_vfs:
+ pci_disable_sriov(pdev);
+
+ kfree(pdsc->vfs);
+ pdsc->vfs = NULL;
+ pdsc->num_vfs = 0;
+
+ return ret;
+}
+
static int pdsc_init_vf(struct pdsc *vf)
{
- return -1;
+ struct devlink *dl;
+
+ vf->vf_id = pci_iov_vf_id(vf->pdev);
+
+ dl = priv_to_devlink(vf);
+ devl_lock(dl);
+ devl_register(dl);
+ devl_unlock(dl);
+
+ return 0;
}
static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {
@@ -337,6 +380,8 @@ static void pdsc_remove(struct pci_dev *pdev)
devl_unlock(dl);
if (!pdev->is_virtfn) {
+ pdsc_sriov_configure(pdev, 0);
+
del_timer_sync(&pdsc->wdtimer);
if (pdsc->wq) {
flush_workqueue(pdsc->wq);
@@ -371,6 +416,7 @@ static struct pci_driver pdsc_driver = {
.id_table = pdsc_id_table,
.probe = pdsc_probe,
.remove = pdsc_remove,
+ .sriov_configure = pdsc_sriov_configure,
};
static int __init pdsc_init_module(void)
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (8 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 09/14] pds_core: add initial VF device handling Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 12:23 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 11/14] pds_core: devlink params for enabling VIF support Shannon Nelson
` (5 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
An auxiliary_bus device is created for each vDPA type VF at VF
probe and destroyed at VF remove. The aux device name comes
from the driver name + VIF type + the unique id assigned at PCI
probe. The VFs are always removed on PF remove, so there should
be no issues with VFs trying to access missing PF structures.
The auxiliary_device names will look like "pds_core.vDPA.nn"
where 'nn' is the VF's uid.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/Makefile | 1 +
drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++
drivers/net/ethernet/amd/pds_core/core.h | 6 ++
drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++-
include/linux/pds/pds_auxbus.h | 16 +++
include/linux/pds/pds_common.h | 1 +
6 files changed, 170 insertions(+), 2 deletions(-)
create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c
create mode 100644 include/linux/pds/pds_auxbus.h
diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
index 6d1d6c58a1fa..0abc33ce826c 100644
--- a/drivers/net/ethernet/amd/pds_core/Makefile
+++ b/drivers/net/ethernet/amd/pds_core/Makefile
@@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o
pds_core-y := main.o \
devlink.o \
+ auxbus.o \
dev.o \
adminq.o \
core.o \
diff --git a/drivers/net/ethernet/amd/pds_core/auxbus.c b/drivers/net/ethernet/amd/pds_core/auxbus.c
new file mode 100644
index 000000000000..6757a5174eb7
--- /dev/null
+++ b/drivers/net/ethernet/amd/pds_core/auxbus.c
@@ -0,0 +1,112 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#include <linux/pci.h>
+
+#include "core.h"
+#include <linux/pds/pds_auxbus.h>
+
+static void pdsc_auxbus_dev_release(struct device *dev)
+{
+ struct pds_auxiliary_dev *padev =
+ container_of(dev, struct pds_auxiliary_dev, aux_dev.dev);
+
+ kfree(padev);
+}
+
+static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *vf,
+ struct pdsc *pf,
+ char *name)
+{
+ struct auxiliary_device *aux_dev;
+ struct pds_auxiliary_dev *padev;
+ int err;
+
+ padev = kzalloc(sizeof(*padev), GFP_KERNEL);
+ if (!padev)
+ return ERR_PTR(-ENOMEM);
+
+ padev->vf_pdev = vf->pdev;
+ padev->pf_pdev = pf->pdev;
+
+ aux_dev = &padev->aux_dev;
+ aux_dev->name = name;
+ aux_dev->id = vf->uid;
+ aux_dev->dev.parent = vf->dev;
+ aux_dev->dev.release = pdsc_auxbus_dev_release;
+
+ err = auxiliary_device_init(aux_dev);
+ if (err < 0) {
+ dev_warn(vf->dev, "auxiliary_device_init of %s failed: %pe\n",
+ name, ERR_PTR(err));
+ goto err_out;
+ }
+
+ err = auxiliary_device_add(aux_dev);
+ if (err) {
+ dev_warn(vf->dev, "auxiliary_device_add of %s failed: %pe\n",
+ name, ERR_PTR(err));
+ goto err_out_uninit;
+ }
+
+ return padev;
+
+err_out_uninit:
+ auxiliary_device_uninit(aux_dev);
+err_out:
+ kfree(padev);
+ return ERR_PTR(err);
+}
+
+int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf)
+{
+ struct pds_auxiliary_dev *padev;
+ int err = 0;
+
+ mutex_lock(&pf->config_lock);
+
+ padev = pf->vfs[vf->vf_id].padev;
+ if (padev) {
+ auxiliary_device_delete(&padev->aux_dev);
+ auxiliary_device_uninit(&padev->aux_dev);
+ }
+ pf->vfs[vf->vf_id].padev = NULL;
+
+ mutex_unlock(&pf->config_lock);
+ return err;
+}
+
+int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf)
+{
+ struct pds_auxiliary_dev *padev;
+ enum pds_core_vif_types vt;
+ int err = 0;
+
+ mutex_lock(&pf->config_lock);
+
+ for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
+ u16 vt_support;
+
+ /* Verify that the type is supported and enabled */
+ vt_support = !!le16_to_cpu(pf->dev_ident.vif_types[vt]);
+ if (!(vt_support &&
+ pf->viftype_status[vt].supported &&
+ pf->viftype_status[vt].enabled))
+ continue;
+
+ padev = pdsc_auxbus_dev_register(vf, pf,
+ pf->viftype_status[vt].name);
+ if (IS_ERR(padev)) {
+ err = PTR_ERR(padev);
+ goto out_unlock;
+ }
+ pf->vfs[vf->vf_id].padev = padev;
+
+ /* We only support a single type per VF, so jump out here */
+ break;
+ }
+
+out_unlock:
+ mutex_unlock(&pf->config_lock);
+ return err;
+}
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index 5be2b986c4d9..16b20bd705e4 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -30,8 +30,11 @@ struct pdsc_dev_bar {
int res_index;
};
+struct pdsc;
+
struct pdsc_vf {
struct pds_auxiliary_dev *padev;
+ struct pdsc *vf;
u16 index;
__le16 vif_types[PDS_DEV_TYPE_MAX];
};
@@ -300,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);
void pdsc_stop(struct pdsc *pdsc);
void pdsc_health_thread(struct work_struct *work);
+int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf);
+int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf);
+
void pdsc_process_adminq(struct pdsc_qcq *qcq);
void pdsc_work_thread(struct work_struct *work);
irqreturn_t pdsc_adminq_isr(int irq, void *data);
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index 5bda66d2a0df..16a2d8a048a3 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -180,6 +180,12 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)
static int pdsc_init_vf(struct pdsc *vf)
{
struct devlink *dl;
+ struct pdsc *pf;
+ int err;
+
+ pf = pdsc_get_pf_struct(vf->pdev);
+ if (IS_ERR_OR_NULL(pf))
+ return PTR_ERR(pf) ?: -1;
vf->vf_id = pci_iov_vf_id(vf->pdev);
@@ -188,7 +194,15 @@ static int pdsc_init_vf(struct pdsc *vf)
devl_register(dl);
devl_unlock(dl);
- return 0;
+ pf->vfs[vf->vf_id].vf = vf;
+ err = pdsc_auxbus_dev_add_vf(vf, pf);
+ if (err) {
+ devl_lock(dl);
+ devl_unregister(dl);
+ devl_unlock(dl);
+ }
+
+ return err;
}
static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {
@@ -379,7 +393,19 @@ static void pdsc_remove(struct pci_dev *pdev)
}
devl_unlock(dl);
- if (!pdev->is_virtfn) {
+ if (pdev->is_virtfn) {
+ struct pdsc *pf;
+
+ pf = pdsc_get_pf_struct(pdsc->pdev);
+ if (!IS_ERR(pf)) {
+ pdsc_auxbus_dev_del_vf(pdsc, pf);
+ pf->vfs[pdsc->vf_id].vf = NULL;
+ }
+ } else {
+ /* Remove the VFs and their aux_bus connections before other
+ * cleanup so that the clients can use the AdminQ to cleanly
+ * shut themselves down.
+ */
pdsc_sriov_configure(pdev, 0);
del_timer_sync(&pdsc->wdtimer);
@@ -419,6 +445,12 @@ static struct pci_driver pdsc_driver = {
.sriov_configure = pdsc_sriov_configure,
};
+void *pdsc_get_pf_struct(struct pci_dev *vf_pdev)
+{
+ return pci_iov_get_pf_drvdata(vf_pdev, &pdsc_driver);
+}
+EXPORT_SYMBOL_GPL(pdsc_get_pf_struct);
+
static int __init pdsc_init_module(void)
{
pdsc_debugfs_create();
diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h
new file mode 100644
index 000000000000..aa0192af4a29
--- /dev/null
+++ b/include/linux/pds/pds_auxbus.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2023 Advanced Micro Devices, Inc */
+
+#ifndef _PDSC_AUXBUS_H_
+#define _PDSC_AUXBUS_H_
+
+#include <linux/auxiliary_bus.h>
+
+struct pds_auxiliary_dev {
+ struct auxiliary_device aux_dev;
+ struct pci_dev *vf_pdev;
+ struct pci_dev *pf_pdev;
+ u16 client_id;
+ void *priv;
+};
+#endif /* _PDSC_AUXBUS_H_ */
diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
index 350295091d9d..898f3c7b14b7 100644
--- a/include/linux/pds/pds_common.h
+++ b/include/linux/pds/pds_common.h
@@ -91,4 +91,5 @@ enum pds_core_logical_qtype {
PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */
};
+void *pdsc_get_pf_struct(struct pci_dev *vf_pdev);
#endif /* _PDS_COMMON_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 11/14] pds_core: devlink params for enabling VIF support
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (9 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 12/14] pds_core: add the aux client API Shannon Nelson
` (4 subsequent siblings)
15 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Add the devlink parameter switches so the user can enable
the features supported by the VFs. The only feature supported
at the moment is vDPA.
Example:
devlink dev param set pci/0000:2b:00.0 \
name enable_vnet cmode runtime value true
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
.../device_drivers/ethernet/amd/pds_core.rst | 19 +++++
drivers/net/ethernet/amd/pds_core/core.h | 7 ++
drivers/net/ethernet/amd/pds_core/devlink.c | 73 +++++++++++++++++++
drivers/net/ethernet/amd/pds_core/main.c | 34 +++++++--
4 files changed, 127 insertions(+), 6 deletions(-)
diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
index 6faf46390f5f..9449451b538f 100644
--- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
+++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
@@ -73,6 +73,25 @@ The ``pds_core`` driver reports the following versions
- fixed
- The revision of the ASIC for this device
+Parameters
+==========
+
+The ``pds_core`` driver implements the following generic
+parameters for controlling the functionality to be made available
+as auxiliary_bus devices.
+
+.. list-table:: Generic parameters implemented
+ :widths: 5 5 8 82
+
+ * - Name
+ - Mode
+ - Type
+ - Description
+ * - ``enable_vnet``
+ - runtime
+ - Boolean
+ - Enables vDPA functionality through an auxiliary_bus device
+
Firmware Management
===================
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index 16b20bd705e4..aab4986007b9 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -251,6 +251,13 @@ int pdsc_dl_info_get(struct devlink *dl, struct devlink_info_req *req,
int pdsc_dl_flash_update(struct devlink *dl,
struct devlink_flash_update_params *params,
struct netlink_ext_ack *extack);
+int pdsc_dl_enable_get(struct devlink *dl, u32 id,
+ struct devlink_param_gset_ctx *ctx);
+int pdsc_dl_enable_set(struct devlink *dl, u32 id,
+ struct devlink_param_gset_ctx *ctx);
+int pdsc_dl_enable_validate(struct devlink *dl, u32 id,
+ union devlink_param_value val,
+ struct netlink_ext_ack *extack);
void __iomem *pdsc_map_dbpage(struct pdsc *pdsc, int page_num);
diff --git a/drivers/net/ethernet/amd/pds_core/devlink.c b/drivers/net/ethernet/amd/pds_core/devlink.c
index 0e98137cf69b..550ae52eee85 100644
--- a/drivers/net/ethernet/amd/pds_core/devlink.c
+++ b/drivers/net/ethernet/amd/pds_core/devlink.c
@@ -2,6 +2,79 @@
/* Copyright(c) 2023 Advanced Micro Devices, Inc */
#include "core.h"
+#include <linux/pds/pds_auxbus.h>
+
+static struct
+pdsc_viftype *pdsc_dl_find_viftype_by_id(struct pdsc *pdsc,
+ enum devlink_param_type dl_id)
+{
+ int vt;
+
+ for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
+ if (pdsc->viftype_status[vt].dl_id == dl_id)
+ return &pdsc->viftype_status[vt];
+ }
+
+ return NULL;
+}
+
+int pdsc_dl_enable_get(struct devlink *dl, u32 id,
+ struct devlink_param_gset_ctx *ctx)
+{
+ struct pdsc *pdsc = devlink_priv(dl);
+ struct pdsc_viftype *vt_entry;
+
+ vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id);
+ if (!vt_entry)
+ return -ENOENT;
+
+ ctx->val.vbool = vt_entry->enabled;
+
+ return 0;
+}
+
+int pdsc_dl_enable_set(struct devlink *dl, u32 id,
+ struct devlink_param_gset_ctx *ctx)
+{
+ struct pdsc *pdsc = devlink_priv(dl);
+ struct pdsc_viftype *vt_entry;
+ int err = 0;
+ int vf_id;
+
+ vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id);
+ if (!vt_entry || !vt_entry->supported)
+ return -EOPNOTSUPP;
+
+ if (vt_entry->enabled == ctx->val.vbool)
+ return 0;
+
+ vt_entry->enabled = ctx->val.vbool;
+ for (vf_id = 0; vf_id < pdsc->num_vfs; vf_id++) {
+ struct pdsc *vf = pdsc->vfs[vf_id].vf;
+
+ err = ctx->val.vbool ? pdsc_auxbus_dev_add_vf(vf, pdsc) :
+ pdsc_auxbus_dev_del_vf(vf, pdsc);
+ }
+
+ return err;
+}
+
+int pdsc_dl_enable_validate(struct devlink *dl, u32 id,
+ union devlink_param_value val,
+ struct netlink_ext_ack *extack)
+{
+ struct pdsc *pdsc = devlink_priv(dl);
+ struct pdsc_viftype *vt_entry;
+
+ vt_entry = pdsc_dl_find_viftype_by_id(pdsc, id);
+ if (!vt_entry || !vt_entry->supported)
+ return -EOPNOTSUPP;
+
+ if (!pdsc->viftype_status[vt_entry->vif_id].supported)
+ return -ENODEV;
+
+ return 0;
+}
int pdsc_dl_flash_update(struct devlink *dl,
struct devlink_flash_update_params *params,
diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
index 16a2d8a048a3..38c5e902b200 100644
--- a/drivers/net/ethernet/amd/pds_core/main.c
+++ b/drivers/net/ethernet/amd/pds_core/main.c
@@ -210,6 +210,14 @@ static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {
.diagnose = pdsc_fw_reporter_diagnose,
};
+static const struct devlink_param pdsc_dl_params[] = {
+ DEVLINK_PARAM_GENERIC(ENABLE_VNET,
+ BIT(DEVLINK_PARAM_CMODE_RUNTIME),
+ pdsc_dl_enable_get,
+ pdsc_dl_enable_set,
+ pdsc_dl_enable_validate),
+};
+
#define PDSC_WQ_NAME_LEN 24
static int pdsc_init_pf(struct pdsc *pdsc)
@@ -257,13 +265,19 @@ static int pdsc_init_pf(struct pdsc *pdsc)
dl = priv_to_devlink(pdsc);
devl_lock(dl);
+ err = devl_params_register(dl, pdsc_dl_params,
+ ARRAY_SIZE(pdsc_dl_params));
+ if (err) {
+ dev_warn(pdsc->dev, "Failed to register devlink params: %pe\n",
+ ERR_PTR(err));
+ goto err_out_unlock_dl;
+ }
hr = devl_health_reporter_create(dl, &pdsc_fw_reporter_ops, 0, pdsc);
if (IS_ERR(hr)) {
dev_warn(pdsc->dev, "Failed to create fw reporter: %pe\n", hr);
err = PTR_ERR(hr);
- devl_unlock(dl);
- goto err_out_stop;
+ goto err_out_unreg_params;
}
pdsc->fw_reporter = hr;
@@ -275,7 +289,11 @@ static int pdsc_init_pf(struct pdsc *pdsc)
return 0;
-err_out_stop:
+err_out_unreg_params:
+ devl_params_unregister(dl, pdsc_dl_params,
+ ARRAY_SIZE(pdsc_dl_params));
+err_out_unlock_dl:
+ devl_unlock(dl);
pdsc_stop(pdsc);
err_out_teardown:
pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
@@ -387,9 +405,13 @@ static void pdsc_remove(struct pci_dev *pdev)
dl = priv_to_devlink(pdsc);
devl_lock(dl);
devl_unregister(dl);
- if (pdsc->fw_reporter) {
- devl_health_reporter_destroy(pdsc->fw_reporter);
- pdsc->fw_reporter = NULL;
+ if (!pdev->is_virtfn) {
+ if (pdsc->fw_reporter) {
+ devl_health_reporter_destroy(pdsc->fw_reporter);
+ pdsc->fw_reporter = NULL;
+ }
+ devl_params_unregister(dl, pdsc_dl_params,
+ ARRAY_SIZE(pdsc_dl_params));
}
devl_unlock(dl);
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 12/14] pds_core: add the aux client API
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (10 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 11/14] pds_core: devlink params for enabling VIF support Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 17:07 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 13/14] pds_core: publish events to the clients Shannon Nelson
` (3 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Add the client API operations for running adminq commands.
The core registers the client with the FW, then the client
has a context for requesting adminq services. We expect
to add additional operations for other clients, including
requesting additional private adminqs and IRQs, but don't have
the need yet.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++-
include/linux/pds/pds_auxbus.h | 28 +++++
2 files changed, 160 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/amd/pds_core/auxbus.c b/drivers/net/ethernet/amd/pds_core/auxbus.c
index 6757a5174eb7..c8c5cc5c9ca6 100644
--- a/drivers/net/ethernet/amd/pds_core/auxbus.c
+++ b/drivers/net/ethernet/amd/pds_core/auxbus.c
@@ -6,6 +6,115 @@
#include "core.h"
#include <linux/pds/pds_auxbus.h>
+static int pds_client_register(struct pdsc *vf, struct pdsc *pf, char *name)
+{
+ union pds_core_adminq_comp comp = { 0 };
+ union pds_core_adminq_cmd cmd = { 0 };
+ int err;
+ u16 ci;
+
+ if (pf->state)
+ return -ENXIO;
+
+ cmd.client_reg.opcode = PDS_AQ_CMD_CLIENT_REG;
+ snprintf(cmd.client_reg.devname, sizeof(cmd.client_reg.devname),
+ "%s.%s.%d", PDS_CORE_DRV_NAME, name, vf->uid);
+
+ err = pdsc_adminq_post(pf, &cmd, &comp, false);
+ if (err) {
+ dev_info(pf->dev, "register dev_name %s with DSC failed, status %d: %pe\n",
+ name, comp.status, ERR_PTR(err));
+ return err;
+ }
+
+ ci = le16_to_cpu(comp.client_reg.client_id);
+ if (!ci) {
+ dev_err(pf->dev, "%s: device returned null client_id\n",
+ __func__);
+ return -EIO;
+ }
+
+ dev_dbg(pf->dev, "%s: device returned client_id %d for %s\n",
+ __func__, ci, cmd.client_reg.devname);
+
+ return ci;
+}
+
+static int pds_client_unregister(struct pdsc *pf, u16 client_id)
+{
+ union pds_core_adminq_comp comp = { 0 };
+ union pds_core_adminq_cmd cmd = { 0 };
+ int err;
+
+ if (pf->state)
+ return -ENXIO;
+
+ cmd.client_unreg.opcode = PDS_AQ_CMD_CLIENT_UNREG;
+ cmd.client_unreg.client_id = cpu_to_le16(client_id);
+
+ err = pdsc_adminq_post(pf, &cmd, &comp, false);
+ if (err)
+ dev_info(pf->dev, "unregister client_id %d failed, status %d: %pe\n",
+ client_id, comp.status, ERR_PTR(err));
+
+ return err;
+}
+
+/**
+ * pds_client_adminq_cmd - Process an adminq request for the client
+ * @padev: ptr to the client device
+ * @req: ptr to buffer with request
+ * @req_len: length of actual struct used for request
+ * @resp: ptr to buffer where answer is to be copied
+ * @flags: optional flags from pds_core_adminq_flags
+ *
+ * Return: 0 on success, or
+ * negative for error
+ *
+ * Client sends pointers to request and response buffers
+ * Core copies request data into pds_core_client_request_cmd
+ * Core sets other fields as needed
+ * Core posts to AdminQ
+ * Core copies completion data into response buffer
+ */
+static int pds_client_adminq_cmd(struct pds_auxiliary_dev *padev,
+ union pds_core_adminq_cmd *req,
+ size_t req_len,
+ union pds_core_adminq_comp *resp,
+ u64 flags)
+{
+ union pds_core_adminq_cmd cmd = { 0 };
+ struct pdsc *pf;
+ size_t cp_len;
+ int err;
+
+ pf = pci_get_drvdata(padev->pf_pdev);
+
+ dev_dbg(pf->dev, "%s: %s opcode %d\n",
+ __func__, dev_name(&padev->aux_dev.dev), req->opcode);
+
+ if (pf->state)
+ return -ENXIO;
+
+ /* Wrap the client's request */
+ cmd.client_request.opcode = PDS_AQ_CMD_CLIENT_CMD;
+ cmd.client_request.client_id = cpu_to_le16(padev->client_id);
+ cp_len = min_t(size_t, req_len, sizeof(cmd.client_request.client_cmd));
+ memcpy(cmd.client_request.client_cmd, req, cp_len);
+
+ err = pdsc_adminq_post(pf, &cmd, resp,
+ !!(flags & PDS_AQ_FLAG_FASTPOLL));
+ if (err && err != -EAGAIN)
+ dev_info(pf->dev, "client admin cmd failed: %pe\n",
+ ERR_PTR(err));
+
+ return err;
+}
+
+static struct pds_core_ops pds_core_ops = {
+ .adminq_cmd = pds_client_adminq_cmd,
+};
+
static void pdsc_auxbus_dev_release(struct device *dev)
{
struct pds_auxiliary_dev *padev =
@@ -16,7 +125,9 @@ static void pdsc_auxbus_dev_release(struct device *dev)
static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *vf,
struct pdsc *pf,
- char *name)
+ u16 client_id,
+ char *name,
+ struct pds_core_ops *ops)
{
struct auxiliary_device *aux_dev;
struct pds_auxiliary_dev *padev;
@@ -28,6 +139,8 @@ static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *vf,
padev->vf_pdev = vf->pdev;
padev->pf_pdev = pf->pdev;
+ padev->ops = ops;
+ padev->client_id = client_id;
aux_dev = &padev->aux_dev;
aux_dev->name = name;
@@ -67,8 +180,10 @@ int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf)
padev = pf->vfs[vf->vf_id].padev;
if (padev) {
+ pds_client_unregister(pf, padev->client_id);
auxiliary_device_delete(&padev->aux_dev);
auxiliary_device_uninit(&padev->aux_dev);
+ padev->client_id = 0;
}
pf->vfs[vf->vf_id].padev = NULL;
@@ -80,6 +195,7 @@ int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf)
{
struct pds_auxiliary_dev *padev;
enum pds_core_vif_types vt;
+ int client_id;
int err = 0;
mutex_lock(&pf->config_lock);
@@ -94,9 +210,22 @@ int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf)
pf->viftype_status[vt].enabled))
continue;
- padev = pdsc_auxbus_dev_register(vf, pf,
- pf->viftype_status[vt].name);
+ /* need to register with FW and get the client_id before
+ * creating the aux device so that the aux client can run
+ * adminq commands as part its probe
+ */
+ client_id = pds_client_register(vf, pf,
+ pf->viftype_status[vt].name);
+ if (client_id < 0) {
+ err = client_id;
+ goto out_unlock;
+ }
+
+ padev = pdsc_auxbus_dev_register(vf, pf, client_id,
+ pf->viftype_status[vt].name,
+ &pds_core_ops);
if (IS_ERR(padev)) {
+ pds_client_unregister(pf, client_id);
err = PTR_ERR(padev);
goto out_unlock;
}
diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h
index aa0192af4a29..f98efd578e1c 100644
--- a/include/linux/pds/pds_auxbus.h
+++ b/include/linux/pds/pds_auxbus.h
@@ -10,7 +10,35 @@ struct pds_auxiliary_dev {
struct auxiliary_device aux_dev;
struct pci_dev *vf_pdev;
struct pci_dev *pf_pdev;
+ struct pds_core_ops *ops;
u16 client_id;
void *priv;
};
+
+/*
+ * ptrs to functions to be used by the client for core services
+ */
+struct pds_core_ops {
+ /* .adminq_cmd() - process an adminq request for the client
+ * padev: ptr to the client device
+ * req: ptr to buffer with request
+ * req_len: length of actual struct used for request
+ * resp: ptr to buffer where answer is to be copied
+ * flags: optional flags defined by enum pds_core_adminq_flags
+ * and used for more flexible adminq behvior
+ *
+ * returns 0 on success, or
+ * negative for error
+ * Client sends pointers to request and response buffers
+ * Core copies request data into pds_core_client_request_cmd
+ * Core sets other fields as needed
+ * Core posts to AdminQ
+ * Core copies completion data into response buffer
+ */
+ int (*adminq_cmd)(struct pds_auxiliary_dev *padev,
+ union pds_core_adminq_cmd *req,
+ size_t req_len,
+ union pds_core_adminq_comp *resp,
+ u64 flags);
+};
#endif /* _PDSC_AUXBUS_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (11 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 12/14] pds_core: add the aux client API Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 17:11 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst Shannon Nelson
` (2 subsequent siblings)
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
When the Core device gets an event from the device, or notices
the device FW to be up or down, it needs to send those events
on to the clients that have an event handler. Add the code to
pass along the events to the clients.
The entry points pdsc_register_notify() and pdsc_unregister_notify()
are EXPORTed for other drivers that want to listen for these events.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++
drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++
drivers/net/ethernet/amd/pds_core/core.h | 3 ++
include/linux/pds/pds_common.h | 2 ++
4 files changed, 39 insertions(+)
diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c
index 25c7dd0d37e5..bb18ac1aabab 100644
--- a/drivers/net/ethernet/amd/pds_core/adminq.c
+++ b/drivers/net/ethernet/amd/pds_core/adminq.c
@@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
case PDS_EVENT_LINK_CHANGE:
dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
ecode, eid);
+ pdsc_notify(PDS_EVENT_LINK_CHANGE, comp);
break;
case PDS_EVENT_RESET:
dev_info(pdsc->dev, "NotifyQ RESET ecode %d eid %lld\n",
ecode, eid);
+ pdsc_notify(PDS_EVENT_RESET, comp);
break;
case PDS_EVENT_XCVR:
diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
index ec088d490d34..b2790be0fc46 100644
--- a/drivers/net/ethernet/amd/pds_core/core.c
+++ b/drivers/net/ethernet/amd/pds_core/core.c
@@ -6,6 +6,25 @@
#include "core.h"
+static BLOCKING_NOTIFIER_HEAD(pds_notify_chain);
+
+int pdsc_register_notify(struct notifier_block *nb)
+{
+ return blocking_notifier_chain_register(&pds_notify_chain, nb);
+}
+EXPORT_SYMBOL_GPL(pdsc_register_notify);
+
+void pdsc_unregister_notify(struct notifier_block *nb)
+{
+ blocking_notifier_chain_unregister(&pds_notify_chain, nb);
+}
+EXPORT_SYMBOL_GPL(pdsc_unregister_notify);
+
+void pdsc_notify(unsigned long event, void *data)
+{
+ blocking_notifier_call_chain(&pds_notify_chain, event, data);
+}
+
void pdsc_intr_free(struct pdsc *pdsc, int index)
{
struct pdsc_intr_info *intr_info;
@@ -513,12 +532,19 @@ void pdsc_stop(struct pdsc *pdsc)
static void pdsc_fw_down(struct pdsc *pdsc)
{
+ union pds_core_notifyq_comp reset_event = {
+ .reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
+ .reset.state = 0,
+ };
+
if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
dev_err(pdsc->dev, "%s: already happening\n", __func__);
return;
}
+ /* Notify clients of fw_down */
devlink_health_report(pdsc->fw_reporter, "FW down reported", pdsc);
+ pdsc_notify(PDS_EVENT_RESET, &reset_event);
pdsc_mask_interrupts(pdsc);
pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
@@ -526,6 +552,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)
static void pdsc_fw_up(struct pdsc *pdsc)
{
+ union pds_core_notifyq_comp reset_event = {
+ .reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
+ .reset.state = 1,
+ };
int err;
if (!test_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
@@ -541,9 +571,11 @@ static void pdsc_fw_up(struct pdsc *pdsc)
if (err)
goto err_out;
+ /* Notify clients of fw_up */
pdsc->fw_recoveries++;
devlink_health_reporter_state_update(pdsc->fw_reporter,
DEVLINK_HEALTH_REPORTER_STATE_HEALTHY);
+ pdsc_notify(PDS_EVENT_RESET, &reset_event);
return;
diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
index aab4986007b9..2215e4915e6a 100644
--- a/drivers/net/ethernet/amd/pds_core/core.h
+++ b/drivers/net/ethernet/amd/pds_core/core.h
@@ -310,6 +310,9 @@ int pdsc_start(struct pdsc *pdsc);
void pdsc_stop(struct pdsc *pdsc);
void pdsc_health_thread(struct work_struct *work);
+int pdsc_register_notify(struct notifier_block *nb);
+void pdsc_unregister_notify(struct notifier_block *nb);
+void pdsc_notify(unsigned long event, void *data);
int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf);
int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf);
diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
index 898f3c7b14b7..17708a142349 100644
--- a/include/linux/pds/pds_common.h
+++ b/include/linux/pds/pds_common.h
@@ -91,5 +91,7 @@ enum pds_core_logical_qtype {
PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */
};
+int pdsc_register_notify(struct notifier_block *nb);
+void pdsc_unregister_notify(struct notifier_block *nb);
void *pdsc_get_pf_struct(struct pci_dev *vf_pdev);
#endif /* _PDS_COMMON_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (12 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 13/14] pds_core: publish events to the clients Shannon Nelson
@ 2023-04-06 23:41 ` Shannon Nelson
2023-04-09 17:17 ` Leon Romanovsky
2023-04-08 3:18 ` [PATCH v9 net-next 00/14] pds_core driver Jakub Kicinski
2023-04-10 20:00 ` Alex Williamson
15 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-06 23:41 UTC (permalink / raw)
To: shannon.nelson, brett.creeley, davem, netdev, kuba; +Cc: drivers, leon, jiri
Documentation and Kconfig hook for building the driver.
Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
---
.../device_drivers/ethernet/amd/pds_core.rst | 16 ++++++++++++++++
MAINTAINERS | 9 +++++++++
drivers/net/ethernet/amd/Kconfig | 12 ++++++++++++
drivers/net/ethernet/amd/Makefile | 1 +
4 files changed, 38 insertions(+)
diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
index 9449451b538f..c5ef20f361da 100644
--- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
+++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
@@ -114,6 +114,22 @@ The driver supports a devlink health reporter for FW status::
# devlink health diagnose pci/0000:2b:00.0 reporter fw
Status: healthy State: 1 Generation: 0 Recoveries: 0
+Enabling the driver
+===================
+
+The driver is enabled via the standard kernel configuration system,
+using the make command::
+
+ make oldconfig/menuconfig/etc.
+
+The driver is located in the menu structure at:
+
+ -> Device Drivers
+ -> Network device support (NETDEVICES [=y])
+ -> Ethernet driver support
+ -> AMD devices
+ -> AMD/Pensando Ethernet PDS_CORE Support
+
Support
=======
diff --git a/MAINTAINERS b/MAINTAINERS
index 30ca644d704f..95b5f25a2c06 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1041,6 +1041,15 @@ F: drivers/gpu/drm/amd/include/vi_structs.h
F: include/uapi/linux/kfd_ioctl.h
F: include/uapi/linux/kfd_sysfs.h
+AMD PDS CORE DRIVER
+M: Shannon Nelson <shannon.nelson@amd.com>
+M: Brett Creeley <brett.creeley@amd.com>
+M: drivers@pensando.io
+L: netdev@vger.kernel.org
+S: Supported
+F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
+F: drivers/net/ethernet/amd/pds_core/
+
AMD SPI DRIVER
M: Sanjay R Mehta <sanju.mehta@amd.com>
S: Maintained
diff --git a/drivers/net/ethernet/amd/Kconfig b/drivers/net/ethernet/amd/Kconfig
index ab42f75b9413..235fcacef5c5 100644
--- a/drivers/net/ethernet/amd/Kconfig
+++ b/drivers/net/ethernet/amd/Kconfig
@@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC
bool
default n
+config PDS_CORE
+ tristate "AMD/Pensando Data Systems Core Device Support"
+ depends on 64BIT && PCI
+ help
+ This enables the support for the AMD/Pensando Core device family of
+ adapters. More specific information on this driver can be
+ found in
+ <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>.
+
+ To compile this driver as a module, choose M here. The module
+ will be called pds_core.
+
endif # NET_VENDOR_AMD
diff --git a/drivers/net/ethernet/amd/Makefile b/drivers/net/ethernet/amd/Makefile
index 42742afe9115..2dcfb84731e1 100644
--- a/drivers/net/ethernet/amd/Makefile
+++ b/drivers/net/ethernet/amd/Makefile
@@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o
obj-$(CONFIG_SUN3LANCE) += sun3lance.o
obj-$(CONFIG_SUNLANCE) += sunlance.o
obj-$(CONFIG_AMD_XGBE) += xgbe/
+obj-$(CONFIG_PDS_CORE) += pds_core/
--
2.17.1
^ permalink raw reply related [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 00/14] pds_core driver
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (13 preceding siblings ...)
2023-04-06 23:41 ` [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst Shannon Nelson
@ 2023-04-08 3:18 ` Jakub Kicinski
2023-04-10 20:00 ` Alex Williamson
15 siblings, 0 replies; 54+ messages in thread
From: Jakub Kicinski @ 2023-04-08 3:18 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, drivers, leon, jiri
On Thu, 6 Apr 2023 16:41:29 -0700 Shannon Nelson wrote:
> This patchset implements a new driver for use with the AMD/Pensando
> Distributed Services Card (DSC), intended to provide core configuration
> services through the auxiliary_bus and through a couple of EXPORTed
> functions for use initially in VFio and vDPA feature specific drivers.
>
> To keep this patchset to a manageable size, the pds_vdpa and pds_vfio
> drivers have been split out into their own patchsets to be reviewed
> separately.
Acked-by: Jakub Kicinski <kuba@kernel.org>
Let's hear from auxdev folks, tho.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver
2023-04-06 23:41 ` [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver Shannon Nelson
@ 2023-04-09 11:26 ` Leon Romanovsky
2023-04-10 18:41 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 11:26 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:30PM -0700, Shannon Nelson wrote:
> This is the initial PCI driver framework for the new pds_core device
> driver and its family of devices. This does the very basics of
> registering for the new PF PCI device 1dd8:100c, setting up debugfs
> entries, and registering with devlink.
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> .../device_drivers/ethernet/amd/pds_core.rst | 40 ++
> .../device_drivers/ethernet/index.rst | 1 +
> drivers/net/ethernet/amd/pds_core/Makefile | 8 +
> drivers/net/ethernet/amd/pds_core/core.h | 63 ++
> drivers/net/ethernet/amd/pds_core/debugfs.c | 34 ++
> drivers/net/ethernet/amd/pds_core/main.c | 285 +++++++++
> include/linux/pds/pds_common.h | 14 +
> include/linux/pds/pds_core_if.h | 540 ++++++++++++++++++
> 8 files changed, 985 insertions(+)
> create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile
> create mode 100644 drivers/net/ethernet/amd/pds_core/core.h
> create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c
> create mode 100644 drivers/net/ethernet/amd/pds_core/main.c
> create mode 100644 include/linux/pds/pds_common.h
> create mode 100644 include/linux/pds/pds_core_if.h
<...>
> +#ifdef CONFIG_DEBUG_FS
> +void pdsc_debugfs_create(void);
> +void pdsc_debugfs_destroy(void);
> +void pdsc_debugfs_add_dev(struct pdsc *pdsc);
> +void pdsc_debugfs_del_dev(struct pdsc *pdsc);
> +#else
> +static inline void pdsc_debugfs_create(void) { }
> +static inline void pdsc_debugfs_destroy(void) { }
> +static inline void pdsc_debugfs_add_dev(struct pdsc *pdsc) { }
> +static inline void pdsc_debugfs_del_dev(struct pdsc *pdsc) { }
> +#endif
I don't think that you need CONFIG_DEBUG_FS guard as debugfs code is
built to handle this case, so you can call to internal debugfs_*() calls
without completed initialization of debugfs.
> +
> +#endif /* _PDSC_H_ */
> diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c
> new file mode 100644
> index 000000000000..9b2385c19c41
> --- /dev/null
> +++ b/drivers/net/ethernet/amd/pds_core/debugfs.c
> @@ -0,0 +1,34 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
> +
> +#ifdef CONFIG_DEBUG_FS
> +
> +#include <linux/pci.h>
> +
> +#include "core.h"
> +
> +static struct dentry *pdsc_dir;
> +
> +void pdsc_debugfs_create(void)
> +{
> + pdsc_dir = debugfs_create_dir(PDS_CORE_DRV_NAME, NULL);
> +}
> +
> +void pdsc_debugfs_destroy(void)
> +{
> + debugfs_remove_recursive(pdsc_dir);
> +}
> +
> +void pdsc_debugfs_add_dev(struct pdsc *pdsc)
> +{
> + pdsc->dentry = debugfs_create_dir(pci_name(pdsc->pdev), pdsc_dir);
> +
> + debugfs_create_ulong("state", 0400, pdsc->dentry, &pdsc->state);
> +}
> +
> +void pdsc_debugfs_del_dev(struct pdsc *pdsc)
> +{
> + debugfs_remove_recursive(pdsc->dentry);
> + pdsc->dentry = NULL;
> +}
> +#endif /* CONFIG_DEBUG_FS */
> diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
> new file mode 100644
> index 000000000000..1c2f3fbaa27c
> --- /dev/null
> +++ b/drivers/net/ethernet/amd/pds_core/main.c
> @@ -0,0 +1,285 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
> +
> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
> +
> +#include <linux/pci.h>
> +
> +#include <linux/pds/pds_common.h>
> +
> +#include "core.h"
> +
> +MODULE_DESCRIPTION(PDSC_DRV_DESCRIPTION);
> +MODULE_AUTHOR("Advanced Micro Devices, Inc");
> +MODULE_LICENSE("GPL");
> +
> +/* Supported devices */
> +static const struct pci_device_id pdsc_id_table[] = {
> + { PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF) },
> + { 0, } /* end of table */
> +};
> +MODULE_DEVICE_TABLE(pci, pdsc_id_table);
> +
> +static void pdsc_unmap_bars(struct pdsc *pdsc)
> +{
> + struct pdsc_dev_bar *bars = pdsc->bars;
> + unsigned int i;
> +
> + for (i = 0; i < PDS_CORE_BARS_MAX; i++) {
> + if (bars[i].vaddr) {
> + pci_iounmap(pdsc->pdev, bars[i].vaddr);
> + bars[i].vaddr = NULL;
> + }
> +
> + bars[i].len = 0;
> + bars[i].bus_addr = 0;
> + bars[i].res_index = 0;
Why are you clearing it? You are going to release bars[] anyway.
It will be great to remove this zeroing pattern from whole driver
as it hides use-after-free bugs.
> + }
> +}
> +
> +static int pdsc_map_bars(struct pdsc *pdsc)
> +{
> + struct pdsc_dev_bar *bar = pdsc->bars;
> + struct pci_dev *pdev = pdsc->pdev;
> + struct device *dev = pdsc->dev;
> + struct pdsc_dev_bar *bars;
> + unsigned int i, j;
> + int num_bars = 0;
> + int err;
> + u32 sig;
> +
> + bars = pdsc->bars;
> + num_bars = 0;
You set it to zero 4 lines above.
> +
<...>
> +module_init(pdsc_init_module);
> +module_exit(pdsc_cleanup_module);
> diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
> new file mode 100644
> index 000000000000..bd041a5170a6
> --- /dev/null
> +++ b/include/linux/pds/pds_common.h
> @@ -0,0 +1,14 @@
> +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */
> +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */
> +
> +#ifndef _PDS_COMMON_H_
> +#define _PDS_COMMON_H_
> +
> +#define PDS_CORE_DRV_NAME "pds_core"
It is KBUILD_MODNAME.
> +
> +/* the device's internal addressing uses up to 52 bits */
> +#define PDS_CORE_ADDR_LEN 52
> +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1)
> +#define PDS_PAGE_SIZE 4096
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
2023-04-06 23:41 ` [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces Shannon Nelson
@ 2023-04-09 11:46 ` Leon Romanovsky
2023-04-10 19:05 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 11:46 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:31PM -0700, Shannon Nelson wrote:
> The devcmd interface is the basic connection to the device through the
> PCI BAR for low level identification and command services. This does
> the early device initialization and finds the identity data, and adds
> devcmd routines to be used by later driver bits.
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> drivers/net/ethernet/amd/pds_core/Makefile | 4 +-
> drivers/net/ethernet/amd/pds_core/core.c | 36 ++
> drivers/net/ethernet/amd/pds_core/core.h | 52 +++
> drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++
> drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++
> drivers/net/ethernet/amd/pds_core/main.c | 33 +-
> include/linux/pds/pds_common.h | 61 ++++
> include/linux/pds/pds_intr.h | 163 +++++++++
> 8 files changed, 763 insertions(+), 3 deletions(-)
> create mode 100644 drivers/net/ethernet/amd/pds_core/core.c
> create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c
> create mode 100644 include/linux/pds/pds_intr.h
<...>
> +int pdsc_setup(struct pdsc *pdsc, bool init)
> +{
> + int err = 0;
You don't need to set value as you overwrite it immediately.
> +
> + if (init)
> + err = pdsc_dev_init(pdsc);
> + else
> + err = pdsc_dev_reinit(pdsc);
> + if (err)
> + return err;
> +
> + clear_bit(PDSC_S_FW_DEAD, &pdsc->state);
> + return 0;
> +}
<...>
> +static int irqs_show(struct seq_file *seq, void *v)
> +{
> + struct pdsc *pdsc = seq->private;
> + struct pdsc_intr_info *intr_info;
> + int i;
> +
> + seq_printf(seq, "index vector name (nintrs %d)\n", pdsc->nintrs);
> +
> + if (!pdsc->intr_info)
> + return 0;
> +
> + for (i = 0; i < pdsc->nintrs; i++) {
> + intr_info = &pdsc->intr_info[i];
> + if (!intr_info->vector)
> + continue;
> +
> + seq_printf(seq, "% 3d % 3d %s\n",
> + i, intr_info->vector, intr_info->name);
> + }
> +
> + return 0;
> +}
> +DEFINE_SHOW_ATTRIBUTE(irqs);
I'm curious why existing IRQ core support is not enough?
> +
> +void pdsc_debugfs_add_irqs(struct pdsc *pdsc)
> +{
> + debugfs_create_file("irqs", 0400, pdsc->dentry, pdsc, &irqs_fops);
> +}
> +
> #endif /* CONFIG_DEBUG_FS */
> diff --git a/drivers/net/ethernet/amd/pds_core/dev.c b/drivers/net/ethernet/amd/pds_core/dev.c
> new file mode 100644
> index 000000000000..52385a72246d
> --- /dev/null
> +++ b/drivers/net/ethernet/amd/pds_core/dev.c
> @@ -0,0 +1,349 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
> +
> +#include <linux/errno.h>
> +#include <linux/pci.h>
> +#include <linux/utsname.h>
> +
> +#include "core.h"
> +
> +int pdsc_err_to_errno(enum pds_core_status_code code)
All users of this function, call to pdsc_devcmd_status() first. Probably
they need to be combined.
> +{
> + switch (code) {
> + case PDS_RC_SUCCESS:
> + return 0;
> + case PDS_RC_EVERSION:
> + case PDS_RC_EQTYPE:
> + case PDS_RC_EQID:
> + case PDS_RC_EINVAL:
> + case PDS_RC_ENOSUPP:
> + return -EINVAL;
> + case PDS_RC_EPERM:
> + return -EPERM;
> + case PDS_RC_ENOENT:
> + return -ENOENT;
> + case PDS_RC_EAGAIN:
> + return -EAGAIN;
> + case PDS_RC_ENOMEM:
> + return -ENOMEM;
> + case PDS_RC_EFAULT:
> + return -EFAULT;
> + case PDS_RC_EBUSY:
> + return -EBUSY;
> + case PDS_RC_EEXIST:
> + return -EEXIST;
> + case PDS_RC_EVFID:
> + return -ENODEV;
> + case PDS_RC_ECLIENT:
> + return -ECHILD;
> + case PDS_RC_ENOSPC:
> + return -ENOSPC;
> + case PDS_RC_ERANGE:
> + return -ERANGE;
> + case PDS_RC_BAD_ADDR:
> + return -EFAULT;
> + case PDS_RC_EOPCODE:
> + case PDS_RC_EINTR:
> + case PDS_RC_DEV_CMD:
> + case PDS_RC_ERROR:
> + case PDS_RC_ERDMA:
> + case PDS_RC_EIO:
> + default:
> + return -EIO;
> + }
> +}
<...>
> +static u8 pdsc_devcmd_status(struct pdsc *pdsc)
> +{
> + return ioread8(&pdsc->cmd_regs->comp.status);
> +}
<...>
> +int pdsc_devcmd_init(struct pdsc *pdsc)
> +{
> + union pds_core_dev_comp comp = { 0 };
There is no need to put 0 while using {} initialization.
> + union pds_core_dev_cmd cmd = {
> + .opcode = PDS_CORE_CMD_INIT,
> + };
> +
> + return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
> +}
<...>
> + /* Get intr_info struct array for tracking */
> + pdsc->intr_info = kcalloc(nintrs, sizeof(*pdsc->intr_info), GFP_KERNEL);
> + if (!pdsc->intr_info) {
> + err = -ENOSPC;
The general convention is to return -ENOMEM for memorly allocation failures.
> + goto err_out;
> + }
<...>
> +/*
> + * enum pds_core_status_code - Device command return codes
> + */
> +enum pds_core_status_code {
> + PDS_RC_SUCCESS = 0, /* Success */
> + PDS_RC_EVERSION = 1, /* Incorrect version for request */
> + PDS_RC_EOPCODE = 2, /* Invalid cmd opcode */
> + PDS_RC_EIO = 3, /* I/O error */
> + PDS_RC_EPERM = 4, /* Permission denied */
> + PDS_RC_EQID = 5, /* Bad qid */
> + PDS_RC_EQTYPE = 6, /* Bad qtype */
> + PDS_RC_ENOENT = 7, /* No such element */
> + PDS_RC_EINTR = 8, /* operation interrupted */
> + PDS_RC_EAGAIN = 9, /* Try again */
> + PDS_RC_ENOMEM = 10, /* Out of memory */
> + PDS_RC_EFAULT = 11, /* Bad address */
> + PDS_RC_EBUSY = 12, /* Device or resource busy */
> + PDS_RC_EEXIST = 13, /* object already exists */
> + PDS_RC_EINVAL = 14, /* Invalid argument */
> + PDS_RC_ENOSPC = 15, /* No space left or alloc failure */
> + PDS_RC_ERANGE = 16, /* Parameter out of range */
> + PDS_RC_BAD_ADDR = 17, /* Descriptor contains a bad ptr */
> + PDS_RC_DEV_CMD = 18, /* Device cmd attempted on AdminQ */
> + PDS_RC_ENOSUPP = 19, /* Operation not supported */
> + PDS_RC_ERROR = 29, /* Generic error */
> + PDS_RC_ERDMA = 30, /* Generic RDMA error */
> + PDS_RC_EVFID = 31, /* VF ID does not exist */
> + PDS_RC_BAD_FW = 32, /* FW file is invalid or corrupted */
> + PDS_RC_ECLIENT = 33, /* No such client id */
> +};
We asked from Intel to remove custom error codes and we would like to
ask it here too. Please use standard in-kernel errors.
> +
> +enum pds_core_driver_type {
> + PDS_DRIVER_LINUX = 1,
This is only relevant here, everything else is not applicable.
> + PDS_DRIVER_WIN = 2,
> + PDS_DRIVER_DPDK = 3,
> + PDS_DRIVER_FREEBSD = 4,
> + PDS_DRIVER_IPXE = 5,
> + PDS_DRIVER_ESXI = 6,
> +};
> +
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 03/14] pds_core: health timer and workqueue
2023-04-06 23:41 ` [PATCH v9 net-next 03/14] pds_core: health timer and workqueue Shannon Nelson
@ 2023-04-09 11:51 ` Leon Romanovsky
2023-04-10 19:12 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 11:51 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:32PM -0700, Shannon Nelson wrote:
> Add in the periodic health check and the related workqueue,
> as well as the handlers for when a FW reset is seen.
>
> The firmware is polled every 5 seconds to be sure that it is
> still alive and that the FW generation didn't change.
>
> The alive check looks to see that the PCI bus is still readable
> and the fw_status still has the RUNNING bit on. If not alive,
> the driver stops activity and tears things down. When the FW
> recovers and the alive check again succeeds, the driver sets
> back up for activity.
>
> The generation check looks at the fw_generation to see if it
> has changed, which can happen if the FW crashed and recovered
> or was updated in between health checks. If changed, the
> driver counts that as though the alive test failed and forces
> the fw_down/fw_up cycle.
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> drivers/net/ethernet/amd/pds_core/core.c | 61 ++++++++++++++++++++++++
> drivers/net/ethernet/amd/pds_core/core.h | 8 ++++
> drivers/net/ethernet/amd/pds_core/dev.c | 3 ++
> drivers/net/ethernet/amd/pds_core/main.c | 37 ++++++++++++++
> 4 files changed, 109 insertions(+)
<...>
>
> err_out_unmap_bars:
> mutex_unlock(&pdsc->config_lock);
> + del_timer_sync(&pdsc->wdtimer);
> + if (pdsc->wq) {
> + flush_workqueue(pdsc->wq);
> + destroy_workqueue(pdsc->wq);
There is no need to call to flush_workqueue() as destroy_workqueue()
will do it.
> + pdsc->wq = NULL;
> + }
> mutex_destroy(&pdsc->config_lock);
> mutex_destroy(&pdsc->devcmd_lock);
> pci_free_irq_vectors(pdsc->pdev);
> @@ -270,6 +300,13 @@ static void pdsc_remove(struct pci_dev *pdev)
> devl_unlock(dl);
>
> if (!pdev->is_virtfn) {
> + del_timer_sync(&pdsc->wdtimer);
> + if (pdsc->wq) {
> + flush_workqueue(pdsc->wq);
> + destroy_workqueue(pdsc->wq);
Same
> + pdsc->wq = NULL;
Not really needed, pdsc is released.
> + }
> +
> mutex_lock(&pdsc->config_lock);
> set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
>
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 04/14] pds_core: add devlink health facilities
2023-04-06 23:41 ` [PATCH v9 net-next 04/14] pds_core: add devlink health facilities Shannon Nelson
@ 2023-04-09 11:54 ` Leon Romanovsky
2023-04-10 19:18 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 11:54 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:33PM -0700, Shannon Nelson wrote:
> Add devlink health reporting on top of our fw watchdog.
>
> Example:
> # devlink health show pci/0000:2b:00.0 reporter fw
> pci/0000:2b:00.0:
> reporter fw
> state healthy error 0 recover 0
> # devlink health diagnose pci/0000:2b:00.0 reporter fw
> Status: healthy State: 1 Generation: 0 Recoveries: 0
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> .../device_drivers/ethernet/amd/pds_core.rst | 12 ++++++
> drivers/net/ethernet/amd/pds_core/Makefile | 1 +
> drivers/net/ethernet/amd/pds_core/core.c | 6 +++
> drivers/net/ethernet/amd/pds_core/core.h | 6 +++
> drivers/net/ethernet/amd/pds_core/devlink.c | 37 +++++++++++++++++++
> drivers/net/ethernet/amd/pds_core/main.c | 22 +++++++++++
> 6 files changed, 84 insertions(+)
> create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
<...>
> +int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
> + struct devlink_fmsg *fmsg,
> + struct netlink_ext_ack *extack)
> +{
> + struct pdsc *pdsc = devlink_health_reporter_priv(reporter);
> + int err = 0;
> +
> + if (test_bit(PDSC_S_FW_DEAD, &pdsc->state))
How is this check protected from race with your health workqueue added
in previous patch?
> + err = devlink_fmsg_string_pair_put(fmsg, "Status", "dead");
> + else if (!pdsc_is_fw_good(pdsc))
Same question.
> + err = devlink_fmsg_string_pair_put(fmsg, "Status", "unhealthy");
> + else
> + err = devlink_fmsg_string_pair_put(fmsg, "Status", "healthy");
> + if (err)
> + return err;
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 05/14] pds_core: set up device and adminq
2023-04-06 23:41 ` [PATCH v9 net-next 05/14] pds_core: set up device and adminq Shannon Nelson
@ 2023-04-09 12:03 ` Leon Romanovsky
2023-04-10 19:27 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 12:03 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:34PM -0700, Shannon Nelson wrote:
> Set up the basic adminq and notifyq queue structures. These are
> used mostly by the client drivers for feature configuration.
> These are essentially the same adminq and notifyq as in the
> ionic driver.
>
> Part of this includes querying for device identity and FW
> information, so we can make that available to devlink dev info.
>
> $ devlink dev info pci/0000:b5:00.0
> pci/0000:b5:00.0:
> driver pds_core
> serial_number FLM18420073
> versions:
> fixed:
> asic.id 0x0
> asic.rev 0x0
> running:
> fw 1.51.0-73
> stored:
> fw.goldfw 1.15.9-C-22
> fw.mainfwa 1.60.0-73
> fw.mainfwb 1.60.0-57
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> .../device_drivers/ethernet/amd/pds_core.rst | 47 ++
> drivers/net/ethernet/amd/pds_core/core.c | 450 ++++++++++++-
> drivers/net/ethernet/amd/pds_core/core.h | 154 +++++
> drivers/net/ethernet/amd/pds_core/debugfs.c | 76 +++
> drivers/net/ethernet/amd/pds_core/devlink.c | 61 ++
> drivers/net/ethernet/amd/pds_core/main.c | 17 +-
> include/linux/pds/pds_adminq.h | 637 ++++++++++++++++++
> 7 files changed, 1438 insertions(+), 4 deletions(-)
> create mode 100644 include/linux/pds/pds_adminq.h
<...>
> +void pdsc_stop(struct pdsc *pdsc)
> +{
> + if (pdsc->wq)
> + flush_workqueue(pdsc->wq);
> +
> + pdsc_mask_interrupts(pdsc);
> +}
> +
<...>
> static const struct devlink_ops pdsc_dl_vf_ops = {
> @@ -332,6 +346,7 @@ static void pdsc_remove(struct pci_dev *pdev)
> mutex_lock(&pdsc->config_lock);
> set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
>
> + pdsc_stop(pdsc);
You are calling to flush this workqueue in a couple of line above.
> pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
> mutex_unlock(&pdsc->config_lock);
> mutex_destroy(&pdsc->config_lock);
> diff --git a/include/linux/pds/pds_adminq.h b/include/linux/pds/pds_adminq.h
> new file mode 100644
> index 000000000000..9cd58b7f5fb2
> --- /dev/null
> +++ b/include/linux/pds/pds_adminq.h
> @@ -0,0 +1,637 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
> +
<...>
> +/* The color bit is a 'done' bit for the completion descriptors
> + * where the meaning alternates between '1' and '0' for alternating
> + * passes through the completion descriptor ring.
> + */
> +static inline u8 pdsc_color_match(u8 color, u8 done_color)
static inline bool?
> +{
> + return (!!(color & PDS_COMP_COLOR_MASK)) == done_color;
> +}
> +#endif /* _PDS_CORE_ADMINQ_H_ */
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults
2023-04-06 23:41 ` [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults Shannon Nelson
@ 2023-04-09 12:08 ` Leon Romanovsky
2023-04-10 19:36 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 12:08 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:37PM -0700, Shannon Nelson wrote:
> The Virtual Interfaces (VIFs) supported by the DSC's
> configuration (vDPA, Eth, RDMA, etc) are reported in the
> dev_ident struct and made visible in debugfs. At this point
> only vDPA is supported in this driver - the other interfaces
> are defined for future use but not yet realized.
Let's add only supported modes for now.
<...>
> +static int viftype_show(struct seq_file *seq, void *v)
> +{
> + struct pdsc *pdsc = seq->private;
> + int vt;
> +
> + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
> + if (!pdsc->viftype_status[vt].name)
> + continue;
> +
> + seq_printf(seq, "%s\t%d supported %d enabled\n",
> + pdsc->viftype_status[vt].name,
> + pdsc->viftype_status[vt].supported,
> + pdsc->viftype_status[vt].enabled);
> + }
> + return 0;
> +}
> +DEFINE_SHOW_ATTRIBUTE(viftype);
I think that it is handled by devlink.
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices
2023-04-06 23:41 ` [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices Shannon Nelson
@ 2023-04-09 12:23 ` Leon Romanovsky
2023-04-10 20:02 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 12:23 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:39PM -0700, Shannon Nelson wrote:
> An auxiliary_bus device is created for each vDPA type VF at VF
> probe and destroyed at VF remove. The aux device name comes
> from the driver name + VIF type + the unique id assigned at PCI
> probe. The VFs are always removed on PF remove, so there should
> be no issues with VFs trying to access missing PF structures.
>
> The auxiliary_device names will look like "pds_core.vDPA.nn"
> where 'nn' is the VF's uid.
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> drivers/net/ethernet/amd/pds_core/Makefile | 1 +
> drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++
> drivers/net/ethernet/amd/pds_core/core.h | 6 ++
> drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++-
> include/linux/pds/pds_auxbus.h | 16 +++
> include/linux/pds/pds_common.h | 1 +
> 6 files changed, 170 insertions(+), 2 deletions(-)
> create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c
> create mode 100644 include/linux/pds/pds_auxbus.h
>
> diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
> index 6d1d6c58a1fa..0abc33ce826c 100644
> --- a/drivers/net/ethernet/amd/pds_core/Makefile
> +++ b/drivers/net/ethernet/amd/pds_core/Makefile
> @@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o
>
> pds_core-y := main.o \
> devlink.o \
> + auxbus.o \
> dev.o \
> adminq.o \
> core.o \
> diff --git a/drivers/net/ethernet/amd/pds_core/auxbus.c b/drivers/net/ethernet/amd/pds_core/auxbus.c
> new file mode 100644
> index 000000000000..6757a5174eb7
> --- /dev/null
> +++ b/drivers/net/ethernet/amd/pds_core/auxbus.c
> @@ -0,0 +1,112 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
> +
> +#include <linux/pci.h>
> +
> +#include "core.h"
> +#include <linux/pds/pds_auxbus.h>
> +
> +static void pdsc_auxbus_dev_release(struct device *dev)
> +{
> + struct pds_auxiliary_dev *padev =
> + container_of(dev, struct pds_auxiliary_dev, aux_dev.dev);
> +
> + kfree(padev);
> +}
> +
> +static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *vf,
> + struct pdsc *pf,
> + char *name)
> +{
> + struct auxiliary_device *aux_dev;
> + struct pds_auxiliary_dev *padev;
> + int err;
> +
> + padev = kzalloc(sizeof(*padev), GFP_KERNEL);
> + if (!padev)
> + return ERR_PTR(-ENOMEM);
> +
> + padev->vf_pdev = vf->pdev;
> + padev->pf_pdev = pf->pdev;
Why do you need to store pointer to PF device in your VF devices?
pci_physfn() will return it from vf_pdev for you.
> +
> + aux_dev = &padev->aux_dev;
> + aux_dev->name = name;
> + aux_dev->id = vf->uid;
> + aux_dev->dev.parent = vf->dev;
> + aux_dev->dev.release = pdsc_auxbus_dev_release;
> +
> + err = auxiliary_device_init(aux_dev);
> + if (err < 0) {
> + dev_warn(vf->dev, "auxiliary_device_init of %s failed: %pe\n",
> + name, ERR_PTR(err));
> + goto err_out;
> + }
> +
> + err = auxiliary_device_add(aux_dev);
> + if (err) {
> + dev_warn(vf->dev, "auxiliary_device_add of %s failed: %pe\n",
> + name, ERR_PTR(err));
> + goto err_out_uninit;
> + }
> +
> + return padev;
> +
> +err_out_uninit:
> + auxiliary_device_uninit(aux_dev);
> +err_out:
> + kfree(padev);
> + return ERR_PTR(err);
> +}
> +
> +int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf)
> +{
> + struct pds_auxiliary_dev *padev;
> + int err = 0;
> +
> + mutex_lock(&pf->config_lock);
> +
> + padev = pf->vfs[vf->vf_id].padev;
> + if (padev) {
> + auxiliary_device_delete(&padev->aux_dev);
> + auxiliary_device_uninit(&padev->aux_dev);
> + }
> + pf->vfs[vf->vf_id].padev = NULL;
> +
> + mutex_unlock(&pf->config_lock);
> + return err;
> +}
> +
> +int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf)
> +{
> + struct pds_auxiliary_dev *padev;
> + enum pds_core_vif_types vt;
> + int err = 0;
> +
> + mutex_lock(&pf->config_lock);
> +
> + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
> + u16 vt_support;
> +
> + /* Verify that the type is supported and enabled */
> + vt_support = !!le16_to_cpu(pf->dev_ident.vif_types[vt]);
> + if (!(vt_support &&
> + pf->viftype_status[vt].supported &&
> + pf->viftype_status[vt].enabled))
> + continue;
> +
> + padev = pdsc_auxbus_dev_register(vf, pf,
> + pf->viftype_status[vt].name);
> + if (IS_ERR(padev)) {
> + err = PTR_ERR(padev);
> + goto out_unlock;
> + }
> + pf->vfs[vf->vf_id].padev = padev;
> +
> + /* We only support a single type per VF, so jump out here */
> + break;
You need to decide, or you implement loop correctly (without break and
proper unfolding) or you don't implement loop yet at all.
And can we please find another name for functions and parameters which
don't include VF in it as it is not correct anymore.
In ideal world, it will be great to have same probe flow for PF and VF
while everything is controlled through FW and auxbus. For PF, you won't
advertise any aux devices, but the flow will continue to be the same.
Thanks
> + }
> +
> +out_unlock:
> + mutex_unlock(&pf->config_lock);
> + return err;
> +}
> diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
> index 5be2b986c4d9..16b20bd705e4 100644
> --- a/drivers/net/ethernet/amd/pds_core/core.h
> +++ b/drivers/net/ethernet/amd/pds_core/core.h
> @@ -30,8 +30,11 @@ struct pdsc_dev_bar {
> int res_index;
> };
>
> +struct pdsc;
> +
> struct pdsc_vf {
> struct pds_auxiliary_dev *padev;
> + struct pdsc *vf;
> u16 index;
> __le16 vif_types[PDS_DEV_TYPE_MAX];
> };
> @@ -300,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);
> void pdsc_stop(struct pdsc *pdsc);
> void pdsc_health_thread(struct work_struct *work);
>
> +int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf);
> +int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf);
> +
> void pdsc_process_adminq(struct pdsc_qcq *qcq);
> void pdsc_work_thread(struct work_struct *work);
> irqreturn_t pdsc_adminq_isr(int irq, void *data);
> diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
> index 5bda66d2a0df..16a2d8a048a3 100644
> --- a/drivers/net/ethernet/amd/pds_core/main.c
> +++ b/drivers/net/ethernet/amd/pds_core/main.c
> @@ -180,6 +180,12 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)
> static int pdsc_init_vf(struct pdsc *vf)
> {
> struct devlink *dl;
> + struct pdsc *pf;
> + int err;
> +
> + pf = pdsc_get_pf_struct(vf->pdev);
> + if (IS_ERR_OR_NULL(pf))
> + return PTR_ERR(pf) ?: -1;
>
> vf->vf_id = pci_iov_vf_id(vf->pdev);
>
> @@ -188,7 +194,15 @@ static int pdsc_init_vf(struct pdsc *vf)
> devl_register(dl);
> devl_unlock(dl);
>
> - return 0;
> + pf->vfs[vf->vf_id].vf = vf;
> + err = pdsc_auxbus_dev_add_vf(vf, pf);
> + if (err) {
> + devl_lock(dl);
> + devl_unregister(dl);
> + devl_unlock(dl);
> + }
> +
> + return err;
> }
>
> static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {
> @@ -379,7 +393,19 @@ static void pdsc_remove(struct pci_dev *pdev)
> }
> devl_unlock(dl);
>
> - if (!pdev->is_virtfn) {
> + if (pdev->is_virtfn) {
> + struct pdsc *pf;
> +
> + pf = pdsc_get_pf_struct(pdsc->pdev);
> + if (!IS_ERR(pf)) {
> + pdsc_auxbus_dev_del_vf(pdsc, pf);
> + pf->vfs[pdsc->vf_id].vf = NULL;
> + }
> + } else {
> + /* Remove the VFs and their aux_bus connections before other
> + * cleanup so that the clients can use the AdminQ to cleanly
> + * shut themselves down.
> + */
> pdsc_sriov_configure(pdev, 0);
>
> del_timer_sync(&pdsc->wdtimer);
> @@ -419,6 +445,12 @@ static struct pci_driver pdsc_driver = {
> .sriov_configure = pdsc_sriov_configure,
> };
>
> +void *pdsc_get_pf_struct(struct pci_dev *vf_pdev)
> +{
> + return pci_iov_get_pf_drvdata(vf_pdev, &pdsc_driver);
> +}
> +EXPORT_SYMBOL_GPL(pdsc_get_pf_struct);
> +
> static int __init pdsc_init_module(void)
> {
> pdsc_debugfs_create();
> diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h
> new file mode 100644
> index 000000000000..aa0192af4a29
> --- /dev/null
> +++ b/include/linux/pds/pds_auxbus.h
> @@ -0,0 +1,16 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
> +
> +#ifndef _PDSC_AUXBUS_H_
> +#define _PDSC_AUXBUS_H_
> +
> +#include <linux/auxiliary_bus.h>
> +
> +struct pds_auxiliary_dev {
> + struct auxiliary_device aux_dev;
> + struct pci_dev *vf_pdev;
> + struct pci_dev *pf_pdev;
> + u16 client_id;
> + void *priv;
> +};
> +#endif /* _PDSC_AUXBUS_H_ */
> diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
> index 350295091d9d..898f3c7b14b7 100644
> --- a/include/linux/pds/pds_common.h
> +++ b/include/linux/pds/pds_common.h
> @@ -91,4 +91,5 @@ enum pds_core_logical_qtype {
> PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */
> };
>
> +void *pdsc_get_pf_struct(struct pci_dev *vf_pdev);
> #endif /* _PDS_COMMON_H_ */
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 12/14] pds_core: add the aux client API
2023-04-06 23:41 ` [PATCH v9 net-next 12/14] pds_core: add the aux client API Shannon Nelson
@ 2023-04-09 17:07 ` Leon Romanovsky
2023-04-10 20:50 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 17:07 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:41PM -0700, Shannon Nelson wrote:
> Add the client API operations for running adminq commands.
> The core registers the client with the FW, then the client
> has a context for requesting adminq services. We expect
> to add additional operations for other clients, including
> requesting additional private adminqs and IRQs, but don't have
> the need yet.
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++-
> include/linux/pds/pds_auxbus.h | 28 +++++
> 2 files changed, 160 insertions(+), 3 deletions(-)
<...>
> +static struct pds_core_ops pds_core_ops = {
> + .adminq_cmd = pds_client_adminq_cmd,
> +};
> +
<...>
> diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h
> index aa0192af4a29..f98efd578e1c 100644
> --- a/include/linux/pds/pds_auxbus.h
> +++ b/include/linux/pds/pds_auxbus.h
> @@ -10,7 +10,35 @@ struct pds_auxiliary_dev {
> struct auxiliary_device aux_dev;
> struct pci_dev *vf_pdev;
> struct pci_dev *pf_pdev;
> + struct pds_core_ops *ops;
I honestly don't understand why pds_core functionality is espoused
through ops callbacks on auxdevice. IMHO, they shouldn't be callbacks
and that functionality shouldn't operate on auxdevice.
Thanks
> u16 client_id;
> void *priv;
> };
> +
> +/*
> + * ptrs to functions to be used by the client for core services
> + */
> +struct pds_core_ops {
> + /* .adminq_cmd() - process an adminq request for the client
> + * padev: ptr to the client device
> + * req: ptr to buffer with request
> + * req_len: length of actual struct used for request
> + * resp: ptr to buffer where answer is to be copied
> + * flags: optional flags defined by enum pds_core_adminq_flags
> + * and used for more flexible adminq behvior
> + *
> + * returns 0 on success, or
> + * negative for error
> + * Client sends pointers to request and response buffers
> + * Core copies request data into pds_core_client_request_cmd
> + * Core sets other fields as needed
> + * Core posts to AdminQ
> + * Core copies completion data into response buffer
> + */
> + int (*adminq_cmd)(struct pds_auxiliary_dev *padev,
> + union pds_core_adminq_cmd *req,
> + size_t req_len,
> + union pds_core_adminq_comp *resp,
> + u64 flags);
> +};
> #endif /* _PDSC_AUXBUS_H_ */
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-06 23:41 ` [PATCH v9 net-next 13/14] pds_core: publish events to the clients Shannon Nelson
@ 2023-04-09 17:11 ` Leon Romanovsky
2023-04-10 21:01 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 17:11 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:42PM -0700, Shannon Nelson wrote:
> When the Core device gets an event from the device, or notices
> the device FW to be up or down, it needs to send those events
> on to the clients that have an event handler. Add the code to
> pass along the events to the clients.
>
> The entry points pdsc_register_notify() and pdsc_unregister_notify()
> are EXPORTed for other drivers that want to listen for these events.
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++
> drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++
> drivers/net/ethernet/amd/pds_core/core.h | 3 ++
> include/linux/pds/pds_common.h | 2 ++
> 4 files changed, 39 insertions(+)
>
> diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c
> index 25c7dd0d37e5..bb18ac1aabab 100644
> --- a/drivers/net/ethernet/amd/pds_core/adminq.c
> +++ b/drivers/net/ethernet/amd/pds_core/adminq.c
> @@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
> case PDS_EVENT_LINK_CHANGE:
> dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
> ecode, eid);
> + pdsc_notify(PDS_EVENT_LINK_CHANGE, comp);
Aren't you "resending" standard netdev event?
It will be better to send only custom, specific to pds_core events,
while leaving general ones to netdev.
> break;
>
> case PDS_EVENT_RESET:
> dev_info(pdsc->dev, "NotifyQ RESET ecode %d eid %lld\n",
> ecode, eid);
> + pdsc_notify(PDS_EVENT_RESET, comp);
We can argue if clients should get this event. Once reset is detected,
the pds_core should close devices by deleting aux drivers.
Thanks
> break;
>
> case PDS_EVENT_XCVR:
> diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
> index ec088d490d34..b2790be0fc46 100644
> --- a/drivers/net/ethernet/amd/pds_core/core.c
> +++ b/drivers/net/ethernet/amd/pds_core/core.c
> @@ -6,6 +6,25 @@
>
> #include "core.h"
>
> +static BLOCKING_NOTIFIER_HEAD(pds_notify_chain);
> +
> +int pdsc_register_notify(struct notifier_block *nb)
> +{
> + return blocking_notifier_chain_register(&pds_notify_chain, nb);
> +}
> +EXPORT_SYMBOL_GPL(pdsc_register_notify);
> +
> +void pdsc_unregister_notify(struct notifier_block *nb)
> +{
> + blocking_notifier_chain_unregister(&pds_notify_chain, nb);
> +}
> +EXPORT_SYMBOL_GPL(pdsc_unregister_notify);
> +
> +void pdsc_notify(unsigned long event, void *data)
> +{
> + blocking_notifier_call_chain(&pds_notify_chain, event, data);
> +}
> +
> void pdsc_intr_free(struct pdsc *pdsc, int index)
> {
> struct pdsc_intr_info *intr_info;
> @@ -513,12 +532,19 @@ void pdsc_stop(struct pdsc *pdsc)
>
> static void pdsc_fw_down(struct pdsc *pdsc)
> {
> + union pds_core_notifyq_comp reset_event = {
> + .reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
> + .reset.state = 0,
> + };
> +
> if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
> dev_err(pdsc->dev, "%s: already happening\n", __func__);
> return;
> }
>
> + /* Notify clients of fw_down */
> devlink_health_report(pdsc->fw_reporter, "FW down reported", pdsc);
> + pdsc_notify(PDS_EVENT_RESET, &reset_event);
>
> pdsc_mask_interrupts(pdsc);
> pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
> @@ -526,6 +552,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)
>
> static void pdsc_fw_up(struct pdsc *pdsc)
> {
> + union pds_core_notifyq_comp reset_event = {
> + .reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
> + .reset.state = 1,
> + };
> int err;
>
> if (!test_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
> @@ -541,9 +571,11 @@ static void pdsc_fw_up(struct pdsc *pdsc)
> if (err)
> goto err_out;
>
> + /* Notify clients of fw_up */
> pdsc->fw_recoveries++;
> devlink_health_reporter_state_update(pdsc->fw_reporter,
> DEVLINK_HEALTH_REPORTER_STATE_HEALTHY);
> + pdsc_notify(PDS_EVENT_RESET, &reset_event);
>
> return;
>
> diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
> index aab4986007b9..2215e4915e6a 100644
> --- a/drivers/net/ethernet/amd/pds_core/core.h
> +++ b/drivers/net/ethernet/amd/pds_core/core.h
> @@ -310,6 +310,9 @@ int pdsc_start(struct pdsc *pdsc);
> void pdsc_stop(struct pdsc *pdsc);
> void pdsc_health_thread(struct work_struct *work);
>
> +int pdsc_register_notify(struct notifier_block *nb);
> +void pdsc_unregister_notify(struct notifier_block *nb);
> +void pdsc_notify(unsigned long event, void *data);
> int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf);
> int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf);
>
> diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
> index 898f3c7b14b7..17708a142349 100644
> --- a/include/linux/pds/pds_common.h
> +++ b/include/linux/pds/pds_common.h
> @@ -91,5 +91,7 @@ enum pds_core_logical_qtype {
> PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */
> };
>
> +int pdsc_register_notify(struct notifier_block *nb);
> +void pdsc_unregister_notify(struct notifier_block *nb);
> void *pdsc_get_pf_struct(struct pci_dev *vf_pdev);
> #endif /* _PDS_COMMON_H_ */
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst
2023-04-06 23:41 ` [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst Shannon Nelson
@ 2023-04-09 17:17 ` Leon Romanovsky
2023-04-10 21:05 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-09 17:17 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Thu, Apr 06, 2023 at 04:41:43PM -0700, Shannon Nelson wrote:
> Documentation and Kconfig hook for building the driver.
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> ---
> .../device_drivers/ethernet/amd/pds_core.rst | 16 ++++++++++++++++
> MAINTAINERS | 9 +++++++++
> drivers/net/ethernet/amd/Kconfig | 12 ++++++++++++
> drivers/net/ethernet/amd/Makefile | 1 +
> 4 files changed, 38 insertions(+)
>
> diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> index 9449451b538f..c5ef20f361da 100644
> --- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> +++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> @@ -114,6 +114,22 @@ The driver supports a devlink health reporter for FW status::
> # devlink health diagnose pci/0000:2b:00.0 reporter fw
> Status: healthy State: 1 Generation: 0 Recoveries: 0
>
> +Enabling the driver
> +===================
> +
> +The driver is enabled via the standard kernel configuration system,
> +using the make command::
> +
> + make oldconfig/menuconfig/etc.
> +
> +The driver is located in the menu structure at:
> +
> + -> Device Drivers
> + -> Network device support (NETDEVICES [=y])
> + -> Ethernet driver support
> + -> AMD devices
> + -> AMD/Pensando Ethernet PDS_CORE Support
> +
> Support
> =======
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 30ca644d704f..95b5f25a2c06 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1041,6 +1041,15 @@ F: drivers/gpu/drm/amd/include/vi_structs.h
> F: include/uapi/linux/kfd_ioctl.h
> F: include/uapi/linux/kfd_sysfs.h
>
> +AMD PDS CORE DRIVER
> +M: Shannon Nelson <shannon.nelson@amd.com>
> +M: Brett Creeley <brett.creeley@amd.com>
> +M: drivers@pensando.io
I don't know if we have any policy here, but prefer if we won't add
private distribution lists to MAINTAINERS file. It is very annoying
to send emails to these lists and get responses from random people.
> +L: netdev@vger.kernel.org
> +S: Supported
> +F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> +F: drivers/net/ethernet/amd/pds_core/
You forgot to add includes to this list.
Thanks
> +
> AMD SPI DRIVER
> M: Sanjay R Mehta <sanju.mehta@amd.com>
> S: Maintained
> diff --git a/drivers/net/ethernet/amd/Kconfig b/drivers/net/ethernet/amd/Kconfig
> index ab42f75b9413..235fcacef5c5 100644
> --- a/drivers/net/ethernet/amd/Kconfig
> +++ b/drivers/net/ethernet/amd/Kconfig
> @@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC
> bool
> default n
>
> +config PDS_CORE
> + tristate "AMD/Pensando Data Systems Core Device Support"
> + depends on 64BIT && PCI
> + help
> + This enables the support for the AMD/Pensando Core device family of
> + adapters. More specific information on this driver can be
> + found in
> + <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>.
> +
> + To compile this driver as a module, choose M here. The module
> + will be called pds_core.
> +
> endif # NET_VENDOR_AMD
> diff --git a/drivers/net/ethernet/amd/Makefile b/drivers/net/ethernet/amd/Makefile
> index 42742afe9115..2dcfb84731e1 100644
> --- a/drivers/net/ethernet/amd/Makefile
> +++ b/drivers/net/ethernet/amd/Makefile
> @@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o
> obj-$(CONFIG_SUN3LANCE) += sun3lance.o
> obj-$(CONFIG_SUNLANCE) += sunlance.o
> obj-$(CONFIG_AMD_XGBE) += xgbe/
> +obj-$(CONFIG_PDS_CORE) += pds_core/
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink
2023-04-06 23:41 ` [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink Shannon Nelson
@ 2023-04-10 15:44 ` Simon Horman
2023-04-10 22:59 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Simon Horman @ 2023-04-10 15:44 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, leon, jiri
On Thu, Apr 06, 2023 at 04:41:36PM -0700, Shannon Nelson wrote:
> Add in the support for doing firmware updates. Of the two
> main banks available, a and b, this updates the one not in
> use and then selects it for the next boot.
>
> Example:
> devlink dev flash pci/0000:b2:00.0 \
> file pensando/dsc_fw_1.63.0-22.tar
>
> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
Hi Shannon,
some minor feedback from my side.
...
> diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> index 265d948a8c02..6faf46390f5f 100644
> --- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> +++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
> @@ -73,6 +73,16 @@ The ``pds_core`` driver reports the following versions
> - fixed
> - The revision of the ASIC for this device
>
> +Firmware Management
> +===================
> +
> +The ``flash`` command can update a the DSC firmware. The downloaded firmware
> +will be saved into either of firmware bank 1 or bank 2, whichever is not
> +currrently in use, and that bank will used for the next boot::
nit: s/currrently/currently/
...
> diff --git a/drivers/net/ethernet/amd/pds_core/fw.c b/drivers/net/ethernet/amd/pds_core/fw.c
...
> +int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw,
> + struct netlink_ext_ack *extack)
> +{
> + u32 buf_sz, copy_sz, offset;
> + struct devlink *dl;
> + int next_interval;
> + u64 data_addr;
> + int err = 0;
> + u8 fw_slot;
> +
> + dev_info(pdsc->dev, "Installing firmware\n");
> +
> + dl = priv_to_devlink(pdsc);
> + devlink_flash_update_status_notify(dl, "Preparing to flash",
> + NULL, 0, 0);
> +
> + buf_sz = sizeof(pdsc->cmd_regs->data);
> +
> + dev_dbg(pdsc->dev,
> + "downloading firmware - size %d part_sz %d nparts %lu\n",
> + (int)fw->size, buf_sz, DIV_ROUND_UP(fw->size, buf_sz));
> +
> + offset = 0;
> + next_interval = 0;
> + data_addr = offsetof(struct pds_core_dev_cmd_regs, data);
> + while (offset < fw->size) {
> + if (offset >= next_interval) {
> + devlink_flash_update_status_notify(dl, "Downloading",
> + NULL, offset,
> + fw->size);
> + next_interval = offset +
> + (fw->size / PDSC_FW_INTERVAL_FRACTION);
> + }
> +
> + copy_sz = min_t(unsigned int, buf_sz, fw->size - offset);
> + mutex_lock(&pdsc->devcmd_lock);
> + memcpy_toio(&pdsc->cmd_regs->data, fw->data + offset, copy_sz);
> + err = pdsc_devcmd_fw_download_locked(pdsc, data_addr,
> + offset, copy_sz);
> + mutex_unlock(&pdsc->devcmd_lock);
> + if (err) {
> + dev_err(pdsc->dev,
> + "download failed offset 0x%x addr 0x%llx len 0x%x: %pe\n",
> + offset, data_addr, copy_sz, ERR_PTR(err));
> + NL_SET_ERR_MSG_MOD(extack, "Segment download failed");
> + goto err_out;
> + }
> + offset += copy_sz;
> + }
> + devlink_flash_update_status_notify(dl, "Downloading", NULL,
> + fw->size, fw->size);
> +
> + devlink_flash_update_timeout_notify(dl, "Installing", NULL,
> + PDSC_FW_INSTALL_TIMEOUT);
> +
> + fw_slot = pdsc_devcmd_fw_install(pdsc);
> + if (fw_slot < 0) {
The type of fs_slot is u8.
But the return type of pdsc_devcmd_fw_install is int,
(I think) it can return negative error values,
and that case is checked on the line above.
Perhaps the type of fw_slot should be int?
Flagged by Coccinelle as:
drivers/net/ethernet/amd/pds_core/fw.c:154:5-12: WARNING: Unsigned expression compared with zero: fw_slot < 0
And Smatch as:
drivers/net/ethernet/amd/pds_core/fw.c:154 pdsc_firmware_update() warn: impossible condition '(fw_slot < 0) => (0-255 < 0)'
> + err = fw_slot;
> + dev_err(pdsc->dev, "install failed: %pe\n", ERR_PTR(err));
> + NL_SET_ERR_MSG_MOD(extack, "Failed to start firmware install");
> + goto err_out;
> + }
...
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver
2023-04-09 11:26 ` Leon Romanovsky
@ 2023-04-10 18:41 ` Shannon Nelson
0 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 18:41 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 4:26 AM, Leon Romanovsky wrote:
>
Thanks for the time you put into these review notes, I appreciate it.
> On Thu, Apr 06, 2023 at 04:41:30PM -0700, Shannon Nelson wrote:
>> This is the initial PCI driver framework for the new pds_core device
>> driver and its family of devices. This does the very basics of
>> registering for the new PF PCI device 1dd8:100c, setting up debugfs
>> entries, and registering with devlink.
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> .../device_drivers/ethernet/amd/pds_core.rst | 40 ++
>> .../device_drivers/ethernet/index.rst | 1 +
>> drivers/net/ethernet/amd/pds_core/Makefile | 8 +
>> drivers/net/ethernet/amd/pds_core/core.h | 63 ++
>> drivers/net/ethernet/amd/pds_core/debugfs.c | 34 ++
>> drivers/net/ethernet/amd/pds_core/main.c | 285 +++++++++
>> include/linux/pds/pds_common.h | 14 +
>> include/linux/pds/pds_core_if.h | 540 ++++++++++++++++++
>> 8 files changed, 985 insertions(+)
>> create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> create mode 100644 drivers/net/ethernet/amd/pds_core/Makefile
>> create mode 100644 drivers/net/ethernet/amd/pds_core/core.h
>> create mode 100644 drivers/net/ethernet/amd/pds_core/debugfs.c
>> create mode 100644 drivers/net/ethernet/amd/pds_core/main.c
>> create mode 100644 include/linux/pds/pds_common.h
>> create mode 100644 include/linux/pds/pds_core_if.h
>
> <...>
>
>> +#ifdef CONFIG_DEBUG_FS
>> +void pdsc_debugfs_create(void);
>> +void pdsc_debugfs_destroy(void);
>> +void pdsc_debugfs_add_dev(struct pdsc *pdsc);
>> +void pdsc_debugfs_del_dev(struct pdsc *pdsc);
>> +#else
>> +static inline void pdsc_debugfs_create(void) { }
>> +static inline void pdsc_debugfs_destroy(void) { }
>> +static inline void pdsc_debugfs_add_dev(struct pdsc *pdsc) { }
>> +static inline void pdsc_debugfs_del_dev(struct pdsc *pdsc) { }
>> +#endif
>
> I don't think that you need CONFIG_DEBUG_FS guard as debugfs code is
> built to handle this case, so you can call to internal debugfs_*() calls
> without completed initialization of debugfs.
Old habits... sure, we can pull that out.
>
>> +
>> +#endif /* _PDSC_H_ */
>> diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c
>> new file mode 100644
>> index 000000000000..9b2385c19c41
>> --- /dev/null
>> +++ b/drivers/net/ethernet/amd/pds_core/debugfs.c
>> @@ -0,0 +1,34 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
>> +
>> +#ifdef CONFIG_DEBUG_FS
>> +
>> +#include <linux/pci.h>
>> +
>> +#include "core.h"
>> +
>> +static struct dentry *pdsc_dir;
>> +
>> +void pdsc_debugfs_create(void)
>> +{
>> + pdsc_dir = debugfs_create_dir(PDS_CORE_DRV_NAME, NULL);
>> +}
>> +
>> +void pdsc_debugfs_destroy(void)
>> +{
>> + debugfs_remove_recursive(pdsc_dir);
>> +}
>> +
>> +void pdsc_debugfs_add_dev(struct pdsc *pdsc)
>> +{
>> + pdsc->dentry = debugfs_create_dir(pci_name(pdsc->pdev), pdsc_dir);
>> +
>> + debugfs_create_ulong("state", 0400, pdsc->dentry, &pdsc->state);
>> +}
>> +
>> +void pdsc_debugfs_del_dev(struct pdsc *pdsc)
>> +{
>> + debugfs_remove_recursive(pdsc->dentry);
>> + pdsc->dentry = NULL;
>> +}
>> +#endif /* CONFIG_DEBUG_FS */
>> diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
>> new file mode 100644
>> index 000000000000..1c2f3fbaa27c
>> --- /dev/null
>> +++ b/drivers/net/ethernet/amd/pds_core/main.c
>> @@ -0,0 +1,285 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
>> +
>> +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
>> +
>> +#include <linux/pci.h>
>> +
>> +#include <linux/pds/pds_common.h>
>> +
>> +#include "core.h"
>> +
>> +MODULE_DESCRIPTION(PDSC_DRV_DESCRIPTION);
>> +MODULE_AUTHOR("Advanced Micro Devices, Inc");
>> +MODULE_LICENSE("GPL");
>> +
>> +/* Supported devices */
>> +static const struct pci_device_id pdsc_id_table[] = {
>> + { PCI_VDEVICE(PENSANDO, PCI_DEVICE_ID_PENSANDO_CORE_PF) },
>> + { 0, } /* end of table */
>> +};
>> +MODULE_DEVICE_TABLE(pci, pdsc_id_table);
>> +
>> +static void pdsc_unmap_bars(struct pdsc *pdsc)
>> +{
>> + struct pdsc_dev_bar *bars = pdsc->bars;
>> + unsigned int i;
>> +
>> + for (i = 0; i < PDS_CORE_BARS_MAX; i++) {
>> + if (bars[i].vaddr) {
>> + pci_iounmap(pdsc->pdev, bars[i].vaddr);
>> + bars[i].vaddr = NULL;
>> + }
>> +
>> + bars[i].len = 0;
>> + bars[i].bus_addr = 0;
>> + bars[i].res_index = 0;
>
> Why are you clearing it? You are going to release bars[] anyway.
> It will be great to remove this zeroing pattern from whole driver
> as it hides use-after-free bugs.
These are from old habits of cleaning up when done with something. Some
of these kinds of zeroing are useful for checks later to see if
something is still valid, but you are right, not all of it is really
necessary.
>
>> + }
>> +}
>> +
>> +static int pdsc_map_bars(struct pdsc *pdsc)
>> +{
>> + struct pdsc_dev_bar *bar = pdsc->bars;
>> + struct pci_dev *pdev = pdsc->pdev;
>> + struct device *dev = pdsc->dev;
>> + struct pdsc_dev_bar *bars;
>> + unsigned int i, j;
>> + int num_bars = 0;
>> + int err;
>> + u32 sig;
>> +
>> + bars = pdsc->bars;
>> + num_bars = 0;
>
> You set it to zero 4 lines above.
Will fix
>
>> +
>
> <...>
>
>> +module_init(pdsc_init_module);
>> +module_exit(pdsc_cleanup_module);
>> diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
>> new file mode 100644
>> index 000000000000..bd041a5170a6
>> --- /dev/null
>> +++ b/include/linux/pds/pds_common.h
>> @@ -0,0 +1,14 @@
>> +/* SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) OR BSD-2-Clause */
>> +/* Copyright(c) 2023 Advanced Micro Devices, Inc. */
>> +
>> +#ifndef _PDS_COMMON_H_
>> +#define _PDS_COMMON_H_
>> +
>> +#define PDS_CORE_DRV_NAME "pds_core"
>
> It is KBUILD_MODNAME.
yep, will fix
>
>> +
>> +/* the device's internal addressing uses up to 52 bits */
>> +#define PDS_CORE_ADDR_LEN 52
>> +#define PDS_CORE_ADDR_MASK (BIT_ULL(PDS_ADDR_LEN) - 1)
>> +#define PDS_PAGE_SIZE 4096
>
> Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
2023-04-09 11:46 ` Leon Romanovsky
@ 2023-04-10 19:05 ` Shannon Nelson
2023-04-13 8:33 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 19:05 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 4:46 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:31PM -0700, Shannon Nelson wrote:
>> The devcmd interface is the basic connection to the device through the
>> PCI BAR for low level identification and command services. This does
>> the early device initialization and finds the identity data, and adds
>> devcmd routines to be used by later driver bits.
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> drivers/net/ethernet/amd/pds_core/Makefile | 4 +-
>> drivers/net/ethernet/amd/pds_core/core.c | 36 ++
>> drivers/net/ethernet/amd/pds_core/core.h | 52 +++
>> drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++
>> drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++
>> drivers/net/ethernet/amd/pds_core/main.c | 33 +-
>> include/linux/pds/pds_common.h | 61 ++++
>> include/linux/pds/pds_intr.h | 163 +++++++++
>> 8 files changed, 763 insertions(+), 3 deletions(-)
>> create mode 100644 drivers/net/ethernet/amd/pds_core/core.c
>> create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c
>> create mode 100644 include/linux/pds/pds_intr.h
>
> <...>
>
>> +int pdsc_setup(struct pdsc *pdsc, bool init)
>> +{
>> + int err = 0;
>
> You don't need to set value as you overwrite it immediately.
will fix
>
>> +
>> + if (init)
>> + err = pdsc_dev_init(pdsc);
>> + else
>> + err = pdsc_dev_reinit(pdsc);
>> + if (err)
>> + return err;
>> +
>> + clear_bit(PDSC_S_FW_DEAD, &pdsc->state);
>> + return 0;
>> +}
>
> <...>
>
>> +static int irqs_show(struct seq_file *seq, void *v)
>> +{
>> + struct pdsc *pdsc = seq->private;
>> + struct pdsc_intr_info *intr_info;
>> + int i;
>> +
>> + seq_printf(seq, "index vector name (nintrs %d)\n", pdsc->nintrs);
>> +
>> + if (!pdsc->intr_info)
>> + return 0;
>> +
>> + for (i = 0; i < pdsc->nintrs; i++) {
>> + intr_info = &pdsc->intr_info[i];
>> + if (!intr_info->vector)
>> + continue;
>> +
>> + seq_printf(seq, "% 3d % 3d %s\n",
>> + i, intr_info->vector, intr_info->name);
>> + }
>> +
>> + return 0;
>> +}
>> +DEFINE_SHOW_ATTRIBUTE(irqs);
>
> I'm curious why existing IRQ core support is not enough?
Yes, that one is not necessary, will remove.
>
>> +
>> +void pdsc_debugfs_add_irqs(struct pdsc *pdsc)
>> +{
>> + debugfs_create_file("irqs", 0400, pdsc->dentry, pdsc, &irqs_fops);
>> +}
>> +
>> #endif /* CONFIG_DEBUG_FS */
>> diff --git a/drivers/net/ethernet/amd/pds_core/dev.c b/drivers/net/ethernet/amd/pds_core/dev.c
>> new file mode 100644
>> index 000000000000..52385a72246d
>> --- /dev/null
>> +++ b/drivers/net/ethernet/amd/pds_core/dev.c
>> @@ -0,0 +1,349 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
>> +
>> +#include <linux/errno.h>
>> +#include <linux/pci.h>
>> +#include <linux/utsname.h>
>> +
>> +#include "core.h"
>> +
>> +int pdsc_err_to_errno(enum pds_core_status_code code)
>
> All users of this function, call to pdsc_devcmd_status() first. Probably
> they need to be combined.
This is also called from pdsc_adminq_post() which doesn't use
pdsc_devcmd_status().
>
>> +{
>> + switch (code) {
>> + case PDS_RC_SUCCESS:
>> + return 0;
>> + case PDS_RC_EVERSION:
>> + case PDS_RC_EQTYPE:
>> + case PDS_RC_EQID:
>> + case PDS_RC_EINVAL:
>> + case PDS_RC_ENOSUPP:
>> + return -EINVAL;
>> + case PDS_RC_EPERM:
>> + return -EPERM;
>> + case PDS_RC_ENOENT:
>> + return -ENOENT;
>> + case PDS_RC_EAGAIN:
>> + return -EAGAIN;
>> + case PDS_RC_ENOMEM:
>> + return -ENOMEM;
>> + case PDS_RC_EFAULT:
>> + return -EFAULT;
>> + case PDS_RC_EBUSY:
>> + return -EBUSY;
>> + case PDS_RC_EEXIST:
>> + return -EEXIST;
>> + case PDS_RC_EVFID:
>> + return -ENODEV;
>> + case PDS_RC_ECLIENT:
>> + return -ECHILD;
>> + case PDS_RC_ENOSPC:
>> + return -ENOSPC;
>> + case PDS_RC_ERANGE:
>> + return -ERANGE;
>> + case PDS_RC_BAD_ADDR:
>> + return -EFAULT;
>> + case PDS_RC_EOPCODE:
>> + case PDS_RC_EINTR:
>> + case PDS_RC_DEV_CMD:
>> + case PDS_RC_ERROR:
>> + case PDS_RC_ERDMA:
>> + case PDS_RC_EIO:
>> + default:
>> + return -EIO;
>> + }
>> +}
>
> <...>
>
>> +static u8 pdsc_devcmd_status(struct pdsc *pdsc)
>> +{
>> + return ioread8(&pdsc->cmd_regs->comp.status);
>> +}
>
> <...>
>
>> +int pdsc_devcmd_init(struct pdsc *pdsc)
>> +{
>> + union pds_core_dev_comp comp = { 0 };
>
> There is no need to put 0 while using {} initialization.
will fix.
>
>> + union pds_core_dev_cmd cmd = {
>> + .opcode = PDS_CORE_CMD_INIT,
>> + };
>> +
>> + return pdsc_devcmd(pdsc, &cmd, &comp, pdsc->devcmd_timeout);
>> +}
>
> <...>
>
>> + /* Get intr_info struct array for tracking */
>> + pdsc->intr_info = kcalloc(nintrs, sizeof(*pdsc->intr_info), GFP_KERNEL);
>> + if (!pdsc->intr_info) {
>> + err = -ENOSPC;
>
> The general convention is to return -ENOMEM for memorly allocation failures.
will fix
>
>> + goto err_out;
>> + }
>
> <...>
>
>> +/*
>> + * enum pds_core_status_code - Device command return codes
>> + */
>> +enum pds_core_status_code {
>> + PDS_RC_SUCCESS = 0, /* Success */
>> + PDS_RC_EVERSION = 1, /* Incorrect version for request */
>> + PDS_RC_EOPCODE = 2, /* Invalid cmd opcode */
>> + PDS_RC_EIO = 3, /* I/O error */
>> + PDS_RC_EPERM = 4, /* Permission denied */
>> + PDS_RC_EQID = 5, /* Bad qid */
>> + PDS_RC_EQTYPE = 6, /* Bad qtype */
>> + PDS_RC_ENOENT = 7, /* No such element */
>> + PDS_RC_EINTR = 8, /* operation interrupted */
>> + PDS_RC_EAGAIN = 9, /* Try again */
>> + PDS_RC_ENOMEM = 10, /* Out of memory */
>> + PDS_RC_EFAULT = 11, /* Bad address */
>> + PDS_RC_EBUSY = 12, /* Device or resource busy */
>> + PDS_RC_EEXIST = 13, /* object already exists */
>> + PDS_RC_EINVAL = 14, /* Invalid argument */
>> + PDS_RC_ENOSPC = 15, /* No space left or alloc failure */
>> + PDS_RC_ERANGE = 16, /* Parameter out of range */
>> + PDS_RC_BAD_ADDR = 17, /* Descriptor contains a bad ptr */
>> + PDS_RC_DEV_CMD = 18, /* Device cmd attempted on AdminQ */
>> + PDS_RC_ENOSUPP = 19, /* Operation not supported */
>> + PDS_RC_ERROR = 29, /* Generic error */
>> + PDS_RC_ERDMA = 30, /* Generic RDMA error */
>> + PDS_RC_EVFID = 31, /* VF ID does not exist */
>> + PDS_RC_BAD_FW = 32, /* FW file is invalid or corrupted */
>> + PDS_RC_ECLIENT = 33, /* No such client id */
>> +};
>
> We asked from Intel to remove custom error codes and we would like to
> ask it here too. Please use standard in-kernel errors.
These are part of the device interface defined by the device firmware
and include some that aren't in the errno set. This is why we use
pdsc_err_to_errno() in pdsc_devcmd_wait() and pdsc_adminq_post(), so
that we can change these status codes that we get from the device into
standard kernel error codes. We try to report both in error messages,
but only return the kernel errno.
However, I see in one place in pdsc_devcmd_wait() we're using the status
codes where we could use the errno, so I'll fix that up.
>
>> +
>> +enum pds_core_driver_type {
>> + PDS_DRIVER_LINUX = 1,
>
> This is only relevant here, everything else is not applicable.
>
>> + PDS_DRIVER_WIN = 2,
>> + PDS_DRIVER_DPDK = 3,
>> + PDS_DRIVER_FREEBSD = 4,
>> + PDS_DRIVER_IPXE = 5,
>> + PDS_DRIVER_ESXI = 6,
>> +};
Yes, they are rather pointless for the Linux kernel, but it is part of
documenting the device interface.
>> +
>
> Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 03/14] pds_core: health timer and workqueue
2023-04-09 11:51 ` Leon Romanovsky
@ 2023-04-10 19:12 ` Shannon Nelson
0 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 19:12 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 4:51 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:32PM -0700, Shannon Nelson wrote:
>> Add in the periodic health check and the related workqueue,
>> as well as the handlers for when a FW reset is seen.
>>
>> The firmware is polled every 5 seconds to be sure that it is
>> still alive and that the FW generation didn't change.
>>
>> The alive check looks to see that the PCI bus is still readable
>> and the fw_status still has the RUNNING bit on. If not alive,
>> the driver stops activity and tears things down. When the FW
>> recovers and the alive check again succeeds, the driver sets
>> back up for activity.
>>
>> The generation check looks at the fw_generation to see if it
>> has changed, which can happen if the FW crashed and recovered
>> or was updated in between health checks. If changed, the
>> driver counts that as though the alive test failed and forces
>> the fw_down/fw_up cycle.
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> drivers/net/ethernet/amd/pds_core/core.c | 61 ++++++++++++++++++++++++
>> drivers/net/ethernet/amd/pds_core/core.h | 8 ++++
>> drivers/net/ethernet/amd/pds_core/dev.c | 3 ++
>> drivers/net/ethernet/amd/pds_core/main.c | 37 ++++++++++++++
>> 4 files changed, 109 insertions(+)
>
> <...>
>
>>
>> err_out_unmap_bars:
>> mutex_unlock(&pdsc->config_lock);
>> + del_timer_sync(&pdsc->wdtimer);
>> + if (pdsc->wq) {
>> + flush_workqueue(pdsc->wq);
>> + destroy_workqueue(pdsc->wq);
>
> There is no need to call to flush_workqueue() as destroy_workqueue()
> will do it.
will fix
>
>> + pdsc->wq = NULL;
>> + }
>> mutex_destroy(&pdsc->config_lock);
>> mutex_destroy(&pdsc->devcmd_lock);
>> pci_free_irq_vectors(pdsc->pdev);
>> @@ -270,6 +300,13 @@ static void pdsc_remove(struct pci_dev *pdev)
>> devl_unlock(dl);
>>
>> if (!pdev->is_virtfn) {
>> + del_timer_sync(&pdsc->wdtimer);
>> + if (pdsc->wq) {
>> + flush_workqueue(pdsc->wq);
>> + destroy_workqueue(pdsc->wq);
>
> Same
>
>> + pdsc->wq = NULL;
>
> Not really needed, pdsc is released.
will fix
>
>> + }
>> +
>> mutex_lock(&pdsc->config_lock);
>> set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
>>
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 04/14] pds_core: add devlink health facilities
2023-04-09 11:54 ` Leon Romanovsky
@ 2023-04-10 19:18 ` Shannon Nelson
0 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 19:18 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 4:54 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:33PM -0700, Shannon Nelson wrote:
>> Add devlink health reporting on top of our fw watchdog.
>>
>> Example:
>> # devlink health show pci/0000:2b:00.0 reporter fw
>> pci/0000:2b:00.0:
>> reporter fw
>> state healthy error 0 recover 0
>> # devlink health diagnose pci/0000:2b:00.0 reporter fw
>> Status: healthy State: 1 Generation: 0 Recoveries: 0
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> .../device_drivers/ethernet/amd/pds_core.rst | 12 ++++++
>> drivers/net/ethernet/amd/pds_core/Makefile | 1 +
>> drivers/net/ethernet/amd/pds_core/core.c | 6 +++
>> drivers/net/ethernet/amd/pds_core/core.h | 6 +++
>> drivers/net/ethernet/amd/pds_core/devlink.c | 37 +++++++++++++++++++
>> drivers/net/ethernet/amd/pds_core/main.c | 22 +++++++++++
>> 6 files changed, 84 insertions(+)
>> create mode 100644 drivers/net/ethernet/amd/pds_core/devlink.c
>
> <...>
>
>> +int pdsc_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
>> + struct devlink_fmsg *fmsg,
>> + struct netlink_ext_ack *extack)
>> +{
>> + struct pdsc *pdsc = devlink_health_reporter_priv(reporter);
>> + int err = 0;
>> +
>> + if (test_bit(PDSC_S_FW_DEAD, &pdsc->state))
>
> How is this check protected from race with your health workqueue added
> in previous patch?
>
>> + err = devlink_fmsg_string_pair_put(fmsg, "Status", "dead");
>> + else if (!pdsc_is_fw_good(pdsc))
>
> Same question.
Yes, it would be good to wrap these in the config_lock.
>
>> + err = devlink_fmsg_string_pair_put(fmsg, "Status", "unhealthy");
>> + else
>> + err = devlink_fmsg_string_pair_put(fmsg, "Status", "healthy");
>> + if (err)
>> + return err;
>
> Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 05/14] pds_core: set up device and adminq
2023-04-09 12:03 ` Leon Romanovsky
@ 2023-04-10 19:27 ` Shannon Nelson
0 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 19:27 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 5:03 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:34PM -0700, Shannon Nelson wrote:
>> Set up the basic adminq and notifyq queue structures. These are
>> used mostly by the client drivers for feature configuration.
>> These are essentially the same adminq and notifyq as in the
>> ionic driver.
>>
>> Part of this includes querying for device identity and FW
>> information, so we can make that available to devlink dev info.
>>
>> $ devlink dev info pci/0000:b5:00.0
>> pci/0000:b5:00.0:
>> driver pds_core
>> serial_number FLM18420073
>> versions:
>> fixed:
>> asic.id 0x0
>> asic.rev 0x0
>> running:
>> fw 1.51.0-73
>> stored:
>> fw.goldfw 1.15.9-C-22
>> fw.mainfwa 1.60.0-73
>> fw.mainfwb 1.60.0-57
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> .../device_drivers/ethernet/amd/pds_core.rst | 47 ++
>> drivers/net/ethernet/amd/pds_core/core.c | 450 ++++++++++++-
>> drivers/net/ethernet/amd/pds_core/core.h | 154 +++++
>> drivers/net/ethernet/amd/pds_core/debugfs.c | 76 +++
>> drivers/net/ethernet/amd/pds_core/devlink.c | 61 ++
>> drivers/net/ethernet/amd/pds_core/main.c | 17 +-
>> include/linux/pds/pds_adminq.h | 637 ++++++++++++++++++
>> 7 files changed, 1438 insertions(+), 4 deletions(-)
>> create mode 100644 include/linux/pds/pds_adminq.h
>
> <...>
>
>> +void pdsc_stop(struct pdsc *pdsc)
>> +{
>> + if (pdsc->wq)
>> + flush_workqueue(pdsc->wq);
>> +
>> + pdsc_mask_interrupts(pdsc);
>> +}
>> +
>
> <...>
>
>> static const struct devlink_ops pdsc_dl_vf_ops = {
>> @@ -332,6 +346,7 @@ static void pdsc_remove(struct pci_dev *pdev)
>> mutex_lock(&pdsc->config_lock);
>> set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state);
>>
>> + pdsc_stop(pdsc);
>
> You are calling to flush this workqueue in a couple of line above.
I guess we really want it flushed...
I'll look at that to see if we really need the extra flushing.
>
>> pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING);
>> mutex_unlock(&pdsc->config_lock);
>> mutex_destroy(&pdsc->config_lock);
>> diff --git a/include/linux/pds/pds_adminq.h b/include/linux/pds/pds_adminq.h
>> new file mode 100644
>> index 000000000000..9cd58b7f5fb2
>> --- /dev/null
>> +++ b/include/linux/pds/pds_adminq.h
>> @@ -0,0 +1,637 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
>> +
>
> <...>
>
>> +/* The color bit is a 'done' bit for the completion descriptors
>> + * where the meaning alternates between '1' and '0' for alternating
>> + * passes through the completion descriptor ring.
>> + */
>> +static inline u8 pdsc_color_match(u8 color, u8 done_color)
>
> static inline bool?
Sure.
>
>> +{
>> + return (!!(color & PDS_COMP_COLOR_MASK)) == done_color;
>> +}
>> +#endif /* _PDS_CORE_ADMINQ_H_ */
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults
2023-04-09 12:08 ` Leon Romanovsky
@ 2023-04-10 19:36 ` Shannon Nelson
2023-04-13 8:36 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 19:36 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 5:08 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:37PM -0700, Shannon Nelson wrote:
>> The Virtual Interfaces (VIFs) supported by the DSC's
>> configuration (vDPA, Eth, RDMA, etc) are reported in the
>> dev_ident struct and made visible in debugfs. At this point
>> only vDPA is supported in this driver - the other interfaces
>> are defined for future use but not yet realized.
>
> Let's add only supported modes for now.
As stated, only the vDPA feature is supported in the driver.
>
> <...>
>
>> +static int viftype_show(struct seq_file *seq, void *v)
>> +{
>> + struct pdsc *pdsc = seq->private;
>> + int vt;
>> +
>> + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
>> + if (!pdsc->viftype_status[vt].name)
>> + continue;
>> +
>> + seq_printf(seq, "%s\t%d supported %d enabled\n",
>> + pdsc->viftype_status[vt].name,
>> + pdsc->viftype_status[vt].supported,
>> + pdsc->viftype_status[vt].enabled);
>> + }
>> + return 0;
>> +}
>> +DEFINE_SHOW_ATTRIBUTE(viftype);
>
> I think that it is handled by devlink.
Yes, this is handled for those for which the driver sets up parameters
(only vDPA), but that doesn't help when we want to see what the device
is actually advertising. This shows us what the device is thinking,
which we find useful in debugging, but it doesn't belong in devlink
information.
>
> Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 00/14] pds_core driver
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
` (14 preceding siblings ...)
2023-04-08 3:18 ` [PATCH v9 net-next 00/14] pds_core driver Jakub Kicinski
@ 2023-04-10 20:00 ` Alex Williamson
2023-04-10 21:05 ` Shannon Nelson
15 siblings, 1 reply; 54+ messages in thread
From: Alex Williamson @ 2023-04-10 20:00 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, leon, jiri
On Thu, 6 Apr 2023 16:41:29 -0700
Shannon Nelson <shannon.nelson@amd.com> wrote:
> Summary:
> --------
> This patchset implements a new driver for use with the AMD/Pensando
> Distributed Services Card (DSC), intended to provide core configuration
> services through the auxiliary_bus and through a couple of EXPORTed
> functions for use initially in VFio and vDPA feature specific drivers.
>
> To keep this patchset to a manageable size, the pds_vdpa and pds_vfio
> drivers have been split out into their own patchsets to be reviewed
> separately.
FYI, this fails to build w/o DYNAMIC_DEBUG set due to implicit
declaration of dynamic_hex_dump(). Seems the ionic driver avoids this
by including linux/dynamic_debug.h. Other use cases have #ifdef around
their uses. Thanks,
Alex
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices
2023-04-09 12:23 ` Leon Romanovsky
@ 2023-04-10 20:02 ` Shannon Nelson
2023-04-13 8:43 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 20:02 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 5:23 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:39PM -0700, Shannon Nelson wrote:
>> An auxiliary_bus device is created for each vDPA type VF at VF
>> probe and destroyed at VF remove. The aux device name comes
>> from the driver name + VIF type + the unique id assigned at PCI
>> probe. The VFs are always removed on PF remove, so there should
>> be no issues with VFs trying to access missing PF structures.
>>
>> The auxiliary_device names will look like "pds_core.vDPA.nn"
>> where 'nn' is the VF's uid.
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> drivers/net/ethernet/amd/pds_core/Makefile | 1 +
>> drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++
>> drivers/net/ethernet/amd/pds_core/core.h | 6 ++
>> drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++-
>> include/linux/pds/pds_auxbus.h | 16 +++
>> include/linux/pds/pds_common.h | 1 +
>> 6 files changed, 170 insertions(+), 2 deletions(-)
>> create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c
>> create mode 100644 include/linux/pds/pds_auxbus.h
>>
>> diff --git a/drivers/net/ethernet/amd/pds_core/Makefile b/drivers/net/ethernet/amd/pds_core/Makefile
>> index 6d1d6c58a1fa..0abc33ce826c 100644
>> --- a/drivers/net/ethernet/amd/pds_core/Makefile
>> +++ b/drivers/net/ethernet/amd/pds_core/Makefile
>> @@ -5,6 +5,7 @@ obj-$(CONFIG_PDS_CORE) := pds_core.o
>>
>> pds_core-y := main.o \
>> devlink.o \
>> + auxbus.o \
>> dev.o \
>> adminq.o \
>> core.o \
>> diff --git a/drivers/net/ethernet/amd/pds_core/auxbus.c b/drivers/net/ethernet/amd/pds_core/auxbus.c
>> new file mode 100644
>> index 000000000000..6757a5174eb7
>> --- /dev/null
>> +++ b/drivers/net/ethernet/amd/pds_core/auxbus.c
>> @@ -0,0 +1,112 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
>> +
>> +#include <linux/pci.h>
>> +
>> +#include "core.h"
>> +#include <linux/pds/pds_auxbus.h>
>> +
>> +static void pdsc_auxbus_dev_release(struct device *dev)
>> +{
>> + struct pds_auxiliary_dev *padev =
>> + container_of(dev, struct pds_auxiliary_dev, aux_dev.dev);
>> +
>> + kfree(padev);
>> +}
>> +
>> +static struct pds_auxiliary_dev *pdsc_auxbus_dev_register(struct pdsc *vf,
>> + struct pdsc *pf,
>> + char *name)
>> +{
>> + struct auxiliary_device *aux_dev;
>> + struct pds_auxiliary_dev *padev;
>> + int err;
>> +
>> + padev = kzalloc(sizeof(*padev), GFP_KERNEL);
>> + if (!padev)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + padev->vf_pdev = vf->pdev;
>> + padev->pf_pdev = pf->pdev;
>
> Why do you need to store pointer to PF device in your VF devices?
> pci_physfn() will return it from vf_pdev for you.
will fix
>
>> +
>> + aux_dev = &padev->aux_dev;
>> + aux_dev->name = name;
>> + aux_dev->id = vf->uid;
>> + aux_dev->dev.parent = vf->dev;
>> + aux_dev->dev.release = pdsc_auxbus_dev_release;
>> +
>> + err = auxiliary_device_init(aux_dev);
>> + if (err < 0) {
>> + dev_warn(vf->dev, "auxiliary_device_init of %s failed: %pe\n",
>> + name, ERR_PTR(err));
>> + goto err_out;
>> + }
>> +
>> + err = auxiliary_device_add(aux_dev);
>> + if (err) {
>> + dev_warn(vf->dev, "auxiliary_device_add of %s failed: %pe\n",
>> + name, ERR_PTR(err));
>> + goto err_out_uninit;
>> + }
>> +
>> + return padev;
>> +
>> +err_out_uninit:
>> + auxiliary_device_uninit(aux_dev);
>> +err_out:
>> + kfree(padev);
>> + return ERR_PTR(err);
>> +}
>> +
>> +int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf)
>> +{
>> + struct pds_auxiliary_dev *padev;
>> + int err = 0;
>> +
>> + mutex_lock(&pf->config_lock);
>> +
>> + padev = pf->vfs[vf->vf_id].padev;
>> + if (padev) {
>> + auxiliary_device_delete(&padev->aux_dev);
>> + auxiliary_device_uninit(&padev->aux_dev);
>> + }
>> + pf->vfs[vf->vf_id].padev = NULL;
>> +
>> + mutex_unlock(&pf->config_lock);
>> + return err;
>> +}
>> +
>> +int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf)
>> +{
>> + struct pds_auxiliary_dev *padev;
>> + enum pds_core_vif_types vt;
>> + int err = 0;
>> +
>> + mutex_lock(&pf->config_lock);
>> +
>> + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
>> + u16 vt_support;
>> +
>> + /* Verify that the type is supported and enabled */
>> + vt_support = !!le16_to_cpu(pf->dev_ident.vif_types[vt]);
>> + if (!(vt_support &&
>> + pf->viftype_status[vt].supported &&
>> + pf->viftype_status[vt].enabled))
>> + continue;
>> +
>> + padev = pdsc_auxbus_dev_register(vf, pf,
>> + pf->viftype_status[vt].name);
>> + if (IS_ERR(padev)) {
>> + err = PTR_ERR(padev);
>> + goto out_unlock;
>> + }
>> + pf->vfs[vf->vf_id].padev = padev;
>> +
>> + /* We only support a single type per VF, so jump out here */
>> + break;
>
> You need to decide, or you implement loop correctly (without break and
> proper unfolding) or you don't implement loop yet at all.
Sure - we can simplify specifically for the single case we have for now
and add the loop later when new features are added.
>
> And can we please find another name for functions and parameters which
> don't include VF in it as it is not correct anymore.
>
> In ideal world, it will be great to have same probe flow for PF and VF
> while everything is controlled through FW and auxbus. For PF, you won't
> advertise any aux devices, but the flow will continue to be the same.
Since we currently only have VFs and not more finely grained
sub-functions, these seem to still make sense and help define the
context of the operations. I can find places where we can reduce the
use of 'VF'. Would you prefer PF and SF to PF and VF where the
difference is important?
>
> Thanks
>
>> + }
>> +
>> +out_unlock:
>> + mutex_unlock(&pf->config_lock);
>> + return err;
>> +}
>> diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
>> index 5be2b986c4d9..16b20bd705e4 100644
>> --- a/drivers/net/ethernet/amd/pds_core/core.h
>> +++ b/drivers/net/ethernet/amd/pds_core/core.h
>> @@ -30,8 +30,11 @@ struct pdsc_dev_bar {
>> int res_index;
>> };
>>
>> +struct pdsc;
>> +
>> struct pdsc_vf {
>> struct pds_auxiliary_dev *padev;
>> + struct pdsc *vf;
>> u16 index;
>> __le16 vif_types[PDS_DEV_TYPE_MAX];
>> };
>> @@ -300,6 +303,9 @@ int pdsc_start(struct pdsc *pdsc);
>> void pdsc_stop(struct pdsc *pdsc);
>> void pdsc_health_thread(struct work_struct *work);
>>
>> +int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf);
>> +int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf);
>> +
>> void pdsc_process_adminq(struct pdsc_qcq *qcq);
>> void pdsc_work_thread(struct work_struct *work);
>> irqreturn_t pdsc_adminq_isr(int irq, void *data);
>> diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c
>> index 5bda66d2a0df..16a2d8a048a3 100644
>> --- a/drivers/net/ethernet/amd/pds_core/main.c
>> +++ b/drivers/net/ethernet/amd/pds_core/main.c
>> @@ -180,6 +180,12 @@ static int pdsc_sriov_configure(struct pci_dev *pdev, int num_vfs)
>> static int pdsc_init_vf(struct pdsc *vf)
>> {
>> struct devlink *dl;
>> + struct pdsc *pf;
>> + int err;
>> +
>> + pf = pdsc_get_pf_struct(vf->pdev);
>> + if (IS_ERR_OR_NULL(pf))
>> + return PTR_ERR(pf) ?: -1;
>>
>> vf->vf_id = pci_iov_vf_id(vf->pdev);
>>
>> @@ -188,7 +194,15 @@ static int pdsc_init_vf(struct pdsc *vf)
>> devl_register(dl);
>> devl_unlock(dl);
>>
>> - return 0;
>> + pf->vfs[vf->vf_id].vf = vf;
>> + err = pdsc_auxbus_dev_add_vf(vf, pf);
>> + if (err) {
>> + devl_lock(dl);
>> + devl_unregister(dl);
>> + devl_unlock(dl);
>> + }
>> +
>> + return err;
>> }
>>
>> static const struct devlink_health_reporter_ops pdsc_fw_reporter_ops = {
>> @@ -379,7 +393,19 @@ static void pdsc_remove(struct pci_dev *pdev)
>> }
>> devl_unlock(dl);
>>
>> - if (!pdev->is_virtfn) {
>> + if (pdev->is_virtfn) {
>> + struct pdsc *pf;
>> +
>> + pf = pdsc_get_pf_struct(pdsc->pdev);
>> + if (!IS_ERR(pf)) {
>> + pdsc_auxbus_dev_del_vf(pdsc, pf);
>> + pf->vfs[pdsc->vf_id].vf = NULL;
>> + }
>> + } else {
>> + /* Remove the VFs and their aux_bus connections before other
>> + * cleanup so that the clients can use the AdminQ to cleanly
>> + * shut themselves down.
>> + */
>> pdsc_sriov_configure(pdev, 0);
>>
>> del_timer_sync(&pdsc->wdtimer);
>> @@ -419,6 +445,12 @@ static struct pci_driver pdsc_driver = {
>> .sriov_configure = pdsc_sriov_configure,
>> };
>>
>> +void *pdsc_get_pf_struct(struct pci_dev *vf_pdev)
>> +{
>> + return pci_iov_get_pf_drvdata(vf_pdev, &pdsc_driver);
>> +}
>> +EXPORT_SYMBOL_GPL(pdsc_get_pf_struct);
>> +
>> static int __init pdsc_init_module(void)
>> {
>> pdsc_debugfs_create();
>> diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h
>> new file mode 100644
>> index 000000000000..aa0192af4a29
>> --- /dev/null
>> +++ b/include/linux/pds/pds_auxbus.h
>> @@ -0,0 +1,16 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
>> +
>> +#ifndef _PDSC_AUXBUS_H_
>> +#define _PDSC_AUXBUS_H_
>> +
>> +#include <linux/auxiliary_bus.h>
>> +
>> +struct pds_auxiliary_dev {
>> + struct auxiliary_device aux_dev;
>> + struct pci_dev *vf_pdev;
>> + struct pci_dev *pf_pdev;
>> + u16 client_id;
>> + void *priv;
>> +};
>> +#endif /* _PDSC_AUXBUS_H_ */
>> diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
>> index 350295091d9d..898f3c7b14b7 100644
>> --- a/include/linux/pds/pds_common.h
>> +++ b/include/linux/pds/pds_common.h
>> @@ -91,4 +91,5 @@ enum pds_core_logical_qtype {
>> PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */
>> };
>>
>> +void *pdsc_get_pf_struct(struct pci_dev *vf_pdev);
>> #endif /* _PDS_COMMON_H_ */
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 12/14] pds_core: add the aux client API
2023-04-09 17:07 ` Leon Romanovsky
@ 2023-04-10 20:50 ` Shannon Nelson
2023-04-13 8:45 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 20:50 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 10:07 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:41PM -0700, Shannon Nelson wrote:
>> Add the client API operations for running adminq commands.
>> The core registers the client with the FW, then the client
>> has a context for requesting adminq services. We expect
>> to add additional operations for other clients, including
>> requesting additional private adminqs and IRQs, but don't have
>> the need yet.
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++-
>> include/linux/pds/pds_auxbus.h | 28 +++++
>> 2 files changed, 160 insertions(+), 3 deletions(-)
>
> <...>
>
>> +static struct pds_core_ops pds_core_ops = {
>> + .adminq_cmd = pds_client_adminq_cmd,
>> +};
>> +
>
> <...>
>
>> diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h
>> index aa0192af4a29..f98efd578e1c 100644
>> --- a/include/linux/pds/pds_auxbus.h
>> +++ b/include/linux/pds/pds_auxbus.h
>> @@ -10,7 +10,35 @@ struct pds_auxiliary_dev {
>> struct auxiliary_device aux_dev;
>> struct pci_dev *vf_pdev;
>> struct pci_dev *pf_pdev;
>> + struct pds_core_ops *ops;
>
> I honestly don't understand why pds_core functionality is espoused
> through ops callbacks on auxdevice. IMHO, they shouldn't be callbacks
> and that functionality shouldn't operate on auxdevice.
The original design had several more operations and wrapped all the
interaction into a single defined interface rather that polluting the
kernel with additional direct EXPORTed functions from the PF. Since
much has disappeared as we simplified the interface and don't yet have
use for some of them, this ops struct with its single entry is the last
vestige of that idea.
Perhaps it is time to put it away and add one more EXPORTed function.
We can revisit this idea if/when the interface grows again.
>
> Thanks
>
>> u16 client_id;
>> void *priv;
>> };
>> +
>> +/*
>> + * ptrs to functions to be used by the client for core services
>> + */
>> +struct pds_core_ops {
>> + /* .adminq_cmd() - process an adminq request for the client
>> + * padev: ptr to the client device
>> + * req: ptr to buffer with request
>> + * req_len: length of actual struct used for request
>> + * resp: ptr to buffer where answer is to be copied
>> + * flags: optional flags defined by enum pds_core_adminq_flags
>> + * and used for more flexible adminq behvior
>> + *
>> + * returns 0 on success, or
>> + * negative for error
>> + * Client sends pointers to request and response buffers
>> + * Core copies request data into pds_core_client_request_cmd
>> + * Core sets other fields as needed
>> + * Core posts to AdminQ
>> + * Core copies completion data into response buffer
>> + */
>> + int (*adminq_cmd)(struct pds_auxiliary_dev *padev,
>> + union pds_core_adminq_cmd *req,
>> + size_t req_len,
>> + union pds_core_adminq_comp *resp,
>> + u64 flags);
>> +};
>> #endif /* _PDSC_AUXBUS_H_ */
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-09 17:11 ` Leon Romanovsky
@ 2023-04-10 21:01 ` Shannon Nelson
2023-04-13 8:55 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 21:01 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 10:11 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:42PM -0700, Shannon Nelson wrote:
>> When the Core device gets an event from the device, or notices
>> the device FW to be up or down, it needs to send those events
>> on to the clients that have an event handler. Add the code to
>> pass along the events to the clients.
>>
>> The entry points pdsc_register_notify() and pdsc_unregister_notify()
>> are EXPORTed for other drivers that want to listen for these events.
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++
>> drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++
>> drivers/net/ethernet/amd/pds_core/core.h | 3 ++
>> include/linux/pds/pds_common.h | 2 ++
>> 4 files changed, 39 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c
>> index 25c7dd0d37e5..bb18ac1aabab 100644
>> --- a/drivers/net/ethernet/amd/pds_core/adminq.c
>> +++ b/drivers/net/ethernet/amd/pds_core/adminq.c
>> @@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
>> case PDS_EVENT_LINK_CHANGE:
>> dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
>> ecode, eid);
>> + pdsc_notify(PDS_EVENT_LINK_CHANGE, comp);
>
> Aren't you "resending" standard netdev event?
> It will be better to send only custom, specific to pds_core events,
> while leaving general ones to netdev.
We have no netdev in pds_core, so we have to publish this to clients
that might have a netdev or some other need to know.
>
>> break;
>>
>> case PDS_EVENT_RESET:
>> dev_info(pdsc->dev, "NotifyQ RESET ecode %d eid %lld\n",
>> ecode, eid);
>> + pdsc_notify(PDS_EVENT_RESET, comp);
>
> We can argue if clients should get this event. Once reset is detected,
> the pds_core should close devices by deleting aux drivers.
We can get a reset signal from the device when it has done a crash
recovery or when it is preparing to do an update, and this allows
clients to quiesce their operations when reset.state==0 and restart when
they see reset.state==1
>
> Thanks
>
>> break;
>>
>> case PDS_EVENT_XCVR:
>> diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c
>> index ec088d490d34..b2790be0fc46 100644
>> --- a/drivers/net/ethernet/amd/pds_core/core.c
>> +++ b/drivers/net/ethernet/amd/pds_core/core.c
>> @@ -6,6 +6,25 @@
>>
>> #include "core.h"
>>
>> +static BLOCKING_NOTIFIER_HEAD(pds_notify_chain);
>> +
>> +int pdsc_register_notify(struct notifier_block *nb)
>> +{
>> + return blocking_notifier_chain_register(&pds_notify_chain, nb);
>> +}
>> +EXPORT_SYMBOL_GPL(pdsc_register_notify);
>> +
>> +void pdsc_unregister_notify(struct notifier_block *nb)
>> +{
>> + blocking_notifier_chain_unregister(&pds_notify_chain, nb);
>> +}
>> +EXPORT_SYMBOL_GPL(pdsc_unregister_notify);
>> +
>> +void pdsc_notify(unsigned long event, void *data)
>> +{
>> + blocking_notifier_call_chain(&pds_notify_chain, event, data);
>> +}
>> +
>> void pdsc_intr_free(struct pdsc *pdsc, int index)
>> {
>> struct pdsc_intr_info *intr_info;
>> @@ -513,12 +532,19 @@ void pdsc_stop(struct pdsc *pdsc)
>>
>> static void pdsc_fw_down(struct pdsc *pdsc)
>> {
>> + union pds_core_notifyq_comp reset_event = {
>> + .reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
>> + .reset.state = 0,
>> + };
>> +
>> if (test_and_set_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
>> dev_err(pdsc->dev, "%s: already happening\n", __func__);
>> return;
>> }
>>
>> + /* Notify clients of fw_down */
>> devlink_health_report(pdsc->fw_reporter, "FW down reported", pdsc);
>> + pdsc_notify(PDS_EVENT_RESET, &reset_event);
>>
>> pdsc_mask_interrupts(pdsc);
>> pdsc_teardown(pdsc, PDSC_TEARDOWN_RECOVERY);
>> @@ -526,6 +552,10 @@ static void pdsc_fw_down(struct pdsc *pdsc)
>>
>> static void pdsc_fw_up(struct pdsc *pdsc)
>> {
>> + union pds_core_notifyq_comp reset_event = {
>> + .reset.ecode = cpu_to_le16(PDS_EVENT_RESET),
>> + .reset.state = 1,
>> + };
>> int err;
>>
>> if (!test_bit(PDSC_S_FW_DEAD, &pdsc->state)) {
>> @@ -541,9 +571,11 @@ static void pdsc_fw_up(struct pdsc *pdsc)
>> if (err)
>> goto err_out;
>>
>> + /* Notify clients of fw_up */
>> pdsc->fw_recoveries++;
>> devlink_health_reporter_state_update(pdsc->fw_reporter,
>> DEVLINK_HEALTH_REPORTER_STATE_HEALTHY);
>> + pdsc_notify(PDS_EVENT_RESET, &reset_event);
>>
>> return;
>>
>> diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h
>> index aab4986007b9..2215e4915e6a 100644
>> --- a/drivers/net/ethernet/amd/pds_core/core.h
>> +++ b/drivers/net/ethernet/amd/pds_core/core.h
>> @@ -310,6 +310,9 @@ int pdsc_start(struct pdsc *pdsc);
>> void pdsc_stop(struct pdsc *pdsc);
>> void pdsc_health_thread(struct work_struct *work);
>>
>> +int pdsc_register_notify(struct notifier_block *nb);
>> +void pdsc_unregister_notify(struct notifier_block *nb);
>> +void pdsc_notify(unsigned long event, void *data);
>> int pdsc_auxbus_dev_add_vf(struct pdsc *vf, struct pdsc *pf);
>> int pdsc_auxbus_dev_del_vf(struct pdsc *vf, struct pdsc *pf);
>>
>> diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h
>> index 898f3c7b14b7..17708a142349 100644
>> --- a/include/linux/pds/pds_common.h
>> +++ b/include/linux/pds/pds_common.h
>> @@ -91,5 +91,7 @@ enum pds_core_logical_qtype {
>> PDS_CORE_QTYPE_MAX = 16 /* don't change - used in struct size */
>> };
>>
>> +int pdsc_register_notify(struct notifier_block *nb);
>> +void pdsc_unregister_notify(struct notifier_block *nb);
>> void *pdsc_get_pf_struct(struct pci_dev *vf_pdev);
>> #endif /* _PDS_COMMON_H_ */
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst
2023-04-09 17:17 ` Leon Romanovsky
@ 2023-04-10 21:05 ` Shannon Nelson
0 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 21:05 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/9/23 10:17 AM, Leon Romanovsky wrote:
>
> On Thu, Apr 06, 2023 at 04:41:43PM -0700, Shannon Nelson wrote:
>> Documentation and Kconfig hook for building the driver.
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>> ---
>> .../device_drivers/ethernet/amd/pds_core.rst | 16 ++++++++++++++++
>> MAINTAINERS | 9 +++++++++
>> drivers/net/ethernet/amd/Kconfig | 12 ++++++++++++
>> drivers/net/ethernet/amd/Makefile | 1 +
>> 4 files changed, 38 insertions(+)
>>
>> diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> index 9449451b538f..c5ef20f361da 100644
>> --- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> +++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> @@ -114,6 +114,22 @@ The driver supports a devlink health reporter for FW status::
>> # devlink health diagnose pci/0000:2b:00.0 reporter fw
>> Status: healthy State: 1 Generation: 0 Recoveries: 0
>>
>> +Enabling the driver
>> +===================
>> +
>> +The driver is enabled via the standard kernel configuration system,
>> +using the make command::
>> +
>> + make oldconfig/menuconfig/etc.
>> +
>> +The driver is located in the menu structure at:
>> +
>> + -> Device Drivers
>> + -> Network device support (NETDEVICES [=y])
>> + -> Ethernet driver support
>> + -> AMD devices
>> + -> AMD/Pensando Ethernet PDS_CORE Support
>> +
>> Support
>> =======
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 30ca644d704f..95b5f25a2c06 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -1041,6 +1041,15 @@ F: drivers/gpu/drm/amd/include/vi_structs.h
>> F: include/uapi/linux/kfd_ioctl.h
>> F: include/uapi/linux/kfd_sysfs.h
>>
>> +AMD PDS CORE DRIVER
>> +M: Shannon Nelson <shannon.nelson@amd.com>
>> +M: Brett Creeley <brett.creeley@amd.com>
>> +M: drivers@pensando.io
>
> I don't know if we have any policy here, but prefer if we won't add
> private distribution lists to MAINTAINERS file. It is very annoying
> to send emails to these lists and get responses from random people.
will fix
>
>> +L: netdev@vger.kernel.org
>> +S: Supported
>> +F: Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> +F: drivers/net/ethernet/amd/pds_core/
>
> You forgot to add includes to this list.
will fix
>
> Thanks
>
>> +
>> AMD SPI DRIVER
>> M: Sanjay R Mehta <sanju.mehta@amd.com>
>> S: Maintained
>> diff --git a/drivers/net/ethernet/amd/Kconfig b/drivers/net/ethernet/amd/Kconfig
>> index ab42f75b9413..235fcacef5c5 100644
>> --- a/drivers/net/ethernet/amd/Kconfig
>> +++ b/drivers/net/ethernet/amd/Kconfig
>> @@ -186,4 +186,16 @@ config AMD_XGBE_HAVE_ECC
>> bool
>> default n
>>
>> +config PDS_CORE
>> + tristate "AMD/Pensando Data Systems Core Device Support"
>> + depends on 64BIT && PCI
>> + help
>> + This enables the support for the AMD/Pensando Core device family of
>> + adapters. More specific information on this driver can be
>> + found in
>> + <file:Documentation/networking/device_drivers/ethernet/amd/pds_core.rst>.
>> +
>> + To compile this driver as a module, choose M here. The module
>> + will be called pds_core.
>> +
>> endif # NET_VENDOR_AMD
>> diff --git a/drivers/net/ethernet/amd/Makefile b/drivers/net/ethernet/amd/Makefile
>> index 42742afe9115..2dcfb84731e1 100644
>> --- a/drivers/net/ethernet/amd/Makefile
>> +++ b/drivers/net/ethernet/amd/Makefile
>> @@ -17,3 +17,4 @@ obj-$(CONFIG_PCNET32) += pcnet32.o
>> obj-$(CONFIG_SUN3LANCE) += sun3lance.o
>> obj-$(CONFIG_SUNLANCE) += sunlance.o
>> obj-$(CONFIG_AMD_XGBE) += xgbe/
>> +obj-$(CONFIG_PDS_CORE) += pds_core/
>> --
>> 2.17.1
>>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 00/14] pds_core driver
2023-04-10 20:00 ` Alex Williamson
@ 2023-04-10 21:05 ` Shannon Nelson
0 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 21:05 UTC (permalink / raw)
To: Alex Williamson; +Cc: brett.creeley, davem, netdev, kuba, drivers, leon, jiri
On 4/10/23 1:00 PM, Alex Williamson wrote:
>
> On Thu, 6 Apr 2023 16:41:29 -0700
> Shannon Nelson <shannon.nelson@amd.com> wrote:
>
>> Summary:
>> --------
>> This patchset implements a new driver for use with the AMD/Pensando
>> Distributed Services Card (DSC), intended to provide core configuration
>> services through the auxiliary_bus and through a couple of EXPORTed
>> functions for use initially in VFio and vDPA feature specific drivers.
>>
>> To keep this patchset to a manageable size, the pds_vdpa and pds_vfio
>> drivers have been split out into their own patchsets to be reviewed
>> separately.
>
> FYI, this fails to build w/o DYNAMIC_DEBUG set due to implicit
> declaration of dynamic_hex_dump(). Seems the ionic driver avoids this
> by including linux/dynamic_debug.h. Other use cases have #ifdef around
> their uses. Thanks,
>
> Alex
>
Oh, good catch, thanks.
sln
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink
2023-04-10 15:44 ` Simon Horman
@ 2023-04-10 22:59 ` Shannon Nelson
0 siblings, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-10 22:59 UTC (permalink / raw)
To: Simon Horman; +Cc: brett.creeley, davem, netdev, kuba, drivers, leon, jiri
On 4/10/23 8:44 AM, Simon Horman wrote:
>
> On Thu, Apr 06, 2023 at 04:41:36PM -0700, Shannon Nelson wrote:
>> Add in the support for doing firmware updates. Of the two
>> main banks available, a and b, this updates the one not in
>> use and then selects it for the next boot.
>>
>> Example:
>> devlink dev flash pci/0000:b2:00.0 \
>> file pensando/dsc_fw_1.63.0-22.tar
>>
>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
>
> Hi Shannon,
>
> some minor feedback from my side.
Thanks, I'll take care of these.
sln
>
> ...
>
>> diff --git a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> index 265d948a8c02..6faf46390f5f 100644
>> --- a/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> +++ b/Documentation/networking/device_drivers/ethernet/amd/pds_core.rst
>> @@ -73,6 +73,16 @@ The ``pds_core`` driver reports the following versions
>> - fixed
>> - The revision of the ASIC for this device
>>
>> +Firmware Management
>> +===================
>> +
>> +The ``flash`` command can update a the DSC firmware. The downloaded firmware
>> +will be saved into either of firmware bank 1 or bank 2, whichever is not
>> +currrently in use, and that bank will used for the next boot::
>
> nit: s/currrently/currently/
>
> ...
>
>> diff --git a/drivers/net/ethernet/amd/pds_core/fw.c b/drivers/net/ethernet/amd/pds_core/fw.c
>
> ...
>
>> +int pdsc_firmware_update(struct pdsc *pdsc, const struct firmware *fw,
>> + struct netlink_ext_ack *extack)
>> +{
>> + u32 buf_sz, copy_sz, offset;
>> + struct devlink *dl;
>> + int next_interval;
>> + u64 data_addr;
>> + int err = 0;
>> + u8 fw_slot;
>> +
>> + dev_info(pdsc->dev, "Installing firmware\n");
>> +
>> + dl = priv_to_devlink(pdsc);
>> + devlink_flash_update_status_notify(dl, "Preparing to flash",
>> + NULL, 0, 0);
>> +
>> + buf_sz = sizeof(pdsc->cmd_regs->data);
>> +
>> + dev_dbg(pdsc->dev,
>> + "downloading firmware - size %d part_sz %d nparts %lu\n",
>> + (int)fw->size, buf_sz, DIV_ROUND_UP(fw->size, buf_sz));
>> +
>> + offset = 0;
>> + next_interval = 0;
>> + data_addr = offsetof(struct pds_core_dev_cmd_regs, data);
>> + while (offset < fw->size) {
>> + if (offset >= next_interval) {
>> + devlink_flash_update_status_notify(dl, "Downloading",
>> + NULL, offset,
>> + fw->size);
>> + next_interval = offset +
>> + (fw->size / PDSC_FW_INTERVAL_FRACTION);
>> + }
>> +
>> + copy_sz = min_t(unsigned int, buf_sz, fw->size - offset);
>> + mutex_lock(&pdsc->devcmd_lock);
>> + memcpy_toio(&pdsc->cmd_regs->data, fw->data + offset, copy_sz);
>> + err = pdsc_devcmd_fw_download_locked(pdsc, data_addr,
>> + offset, copy_sz);
>> + mutex_unlock(&pdsc->devcmd_lock);
>> + if (err) {
>> + dev_err(pdsc->dev,
>> + "download failed offset 0x%x addr 0x%llx len 0x%x: %pe\n",
>> + offset, data_addr, copy_sz, ERR_PTR(err));
>> + NL_SET_ERR_MSG_MOD(extack, "Segment download failed");
>> + goto err_out;
>> + }
>> + offset += copy_sz;
>> + }
>> + devlink_flash_update_status_notify(dl, "Downloading", NULL,
>> + fw->size, fw->size);
>> +
>> + devlink_flash_update_timeout_notify(dl, "Installing", NULL,
>> + PDSC_FW_INSTALL_TIMEOUT);
>> +
>> + fw_slot = pdsc_devcmd_fw_install(pdsc);
>> + if (fw_slot < 0) {
>
> The type of fs_slot is u8.
> But the return type of pdsc_devcmd_fw_install is int,
> (I think) it can return negative error values,
> and that case is checked on the line above.
>
> Perhaps the type of fw_slot should be int?
>
> Flagged by Coccinelle as:
>
> drivers/net/ethernet/amd/pds_core/fw.c:154:5-12: WARNING: Unsigned expression compared with zero: fw_slot < 0
>
> And Smatch as:
>
> drivers/net/ethernet/amd/pds_core/fw.c:154 pdsc_firmware_update() warn: impossible condition '(fw_slot < 0) => (0-255 < 0)'
>
>> + err = fw_slot;
>> + dev_err(pdsc->dev, "install failed: %pe\n", ERR_PTR(err));
>> + NL_SET_ERR_MSG_MOD(extack, "Failed to start firmware install");
>> + goto err_out;
>> + }
>
> ...
>
> --
> You received this message because you are subscribed to the Google Groups "Pensando Drivers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to drivers+unsubscribe@pensando.io.
> To view this discussion on the web visit https://groups.google.com/a/pensando.io/d/msgid/drivers/ZDQuxBlfH5foSEFA%40corigine.com.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
2023-04-10 19:05 ` Shannon Nelson
@ 2023-04-13 8:33 ` Leon Romanovsky
2023-04-13 15:08 ` Jakub Kicinski
2023-04-14 0:00 ` Shannon Nelson
0 siblings, 2 replies; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-13 8:33 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Mon, Apr 10, 2023 at 12:05:20PM -0700, Shannon Nelson wrote:
> On 4/9/23 4:46 AM, Leon Romanovsky wrote:
> >
> > On Thu, Apr 06, 2023 at 04:41:31PM -0700, Shannon Nelson wrote:
> > > The devcmd interface is the basic connection to the device through the
> > > PCI BAR for low level identification and command services. This does
> > > the early device initialization and finds the identity data, and adds
> > > devcmd routines to be used by later driver bits.
> > >
> > > Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> > > ---
> > > drivers/net/ethernet/amd/pds_core/Makefile | 4 +-
> > > drivers/net/ethernet/amd/pds_core/core.c | 36 ++
> > > drivers/net/ethernet/amd/pds_core/core.h | 52 +++
> > > drivers/net/ethernet/amd/pds_core/debugfs.c | 68 ++++
> > > drivers/net/ethernet/amd/pds_core/dev.c | 349 ++++++++++++++++++++
> > > drivers/net/ethernet/amd/pds_core/main.c | 33 +-
> > > include/linux/pds/pds_common.h | 61 ++++
> > > include/linux/pds/pds_intr.h | 163 +++++++++
> > > 8 files changed, 763 insertions(+), 3 deletions(-)
> > > create mode 100644 drivers/net/ethernet/amd/pds_core/core.c
> > > create mode 100644 drivers/net/ethernet/amd/pds_core/dev.c
> > > create mode 100644 include/linux/pds/pds_intr.h
> >
> > <...>
<...>
> > > #endif /* CONFIG_DEBUG_FS */
> > > diff --git a/drivers/net/ethernet/amd/pds_core/dev.c b/drivers/net/ethernet/amd/pds_core/dev.c
> > > new file mode 100644
> > > index 000000000000..52385a72246d
> > > --- /dev/null
> > > +++ b/drivers/net/ethernet/amd/pds_core/dev.c
> > > @@ -0,0 +1,349 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/* Copyright(c) 2023 Advanced Micro Devices, Inc */
> > > +
> > > +#include <linux/errno.h>
> > > +#include <linux/pci.h>
> > > +#include <linux/utsname.h>
> > > +
> > > +#include "core.h"
> > > +
> > > +int pdsc_err_to_errno(enum pds_core_status_code code)
> >
> > All users of this function, call to pdsc_devcmd_status() first. Probably
> > they need to be combined.
>
> This is also called from pdsc_adminq_post() which doesn't use
> pdsc_devcmd_status().
I probably missed that.
>
> >
> > > +{
> > > + switch (code) {
> > > + case PDS_RC_SUCCESS:
> > > + return 0;
> > > + case PDS_RC_EVERSION:
> > > + case PDS_RC_EQTYPE:
> > > + case PDS_RC_EQID:
> > > + case PDS_RC_EINVAL:
> > > + case PDS_RC_ENOSUPP:
> > > + return -EINVAL;
> > > + case PDS_RC_EPERM:
> > > + return -EPERM;
> > > + case PDS_RC_ENOENT:
> > > + return -ENOENT;
> > > + case PDS_RC_EAGAIN:
> > > + return -EAGAIN;
> > > + case PDS_RC_ENOMEM:
> > > + return -ENOMEM;
> > > + case PDS_RC_EFAULT:
> > > + return -EFAULT;
> > > + case PDS_RC_EBUSY:
> > > + return -EBUSY;
> > > + case PDS_RC_EEXIST:
> > > + return -EEXIST;
> > > + case PDS_RC_EVFID:
> > > + return -ENODEV;
> > > + case PDS_RC_ECLIENT:
> > > + return -ECHILD;
> > > + case PDS_RC_ENOSPC:
> > > + return -ENOSPC;
> > > + case PDS_RC_ERANGE:
> > > + return -ERANGE;
> > > + case PDS_RC_BAD_ADDR:
> > > + return -EFAULT;
> > > + case PDS_RC_EOPCODE:
> > > + case PDS_RC_EINTR:
> > > + case PDS_RC_DEV_CMD:
> > > + case PDS_RC_ERROR:
> > > + case PDS_RC_ERDMA:
> > > + case PDS_RC_EIO:
> > > + default:
> > > + return -EIO;
> > > + }
> > > +}
> >
<...>
> > > +/*
> > > + * enum pds_core_status_code - Device command return codes
> > > + */
> > > +enum pds_core_status_code {
> > > + PDS_RC_SUCCESS = 0, /* Success */
> > > + PDS_RC_EVERSION = 1, /* Incorrect version for request */
> > > + PDS_RC_EOPCODE = 2, /* Invalid cmd opcode */
> > > + PDS_RC_EIO = 3, /* I/O error */
> > > + PDS_RC_EPERM = 4, /* Permission denied */
> > > + PDS_RC_EQID = 5, /* Bad qid */
> > > + PDS_RC_EQTYPE = 6, /* Bad qtype */
> > > + PDS_RC_ENOENT = 7, /* No such element */
> > > + PDS_RC_EINTR = 8, /* operation interrupted */
> > > + PDS_RC_EAGAIN = 9, /* Try again */
> > > + PDS_RC_ENOMEM = 10, /* Out of memory */
> > > + PDS_RC_EFAULT = 11, /* Bad address */
> > > + PDS_RC_EBUSY = 12, /* Device or resource busy */
> > > + PDS_RC_EEXIST = 13, /* object already exists */
> > > + PDS_RC_EINVAL = 14, /* Invalid argument */
> > > + PDS_RC_ENOSPC = 15, /* No space left or alloc failure */
> > > + PDS_RC_ERANGE = 16, /* Parameter out of range */
> > > + PDS_RC_BAD_ADDR = 17, /* Descriptor contains a bad ptr */
> > > + PDS_RC_DEV_CMD = 18, /* Device cmd attempted on AdminQ */
> > > + PDS_RC_ENOSUPP = 19, /* Operation not supported */
> > > + PDS_RC_ERROR = 29, /* Generic error */
> > > + PDS_RC_ERDMA = 30, /* Generic RDMA error */
> > > + PDS_RC_EVFID = 31, /* VF ID does not exist */
> > > + PDS_RC_BAD_FW = 32, /* FW file is invalid or corrupted */
> > > + PDS_RC_ECLIENT = 33, /* No such client id */
> > > +};
> >
> > We asked from Intel to remove custom error codes and we would like to
> > ask it here too. Please use standard in-kernel errors.
>
> These are part of the device interface defined by the device firmware and
> include some that aren't in the errno set. This is why we use
> pdsc_err_to_errno() in pdsc_devcmd_wait() and pdsc_adminq_post(), so that we
> can change these status codes that we get from the device into standard
> kernel error codes. We try to report both in error messages, but only
> return the kernel errno.
You don't really need to create separate enum for that and place
it in include/linux/pds/pds_common.h like you did.
You FW returns u8 status, which you can feed to pdsc_err_to_errno().
In the latter function, you will declare this translation enum.
Such core reorg will esnure that you still have meaningful FW statuses
while keeping them in limited code namespace.
>
> However, I see in one place in pdsc_devcmd_wait() we're using the status
> codes where we could use the errno, so I'll fix that up.
>
>
> >
> > > +
> > > +enum pds_core_driver_type {
> > > + PDS_DRIVER_LINUX = 1,
> >
> > This is only relevant here, everything else is not applicable.
> >
> > > + PDS_DRIVER_WIN = 2,
> > > + PDS_DRIVER_DPDK = 3,
> > > + PDS_DRIVER_FREEBSD = 4,
> > > + PDS_DRIVER_IPXE = 5,
> > > + PDS_DRIVER_ESXI = 6,
> > > +};
>
> Yes, they are rather pointless for the Linux kernel, but it is part of
> documenting the device interface.
It is not used in upstream kernel.
Thanks
>
> > > +
> >
> > Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults
2023-04-10 19:36 ` Shannon Nelson
@ 2023-04-13 8:36 ` Leon Romanovsky
0 siblings, 0 replies; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-13 8:36 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Mon, Apr 10, 2023 at 12:36:31PM -0700, Shannon Nelson wrote:
> On 4/9/23 5:08 AM, Leon Romanovsky wrote:
> >
> > On Thu, Apr 06, 2023 at 04:41:37PM -0700, Shannon Nelson wrote:
> > > The Virtual Interfaces (VIFs) supported by the DSC's
> > > configuration (vDPA, Eth, RDMA, etc) are reported in the
> > > dev_ident struct and made visible in debugfs. At this point
> > > only vDPA is supported in this driver - the other interfaces
> > > are defined for future use but not yet realized.
> >
> > Let's add only supported modes for now.
>
> As stated, only the vDPA feature is supported in the driver.
>
> >
> > <...>
> >
> > > +static int viftype_show(struct seq_file *seq, void *v)
> > > +{
> > > + struct pdsc *pdsc = seq->private;
> > > + int vt;
> > > +
> > > + for (vt = 0; vt < PDS_DEV_TYPE_MAX; vt++) {
> > > + if (!pdsc->viftype_status[vt].name)
> > > + continue;
> > > +
> > > + seq_printf(seq, "%s\t%d supported %d enabled\n",
> > > + pdsc->viftype_status[vt].name,
> > > + pdsc->viftype_status[vt].supported,
> > > + pdsc->viftype_status[vt].enabled);
> > > + }
> > > + return 0;
> > > +}
> > > +DEFINE_SHOW_ATTRIBUTE(viftype);
> >
> > I think that it is handled by devlink.
>
> Yes, this is handled for those for which the driver sets up parameters (only
> vDPA), but that doesn't help when we want to see what the device is actually
> advertising. This shows us what the device is thinking, which we find
> useful in debugging, but it doesn't belong in devlink information.
I'm not sure, users somehow need to know if they can or can't enable
certain feature and it is impossible without advertising it.
All drivers which support auxbus need this feature.
Thanks
>
> >
> > Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices
2023-04-10 20:02 ` Shannon Nelson
@ 2023-04-13 8:43 ` Leon Romanovsky
0 siblings, 0 replies; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-13 8:43 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Mon, Apr 10, 2023 at 01:02:10PM -0700, Shannon Nelson wrote:
> On 4/9/23 5:23 AM, Leon Romanovsky wrote:
> >
> > On Thu, Apr 06, 2023 at 04:41:39PM -0700, Shannon Nelson wrote:
> > > An auxiliary_bus device is created for each vDPA type VF at VF
> > > probe and destroyed at VF remove. The aux device name comes
> > > from the driver name + VIF type + the unique id assigned at PCI
> > > probe. The VFs are always removed on PF remove, so there should
> > > be no issues with VFs trying to access missing PF structures.
> > >
> > > The auxiliary_device names will look like "pds_core.vDPA.nn"
> > > where 'nn' is the VF's uid.
> > >
> > > Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> > > ---
> > > drivers/net/ethernet/amd/pds_core/Makefile | 1 +
> > > drivers/net/ethernet/amd/pds_core/auxbus.c | 112 +++++++++++++++++++++
> > > drivers/net/ethernet/amd/pds_core/core.h | 6 ++
> > > drivers/net/ethernet/amd/pds_core/main.c | 36 ++++++-
> > > include/linux/pds/pds_auxbus.h | 16 +++
> > > include/linux/pds/pds_common.h | 1 +
> > > 6 files changed, 170 insertions(+), 2 deletions(-)
> > > create mode 100644 drivers/net/ethernet/amd/pds_core/auxbus.c
> > > create mode 100644 include/linux/pds/pds_auxbus.h
<...>
> >
> > And can we please find another name for functions and parameters which
> > don't include VF in it as it is not correct anymore.
> >
> > In ideal world, it will be great to have same probe flow for PF and VF
> > while everything is controlled through FW and auxbus. For PF, you won't
> > advertise any aux devices, but the flow will continue to be the same.
>
> Since we currently only have VFs and not more finely grained sub-functions,
> these seem to still make sense and help define the context of the
> operations. I can find places where we can reduce the use of 'VF'. Would
> you prefer PF and SF to PF and VF where the difference is important?
I'm talking about VF names in auxbus code. It simply doesn't belong there as you
are creating/deleting auxiliary devices for specific physical devices. They by
chance your VFs.
"PF/VF probe flow" is my wishful thinking, it is fine to leave it as is.
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 12/14] pds_core: add the aux client API
2023-04-10 20:50 ` Shannon Nelson
@ 2023-04-13 8:45 ` Leon Romanovsky
0 siblings, 0 replies; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-13 8:45 UTC (permalink / raw)
To: Shannon Nelson; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On Mon, Apr 10, 2023 at 01:50:52PM -0700, Shannon Nelson wrote:
> On 4/9/23 10:07 AM, Leon Romanovsky wrote:
> >
> > On Thu, Apr 06, 2023 at 04:41:41PM -0700, Shannon Nelson wrote:
> > > Add the client API operations for running adminq commands.
> > > The core registers the client with the FW, then the client
> > > has a context for requesting adminq services. We expect
> > > to add additional operations for other clients, including
> > > requesting additional private adminqs and IRQs, but don't have
> > > the need yet.
> > >
> > > Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> > > ---
> > > drivers/net/ethernet/amd/pds_core/auxbus.c | 135 ++++++++++++++++++++-
> > > include/linux/pds/pds_auxbus.h | 28 +++++
> > > 2 files changed, 160 insertions(+), 3 deletions(-)
<...>
> >
> > > diff --git a/include/linux/pds/pds_auxbus.h b/include/linux/pds/pds_auxbus.h
> > > index aa0192af4a29..f98efd578e1c 100644
> > > --- a/include/linux/pds/pds_auxbus.h
> > > +++ b/include/linux/pds/pds_auxbus.h
> > > @@ -10,7 +10,35 @@ struct pds_auxiliary_dev {
> > > struct auxiliary_device aux_dev;
> > > struct pci_dev *vf_pdev;
> > > struct pci_dev *pf_pdev;
> > > + struct pds_core_ops *ops;
> >
> > I honestly don't understand why pds_core functionality is espoused
> > through ops callbacks on auxdevice. IMHO, they shouldn't be callbacks
> > and that functionality shouldn't operate on auxdevice.
>
> The original design had several more operations and wrapped all the
> interaction into a single defined interface rather that polluting the kernel
> with additional direct EXPORTed functions from the PF. Since much has
> disappeared as we simplified the interface and don't yet have use for some
> of them, this ops struct with its single entry is the last vestige of that
> idea.
>
> Perhaps it is time to put it away and add one more EXPORTed function. We can
> revisit this idea if/when the interface grows again.
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-10 21:01 ` Shannon Nelson
@ 2023-04-13 8:55 ` Leon Romanovsky
2023-04-13 15:14 ` Jakub Kicinski
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-13 8:55 UTC (permalink / raw)
To: Shannon Nelson, kuba; +Cc: brett.creeley, davem, netdev, drivers, jiri
On Mon, Apr 10, 2023 at 02:01:13PM -0700, Shannon Nelson wrote:
> On 4/9/23 10:11 AM, Leon Romanovsky wrote:
> >
> > On Thu, Apr 06, 2023 at 04:41:42PM -0700, Shannon Nelson wrote:
> > > When the Core device gets an event from the device, or notices
> > > the device FW to be up or down, it needs to send those events
> > > on to the clients that have an event handler. Add the code to
> > > pass along the events to the clients.
> > >
> > > The entry points pdsc_register_notify() and pdsc_unregister_notify()
> > > are EXPORTed for other drivers that want to listen for these events.
> > >
> > > Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
> > > ---
> > > drivers/net/ethernet/amd/pds_core/adminq.c | 2 ++
> > > drivers/net/ethernet/amd/pds_core/core.c | 32 ++++++++++++++++++++++
> > > drivers/net/ethernet/amd/pds_core/core.h | 3 ++
> > > include/linux/pds/pds_common.h | 2 ++
> > > 4 files changed, 39 insertions(+)
> > >
> > > diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c
> > > index 25c7dd0d37e5..bb18ac1aabab 100644
> > > --- a/drivers/net/ethernet/amd/pds_core/adminq.c
> > > +++ b/drivers/net/ethernet/amd/pds_core/adminq.c
> > > @@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
> > > case PDS_EVENT_LINK_CHANGE:
> > > dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
> > > ecode, eid);
> > > + pdsc_notify(PDS_EVENT_LINK_CHANGE, comp);
> >
> > Aren't you "resending" standard netdev event?
> > It will be better to send only custom, specific to pds_core events,
> > while leaving general ones to netdev.
>
> We have no netdev in pds_core, so we have to publish this to clients that
> might have a netdev or some other need to know.
I don't know netdev well enough if it is ok or not and maybe netdev will
sent this LINK_CHANGE by itself anyway.
Jakub???
>
> >
> > > break;
> > >
> > > case PDS_EVENT_RESET:
> > > dev_info(pdsc->dev, "NotifyQ RESET ecode %d eid %lld\n",
> > > ecode, eid);
> > > + pdsc_notify(PDS_EVENT_RESET, comp);
> >
> > We can argue if clients should get this event. Once reset is detected,
> > the pds_core should close devices by deleting aux drivers.
>
> We can get a reset signal from the device when it has done a crash recovery
> or when it is preparing to do an update, and this allows clients to quiesce
> their operations when reset.state==0 and restart when they see
> reset.state==1
I don't think that it is safe behaviour from user POV. If FW resets
itself under the hood, how can client be sure that nothing changes
in its operation? Once FW reset occurs, it is much safer for the clients
to reconfigure everything.
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
2023-04-13 8:33 ` Leon Romanovsky
@ 2023-04-13 15:08 ` Jakub Kicinski
2023-04-14 0:00 ` Shannon Nelson
1 sibling, 0 replies; 54+ messages in thread
From: Jakub Kicinski @ 2023-04-13 15:08 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Shannon Nelson, brett.creeley, davem, netdev, drivers, jiri
On Thu, 13 Apr 2023 11:33:25 +0300 Leon Romanovsky wrote:
> > > This is only relevant here, everything else is not applicable.
> > >
> > > > + PDS_DRIVER_WIN = 2,
> > > > + PDS_DRIVER_DPDK = 3,
> > > > + PDS_DRIVER_FREEBSD = 4,
> > > > + PDS_DRIVER_IPXE = 5,
> > > > + PDS_DRIVER_ESXI = 6,
> > > > +};
> >
> > Yes, they are rather pointless for the Linux kernel, but it is part of
> > documenting the device interface.
>
> It is not used in upstream kernel.
If Shannon prefers to keep the full enum, I think that's fine.
Declaring the full interface is often the only way a user can
try to figure out how the API works, as most vendors don't publish
their APIs.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-13 8:55 ` Leon Romanovsky
@ 2023-04-13 15:14 ` Jakub Kicinski
2023-04-13 16:44 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Jakub Kicinski @ 2023-04-13 15:14 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Shannon Nelson, brett.creeley, davem, netdev, drivers, jiri
On Thu, 13 Apr 2023 11:55:01 +0300 Leon Romanovsky wrote:
> > > > diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c
> > > > index 25c7dd0d37e5..bb18ac1aabab 100644
> > > > --- a/drivers/net/ethernet/amd/pds_core/adminq.c
> > > > +++ b/drivers/net/ethernet/amd/pds_core/adminq.c
> > > > @@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
> > > > case PDS_EVENT_LINK_CHANGE:
> > > > dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
> > > > ecode, eid);
> > > > + pdsc_notify(PDS_EVENT_LINK_CHANGE, comp);
> > >
> > > Aren't you "resending" standard netdev event?
> > > It will be better to send only custom, specific to pds_core events,
> > > while leaving general ones to netdev.
> >
> > We have no netdev in pds_core, so we have to publish this to clients that
> > might have a netdev or some other need to know.
>
> I don't know netdev well enough if it is ok or not and maybe netdev will
> sent this LINK_CHANGE by itself anyway.
>
> Jakub???
I actually prefer for the driver to distribute the event via its own
means than some random borderline proprietary stuff outside of netdev
using netdev events.
> > > We can argue if clients should get this event. Once reset is detected,
> > > the pds_core should close devices by deleting aux drivers.
> >
> > We can get a reset signal from the device when it has done a crash recovery
> > or when it is preparing to do an update, and this allows clients to quiesce
> > their operations when reset.state==0 and restart when they see
> > reset.state==1
>
> I don't think that it is safe behaviour from user POV. If FW resets
> itself under the hood, how can client be sure that nothing changes
> in its operation? Once FW reset occurs, it is much safer for the clients
> to reconfigure everything.
What's the argument exactly? We do have async resets including in mlx5,
grep for enable_remote_dev_reset
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-13 15:14 ` Jakub Kicinski
@ 2023-04-13 16:44 ` Leon Romanovsky
2023-04-13 16:55 ` Jakub Kicinski
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-13 16:44 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Shannon Nelson, brett.creeley, davem, netdev, drivers, jiri
On Thu, Apr 13, 2023 at 08:14:10AM -0700, Jakub Kicinski wrote:
> On Thu, 13 Apr 2023 11:55:01 +0300 Leon Romanovsky wrote:
> > > > > diff --git a/drivers/net/ethernet/amd/pds_core/adminq.c b/drivers/net/ethernet/amd/pds_core/adminq.c
> > > > > index 25c7dd0d37e5..bb18ac1aabab 100644
> > > > > --- a/drivers/net/ethernet/amd/pds_core/adminq.c
> > > > > +++ b/drivers/net/ethernet/amd/pds_core/adminq.c
> > > > > @@ -27,11 +27,13 @@ static int pdsc_process_notifyq(struct pdsc_qcq *qcq)
> > > > > case PDS_EVENT_LINK_CHANGE:
> > > > > dev_info(pdsc->dev, "NotifyQ LINK_CHANGE ecode %d eid %lld\n",
> > > > > ecode, eid);
> > > > > + pdsc_notify(PDS_EVENT_LINK_CHANGE, comp);
> > > >
> > > > Aren't you "resending" standard netdev event?
> > > > It will be better to send only custom, specific to pds_core events,
> > > > while leaving general ones to netdev.
> > >
> > > We have no netdev in pds_core, so we have to publish this to clients that
> > > might have a netdev or some other need to know.
> >
> > I don't know netdev well enough if it is ok or not and maybe netdev will
> > sent this LINK_CHANGE by itself anyway.
> >
> > Jakub???
>
> I actually prefer for the driver to distribute the event via its own
> means than some random borderline proprietary stuff outside of netdev
> using netdev events.
ok
>
> > > > We can argue if clients should get this event. Once reset is detected,
> > > > the pds_core should close devices by deleting aux drivers.
> > >
> > > We can get a reset signal from the device when it has done a crash recovery
> > > or when it is preparing to do an update, and this allows clients to quiesce
> > > their operations when reset.state==0 and restart when they see
> > > reset.state==1
> >
> > I don't think that it is safe behaviour from user POV. If FW resets
> > itself under the hood, how can client be sure that nothing changes
> > in its operation? Once FW reset occurs, it is much safer for the clients
> > to reconfigure everything.
>
> What's the argument exactly? We do have async resets including in mlx5,
> grep for enable_remote_dev_reset
I think that it is different. I'm complaining that during FW reset,
auxiliary devices are not recreated and continue to be connected to
physical device with a hope that everything will continue to work from
kernel and FW perspective.
It is different from enable_remote_dev_reset, where someone externally
resets device which will trigger mlx5_device_rescan() routine through
mlx5_unload_one->mlx5_load_one sequence.
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-13 16:44 ` Leon Romanovsky
@ 2023-04-13 16:55 ` Jakub Kicinski
2023-04-13 17:07 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Jakub Kicinski @ 2023-04-13 16:55 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Shannon Nelson, brett.creeley, davem, netdev, drivers, jiri
On Thu, 13 Apr 2023 19:44:34 +0300 Leon Romanovsky wrote:
> > > I don't think that it is safe behaviour from user POV. If FW resets
> > > itself under the hood, how can client be sure that nothing changes
> > > in its operation? Once FW reset occurs, it is much safer for the clients
> > > to reconfigure everything.
> >
> > What's the argument exactly? We do have async resets including in mlx5,
> > grep for enable_remote_dev_reset
>
> I think that it is different. I'm complaining that during FW reset,
> auxiliary devices are not recreated and continue to be connected to
> physical device with a hope that everything will continue to work from
> kernel and FW perspective.
>
> It is different from enable_remote_dev_reset, where someone externally
> resets device which will trigger mlx5_device_rescan() routine through
> mlx5_unload_one->mlx5_load_one sequence.
Hm, my memory may be incorrect and I didn't look at the code but
I thought that knob came from the "hit-less upgrade" effort.
And for "hit-less upgrade" not respawning the devices was the whole
point.
Which is not to disagree with you. What I'm trying to get at is that
there are different types of reset which deserve different treatment.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-13 16:55 ` Jakub Kicinski
@ 2023-04-13 17:07 ` Leon Romanovsky
2023-04-13 17:10 ` Jakub Kicinski
0 siblings, 1 reply; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-13 17:07 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Shannon Nelson, brett.creeley, davem, netdev, drivers, jiri
On Thu, Apr 13, 2023 at 09:55:09AM -0700, Jakub Kicinski wrote:
> On Thu, 13 Apr 2023 19:44:34 +0300 Leon Romanovsky wrote:
> > > > I don't think that it is safe behaviour from user POV. If FW resets
> > > > itself under the hood, how can client be sure that nothing changes
> > > > in its operation? Once FW reset occurs, it is much safer for the clients
> > > > to reconfigure everything.
> > >
> > > What's the argument exactly? We do have async resets including in mlx5,
> > > grep for enable_remote_dev_reset
> >
> > I think that it is different. I'm complaining that during FW reset,
> > auxiliary devices are not recreated and continue to be connected to
> > physical device with a hope that everything will continue to work from
> > kernel and FW perspective.
> >
> > It is different from enable_remote_dev_reset, where someone externally
> > resets device which will trigger mlx5_device_rescan() routine through
> > mlx5_unload_one->mlx5_load_one sequence.
>
> Hm, my memory may be incorrect and I didn't look at the code but
> I thought that knob came from the "hit-less upgrade" effort.
> And for "hit-less upgrade" not respawning the devices was the whole
> point.
>
> Which is not to disagree with you. What I'm trying to get at is that
> there are different types of reset which deserve different treatment.
I don't disagree with you either, just have a feeling that proposed
behaviour is wrong.
Thanks
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-13 17:07 ` Leon Romanovsky
@ 2023-04-13 17:10 ` Jakub Kicinski
2023-04-13 23:42 ` Shannon Nelson
0 siblings, 1 reply; 54+ messages in thread
From: Jakub Kicinski @ 2023-04-13 17:10 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Shannon Nelson, brett.creeley, davem, netdev, drivers, jiri
On Thu, 13 Apr 2023 20:07:04 +0300 Leon Romanovsky wrote:
> > Hm, my memory may be incorrect and I didn't look at the code but
> > I thought that knob came from the "hit-less upgrade" effort.
> > And for "hit-less upgrade" not respawning the devices was the whole
> > point.
> >
> > Which is not to disagree with you. What I'm trying to get at is that
> > there are different types of reset which deserve different treatment.
>
> I don't disagree with you either, just have a feeling that proposed
> behaviour is wrong.
Shannon, can you elaborate on what the impact of the reset is?
What loss of state and/or configuration is possible?
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-13 17:10 ` Jakub Kicinski
@ 2023-04-13 23:42 ` Shannon Nelson
2023-04-14 8:51 ` Leon Romanovsky
0 siblings, 1 reply; 54+ messages in thread
From: Shannon Nelson @ 2023-04-13 23:42 UTC (permalink / raw)
To: Jakub Kicinski, Leon Romanovsky
Cc: brett.creeley, davem, netdev, drivers, jiri
On 4/13/23 10:10 AM, Jakub Kicinski wrote:
>
> On Thu, 13 Apr 2023 20:07:04 +0300 Leon Romanovsky wrote:
>>> Hm, my memory may be incorrect and I didn't look at the code but
>>> I thought that knob came from the "hit-less upgrade" effort.
>>> And for "hit-less upgrade" not respawning the devices was the whole
>>> point.
>>>
>>> Which is not to disagree with you. What I'm trying to get at is that
>>> there are different types of reset which deserve different treatment.
>>
>> I don't disagree with you either, just have a feeling that proposed
>> behaviour is wrong.
>
> Shannon, can you elaborate on what the impact of the reset is?
> What loss of state and/or configuration is possible?
The device has a couple different cases that might generate the RESET
message:
- crashed and recovered, no state saved
- FW restarted, some or all state saved
There are some variations on this, but these are the two general cases.
We can see in the existing ionic driver there already is some handling
of this where the driver sees the FW come back and is able to replay the
Rx filters and Rx mode configurations. If and when we are able to add
an Eth client through pds_core it will want this message so that it can
replay configuration in the same way. This case will also want the Link
Down event so that it can do the right thing with the netdev.
For the VFio/Migration support (pds_vfio) the RESET message is
essentially a no-op if nothing is happening. But if the system is in
the middle of a migration it offers the ability to "cleanly" fail the
migration and let the system get ready to try again.
For the vDPA case (pds_vdpa) we can trigger the config_cb callback to
get the attention of the stack above us to suggest it look at current
status and respond as needed, whether that is a Link Change or a Reset.
sln
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces
2023-04-13 8:33 ` Leon Romanovsky
2023-04-13 15:08 ` Jakub Kicinski
@ 2023-04-14 0:00 ` Shannon Nelson
1 sibling, 0 replies; 54+ messages in thread
From: Shannon Nelson @ 2023-04-14 0:00 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: brett.creeley, davem, netdev, kuba, drivers, jiri
On 4/13/23 1:33 AM, Leon Romanovsky wrote:
>
> On Mon, Apr 10, 2023 at 12:05:20PM -0700, Shannon Nelson wrote:
>> On 4/9/23 4:46 AM, Leon Romanovsky wrote:
>>>
>>> On Thu, Apr 06, 2023 at 04:41:31PM -0700, Shannon Nelson wrote:
>>>> The devcmd interface is the basic connection to the device through the
>>>> PCI BAR for low level identification and command services. This does
>>>> the early device initialization and finds the identity data, and adds
>>>> devcmd routines to be used by later driver bits.
>>>>
>>>> Signed-off-by: Shannon Nelson <shannon.nelson@amd.com>
[...]
>
>>>> +/*
>>>> + * enum pds_core_status_code - Device command return codes
>>>> + */
>>>> +enum pds_core_status_code {
>>>> + PDS_RC_SUCCESS = 0, /* Success */
>>>> + PDS_RC_EVERSION = 1, /* Incorrect version for request */
>>>> + PDS_RC_EOPCODE = 2, /* Invalid cmd opcode */
>>>> + PDS_RC_EIO = 3, /* I/O error */
>>>> + PDS_RC_EPERM = 4, /* Permission denied */
>>>> + PDS_RC_EQID = 5, /* Bad qid */
>>>> + PDS_RC_EQTYPE = 6, /* Bad qtype */
>>>> + PDS_RC_ENOENT = 7, /* No such element */
>>>> + PDS_RC_EINTR = 8, /* operation interrupted */
>>>> + PDS_RC_EAGAIN = 9, /* Try again */
>>>> + PDS_RC_ENOMEM = 10, /* Out of memory */
>>>> + PDS_RC_EFAULT = 11, /* Bad address */
>>>> + PDS_RC_EBUSY = 12, /* Device or resource busy */
>>>> + PDS_RC_EEXIST = 13, /* object already exists */
>>>> + PDS_RC_EINVAL = 14, /* Invalid argument */
>>>> + PDS_RC_ENOSPC = 15, /* No space left or alloc failure */
>>>> + PDS_RC_ERANGE = 16, /* Parameter out of range */
>>>> + PDS_RC_BAD_ADDR = 17, /* Descriptor contains a bad ptr */
>>>> + PDS_RC_DEV_CMD = 18, /* Device cmd attempted on AdminQ */
>>>> + PDS_RC_ENOSUPP = 19, /* Operation not supported */
>>>> + PDS_RC_ERROR = 29, /* Generic error */
>>>> + PDS_RC_ERDMA = 30, /* Generic RDMA error */
>>>> + PDS_RC_EVFID = 31, /* VF ID does not exist */
>>>> + PDS_RC_BAD_FW = 32, /* FW file is invalid or corrupted */
>>>> + PDS_RC_ECLIENT = 33, /* No such client id */
>>>> +};
>>>
>>> We asked from Intel to remove custom error codes and we would like to
>>> ask it here too. Please use standard in-kernel errors.
>>
>> These are part of the device interface defined by the device firmware and
>> include some that aren't in the errno set. This is why we use
>> pdsc_err_to_errno() in pdsc_devcmd_wait() and pdsc_adminq_post(), so that we
>> can change these status codes that we get from the device into standard
>> kernel error codes. We try to report both in error messages, but only
>> return the kernel errno.
>
> You don't really need to create separate enum for that and place
> it in include/linux/pds/pds_common.h like you did.
>
> You FW returns u8 status, which you can feed to pdsc_err_to_errno().
> In the latter function, you will declare this translation enum.
> Such core reorg will esnure that you still have meaningful FW statuses
> while keeping them in limited code namespace.
Yes, I can see moving it down into the decode function itself can work
and keep it limited in scope. I'll take care of that.
Thanks,
sln
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v9 net-next 13/14] pds_core: publish events to the clients
2023-04-13 23:42 ` Shannon Nelson
@ 2023-04-14 8:51 ` Leon Romanovsky
0 siblings, 0 replies; 54+ messages in thread
From: Leon Romanovsky @ 2023-04-14 8:51 UTC (permalink / raw)
To: Shannon Nelson
Cc: Jakub Kicinski, brett.creeley, davem, netdev, drivers, jiri
On Thu, Apr 13, 2023 at 04:42:08PM -0700, Shannon Nelson wrote:
> On 4/13/23 10:10 AM, Jakub Kicinski wrote:
> >
> > On Thu, 13 Apr 2023 20:07:04 +0300 Leon Romanovsky wrote:
> > > > Hm, my memory may be incorrect and I didn't look at the code but
> > > > I thought that knob came from the "hit-less upgrade" effort.
> > > > And for "hit-less upgrade" not respawning the devices was the whole
> > > > point.
> > > >
> > > > Which is not to disagree with you. What I'm trying to get at is that
> > > > there are different types of reset which deserve different treatment.
> > >
> > > I don't disagree with you either, just have a feeling that proposed
> > > behaviour is wrong.
> >
> > Shannon, can you elaborate on what the impact of the reset is?
> > What loss of state and/or configuration is possible?
>
> The device has a couple different cases that might generate the RESET
> message:
> - crashed and recovered, no state saved
> - FW restarted, some or all state saved
> There are some variations on this, but these are the two general cases.
>
> We can see in the existing ionic driver there already is some handling of
> this where the driver sees the FW come back and is able to replay the Rx
> filters and Rx mode configurations. If and when we are able to add an Eth
> client through pds_core it will want this message so that it can replay
> configuration in the same way. This case will also want the Link Down event
> so that it can do the right thing with the netdev.
I don't see how you can replay ALL (ethtool, devlink, ip, e.t.c) states
without net core involvement. The real fun will be with many offloaded
features, where you must preserve everything while keeping upper layer
in-sync with HW.
Thanks
>
> For the VFio/Migration support (pds_vfio) the RESET message is essentially a
> no-op if nothing is happening. But if the system is in the middle of a
> migration it offers the ability to "cleanly" fail the migration and let the
> system get ready to try again.
>
> For the vDPA case (pds_vdpa) we can trigger the config_cb callback to get
> the attention of the stack above us to suggest it look at current status and
> respond as needed, whether that is a Link Change or a Reset.
>
> sln
^ permalink raw reply [flat|nested] 54+ messages in thread
end of thread, other threads:[~2023-04-14 8:51 UTC | newest]
Thread overview: 54+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-04-06 23:41 [PATCH v9 net-next 00/14] pds_core driver Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 01/14] pds_core: initial framework for pds_core PF driver Shannon Nelson
2023-04-09 11:26 ` Leon Romanovsky
2023-04-10 18:41 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 02/14] pds_core: add devcmd device interfaces Shannon Nelson
2023-04-09 11:46 ` Leon Romanovsky
2023-04-10 19:05 ` Shannon Nelson
2023-04-13 8:33 ` Leon Romanovsky
2023-04-13 15:08 ` Jakub Kicinski
2023-04-14 0:00 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 03/14] pds_core: health timer and workqueue Shannon Nelson
2023-04-09 11:51 ` Leon Romanovsky
2023-04-10 19:12 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 04/14] pds_core: add devlink health facilities Shannon Nelson
2023-04-09 11:54 ` Leon Romanovsky
2023-04-10 19:18 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 05/14] pds_core: set up device and adminq Shannon Nelson
2023-04-09 12:03 ` Leon Romanovsky
2023-04-10 19:27 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 06/14] pds_core: Add adminq processing and commands Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 07/14] pds_core: add FW update feature to devlink Shannon Nelson
2023-04-10 15:44 ` Simon Horman
2023-04-10 22:59 ` Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 08/14] pds_core: set up the VIF definitions and defaults Shannon Nelson
2023-04-09 12:08 ` Leon Romanovsky
2023-04-10 19:36 ` Shannon Nelson
2023-04-13 8:36 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 09/14] pds_core: add initial VF device handling Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 10/14] pds_core: add auxiliary_bus devices Shannon Nelson
2023-04-09 12:23 ` Leon Romanovsky
2023-04-10 20:02 ` Shannon Nelson
2023-04-13 8:43 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 11/14] pds_core: devlink params for enabling VIF support Shannon Nelson
2023-04-06 23:41 ` [PATCH v9 net-next 12/14] pds_core: add the aux client API Shannon Nelson
2023-04-09 17:07 ` Leon Romanovsky
2023-04-10 20:50 ` Shannon Nelson
2023-04-13 8:45 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 13/14] pds_core: publish events to the clients Shannon Nelson
2023-04-09 17:11 ` Leon Romanovsky
2023-04-10 21:01 ` Shannon Nelson
2023-04-13 8:55 ` Leon Romanovsky
2023-04-13 15:14 ` Jakub Kicinski
2023-04-13 16:44 ` Leon Romanovsky
2023-04-13 16:55 ` Jakub Kicinski
2023-04-13 17:07 ` Leon Romanovsky
2023-04-13 17:10 ` Jakub Kicinski
2023-04-13 23:42 ` Shannon Nelson
2023-04-14 8:51 ` Leon Romanovsky
2023-04-06 23:41 ` [PATCH v9 net-next 14/14] pds_core: Kconfig and pds_core.rst Shannon Nelson
2023-04-09 17:17 ` Leon Romanovsky
2023-04-10 21:05 ` Shannon Nelson
2023-04-08 3:18 ` [PATCH v9 net-next 00/14] pds_core driver Jakub Kicinski
2023-04-10 20:00 ` Alex Williamson
2023-04-10 21:05 ` Shannon Nelson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.