* [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
@ 2020-11-29 0:00 Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers Sonal Santan
` (9 more replies)
0 siblings, 10 replies; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
Hello,
This patch series adds management physical function driver for Xilinx Alveo PCIe
accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
This driver is part of Xilinx Runtime (XRT) open source stack.
The patch depends on the "PATCH Xilinx Alveo libfdt prep" which was posted
before.
ALVEO PLATFORM ARCHITECTURE
Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
re-configurable *user* partition. The shell partition is automatically loaded from
flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
till the next cold reboot. The shell exposes two PCIe physical functions:
1. management physical function
2. user physical function
The patch series includes Documentation/xrt.rst which describes Alveo
platform, xmgmt driver architecture and deployment model in more more detail.
Users compile their high level design in C/C++/OpenCL or RTL into FPGA image
using Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
tools. The image is packaged as xclbin and contains partial bitstream for the
user partition and necessary metadata. Users can dynamically swap the image
running on the user partition in order to switch between different workloads.
ALVEO DRIVERS
Alveo Linux kernel driver *xmgmt* binds to management physical function of
Alveo platform. The modular driver framework is organized into several
platform drivers which primarily handle the following functionality:
1. Loading firmware container also called xsabin at driver attach time
2. Loading of user compiled xclbin with FPGA Manager integration
3. Clock scaling of image running on user partition
4. In-band sensors: temp, voltage, power, etc.
5. Device reset and rescan
6. Flashing static *shell* partition
The platform drivers are packaged into *xrt-lib* helper module with a well
defined interfaces the details of which can be found in Documentation/xrt.rst.
xmgmt driver is second generation Alveo management driver and evolution of
the first generation (out of tree) Alveo management driver, xclmgmt. The
sources of the first generation drivers were posted on LKML last year--
https://lore.kernel.org/lkml/20190319215401.6562-1-sonal.santan@xilinx.com/
Changes since the first generation driver include the following: the driver
has been re-architected as data driven modular driver; the driver has been
split into xmgmt and xrt-lib; user physical function driver has been removed
from the patch series.
Alveo/XRT security and platform architecture is documented on the following
GitHub pages:
https://xilinx.github.io/XRT/master/html/security.html
https://xilinx.github.io/XRT/master/html/platforms_partitions.html
User physical function driver is not included in this patch series.
TESTING AND VALIDATION
xmgmt driver can be tested with full XRT open source stack which includes
user space libraries, board utilities and (out of tree) first generation
user physical function driver xocl. XRT open source runtime stack is
available at https://github.com/Xilinx/XRT. This patch series has been
validated on Alveo U50 platform.
Complete documentation for XRT open source stack can be found here--
https://xilinx.github.io/XRT/master/html/index.html
Thanks,
-Sonal
Sonal Santan (8):
Documentation: fpga: Add a document describing Alveo XRT drivers
fpga: xrt: Add UAPI header files
fpga: xrt: infrastructure support for xmgmt driver
fpga: xrt: core infrastructure for xrt-lib module
fpga: xrt: platform drivers for subsystems in shell partition
fpga: xrt: header file for platform and parent drivers
fpga: xrt: Alveo management physical function driver
fpga: xrt: Kconfig and Makefile updates for XRT drivers
Documentation/fpga/index.rst | 1 +
Documentation/fpga/xrt.rst | 588 +++++
drivers/fpga/Kconfig | 2 +
drivers/fpga/Makefile | 3 +
drivers/fpga/alveo/Kconfig | 7 +
drivers/fpga/alveo/common/xrt-metadata.c | 590 +++++
drivers/fpga/alveo/common/xrt-root.c | 744 +++++++
drivers/fpga/alveo/common/xrt-root.h | 24 +
drivers/fpga/alveo/common/xrt-xclbin.c | 387 ++++
drivers/fpga/alveo/common/xrt-xclbin.h | 46 +
drivers/fpga/alveo/include/xmgmt-main.h | 34 +
drivers/fpga/alveo/include/xrt-axigate.h | 31 +
drivers/fpga/alveo/include/xrt-calib.h | 28 +
drivers/fpga/alveo/include/xrt-clkfreq.h | 21 +
drivers/fpga/alveo/include/xrt-clock.h | 29 +
drivers/fpga/alveo/include/xrt-cmc.h | 23 +
drivers/fpga/alveo/include/xrt-ddr-srsr.h | 29 +
drivers/fpga/alveo/include/xrt-flash.h | 28 +
drivers/fpga/alveo/include/xrt-gpio.h | 41 +
drivers/fpga/alveo/include/xrt-icap.h | 27 +
drivers/fpga/alveo/include/xrt-mailbox.h | 44 +
drivers/fpga/alveo/include/xrt-metadata.h | 184 ++
drivers/fpga/alveo/include/xrt-parent.h | 103 +
drivers/fpga/alveo/include/xrt-partition.h | 33 +
drivers/fpga/alveo/include/xrt-subdev.h | 333 +++
drivers/fpga/alveo/include/xrt-ucs.h | 22 +
drivers/fpga/alveo/lib/Kconfig | 11 +
drivers/fpga/alveo/lib/Makefile | 42 +
drivers/fpga/alveo/lib/subdevs/xrt-axigate.c | 298 +++
drivers/fpga/alveo/lib/subdevs/xrt-calib.c | 291 +++
drivers/fpga/alveo/lib/subdevs/xrt-clkfreq.c | 214 ++
drivers/fpga/alveo/lib/subdevs/xrt-clock.c | 638 ++++++
.../fpga/alveo/lib/subdevs/xrt-cmc-bdinfo.c | 343 +++
drivers/fpga/alveo/lib/subdevs/xrt-cmc-ctrl.c | 322 +++
drivers/fpga/alveo/lib/subdevs/xrt-cmc-impl.h | 135 ++
.../fpga/alveo/lib/subdevs/xrt-cmc-mailbox.c | 320 +++
drivers/fpga/alveo/lib/subdevs/xrt-cmc-sc.c | 361 ++++
.../fpga/alveo/lib/subdevs/xrt-cmc-sensors.c | 445 ++++
drivers/fpga/alveo/lib/subdevs/xrt-cmc.c | 239 +++
drivers/fpga/alveo/lib/subdevs/xrt-gpio.c | 198 ++
drivers/fpga/alveo/lib/subdevs/xrt-icap.c | 306 +++
drivers/fpga/alveo/lib/subdevs/xrt-mailbox.c | 1905 +++++++++++++++++
.../fpga/alveo/lib/subdevs/xrt-partition.c | 261 +++
drivers/fpga/alveo/lib/subdevs/xrt-qspi.c | 1347 ++++++++++++
drivers/fpga/alveo/lib/subdevs/xrt-srsr.c | 322 +++
drivers/fpga/alveo/lib/subdevs/xrt-test.c | 274 +++
drivers/fpga/alveo/lib/subdevs/xrt-ucs.c | 238 ++
.../fpga/alveo/lib/subdevs/xrt-vsec-golden.c | 238 ++
drivers/fpga/alveo/lib/subdevs/xrt-vsec.c | 337 +++
drivers/fpga/alveo/lib/xrt-cdev.c | 234 ++
drivers/fpga/alveo/lib/xrt-main.c | 275 +++
drivers/fpga/alveo/lib/xrt-main.h | 46 +
drivers/fpga/alveo/lib/xrt-subdev.c | 1007 +++++++++
drivers/fpga/alveo/mgmt/Kconfig | 11 +
drivers/fpga/alveo/mgmt/Makefile | 28 +
drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c | 194 ++
drivers/fpga/alveo/mgmt/xmgmt-fmgr.h | 29 +
drivers/fpga/alveo/mgmt/xmgmt-main-impl.h | 36 +
drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c | 930 ++++++++
drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c | 190 ++
drivers/fpga/alveo/mgmt/xmgmt-main.c | 843 ++++++++
drivers/fpga/alveo/mgmt/xmgmt-root.c | 375 ++++
include/uapi/linux/xrt/flash_xrt_data.h | 67 +
include/uapi/linux/xrt/mailbox_proto.h | 394 ++++
include/uapi/linux/xrt/mailbox_transport.h | 74 +
include/uapi/linux/xrt/xclbin.h | 418 ++++
include/uapi/linux/xrt/xmgmt-ioctl.h | 72 +
67 files changed, 17710 insertions(+)
create mode 100644 Documentation/fpga/xrt.rst
create mode 100644 drivers/fpga/alveo/Kconfig
create mode 100644 drivers/fpga/alveo/common/xrt-metadata.c
create mode 100644 drivers/fpga/alveo/common/xrt-root.c
create mode 100644 drivers/fpga/alveo/common/xrt-root.h
create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.c
create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.h
create mode 100644 drivers/fpga/alveo/include/xmgmt-main.h
create mode 100644 drivers/fpga/alveo/include/xrt-axigate.h
create mode 100644 drivers/fpga/alveo/include/xrt-calib.h
create mode 100644 drivers/fpga/alveo/include/xrt-clkfreq.h
create mode 100644 drivers/fpga/alveo/include/xrt-clock.h
create mode 100644 drivers/fpga/alveo/include/xrt-cmc.h
create mode 100644 drivers/fpga/alveo/include/xrt-ddr-srsr.h
create mode 100644 drivers/fpga/alveo/include/xrt-flash.h
create mode 100644 drivers/fpga/alveo/include/xrt-gpio.h
create mode 100644 drivers/fpga/alveo/include/xrt-icap.h
create mode 100644 drivers/fpga/alveo/include/xrt-mailbox.h
create mode 100644 drivers/fpga/alveo/include/xrt-metadata.h
create mode 100644 drivers/fpga/alveo/include/xrt-parent.h
create mode 100644 drivers/fpga/alveo/include/xrt-partition.h
create mode 100644 drivers/fpga/alveo/include/xrt-subdev.h
create mode 100644 drivers/fpga/alveo/include/xrt-ucs.h
create mode 100644 drivers/fpga/alveo/lib/Kconfig
create mode 100644 drivers/fpga/alveo/lib/Makefile
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-axigate.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-calib.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-clkfreq.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-clock.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-bdinfo.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-ctrl.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-impl.h
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-mailbox.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-sc.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-sensors.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-gpio.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-icap.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-mailbox.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-partition.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-qspi.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-srsr.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-test.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-ucs.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-vsec-golden.c
create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-vsec.c
create mode 100644 drivers/fpga/alveo/lib/xrt-cdev.c
create mode 100644 drivers/fpga/alveo/lib/xrt-main.c
create mode 100644 drivers/fpga/alveo/lib/xrt-main.h
create mode 100644 drivers/fpga/alveo/lib/xrt-subdev.c
create mode 100644 drivers/fpga/alveo/mgmt/Kconfig
create mode 100644 drivers/fpga/alveo/mgmt/Makefile
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-root.c
create mode 100644 include/uapi/linux/xrt/flash_xrt_data.h
create mode 100644 include/uapi/linux/xrt/mailbox_proto.h
create mode 100644 include/uapi/linux/xrt/mailbox_transport.h
create mode 100644 include/uapi/linux/xrt/xclbin.h
create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
--
2.17.1
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
@ 2020-11-29 0:00 ` Sonal Santan
2020-12-01 4:54 ` Moritz Fischer
2020-11-29 0:00 ` [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files Sonal Santan
` (8 subsequent siblings)
9 siblings, 1 reply; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
From: Sonal Santan <sonal.santan@xilinx.com>
Describe Alveo XRT driver architecture and provide basic overview
of Xilinx Alveo platform.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
---
Documentation/fpga/index.rst | 1 +
Documentation/fpga/xrt.rst | 588 +++++++++++++++++++++++++++++++++++
2 files changed, 589 insertions(+)
create mode 100644 Documentation/fpga/xrt.rst
diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
index f80f95667ca2..30134357b70d 100644
--- a/Documentation/fpga/index.rst
+++ b/Documentation/fpga/index.rst
@@ -8,6 +8,7 @@ fpga
:maxdepth: 1
dfl
+ xrt
.. only:: subproject and html
diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
new file mode 100644
index 000000000000..9f37d46459b0
--- /dev/null
+++ b/Documentation/fpga/xrt.rst
@@ -0,1 +1,588 @@
+==================================
+XRTV2 Linux Kernel Driver Overview
+==================================
+
+XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_ drivers which
+support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_ PCIe platforms
+from Xilinx.
+
+XRTV2 drivers support *subsystem* style data driven platforms where driver's configuration
+and behavior is determined by meta data provided by platform (in *device tree* format).
+Primary management physical function (MPF) driver is called **xmgmt**. Primary user physical
+function (UPF) driver is called **xuser** and HW subsystem drivers are packaged into a library
+module called **xrt-lib**, which is shared by **xmgmt** and **xuser** (WIP).
+
+Alveo Platform Overview
+=======================
+
+Alveo platforms are architected as two physical FPGA partitions: *Shell* and *User*. Shell
+provides basic infrastructure for the Alveo platform like PCIe connectivity, board management,
+Dynamic Function Exchange (DFX), sensors, clocking, reset, and security. User partition contains
+user compiled binary which is loaded by a process called DFX also known as partial reconfiguration.
+
+Physical partitions require strict HW compatibility with each other for DFX to work properly.
+Every physical partition has two interface UUIDs: *parent* UUID and *child* UUID. For simple
+single stage platforms Shell → User forms parent child relationship. For complex two stage
+platforms Base → Shell → User forms the parent child relationship chain.
+
+.. note::
+ Partition compatibility matching is key design component of Alveo platforms and XRT. Partitions
+ have child and parent relationship. A loaded partition exposes child partition UUID to advertise
+ its compatibility requirement for child partition. When loading a child partition the xmgmt
+ management driver matches parent UUID of the child partition against child UUID exported by the
+ parent. Parent and child partition UUIDs are stored in the *xclbin* (for user) or *xsabin* (for
+ base and shell). Except for root UUID, VSEC, hardware itself does not know about UUIDs. UUIDs are
+ stored in xsabin and xclbin.
+
+
+The physical partitions and their loading is illustrated below::
+
+ SHELL USER
+ +-----------+ +-------------------+
+ | | | |
+ | VSEC UUID | CHILD PARENT | LOGIC UUID |
+ | o------->|<--------o |
+ | | UUID UUID | |
+ +-----+-----+ +--------+----------+
+ | |
+ . .
+ | |
+ +---+---+ +------+--------+
+ | POR | | USER COMPILED |
+ | FLASH | | XCLBIN |
+ +-------+ +---------------+
+
+
+Loading Sequence
+----------------
+
+Shell partition is loaded from flash at system boot time. It establishes the PCIe link and exposes
+two physical functions to the BIOS. After OS boot, xmgmt driver attaches to PCIe physical function
+0 exposed by the Shell and then looks for VSEC in PCIe extended configuration space. Using VSEC it
+determines the logic UUID of Shell and uses the UUID to load matching *xsabin* file from Linux
+firmware directory. The xsabin file contains metadata to discover peripherals that are part of Shell
+and firmware(s) for any embedded soft processors in Shell.
+
+Shell exports child interface UUID which is used for compatibility check when loading user compiled
+xclbin over the User partition as part of DFX. When a user requests loading of a specific xclbin the
+xmgmt management driver reads the parent interface UUID specified in the xclbin and matches it with
+child interface UUID exported by Shell to determine if xclbin is compatible with the Shell. If match
+fails loading of xclbin is denied.
+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command. When loading xclbin xmgmt driver
+performs the following operations:
+
+1. Sanity check the xclbin contents
+2. Isolate the User partition
+3. Download the bitstream using the FPGA config engine (ICAP)
+4. De-isolate the User partition
+5. Program the clocks (ClockWiz) driving the User partition
+6. Wait for memory controller (MIG) calibration
+
+`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
+provides more detailed information on platform loading.
+
+xsabin
+------
+
+Each Alveo platform comes packaged with its own xsabin. The xsabin is trusted component of the
+platform. For format details refer to :ref:`xsabin/xclbin Container Format`. xsabin contains
+basic information like UUIDs, platform name and metadata in the form of device tree. See
+:ref:`Device Tree Usage` for details and example.
+
+xclbin
+------
+
+xclbin is compiled by end user using
+`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_ tool set from
+Xilinx. The xclbin contains sections describing user compiled acceleration engines/kernels, memory
+subsystems, clocking information etc. It also contains bitstream for the user partition, UUIDs,
+platform name, etc. xclbin uses the same container format as xsabin which is described below.
+
+
+xsabin/xclbin Container Format
+------------------------------
+
+xclbin/xsabin is ELF-like binary container format. It is structured as series of sections.
+There is a file header followed by several section headers which is followed by sections.
+A section header points to an actual section. There is an optional signature at the end.
+The format is defined by header file ``xclbin.h``. The following figure illustrates a
+typical xclbin::
+
+
+ +---------------------+
+ | |
+ | HEADER |
+ +---------------------+
+ | SECTION HEADER |
+ | |
+ +---------------------+
+ | ... |
+ | |
+ +---------------------+
+ | SECTION HEADER |
+ | |
+ +---------------------+
+ | SECTION |
+ | |
+ +---------------------+
+ | ... |
+ | |
+ +---------------------+
+ | SECTION |
+ | |
+ +---------------------+
+ | SIGNATURE |
+ | (OPTIONAL) |
+ +---------------------+
+
+
+xclbin/xsabin files can be packaged, un-packaged and inspected using XRT utility called
+**xclbinutil**. xclbinutil is part of XRT open source software stack. The source code for
+xclbinutil can be found at https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
+
+For example to enumerate the contents of a xclbin/xsabin use the *--info* switch as shown
+below::
+
+ xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
+ xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
+
+
+Device Tree Usage
+-----------------
+
+As mentioned previously xsabin stores metadata which advertise HW subsystems present in a partition.
+The metadata is stored in device tree format with well defined schema. Subsystem instantiations are
+captured as children of ``addressable_endpoints`` node. Subsystem nodes have standard attributes like
+``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe specific attributes:
+``pcie_physical_function`` and ``pcie_bar_mapping``. These identify which PCIe physical function and
+which BAR space in that physical function the subsystem resides. XRT management driver uses this
+information to bind *platform drivers* to the subsystem instantiations. The platform drivers are
+found in **xrt-lib.ko** kernel module defined later. Below is an example of device tree for Alveo U50
+platform::
+
+ /dts-v1/;
+
+ /{
+ logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
+
+ schema_version {
+ major = <0x01>;
+ minor = <0x00>;
+ };
+
+ interfaces {
+
+ @0 {
+ interface_uuid = "862c7020a250293e32036f19956669e5";
+ };
+ };
+
+ addressable_endpoints {
+
+ ep_blp_rom_00 {
+ reg = <0x00 0x1f04000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+ };
+
+ ep_card_flash_program_00 {
+ reg = <0x00 0x1f06000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
+ interrupts = <0x03 0x03>;
+ };
+
+ ep_cmc_firmware_mem_00 {
+ reg = <0x00 0x1e20000 0x00 0x20000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+ firmware {
+ firmware_product_name = "cmc";
+ firmware_branch_name = "u50";
+ firmware_version_major = <0x01>;
+ firmware_version_minor = <0x00>;
+ };
+ };
+
+ ep_cmc_intc_00 {
+ reg = <0x00 0x1e03000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+ interrupts = <0x04 0x04>;
+ };
+
+ ep_cmc_mutex_00 {
+ reg = <0x00 0x1e02000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_cmc_regmap_00 {
+ reg = <0x00 0x1e08000 0x00 0x2000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+ firmware {
+ firmware_product_name = "sc-fw";
+ firmware_branch_name = "u50";
+ firmware_version_major = <0x05>;
+ };
+ };
+
+ ep_cmc_reset_00 {
+ reg = <0x00 0x1e01000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_ddr_mem_calib_00 {
+ reg = <0x00 0x63000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_debug_bscan_mgmt_00 {
+ reg = <0x00 0x1e90000 0x00 0x10000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
+ };
+
+ ep_ert_base_address_00 {
+ reg = <0x00 0x21000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_ert_command_queue_mgmt_00 {
+ reg = <0x00 0x40000 0x00 0x10000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+ };
+
+ ep_ert_command_queue_user_00 {
+ reg = <0x00 0x40000 0x00 0x10000>;
+ pcie_physical_function = <0x01>;
+ compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+ };
+
+ ep_ert_firmware_mem_00 {
+ reg = <0x00 0x30000 0x00 0x8000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+ firmware {
+ firmware_product_name = "ert";
+ firmware_branch_name = "v20";
+ firmware_version_major = <0x01>;
+ };
+ };
+
+ ep_ert_intc_00 {
+ reg = <0x00 0x23000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+ interrupts = <0x05 0x05>;
+ };
+
+ ep_ert_reset_00 {
+ reg = <0x00 0x22000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_ert_sched_00 {
+ reg = <0x00 0x50000 0x00 0x1000>;
+ pcie_physical_function = <0x01>;
+ compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
+ interrupts = <0x09 0x0c>;
+ };
+
+ ep_fpga_configuration_00 {
+ reg = <0x00 0x1e88000 0x00 0x8000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
+ interrupts = <0x02 0x02>;
+ };
+
+ ep_icap_reset_00 {
+ reg = <0x00 0x1f07000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_mailbox_mgmt_00 {
+ reg = <0x00 0x1f10000 0x00 0x10000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+ interrupts = <0x00 0x00>;
+ };
+
+ ep_mailbox_user_00 {
+ reg = <0x00 0x1f00000 0x00 0x10000>;
+ pcie_physical_function = <0x01>;
+ compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+ interrupts = <0x08 0x08>;
+ };
+
+ ep_msix_00 {
+ reg = <0x00 0x00 0x00 0x20000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
+ pcie_bar_mapping = <0x02>;
+ };
+
+ ep_pcie_link_mon_00 {
+ reg = <0x00 0x1f05000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_pr_isolate_plp_00 {
+ reg = <0x00 0x1f01000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_pr_isolate_ulp_00 {
+ reg = <0x00 0x1000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_uuid_rom_00 {
+ reg = <0x00 0x64000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+ };
+
+ ep_xdma_00 {
+ reg = <0x00 0x00 0x00 0x10000>;
+ pcie_physical_function = <0x01>;
+ compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
+ pcie_bar_mapping = <0x02>;
+ };
+ };
+
+ }
+
+
+
+Deployment Models
+=================
+
+Baremetal
+---------
+
+In bare-metal deployments both MPF and UPF are visible and accessible. xmgmt driver binds to
+MPF. xmgmt driver operations are privileged and available to system administrator. The full
+stack is illustrated below::
+
+
+ HOST
+
+ [XMGMT] [XUSER]
+ | |
+ | |
+ +-----+ +-----+
+ | MPF | | UPF |
+ | | | |
+ | PF0 | | PF1 |
+ +--+--+ +--+--+
+ ......... ^................. ^..........
+ | |
+ | PCIe DEVICE |
+ | |
+ +--+------------------+--+
+ | SHELL |
+ | |
+ +------------------------+
+ | USER |
+ | |
+ | |
+ | |
+ | |
+ +------------------------+
+
+
+
+Virtualized
+-----------
+
+In virtualized deployments privileged MPF is assigned to host but unprivileged UPF
+is assigned to guest VM via PCIe pass-through. xmgmt driver in host binds to MPF.
+xmgmt driver operations are privileged and only accessible by hosting service provider.
+The full stack is illustrated below::
+
+
+ .............
+ HOST . VM .
+ . .
+ [XMGMT] . [XUSER] .
+ | . | .
+ | . | .
+ +-----+ . +-----+ .
+ | MPF | . | UPF | .
+ | | . | | .
+ | PF0 | . | PF1 | .
+ +--+--+ . +--+--+ .
+ ......... ^................. ^..........
+ | |
+ | PCIe DEVICE |
+ | |
+ +--+------------------+--+
+ | SHELL |
+ | |
+ +------------------------+
+ | USER |
+ | |
+ | |
+ | |
+ | |
+ +------------------------+
+
+
+
+Driver Modules
+==============
+
+xrt-lib.ko
+----------
+
+Repository of all subsystem drivers and pure software modules that can potentially
+be shared between xmgmt and xuser. All these drivers are structured as Linux
+*platform driver* and are instantiated by xmgmt (or xuser in future) based on meta
+data associated with hardware. The metadata is in the form of device tree as
+explained before.
+
+xmgmt.ko
+--------
+
+The xmgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
+PCIE device. It consists of one *root* driver, one or more *partition* drivers
+and one or more *leaf* drivers. The root and MPF specific leaf drivers are in
+xmgmt.ko. The partition driver and other leaf drivers are in xrt-lib.ko.
+
+The instantiation of specific partition driver or leaf driver is completely data
+driven based on meta data (mostly in device tree format) found through VSEC
+capability and inside firmware files, such as xsabin or xclbin file. The root
+driver manages life cycle of multiple partition drivers, which, in turn, manages
+multiple leaf drivers. This allows a single set of driver code to support all
+kinds of subsystems exposed by different shells. The difference among all
+these subsystems will be handled in leaf drivers with root and partition drivers
+being part of the infrastructure and provide common services for all leaves found
+on all platforms.
+
+
+xmgmt-root
+^^^^^^^^^^
+
+The xmgmt-root driver is a PCIe device driver attaches to MPF. It's part of the
+infrastructure of the MPF driver and resides in xmgmt.ko. This driver
+
+* manages one or more partition drivers
+* provides access to functionalities that requires pci_dev, such as PCIE config
+ space access, to other leaf drivers through parent calls
+* together with partition driver, facilities event callbacks for other leaf drivers
+* together with partition driver, facilities inter-leaf driver calls for other leaf
+ drivers
+
+When root driver starts, it will explicitly create an initial partition instance,
+which contains leaf drivers that will trigger the creation of other partition
+instances. The root driver will wait for all partitions and leaves to be created
+before it returns from it's probe routine and claim success of the initialization
+of the entire xmgmt driver.
+
+partition
+^^^^^^^^^
+
+The partition driver is a platform device driver whose life cycle is managed by
+root and does not have real IO mem or IRQ resources. It's part of the
+infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
+
+* manages one or more leaf drivers so that multiple leaves can be managed as a group
+* provides access to root from leaves, so that parent calls, event notifications
+ and inter-leaf calls can happen
+
+In xmgmt, an initial partition driver instance will be created by root, which
+contains leaves that will trigger partition instances to be created to manage
+groups of leaves found on different partitions on hardware, such as VSEC, Shell,
+and User.
+
+leaves
+^^^^^^
+
+The leaf driver is a platform device driver whose life cycle is managed by
+a partition driver and may or may not have real IO mem or IRQ resources. They
+are the real meat of xmgmt and contains platform specific code to Shell and User
+found on a MPF.
+
+A leaf driver may not have real hardware resources when it merely acts as a driver
+that manages certain in-memory states for xmgmt. These in-memory states could be
+shared by multiple other leaves.
+
+Leaf drivers assigned to specific hardware resources drive specific subsystem in
+the device. To manipulate the subsystem or carry out a task, a leaf driver may ask
+help from root via parent calls and/or from other leaves via inter-leaf calls.
+
+A leaf can also broadcast events through infrastructure code for other leaves
+to process. It can also receive event notification from infrastructure about certain
+events, such as post-creation or pre-exit of a particular leaf.
+
+
+Driver Interfaces
+=================
+
+xmgmt Driver Ioctls
+-------------------
+
+Ioctls exposed by xmgmt driver to user space are enumerated in the following table:
+
+== ===================== ============================= ===========================
+# Functionality ioctl request code data format
+== ===================== ============================= ===========================
+1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
+2 CL frequency scaling XMGMT_IOCFREQSCALE xmgmt_ioc_freqscaling
+== ===================== ============================= ===========================
+
+xmgmt Driver Sysfs
+------------------
+
+xmgmt driver exposes a rich set of sysfs interfaces. Subsystem platform drivers
+export sysfs node for every platform instance.
+
+Every partition also exports its UUIDs. See below for examples::
+
+ /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
+ /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
+
+
+hwmon
+-----
+
+xmgmt driver exposes standard hwmon interface to report voltage, current, temperature,
+power, etc. These can easily be viewed using *sensors* command line utility.
+
+
+mailbox
+-------
+
+xmgmt communicates with user physical function driver via HW mailbox. Mailbox opcodes
+are defined in ``mailbox_proto.h``. `Mailbox Inter-domain Communication Protocol
+<https://xilinx.github.io/XRT/master/html/mailbox.proto.html>`_ defines the full
+specification. xmgmt implements subset of the specification. It provides the following
+services to the UPF driver:
+
+1. Responding to *are you there* request including determining if the two drivers are
+ running in the same OS domain
+2. Provide sensor readings, loaded xclbin UUID, clock frequency, shell information, etc.
+3. Perform PCIe hot reset
+4. Download user compiled xclbin
+
+
+Platform Security Considerations
+================================
+
+`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
+discusses the deployment options and security implications in great detail.
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers Sonal Santan
@ 2020-11-29 0:00 ` Sonal Santan
2020-12-01 4:27 ` Moritz Fischer
2020-11-29 0:00 ` [PATCH Xilinx Alveo 3/8] fpga: xrt: infrastructure support for xmgmt driver Sonal Santan
` (7 subsequent siblings)
9 siblings, 1 reply; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
From: Sonal Santan <sonal.santan@xilinx.com>
Add XRT UAPI header files which describe flash layout, XRT
mailbox protocol, xclBin/axlf FPGA image container format and
XRT management physical function driver ioctl interfaces.
flash_xrt_data.h:
Layout used by XRT to store private data on flash.
mailbox_proto.h:
Mailbox opcodes and high level data structures representing
various kinds of information like sensors, clock, etc.
mailbox_transport.h:
Transport protocol used by mailbox.
xclbin.h:
Container format used to store compiled FPGA image which includes
bitstream and metadata.
xmgmt-ioctl.h:
Ioctls defined by management physical function driver:
* XMGMT_IOCICAPDOWNLOAD_AXLF
xclbin download which programs the user partition
* XMGMT_IOCFREQSCALE
Program the clocks driving user partition
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
---
include/uapi/linux/xrt/flash_xrt_data.h | 67 ++++
include/uapi/linux/xrt/mailbox_proto.h | 394 +++++++++++++++++++
include/uapi/linux/xrt/mailbox_transport.h | 74 ++++
include/uapi/linux/xrt/xclbin.h | 418 +++++++++++++++++++++
include/uapi/linux/xrt/xmgmt-ioctl.h | 72 ++++
5 files changed, 1025 insertions(+)
create mode 100644 include/uapi/linux/xrt/flash_xrt_data.h
create mode 100644 include/uapi/linux/xrt/mailbox_proto.h
create mode 100644 include/uapi/linux/xrt/mailbox_transport.h
create mode 100644 include/uapi/linux/xrt/xclbin.h
create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
diff --git a/include/uapi/linux/xrt/flash_xrt_data.h b/include/uapi/linux/xrt/flash_xrt_data.h
new file mode 100644
index 000000000000..0cafc2f38fbe
--- /dev/null
+++ b/include/uapi/linux/xrt/flash_xrt_data.h
@@ -0,0 +1,67 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _FLASH_XRT_DATA_H_
+#define _FLASH_XRT_DATA_H_
+
+#define XRT_DATA_MAGIC "XRTDATA"
+
+/*
+ * This header file contains data structure for xrt meta data on flash. This
+ * file is included in user space utilities and kernel drivers. The data
+ * structure is used to describe on-flash xrt data which is written by utility
+ * and read by driver. Any change of the data structure should either be
+ * backward compatible or cause version to be bumped up.
+ */
+
+struct flash_data_ident {
+ char fdi_magic[7];
+ char fdi_version;
+};
+
+/*
+ * On-flash meta data describing XRT data on flash. Either fdh_id_begin or
+ * fdh_id_end should be at well-known location on flash so that the reader
+ * can easily pick up fdi_version from flash before it tries to interpret
+ * the whole data structure.
+ * E.g., you align header in the end of the flash so that fdh_id_end is at well
+ * known location or align header at the beginning of the flash so that
+ * fdh_id_begin is at well known location.
+ */
+struct flash_data_header {
+ struct flash_data_ident fdh_id_begin;
+ uint32_t fdh_data_offset;
+ uint32_t fdh_data_len;
+ uint32_t fdh_data_parity;
+ uint8_t fdh_reserved[16];
+ struct flash_data_ident fdh_id_end;
+};
+
+static inline uint32_t flash_xrt_data_get_parity32(unsigned char *buf, size_t n)
+{
+ char *p;
+ size_t i;
+ size_t len;
+ uint32_t parity = 0;
+
+ for (len = 0; len < n; len += 4) {
+ uint32_t tmp = 0;
+ size_t thislen = n - len;
+
+ /* One word at a time. */
+ if (thislen > 4)
+ thislen = 4;
+
+ for (i = 0, p = (char *)&tmp; i < thislen; i++)
+ p[i] = buf[len + i];
+ parity ^= tmp;
+ }
+ return parity;
+}
+
+#endif
diff --git a/include/uapi/linux/xrt/mailbox_proto.h b/include/uapi/linux/xrt/mailbox_proto.h
new file mode 100644
index 000000000000..2aa782d86792
--- /dev/null
+++ b/include/uapi/linux/xrt/mailbox_proto.h
@@ -0,0 +1,394 @@
+/* SPDX-License-Identifier: Apache-2.0 OR GPL-2.0 */
+/*
+ * Copyright (C) 2019-2020, Xilinx Inc
+ */
+
+#ifndef _XCL_MB_PROTOCOL_H_
+#define _XCL_MB_PROTOCOL_H_
+
+#ifndef __KERNEL__
+#include <stdint.h>
+#else
+#include <linux/types.h>
+#endif
+
+/*
+ * This header file contains mailbox protocol b/w mgmt and user pfs.
+ * - Any changes made here should maintain backward compatibility.
+ * - If it's not possible, new OP code should be added and version number should
+ * be bumped up.
+ * - Support for old OP code should never be removed.
+ */
+#define XCL_MB_PROTOCOL_VER 0U
+
+/*
+ * UUID_SZ should ALWAYS have the same number
+ * as the MACRO UUID_SIZE defined in linux/uuid.h
+ */
+#define XCL_UUID_SZ 16
+
+/**
+ * enum mailbox_request - List of all mailbox request OPCODE. Some OP code
+ * requires arguments, which is defined as corresponding
+ * data structures below. Response to the request usually
+ * is a int32_t containing the error code. Some responses
+ * are more complicated and require a data structure,
+ * which is also defined below in this file.
+ * @XCL_MAILBOX_REQ_UNKNOWN: invalid OP code
+ * @XCL_MAILBOX_REQ_TEST_READY: test msg is ready (post only, internal test only)
+ * @XCL_MAILBOX_REQ_TEST_READ: fetch test msg from peer (internal test only)
+ * @XCL_MAILBOX_REQ_LOCK_BITSTREAM: lock down xclbin on mgmt pf (not implemented)
+ * @XCL_MAILBOX_REQ_UNLOCK_BITSTREAM: unlock xclbin on mgmt pf (not implemented)
+ * @XCL_MAILBOX_REQ_HOT_RESET: request mgmt pf driver to reset the board
+ * @XCL_MAILBOX_REQ_FIREWALL: firewall trip detected on mgmt pf (post only)
+ * @XCL_MAILBOX_REQ_LOAD_XCLBIN_KADDR: download xclbin (pointed to by a pointer)
+ * @XCL_MAILBOX_REQ_LOAD_XCLBIN: download xclbin (bitstream is in payload)
+ * @XCL_MAILBOX_REQ_RECLOCK: set clock frequency
+ * @XCL_MAILBOX_REQ_PEER_DATA: read specified data from peer
+ * @XCL_MAILBOX_REQ_USER_PROBE: for user pf to probe the peer mgmt pf
+ * @XCL_MAILBOX_REQ_MGMT_STATE: for mgmt pf to notify user pf of its state change
+ * (post only)
+ * @XCL_MAILBOX_REQ_CHG_SHELL: shell change is required on mgmt pf (post only)
+ * @XCL_MAILBOX_REQ_PROGRAM_SHELL: request mgmt pf driver to reprogram shell
+ */
+enum xcl_mailbox_request {
+ XCL_MAILBOX_REQ_UNKNOWN = 0,
+ XCL_MAILBOX_REQ_TEST_READY = 1,
+ XCL_MAILBOX_REQ_TEST_READ = 2,
+ XCL_MAILBOX_REQ_LOCK_BITSTREAM = 3,
+ XCL_MAILBOX_REQ_UNLOCK_BITSTREAM = 4,
+ XCL_MAILBOX_REQ_HOT_RESET = 5,
+ XCL_MAILBOX_REQ_FIREWALL = 6,
+ XCL_MAILBOX_REQ_LOAD_XCLBIN_KADDR = 7,
+ XCL_MAILBOX_REQ_LOAD_XCLBIN = 8,
+ XCL_MAILBOX_REQ_RECLOCK = 9,
+ XCL_MAILBOX_REQ_PEER_DATA = 10,
+ XCL_MAILBOX_REQ_USER_PROBE = 11,
+ XCL_MAILBOX_REQ_MGMT_STATE = 12,
+ XCL_MAILBOX_REQ_CHG_SHELL = 13,
+ XCL_MAILBOX_REQ_PROGRAM_SHELL = 14,
+ XCL_MAILBOX_REQ_READ_P2P_BAR_ADDR = 15,
+ /* Version 0 OP code ends */
+};
+
+static inline const char *mailbox_req2name(enum xcl_mailbox_request req)
+{
+ switch (req) {
+ case XCL_MAILBOX_REQ_TEST_READY: return "XCL_MAILBOX_REQ_TEST_READY";
+ case XCL_MAILBOX_REQ_TEST_READ: return "XCL_MAILBOX_REQ_TEST_READ";
+ case XCL_MAILBOX_REQ_LOCK_BITSTREAM: return "XCL_MAILBOX_REQ_LOCK_BITSTREAM";
+ case XCL_MAILBOX_REQ_UNLOCK_BITSTREAM: return "XCL_MAILBOX_REQ_UNLOCK_BITSTREAM";
+ case XCL_MAILBOX_REQ_HOT_RESET: return "XCL_MAILBOX_REQ_HOT_RESET";
+ case XCL_MAILBOX_REQ_FIREWALL: return "XCL_MAILBOX_REQ_FIREWALL";
+ case XCL_MAILBOX_REQ_LOAD_XCLBIN_KADDR: return "XCL_MAILBOX_REQ_LOAD_XCLBIN_KADDR";
+ case XCL_MAILBOX_REQ_LOAD_XCLBIN: return "XCL_MAILBOX_REQ_LOAD_XCLBIN";
+ case XCL_MAILBOX_REQ_RECLOCK: return "XCL_MAILBOX_REQ_RECLOCK";
+ case XCL_MAILBOX_REQ_PEER_DATA: return "XCL_MAILBOX_REQ_PEER_DATA";
+ case XCL_MAILBOX_REQ_USER_PROBE: return "XCL_MAILBOX_REQ_USER_PROBE";
+ case XCL_MAILBOX_REQ_MGMT_STATE: return "XCL_MAILBOX_REQ_MGMT_STATE";
+ case XCL_MAILBOX_REQ_CHG_SHELL: return "XCL_MAILBOX_REQ_CHG_SHELL";
+ case XCL_MAILBOX_REQ_PROGRAM_SHELL: return "XCL_MAILBOX_REQ_PROGRAM_SHELL";
+ case XCL_MAILBOX_REQ_READ_P2P_BAR_ADDR: return "XCL_MAILBOX_REQ_READ_P2P_BAR_ADDR";
+ default: return "UNKNOWN";
+ }
+}
+
+/**
+ * struct mailbox_req_bitstream_lock - MAILBOX_REQ_LOCK_BITSTREAM and
+ * MAILBOX_REQ_UNLOCK_BITSTREAM payload type
+ * @uuid: uuid of the xclbin
+ */
+struct xcl_mailbox_req_bitstream_lock {
+ uint64_t reserved;
+ uint8_t uuid[XCL_UUID_SZ];
+};
+
+/**
+ * enum group_kind - Groups of data that can be fetched from mgmt side
+ * @SENSOR: all kinds of sensor readings
+ * @ICAP: ICAP IP related information
+ * @BDINFO: Board Info, serial_num, mac_address
+ * @MIG_ECC: ECC statistics
+ * @FIREWALL: AF detected time, status
+ */
+enum xcl_group_kind {
+ XCL_SENSOR = 0,
+ XCL_ICAP,
+ XCL_BDINFO,
+ XCL_MIG_ECC,
+ XCL_FIREWALL,
+ XCL_DNA,
+ XCL_SUBDEV,
+};
+
+static inline const char *mailbox_group_kind2name(enum xcl_group_kind kind)
+{
+ switch (kind) {
+ case XCL_SENSOR: return "XCL_SENSOR";
+ case XCL_ICAP: return "XCL_ICAP";
+ case XCL_BDINFO: return "XCL_BDINFO";
+ case XCL_MIG_ECC: return "XCL_MIG_ECC";
+ case XCL_FIREWALL: return "XCL_FIREWALL";
+ case XCL_DNA: return "XCL_DNA";
+ case XCL_SUBDEV: return "XCL_SUBDEV";
+ default: return "UNKNOWN";
+ }
+}
+
+/**
+ * struct xcl_board_info - Data structure used to fetch BDINFO group
+ */
+#define BOARD_INFO_STR_LEN 256
+#define BOARD_INFO_MAC_LEN 6
+#define BOARD_INFO_PAD_LEN 26
+struct xcl_board_info {
+ char serial_num[BOARD_INFO_STR_LEN];
+ char mac_addr0[BOARD_INFO_MAC_LEN];
+ char padding0[BOARD_INFO_PAD_LEN];
+ char mac_addr1[BOARD_INFO_MAC_LEN];
+ char padding1[BOARD_INFO_PAD_LEN];
+ char mac_addr2[BOARD_INFO_MAC_LEN];
+ char padding2[BOARD_INFO_PAD_LEN];
+ char mac_addr3[BOARD_INFO_MAC_LEN];
+ char padding3[BOARD_INFO_PAD_LEN];
+ char revision[BOARD_INFO_STR_LEN];
+ char bd_name[BOARD_INFO_STR_LEN];
+ char bmc_ver[BOARD_INFO_STR_LEN];
+ uint32_t max_power;
+ uint32_t fan_presence;
+ uint32_t config_mode;
+ char exp_bmc_ver[BOARD_INFO_STR_LEN];
+ uint32_t mac_contiguous_num;
+ char mac_addr_first[BOARD_INFO_MAC_LEN];
+};
+
+/**
+ * struct xcl_sensor - Data structure used to fetch SENSOR group
+ */
+struct xcl_sensor {
+ uint32_t vol_12v_pex;
+ uint32_t vol_12v_aux;
+ uint32_t cur_12v_pex;
+ uint32_t cur_12v_aux;
+ uint32_t vol_3v3_pex;
+ uint32_t vol_3v3_aux;
+ uint32_t cur_3v3_aux;
+ uint32_t ddr_vpp_btm;
+ uint32_t sys_5v5;
+ uint32_t top_1v2;
+ uint32_t vol_1v8;
+ uint32_t vol_0v85;
+ uint32_t ddr_vpp_top;
+ uint32_t mgt0v9avcc;
+ uint32_t vol_12v_sw;
+ uint32_t mgtavtt;
+ uint32_t vcc1v2_btm;
+ uint32_t fpga_temp;
+ uint32_t fan_temp;
+ uint32_t fan_rpm;
+ uint32_t dimm_temp0;
+ uint32_t dimm_temp1;
+ uint32_t dimm_temp2;
+ uint32_t dimm_temp3;
+ uint32_t vccint_vol;
+ uint32_t vccint_curr;
+ uint32_t se98_temp0;
+ uint32_t se98_temp1;
+ uint32_t se98_temp2;
+ uint32_t cage_temp0;
+ uint32_t cage_temp1;
+ uint32_t cage_temp2;
+ uint32_t cage_temp3;
+ uint32_t hbm_temp0;
+ uint32_t cur_3v3_pex;
+ uint32_t cur_0v85;
+ uint32_t vol_3v3_vcc;
+ uint32_t vol_1v2_hbm;
+ uint32_t vol_2v5_vpp;
+ uint32_t vccint_bram;
+ uint32_t version;
+ uint32_t oem_id;
+ uint32_t vccint_temp;
+ uint32_t vol_12v_aux1;
+ uint32_t vol_vcc1v2_i;
+ uint32_t vol_v12_in_i;
+ uint32_t vol_v12_in_aux0_i;
+ uint32_t vol_v12_in_aux1_i;
+ uint32_t vol_vccaux;
+ uint32_t vol_vccaux_pmc;
+ uint32_t vol_vccram;
+};
+
+/**
+ * struct xcl_hwicap - Data structure used to fetch ICAP group
+ */
+struct xcl_pr_region {
+ uint64_t freq_data;
+ uint64_t freq_kernel;
+ uint64_t freq_system;
+ uint64_t freq_unused;
+ uint64_t freq_cntr_data;
+ uint64_t freq_cntr_kernel;
+ uint64_t freq_cntr_system;
+ uint64_t freq_cntr_unused;
+ uint64_t idcode;
+ uint8_t uuid[XCL_UUID_SZ];
+ uint64_t mig_calib;
+ uint64_t data_retention;
+};
+
+/**
+ * struct xcl_mig_ecc - Data structure used to fetch MIG_ECC group
+ */
+struct xcl_mig_ecc {
+ uint64_t mem_type;
+ uint64_t mem_idx;
+ uint64_t ecc_enabled;
+ uint64_t ecc_status;
+ uint64_t ecc_ce_cnt;
+ uint64_t ecc_ue_cnt;
+ uint64_t ecc_ce_ffa;
+ uint64_t ecc_ue_ffa;
+};
+
+/**
+ * struct xcl_firewall - Data structure used to fetch FIREWALL group
+ */
+struct xcl_firewall {
+ uint64_t max_level;
+ uint64_t curr_status;
+ uint64_t curr_level;
+ uint64_t err_detected_status;
+ uint64_t err_detected_level;
+ uint64_t err_detected_time;
+};
+
+
+/**
+ * struct xcl_dna - Data structure used to fetch DNA group
+ */
+struct xcl_dna {
+ uint64_t status;
+ uint32_t dna[4];
+ uint64_t capability;
+ uint64_t dna_version;
+ uint64_t revision;
+};
+/**
+ * Data structure used to fetch SUBDEV group
+ */
+enum xcl_subdev_return_code {
+ XRT_MSG_SUBDEV_RTN_UNCHANGED = 1,
+ XRT_MSG_SUBDEV_RTN_PARTIAL,
+ XRT_MSG_SUBDEV_RTN_COMPLETE,
+ XRT_MSG_SUBDEV_RTN_PENDINGPLP,
+};
+struct xcl_subdev {
+ uint32_t ver;
+ enum xcl_subdev_return_code rtncode;
+ uint64_t checksum;
+ uint64_t size;
+ uint64_t offset;
+ uint64_t data[1];
+};
+/**
+ * struct mailbox_subdev_peer - MAILBOX_REQ_PEER_DATA payload type
+ * @kind: data group
+ * @size: buffer size for receiving response
+ */
+struct xcl_mailbox_peer_data {
+ enum xcl_group_kind kind;
+ uint32_t padding;
+ uint64_t size;
+ uint64_t entries;
+ uint64_t offset;
+};
+
+/**
+ * struct mailbox_conn - MAILBOX_REQ_USER_PROBE payload type
+ * @kaddr: KVA of the verification data buffer
+ * @paddr: physical addresss of the verification data buffer
+ * @crc32: CRC value of the verification data buffer
+ * @version: protocol version supported by peer
+ */
+struct xcl_mailbox_conn {
+ uint64_t kaddr;
+ uint64_t paddr;
+ uint32_t crc32;
+ uint32_t version;
+};
+
+#define XCL_COMM_ID_SIZE 2048
+#define XCL_MB_PEER_READY (1UL << 0)
+#define XCL_MB_PEER_SAME_DOMAIN (1UL << 1)
+/**
+ * struct mailbox_conn_resp - MAILBOX_REQ_USER_PROBE response payload type
+ * @version: protocol version should be used
+ * @conn_flags: connection status
+ * @chan_switch: bitmap to indicate SW / HW channel for each OP code msg
+ * @comm_id: user defined cookie
+ */
+struct xcl_mailbox_conn_resp {
+ uint32_t version;
+ uint32_t reserved;
+ uint64_t conn_flags;
+ uint64_t chan_switch;
+ char comm_id[XCL_COMM_ID_SIZE];
+};
+
+#define XCL_MB_STATE_ONLINE (1UL << 0)
+#define XCL_MB_STATE_OFFLINE (1UL << 1)
+/**
+ * struct mailbox_peer_state - MAILBOX_REQ_MGMT_STATE payload type
+ * @state_flags: peer state flags
+ */
+struct xcl_mailbox_peer_state {
+ uint64_t state_flags;
+};
+
+/**
+ * struct mailbox_bitstream_kaddr - MAILBOX_REQ_LOAD_XCLBIN_KADDR payload type
+ * @addr: pointer to xclbin body
+ */
+struct xcl_mailbox_bitstream_kaddr {
+ uint64_t addr;
+};
+
+/**
+ * struct mailbox_clock_freqscaling - MAILBOX_REQ_RECLOCK payload type
+ * @region: region of clock
+ * @target_freqs: array of target clock frequencies (max clocks: 16)
+ */
+struct xcl_mailbox_clock_freqscaling {
+ unsigned int region;
+ unsigned short target_freqs[16];
+};
+
+/**
+ * struct mailbox_req - mailbox request message header
+ * @req: opcode
+ * @flags: flags of this message
+ * @data: payload of variable length
+ */
+struct xcl_mailbox_req {
+ uint64_t flags;
+ enum xcl_mailbox_request req;
+ char data[1]; /* variable length of payload */
+};
+
+/**
+ * struct mailbox_p2p_bar_addr
+ * @bar_addr: p2p bar address
+ * @bar_len: p2p bar length
+ */
+struct xcl_mailbox_p2p_bar_addr {
+ uint64_t p2p_bar_addr;
+ uint64_t p2p_bar_len;
+};
+
+static inline const char *mailbox_chan2name(bool sw_ch)
+{
+ return sw_ch ? "SW-CHANNEL" : "HW-CHANNEL";
+}
+
+#endif /* _XCL_MB_PROTOCOL_H_ */
diff --git a/include/uapi/linux/xrt/mailbox_transport.h b/include/uapi/linux/xrt/mailbox_transport.h
new file mode 100644
index 000000000000..4823446797a6
--- /dev/null
+++ b/include/uapi/linux/xrt/mailbox_transport.h
@@ -0,0 +1,74 @@
+/* SPDX-License-Identifier: Apache-2.0 OR GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XCL_MB_TRANSPORT_H_
+#define _XCL_MB_TRANSPORT_H_
+
+/*
+ * This header file contains data structures used in mailbox transport layer
+ * b/w mgmt and user pfs. Any changes made here should maintain backward
+ * compatibility.
+ */
+
+/**
+ * struct sw_chan - mailbox software channel message metadata. This defines the
+ * interface between daemons (MPD and MSD) and mailbox's
+ * read or write callbacks. A mailbox message (either a request
+ * or response) is wrapped by this data structure as payload.
+ * A sw_chan is passed between mailbox driver and daemon via
+ * read / write driver callbacks. And it is also passed between
+ * MPD and MSD via vendor defined interface (TCP socket, etc).
+ * @sz: payload size
+ * @flags: flags of this message as in struct mailbox_req
+ * @id: message ID
+ * @data: payload (struct mailbox_req or response data matching the request)
+ */
+struct xcl_sw_chan {
+ uint64_t sz;
+ uint64_t flags;
+ uint64_t id;
+ char data[1]; /* variable length of payload */
+};
+
+/**
+ * A packet transport by mailbox hardware channel.
+ * When extending, only add new data structure to body. Choose to add new flag
+ * if new feature can be safely ignored by peer, other wise, add new type.
+ */
+enum packet_type {
+ PKT_INVALID = 0,
+ PKT_TEST,
+ PKT_MSG_START,
+ PKT_MSG_BODY
+};
+
+#define PACKET_SIZE 16 /* Number of DWORD. */
+
+/* Lower 8 bits for type, the rest for flags. Total packet size is 64 bytes */
+#define PKT_TYPE_MASK 0xff
+#define PKT_TYPE_MSG_END (1 << 31)
+struct mailbox_pkt {
+ struct {
+ u32 type;
+ u32 payload_size;
+ } hdr;
+ union {
+ u32 data[PACKET_SIZE - 2];
+ struct {
+ u64 msg_req_id;
+ u32 msg_flags;
+ u32 msg_size;
+ u32 payload[0];
+ } msg_start;
+ struct {
+ u32 payload[0];
+ } msg_body;
+ } body;
+} __packed;
+
+#endif /* _XCL_MB_TRANSPORT_H_ */
diff --git a/include/uapi/linux/xrt/xclbin.h b/include/uapi/linux/xrt/xclbin.h
new file mode 100644
index 000000000000..885cae1700f9
--- /dev/null
+++ b/include/uapi/linux/xrt/xclbin.h
@@ -0,0 +1,418 @@
+/* SPDX-License-Identifier: Apache-2.0 OR GPL-2.0 */
+/*
+ * Xilinx FPGA compiled binary container format
+ *
+ * Copyright (C) 2015-2020, Xilinx Inc
+ */
+
+
+#ifndef _XCLBIN_H_
+#define _XCLBIN_H_
+
+#ifdef _WIN32
+ #include <cstdint>
+ #include <algorithm>
+ #include "windows/uuid.h"
+#else
+ #if defined(__KERNEL__)
+ #include <linux/types.h>
+ #include <linux/uuid.h>
+ #include <linux/version.h>
+ #elif defined(__cplusplus)
+ #include <cstdlib>
+ #include <cstdint>
+ #include <algorithm>
+ #include <uuid/uuid.h>
+ #else
+ #include <stdlib.h>
+ #include <stdint.h>
+ #include <uuid/uuid.h>
+ #endif
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+ /**
+ * Container format for Xilinx bitstreams, metadata and other
+ * binary blobs.
+ * Every segment must be aligned at 8 byte boundary with null byte padding
+ * between adjacent segments if required.
+ * For segements which are not present both offset and length must be 0 in
+ * the header.
+ * Currently only xclbin0\0 is recognized as file magic. In future if/when file
+ * format is updated the magic string will be changed to xclbin1\0 and so on.
+ */
+ enum XCLBIN_MODE {
+ XCLBIN_FLAT,
+ XCLBIN_PR,
+ XCLBIN_TANDEM_STAGE2,
+ XCLBIN_TANDEM_STAGE2_WITH_PR,
+ XCLBIN_HW_EMU,
+ XCLBIN_SW_EMU,
+ XCLBIN_MODE_MAX
+ };
+
+ /*
+ * AXLF LAYOUT
+ * -----------
+ *
+ * -----------------------------------------
+ * | Magic |
+ * -----------------------------------------
+ * | Header |
+ * -----------------------------------------
+ * | One or more section headers |
+ * -----------------------------------------
+ * | Matching number of sections with data |
+ * -----------------------------------------
+ *
+ */
+
+ enum axlf_section_kind {
+ BITSTREAM = 0,
+ CLEARING_BITSTREAM,
+ EMBEDDED_METADATA,
+ FIRMWARE,
+ DEBUG_DATA,
+ SCHED_FIRMWARE,
+ MEM_TOPOLOGY,
+ CONNECTIVITY,
+ IP_LAYOUT,
+ DEBUG_IP_LAYOUT,
+ DESIGN_CHECK_POINT,
+ CLOCK_FREQ_TOPOLOGY,
+ MCS,
+ BMC,
+ BUILD_METADATA,
+ KEYVALUE_METADATA,
+ USER_METADATA,
+ DNA_CERTIFICATE,
+ PDI,
+ BITSTREAM_PARTIAL_PDI,
+ PARTITION_METADATA,
+ EMULATION_DATA,
+ SYSTEM_METADATA,
+ SOFT_KERNEL,
+ ASK_FLASH,
+ AIE_METADATA,
+ ASK_GROUP_TOPOLOGY,
+ ASK_GROUP_CONNECTIVITY
+ };
+
+ enum MEM_TYPE {
+ MEM_DDR3,
+ MEM_DDR4,
+ MEM_DRAM,
+ MEM_STREAMING,
+ MEM_PREALLOCATED_GLOB,
+ MEM_ARE, //Aurora
+ MEM_HBM,
+ MEM_BRAM,
+ MEM_URAM,
+ MEM_STREAMING_CONNECTION
+ };
+
+ enum IP_TYPE {
+ IP_MB = 0,
+ IP_KERNEL, //kernel instance
+ IP_DNASC,
+ IP_DDR4_CONTROLLER,
+ IP_MEM_DDR4,
+ IP_MEM_HBM
+ };
+
+ struct axlf_section_header {
+ uint32_t m_sectionKind; /* Section type */
+ char m_sectionName[16]; /* Examples: "stage2", "clear1", "clear2", "ocl1", "ocl2, "ublaze", "sched" */
+ uint64_t m_sectionOffset; /* File offset of section data */
+ uint64_t m_sectionSize; /* Size of section data */
+ };
+
+ struct axlf_header {
+ uint64_t m_length; /* Total size of the xclbin file */
+ uint64_t m_timeStamp; /* Number of seconds since epoch when xclbin was created */
+ uint64_t m_featureRomTimeStamp; /* TimeSinceEpoch of the featureRom */
+ uint16_t m_versionPatch; /* Patch Version */
+ uint8_t m_versionMajor; /* Major Version - Version: 2.1.0*/
+ uint8_t m_versionMinor; /* Minor Version */
+ uint32_t m_mode; /* XCLBIN_MODE */
+ union {
+ struct {
+ uint64_t m_platformId; /* 64 bit platform ID: vendor-device-subvendor-subdev */
+ uint64_t m_featureId; /* 64 bit feature id */
+ } rom;
+ unsigned char rom_uuid[16]; /* feature ROM UUID for which this xclbin was generated */
+ };
+ unsigned char m_platformVBNV[64]; /* e.g. xilinx:xil-accel-rd-ku115:4ddr-xpr:3.4: null terminated */
+ union {
+ char m_next_axlf[16]; /* Name of next xclbin file in the daisy chain */
+ uuid_t uuid; /* uuid of this xclbin*/
+ };
+ char m_debug_bin[16]; /* Name of binary with debug information */
+ uint32_t m_numSections; /* Number of section headers */
+ };
+
+ struct axlf {
+ char m_magic[8]; /* Should be "xclbin2\0" */
+ int32_t m_signature_length; /* Length of the signature. -1 indicates no signature */
+ unsigned char reserved[28]; /* Note: Initialized to 0xFFs */
+
+ unsigned char m_keyBlock[256]; /* Signature for validation of binary */
+ uint64_t m_uniqueId; /* axlf's uniqueId, use it to skip redownload etc */
+ struct axlf_header m_header; /* Inline header */
+ struct axlf_section_header m_sections[1]; /* One or more section headers follow */
+ };
+
+ typedef struct axlf xclBin;
+
+ /**** BEGIN : Xilinx internal section *****/
+
+ /* bitstream information */
+ struct xlnx_bitstream {
+ uint8_t m_freq[8];
+ char bits[1];
+ };
+
+ /**** MEMORY TOPOLOGY SECTION ****/
+ struct mem_data {
+ uint8_t m_type; //enum corresponding to mem_type.
+ uint8_t m_used; //if 0 this bank is not present
+ union {
+ uint64_t m_size; //if mem_type DDR, then size in KB;
+ uint64_t route_id; //if streaming then "route_id"
+ };
+ union {
+ uint64_t m_base_address;//if DDR then the base address;
+ uint64_t flow_id; //if streaming then "flow id"
+ };
+ unsigned char m_tag[16]; //DDR: BANK0,1,2,3, has to be null terminated; if streaming then stream0, 1 etc
+ };
+
+ struct mem_topology {
+ int32_t m_count; //Number of mem_data
+ struct mem_data m_mem_data[1]; //Should be sorted on mem_type
+ };
+
+ /**** CONNECTIVITY SECTION ****/
+ /* Connectivity of each argument of Kernel. It will be in terms of argument
+ * index associated. For associating kernel instances with arguments and
+ * banks, start at the connectivity section. Using the m_ip_layout_index
+ * access the ip_data.m_name. Now we can associate this kernel instance
+ * with its original kernel name and get the connectivity as well. This
+ * enables us to form related groups of kernel instances.
+ */
+
+ struct connection {
+ int32_t arg_index; //From 0 to n, may not be contiguous as scalars skipped
+ int32_t m_ip_layout_index; //index into the ip_layout section. ip_layout.m_ip_data[index].m_type == IP_KERNEL
+ int32_t mem_data_index; //index of the m_mem_data . Flag error is m_used false.
+ };
+
+ struct connectivity {
+ int32_t m_count;
+ struct connection m_connection[1];
+ };
+
+
+ /**** IP_LAYOUT SECTION ****/
+
+ // IP Kernel
+ #define IP_INT_ENABLE_MASK 0x0001
+ #define IP_INTERRUPT_ID_MASK 0x00FE
+ #define IP_INTERRUPT_ID_SHIFT 0x1
+
+ enum IP_CONTROL {
+ AP_CTRL_HS = 0,
+ AP_CTRL_CHAIN = 1,
+ AP_CTRL_NONE = 2,
+ AP_CTRL_ME = 3,
+ ACCEL_ADAPTER = 4
+ };
+
+ #define IP_CONTROL_MASK 0xFF00
+ #define IP_CONTROL_SHIFT 0x8
+
+ /* IPs on AXI lite - their types, names, and base addresses.*/
+ struct ip_data {
+ uint32_t m_type; //map to IP_TYPE enum
+ union {
+ uint32_t properties; // Default: 32-bits to indicate ip specific property.
+ // m_type: IP_KERNEL
+ // m_int_enable : Bit - 0x0000_0001;
+ // m_interrupt_id : Bits - 0x0000_00FE;
+ // m_ip_control : Bits = 0x0000_FF00;
+ struct { // m_type: IP_MEM_*
+ uint16_t m_index;
+ uint8_t m_pc_index;
+ uint8_t unused;
+ } indices;
+ };
+ uint64_t m_base_address;
+ uint8_t m_name[64]; //eg Kernel name corresponding to KERNEL instance, can embed CU name in future.
+ };
+
+ struct ip_layout {
+ int32_t m_count;
+ struct ip_data m_ip_data[1]; //All the ip_data needs to be sorted by m_base_address.
+ };
+
+ /*** Debug IP section layout ****/
+ enum DEBUG_IP_TYPE {
+ UNDEFINED = 0,
+ LAPC,
+ ILA,
+ AXI_MM_MONITOR,
+ AXI_TRACE_FUNNEL,
+ AXI_MONITOR_FIFO_LITE,
+ AXI_MONITOR_FIFO_FULL,
+ ACCEL_MONITOR,
+ AXI_STREAM_MONITOR,
+ AXI_STREAM_PROTOCOL_CHECKER,
+ TRACE_S2MM,
+ AXI_DMA,
+ TRACE_S2MM_FULL
+ };
+
+ struct debug_ip_data {
+ uint8_t m_type; // type of enum DEBUG_IP_TYPE
+ uint8_t m_index_lowbyte;
+ uint8_t m_properties;
+ uint8_t m_major;
+ uint8_t m_minor;
+ uint8_t m_index_highbyte;
+ uint8_t m_reserved[2];
+ uint64_t m_base_address;
+ char m_name[128];
+ };
+
+ struct debug_ip_layout {
+ uint16_t m_count;
+ struct debug_ip_data m_debug_ip_data[1];
+ };
+
+ enum CLOCK_TYPE { /* Supported clock frequency types */
+ CT_UNUSED = 0, /* Initialized value */
+ CT_DATA = 1, /* Data clock */
+ CT_KERNEL = 2, /* Kernel clock */
+ CT_SYSTEM = 3 /* System Clock */
+ };
+
+ struct clock_freq { /* Clock Frequency Entry */
+ uint16_t m_freq_Mhz; /* Frequency in MHz */
+ uint8_t m_type; /* Clock type (enum CLOCK_TYPE) */
+ uint8_t m_unused[5]; /* Not used - padding */
+ char m_name[128]; /* Clock Name */
+ };
+
+ struct clock_freq_topology { /* Clock frequency section */
+ int16_t m_count; /* Number of entries */
+ struct clock_freq m_clock_freq[1]; /* Clock array */
+ };
+
+ enum MCS_TYPE { /* Supported MCS file types */
+ MCS_UNKNOWN = 0, /* Initialized value */
+ MCS_PRIMARY = 1, /* The primary mcs file data */
+ MCS_SECONDARY = 2, /* The secondary mcs file data */
+ };
+
+ struct mcs_chunk { /* One chunk of MCS data */
+ uint8_t m_type; /* MCS data type */
+ uint8_t m_unused[7]; /* padding */
+ uint64_t m_offset; /* data offset from the start of the section */
+ uint64_t m_size; /* data size */
+ };
+
+ struct mcs { /* MCS data section */
+ int8_t m_count; /* Number of chunks */
+ int8_t m_unused[7]; /* padding */
+ struct mcs_chunk m_chunk[1]; /* MCS chunks followed by data */
+ };
+
+ struct bmc { /* bmc data section */
+ uint64_t m_offset; /* data offset from the start of the section */
+ uint64_t m_size; /* data size (bytes)*/
+ char m_image_name[64]; /* Name of the image (e.g., MSP432P401R) */
+ char m_device_name[64]; /* Device ID (e.g., VCU1525) */
+ char m_version[64];
+ char m_md5value[33]; /* MD5 Expected Value(e.g., 56027182079c0bd621761b7dab5a27ca)*/
+ char m_padding[7]; /* Padding */
+ };
+
+ struct soft_kernel { /* soft kernel data section */
+ // Prefix Syntax:
+ // mpo - member, pointer, offset
+ // This variable represents a zero terminated string
+ // that is offseted from the beginning of the section.
+ //
+ // The pointer to access the string is initialized as follows:
+ // char * pCharString = (address_of_section) + (mpo value)
+ uint32_t mpo_name; // Name of the soft kernel
+ uint32_t m_image_offset; // Image offset
+ uint32_t m_image_size; // Image size
+ uint32_t mpo_version; // Version
+ uint32_t mpo_md5_value; // MD5 checksum
+ uint32_t mpo_symbol_name; // Symbol name
+ uint32_t m_num_instances; // Number of instances
+ uint8_t padding[36]; // Reserved for future use
+ uint8_t reservedExt[16]; // Reserved for future extended data
+ };
+
+ enum CHECKSUM_TYPE
+ {
+ CST_UNKNOWN = 0,
+ CST_SDBM = 1,
+ CST_LAST
+ };
+
+ /**** END : Xilinx internal section *****/
+
+# ifdef __cplusplus
+ namespace xclbin {
+ inline const axlf_section_header*
+ get_axlf_section(const axlf* top, axlf_section_kind kind)
+ {
+ auto begin = top->m_sections;
+ auto end = begin + top->m_header.m_numSections;
+ auto itr = std::find_if(begin,end,[kind](const axlf_section_header& sec) { return sec.m_sectionKind==(const uint32_t) kind; });
+ return (itr!=end) ? &(*itr) : nullptr;
+ }
+
+ // Helper C++ section iteration
+ // To keep with with the current "coding" them, the function get_axlf_section_next() was
+ // introduced find 'next' common section names.
+ //
+ // Future TODO: Create a custom iterator and refactor the code base to use it.
+ //
+ // Example on how this function may be used:
+ //
+ // const axlf_section_header * pSection;
+ // const axlf* top = <xclbin image in memory>;
+ // for (pSection = xclbin::get_axlf_section( top, SOFT_KERNEL);
+ // pSection != nullptr;
+ // pSection = xclbin::get_axlf_section_next( top, pSection, SOFT_KERNEL)) {
+ // <code to do work>
+ // }
+ inline const axlf_section_header*
+ get_axlf_section_next(const axlf* top, const axlf_section_header* current_section, axlf_section_kind kind)
+ {
+ if (top == nullptr) { return nullptr; }
+ if (current_section == nullptr) { return nullptr; }
+
+ auto end = top->m_sections + top->m_header.m_numSections;
+
+ auto begin = current_section + 1; // Point to the next section
+ if (begin == end) { return nullptr; }
+
+ auto itr = std::find_if(begin, end, [kind](const axlf_section_header &sec) {return sec.m_sectionKind == (const uint32_t)kind;});
+ return (itr!=end) ? &(*itr) : nullptr;
+ }
+ }
+# endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/include/uapi/linux/xrt/xmgmt-ioctl.h b/include/uapi/linux/xrt/xmgmt-ioctl.h
new file mode 100644
index 000000000000..f949a7c21560
--- /dev/null
+++ b/include/uapi/linux/xrt/xmgmt-ioctl.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: Apache-2.0 OR GPL-2.0 */
+/*
+ * Copyright (C) 2015-2020, Xilinx Inc
+ *
+ */
+
+/**
+ * DOC: PCIe Kernel Driver for Managament Physical Function
+ * Interfaces exposed by *xclmgmt* driver are defined in file, *mgmt-ioctl.h*.
+ * Core functionality provided by *xmgmt* driver is described in the following table:
+ *
+ * ==== ====================================== ============================== ==================================
+ * # Functionality ioctl request code data format
+ * ==== ====================================== ============================== ==================================
+ * 1 FPGA image download XCLMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
+ * 2 CL frequency scaling XCLMGMT_IOCFREQSCALE xmgmt_ioc_freqscaling
+ * ==== ====================================== ============================== ==================================
+ *
+ */
+
+#ifndef _XMGMT_IOCALLS_POSIX_H_
+#define _XMGMT_IOCALLS_POSIX_H_
+
+#include <linux/ioctl.h>
+
+#define XMGMT_IOC_MAGIC 'X'
+#define XMGMT_NUM_SUPPORTED_CLOCKS 4
+
+#define XMGMT_IOC_FREQ_SCALE 0x2
+#define XMGMT_IOC_ICAP_DOWNLOAD_AXLF 0x6
+
+
+/**
+ * struct xmgmt_ioc_bitstream_axlf - load xclbin (AXLF) device image
+ * used with XMGMT_IOCICAPDOWNLOAD_AXLF ioctl
+ *
+ * @xclbin: Pointer to user's xclbin structure in memory
+ */
+struct xmgmt_ioc_bitstream_axlf {
+ struct axlf *xclbin;
+};
+
+/**
+ * struct xmgmt_ioc_freqscaling - scale frequencies on the board using Xilinx clock wizard
+ * used with XMGMT_IOCFREQSCALE ioctl
+ *
+ * @ocl_region: PR region (currently only 0 is supported)
+ * @ocl_target_freq: Array of requested frequencies, a value o zero in the array indicates leave untouched
+ */
+struct xmgmt_ioc_freqscaling {
+ unsigned int ocl_region;
+ unsigned short ocl_target_freq[XMGMT_NUM_SUPPORTED_CLOCKS];
+};
+
+#define DATA_CLK 0
+#define KERNEL_CLK 1
+#define SYSTEM_CLK 2
+
+#define XMGMT_IOCICAPDOWNLOAD_AXLF _IOW(XMGMT_IOC_MAGIC, XMGMT_IOC_ICAP_DOWNLOAD_AXLF, struct xmgmt_ioc_bitstream_axlf)
+#define XMGMT_IOCFREQSCALE _IOW(XMGMT_IOC_MAGIC, XMGMT_IOC_FREQ_SCALE, struct xmgmt_ioc_freqscaling)
+
+/*
+ * The following definitions are for binary compatibility with classic XRT management driver
+ */
+
+#define XCLMGMT_IOCICAPDOWNLOAD_AXLF XMGMT_IOCICAPDOWNLOAD_AXLF
+#define XCLMGMT_IOCFREQSCALE XMGMT_IOCFREQSCALE
+
+#define xclmgmt_ioc_bitstream_axlf xmgmt_ioc_bitstream_axlf
+#define xclmgmt_ioc_freqscaling xmgmt_ioc_freqscaling
+
+#endif
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH Xilinx Alveo 3/8] fpga: xrt: infrastructure support for xmgmt driver
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files Sonal Santan
@ 2020-11-29 0:00 ` Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 4/8] fpga: xrt: core infrastructure for xrt-lib module Sonal Santan
` (6 subsequent siblings)
9 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
From: Sonal Santan <sonal.santan@xilinx.com>
Add infrastructure code for XRT management physical function
driver. This provides support for enumerating and extracting
sections from xclbin files, interacting with device tree nodes
found in xclbin and working with Alveo partitions.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
---
drivers/fpga/alveo/common/xrt-metadata.c | 590 ++++++++++++++++++
drivers/fpga/alveo/common/xrt-root.c | 744 +++++++++++++++++++++++
drivers/fpga/alveo/common/xrt-root.h | 24 +
drivers/fpga/alveo/common/xrt-xclbin.c | 387 ++++++++++++
drivers/fpga/alveo/common/xrt-xclbin.h | 46 ++
5 files changed, 1791 insertions(+)
create mode 100644 drivers/fpga/alveo/common/xrt-metadata.c
create mode 100644 drivers/fpga/alveo/common/xrt-root.c
create mode 100644 drivers/fpga/alveo/common/xrt-root.h
create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.c
create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.h
diff --git a/drivers/fpga/alveo/common/xrt-metadata.c b/drivers/fpga/alveo/common/xrt-metadata.c
new file mode 100644
index 000000000000..5596619ed82d
--- /dev/null
+++ b/drivers/fpga/alveo/common/xrt-metadata.c
@@ -0,0 +1,590 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Metadata parse APIs
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#include <linux/libfdt_env.h>
+#include "libfdt.h"
+#include "xrt-metadata.h"
+
+#define MAX_BLOB_SIZE (4096 * 25)
+
+#define md_err(dev, fmt, args...) \
+ dev_err(dev, "%s: "fmt, __func__, ##args)
+#define md_warn(dev, fmt, args...) \
+ dev_warn(dev, "%s: "fmt, __func__, ##args)
+#define md_info(dev, fmt, args...) \
+ dev_info(dev, "%s: "fmt, __func__, ##args)
+#define md_dbg(dev, fmt, args...) \
+ dev_dbg(dev, "%s: "fmt, __func__, ##args)
+
+static int xrt_md_setprop(struct device *dev, char *blob, int offset,
+ const char *prop, const void *val, int size);
+static int xrt_md_overlay(struct device *dev, char *blob, int target,
+ const char *overlay_blob, int overlay_offset);
+static int xrt_md_get_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name, int *ep_offset);
+
+long xrt_md_size(struct device *dev, const char *blob)
+{
+ long len = (long) fdt_totalsize(blob);
+
+ return (len > MAX_BLOB_SIZE) ? -EINVAL : len;
+}
+
+int xrt_md_create(struct device *dev, char **blob)
+{
+ int ret = 0;
+
+ WARN_ON(!blob);
+
+ *blob = vmalloc(MAX_BLOB_SIZE);
+ if (!*blob)
+ return -ENOMEM;
+
+ ret = fdt_create_empty_tree(*blob, MAX_BLOB_SIZE);
+ if (ret) {
+ md_err(dev, "format blob failed, ret = %d", ret);
+ goto failed;
+ }
+
+ ret = fdt_next_node(*blob, -1, NULL);
+ if (ret < 0) {
+ md_err(dev, "No Node, ret = %d", ret);
+ goto failed;
+ }
+
+ ret = fdt_add_subnode(*blob, ret, NODE_ENDPOINTS);
+ if (ret < 0)
+ md_err(dev, "add node failed, ret = %d", ret);
+
+failed:
+ if (ret < 0) {
+ vfree(*blob);
+ *blob = NULL;
+ } else
+ ret = 0;
+
+ return ret;
+}
+
+int xrt_md_add_node(struct device *dev, char *blob, int parent_offset,
+ const char *ep_name)
+{
+ int ret;
+
+ ret = fdt_add_subnode(blob, parent_offset, ep_name);
+ if (ret < 0 && ret != -FDT_ERR_EXISTS)
+ md_err(dev, "failed to add node %s. %d", ep_name, ret);
+
+ return ret;
+}
+
+int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
+ char *regmap_name)
+{
+ int ret;
+ int ep_offset;
+
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name, &ep_offset);
+ if (ret) {
+ md_err(dev, "can not find ep %s", ep_name);
+ return -EINVAL;
+ }
+
+ ret = fdt_del_node(blob, ep_offset);
+ if (ret)
+ md_err(dev, "delete node %s failed, ret %d", ep_name, ret);
+
+ return ret;
+}
+
+static int __xrt_md_add_endpoint(struct device *dev, char *blob,
+ struct xrt_md_endpoint *ep, int *offset, bool root)
+{
+ int ret = 0;
+ int ep_offset;
+ u32 val, count = 0;
+ u64 io_range[2];
+ char comp[128];
+
+ if (!ep->ep_name) {
+ md_err(dev, "empty name");
+ return -EINVAL;
+ }
+
+ if (!root) {
+ ret = xrt_md_get_endpoint(dev, blob, NODE_ENDPOINTS, NULL,
+ &ep_offset);
+ if (ret) {
+ md_err(dev, "invalid blob, ret = %d", ret);
+ return -EINVAL;
+ }
+ } else {
+ ep_offset = 0;
+ }
+
+ ep_offset = xrt_md_add_node(dev, blob, ep_offset, ep->ep_name);
+ if (ep_offset < 0) {
+ md_err(dev, "add endpoint failed, ret = %d", ret);
+ return -EINVAL;
+ }
+ if (offset)
+ *offset = ep_offset;
+
+ if (ep->size != 0) {
+ val = cpu_to_be32(ep->bar);
+ ret = xrt_md_setprop(dev, blob, ep_offset, PROP_BAR_IDX,
+ &val, sizeof(u32));
+ if (ret) {
+ md_err(dev, "set %s failed, ret %d",
+ PROP_BAR_IDX, ret);
+ goto failed;
+ }
+ io_range[0] = cpu_to_be64((u64)ep->bar_off);
+ io_range[1] = cpu_to_be64((u64)ep->size);
+ ret = xrt_md_setprop(dev, blob, ep_offset, PROP_IO_OFFSET,
+ io_range, sizeof(io_range));
+ if (ret) {
+ md_err(dev, "set %s failed, ret %d",
+ PROP_IO_OFFSET, ret);
+ goto failed;
+ }
+ }
+
+ if (ep->regmap) {
+ if (ep->regmap_ver) {
+ count = snprintf(comp, sizeof(comp),
+ "%s-%s", ep->regmap, ep->regmap_ver);
+ count++;
+ }
+
+ count += snprintf(comp + count, sizeof(comp) - count,
+ "%s", ep->regmap);
+ count++;
+
+ ret = xrt_md_setprop(dev, blob, ep_offset, PROP_COMPATIBLE,
+ comp, count);
+ if (ret) {
+ md_err(dev, "set %s failed, ret %d",
+ PROP_COMPATIBLE, ret);
+ goto failed;
+ }
+ }
+
+failed:
+ if (ret)
+ xrt_md_del_endpoint(dev, blob, ep->ep_name, NULL);
+
+ return ret;
+}
+
+int xrt_md_add_endpoint(struct device *dev, char *blob,
+ struct xrt_md_endpoint *ep)
+{
+ return __xrt_md_add_endpoint(dev, blob, ep, NULL, false);
+}
+
+static int xrt_md_get_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name, int *ep_offset)
+{
+ int offset;
+ const char *name;
+
+ for (offset = fdt_next_node(blob, -1, NULL);
+ offset >= 0;
+ offset = fdt_next_node(blob, offset, NULL)) {
+ name = fdt_get_name(blob, offset, NULL);
+ if (!name || strncmp(name, ep_name, strlen(ep_name) + 1))
+ continue;
+ if (!regmap_name ||
+ !fdt_node_check_compatible(blob, offset, regmap_name))
+ break;
+ }
+ if (offset < 0)
+ return -ENODEV;
+
+ *ep_offset = offset;
+
+ return 0;
+}
+
+int xrt_md_get_epname_pointer(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name, const char **epname)
+{
+ int offset;
+ int ret;
+
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
+ &offset);
+ if (!ret && epname && offset >= 0)
+ *epname = fdt_get_name(blob, offset, NULL);
+
+ return ret;
+}
+
+int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
+ const char *regmap_name, const char *prop, const void **val, int *size)
+{
+ int offset;
+ int ret;
+
+ if (val)
+ *val = NULL;
+ if (ep_name) {
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
+ &offset);
+ if (ret) {
+ md_err(dev, "cannot get ep %s, regmap %s, ret = %d",
+ ep_name, regmap_name, ret);
+ return -EINVAL;
+ }
+ } else {
+ offset = fdt_next_node(blob, -1, NULL);
+ if (offset < 0) {
+ md_err(dev, "internal error, ret = %d", offset);
+ return -EINVAL;
+ }
+ }
+
+ if (val) {
+ *val = fdt_getprop(blob, offset, prop, size);
+ if (!*val) {
+ md_dbg(dev, "get ep %s, prop %s failed", ep_name, prop);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int xrt_md_setprop(struct device *dev, char *blob, int offset,
+ const char *prop, const void *val, int size)
+{
+ int ret;
+
+ ret = fdt_setprop(blob, offset, prop, val, size);
+ if (ret)
+ md_err(dev, "failed to set prop %d", ret);
+
+ return ret;
+}
+
+int xrt_md_set_prop(struct device *dev, char *blob,
+ const char *ep_name, const char *regmap_name,
+ const char *prop, const void *val, int size)
+{
+ int offset;
+ int ret;
+
+ if (ep_name) {
+ ret = xrt_md_get_endpoint(dev, blob, ep_name,
+ regmap_name, &offset);
+ if (ret) {
+ md_err(dev, "cannot get node %s, ret = %d",
+ ep_name, ret);
+ return -EINVAL;
+ }
+ } else {
+ offset = fdt_next_node(blob, -1, NULL);
+ if (offset < 0) {
+ md_err(dev, "internal error, ret = %d", offset);
+ return -EINVAL;
+ }
+ }
+
+ ret = xrt_md_setprop(dev, blob, offset, prop, val, size);
+ if (ret)
+ md_err(dev, "set prop %s failed, ret = %d", prop, ret);
+
+ return ret;
+}
+
+int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
+ const char *ep_name, const char *regmap_name, const char *new_ep_name)
+{
+ int offset, target;
+ int ret;
+ struct xrt_md_endpoint ep = {0};
+ const char *newepnm = new_ep_name ? new_ep_name : ep_name;
+
+ ret = xrt_md_get_endpoint(dev, src_blob, ep_name, regmap_name,
+ &offset);
+ if (ret)
+ return -EINVAL;
+
+ ret = xrt_md_get_endpoint(dev, blob, newepnm, regmap_name, &target);
+ if (ret) {
+ ep.ep_name = newepnm;
+ ret = __xrt_md_add_endpoint(dev, blob, &ep, &target,
+ fdt_parent_offset(src_blob, offset) == 0);
+ if (ret)
+ return -EINVAL;
+ }
+
+ ret = xrt_md_overlay(dev, blob, target, src_blob, offset);
+ if (ret)
+ md_err(dev, "overlay failed, ret = %d", ret);
+
+ return ret;
+}
+
+int xrt_md_copy_all_eps(struct device *dev, char *blob, const char *src_blob)
+{
+ return xrt_md_copy_endpoint(dev, blob, src_blob, NODE_ENDPOINTS,
+ NULL, NULL);
+}
+
+char *xrt_md_dup(struct device *dev, const char *blob)
+{
+ int ret;
+ char *dup_blob;
+
+ ret = xrt_md_create(dev, &dup_blob);
+ if (ret)
+ return NULL;
+ ret = xrt_md_overlay(dev, dup_blob, -1, blob, -1);
+ if (ret) {
+ vfree(dup_blob);
+ return NULL;
+ }
+
+ return dup_blob;
+}
+
+static int xrt_md_overlay(struct device *dev, char *blob, int target,
+ const char *overlay_blob, int overlay_offset)
+{
+ int property, subnode;
+ int ret;
+
+ WARN_ON(!blob || !overlay_blob);
+
+ if (!blob) {
+ md_err(dev, "blob is NULL");
+ return -EINVAL;
+ }
+
+ if (target < 0) {
+ target = fdt_next_node(blob, -1, NULL);
+ if (target < 0) {
+ md_err(dev, "invalid target");
+ return -EINVAL;
+ }
+ }
+ if (overlay_offset < 0) {
+ overlay_offset = fdt_next_node(overlay_blob, -1, NULL);
+ if (overlay_offset < 0) {
+ md_err(dev, "invalid overlay");
+ return -EINVAL;
+ }
+ }
+
+ fdt_for_each_property_offset(property, overlay_blob, overlay_offset) {
+ const char *name;
+ const void *prop;
+ int prop_len;
+
+ prop = fdt_getprop_by_offset(overlay_blob, property, &name,
+ &prop_len);
+ if (!prop || prop_len >= MAX_BLOB_SIZE) {
+ md_err(dev, "internal error");
+ return -EINVAL;
+ }
+
+ ret = xrt_md_setprop(dev, blob, target, name, prop,
+ prop_len);
+ if (ret) {
+ md_err(dev, "setprop failed, ret = %d", ret);
+ return ret;
+ }
+ }
+
+ fdt_for_each_subnode(subnode, overlay_blob, overlay_offset) {
+ const char *name = fdt_get_name(overlay_blob, subnode, NULL);
+ int nnode;
+
+ nnode = xrt_md_add_node(dev, blob, target, name);
+ if (nnode == -FDT_ERR_EXISTS)
+ nnode = fdt_subnode_offset(blob, target, name);
+ if (nnode < 0) {
+ md_err(dev, "add node failed, ret = %d", nnode);
+ return nnode;
+ }
+
+ ret = xrt_md_overlay(dev, blob, nnode, overlay_blob, subnode);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ char **next_ep, char **next_regmap)
+{
+ int offset, ret;
+
+ if (!ep_name) {
+ ret = xrt_md_get_endpoint(dev, blob, NODE_ENDPOINTS, NULL,
+ &offset);
+ } else {
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
+ &offset);
+ }
+
+ if (ret) {
+ *next_ep = NULL;
+ *next_regmap = NULL;
+ return -EINVAL;
+ }
+
+ offset = ep_name ? fdt_next_subnode(blob, offset) :
+ fdt_first_subnode(blob, offset);
+ if (offset < 0) {
+ *next_ep = NULL;
+ *next_regmap = NULL;
+ return -EINVAL;
+ }
+
+ *next_ep = (char *)fdt_get_name(blob, offset, NULL);
+ *next_regmap = (char *)fdt_stringlist_get(blob, offset, PROP_COMPATIBLE,
+ 0, NULL);
+
+ return 0;
+}
+
+int xrt_md_get_compatible_epname(struct device *dev, const char *blob,
+ const char *regmap_name, const char **ep_name)
+{
+ int ep_offset;
+
+ ep_offset = fdt_node_offset_by_compatible(blob, -1, regmap_name);
+ if (ep_offset < 0) {
+ *ep_name = NULL;
+ return -ENOENT;
+ }
+
+ *ep_name = (char *)fdt_get_name(blob, ep_offset, NULL);
+
+ return 0;
+}
+
+int xrt_md_uuid_strtoid(struct device *dev, const char *uuidstr, uuid_t *p_uuid)
+{
+ char *p;
+ const char *str;
+ char tmp[3] = { 0 };
+ int i, ret;
+
+ memset(p_uuid, 0, sizeof(*p_uuid));
+ p = (char *)p_uuid;
+ str = uuidstr + strlen(uuidstr) - 2;
+
+ for (i = 0; i < sizeof(*p_uuid) && str >= uuidstr; i++) {
+ tmp[0] = *str;
+ tmp[1] = *(str + 1);
+ ret = kstrtou8(tmp, 16, p);
+ if (ret) {
+ md_err(dev, "Invalid uuid %s", uuidstr);
+ return -EINVAL;
+ }
+ p++;
+ str -= 2;
+ }
+
+ return 0;
+}
+
+void xrt_md_pack(struct device *dev, char *blob)
+{
+ int ret;
+
+ ret = fdt_pack(blob);
+ if (ret)
+ md_err(dev, "pack failed %d", ret);
+}
+
+int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
+ u32 *num_uuids, uuid_t *intf_uuids)
+{
+ int offset, count = 0;
+ int ret;
+ const char *uuid_str;
+
+ ret = xrt_md_get_endpoint(dev, blob, NODE_INTERFACES, NULL, &offset);
+ if (ret)
+ return -ENOENT;
+
+ for (offset = fdt_first_subnode(blob, offset);
+ offset >= 0;
+ offset = fdt_next_subnode(blob, offset)) {
+ uuid_str = fdt_getprop(blob, offset, PROP_INTERFACE_UUID,
+ NULL);
+ if (!uuid_str) {
+ md_err(dev, "empty intf uuid node");
+ return -EINVAL;
+ }
+
+ if (intf_uuids && count < *num_uuids) {
+ ret = xrt_md_uuid_strtoid(dev, uuid_str,
+ &intf_uuids[count]);
+ if (ret)
+ return -EINVAL;
+ }
+ count++;
+ }
+
+ *num_uuids = count;
+
+ return 0;
+}
+
+int xrt_md_check_uuids(struct device *dev, const char *blob, char *subset_blob)
+{
+ const char *subset_int_uuid = NULL;
+ const char *int_uuid = NULL;
+ int offset, subset_offset, off;
+ int ret;
+
+ ret = xrt_md_get_endpoint(dev, subset_blob, NODE_INTERFACES, NULL,
+ &subset_offset);
+ if (ret)
+ return -EINVAL;
+
+ ret = xrt_md_get_endpoint(dev, blob, NODE_INTERFACES, NULL,
+ &offset);
+ if (ret)
+ return -EINVAL;
+
+ for (subset_offset = fdt_first_subnode(subset_blob, subset_offset);
+ subset_offset >= 0;
+ subset_offset = fdt_next_subnode(subset_blob, subset_offset)) {
+ subset_int_uuid = fdt_getprop(subset_blob, subset_offset,
+ PROP_INTERFACE_UUID, NULL);
+ if (!subset_int_uuid)
+ return -EINVAL;
+ off = offset;
+
+ for (off = fdt_first_subnode(blob, off);
+ off >= 0;
+ off = fdt_next_subnode(blob, off)) {
+ int_uuid = fdt_getprop(blob, off,
+ PROP_INTERFACE_UUID, NULL);
+ if (!int_uuid)
+ return -EINVAL;
+ if (!strcmp(int_uuid, subset_int_uuid))
+ break;
+ }
+ if (off < 0)
+ return -ENOENT;
+ }
+
+ return 0;
+}
diff --git a/drivers/fpga/alveo/common/xrt-root.c b/drivers/fpga/alveo/common/xrt-root.c
new file mode 100644
index 000000000000..7cc23dfaf3cf
--- /dev/null
+++ b/drivers/fpga/alveo/common/xrt-root.c
@@ -0,0 +1,744 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/hwmon.h>
+#include "xrt-subdev.h"
+#include "xrt-parent.h"
+#include "xrt-partition.h"
+#include "xrt-root.h"
+#include "xrt-metadata.h"
+#include "xrt-root.h"
+
+#define XROOT_PDEV(xr) ((xr)->pdev)
+#define XROOT_DEV(xr) (&(XROOT_PDEV(xr)->dev))
+#define xroot_err(xr, fmt, args...) \
+ dev_err(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+#define xroot_warn(xr, fmt, args...) \
+ dev_warn(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+#define xroot_info(xr, fmt, args...) \
+ dev_info(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+#define xroot_dbg(xr, fmt, args...) \
+ dev_dbg(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+
+#define XRT_VSEC_ID 0x20
+#define XROOT_PART_FIRST (-1)
+#define XROOT_PART_LAST (-2)
+
+static int xroot_parent_cb(struct device *, void *, u32, void *);
+
+struct xroot_async_evt {
+ struct list_head list;
+ struct xrt_parent_ioctl_async_broadcast_evt evt;
+};
+
+struct xroot_event_cb {
+ struct list_head list;
+ bool initialized;
+ struct xrt_parent_ioctl_evt_cb cb;
+};
+
+struct xroot_events {
+ struct list_head cb_list;
+ struct mutex cb_lock;
+ struct work_struct cb_init_work;
+ struct mutex async_evt_lock;
+ struct list_head async_evt_list;
+ struct work_struct async_evt_work;
+};
+
+struct xroot_parts {
+ struct xrt_subdev_pool pool;
+ struct work_struct bringup_work;
+ atomic_t bringup_pending;
+ atomic_t bringup_failed;
+ struct completion bringup_comp;
+};
+
+struct xroot {
+ struct pci_dev *pdev;
+ struct xroot_events events;
+ struct xroot_parts parts;
+};
+
+struct xroot_part_match_arg {
+ enum xrt_subdev_id id;
+ int instance;
+};
+
+static bool xroot_part_match(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ struct xroot_part_match_arg *a = (struct xroot_part_match_arg *)arg;
+ return id == a->id && pdev->id == a->instance;
+}
+
+static int xroot_get_partition(struct xroot *xr, int instance,
+ struct platform_device **partp)
+{
+ int rc = 0;
+ struct xrt_subdev_pool *parts = &xr->parts.pool;
+ struct device *dev = DEV(xr->pdev);
+ struct xroot_part_match_arg arg = { XRT_SUBDEV_PART, instance };
+
+ if (instance == XROOT_PART_LAST) {
+ rc = xrt_subdev_pool_get(parts, XRT_SUBDEV_MATCH_NEXT,
+ *partp, dev, partp);
+ } else if (instance == XROOT_PART_FIRST) {
+ rc = xrt_subdev_pool_get(parts, XRT_SUBDEV_MATCH_PREV,
+ *partp, dev, partp);
+ } else {
+ rc = xrt_subdev_pool_get(parts, xroot_part_match,
+ &arg, dev, partp);
+ }
+
+ if (rc && rc != -ENOENT)
+ xroot_err(xr, "failed to hold partition %d: %d", instance, rc);
+ return rc;
+}
+
+static void xroot_put_partition(struct xroot *xr, struct platform_device *part)
+{
+ int inst = part->id;
+ int rc = xrt_subdev_pool_put(&xr->parts.pool, part, DEV(xr->pdev));
+
+ if (rc)
+ xroot_err(xr, "failed to release partition %d: %d", inst, rc);
+}
+
+static int
+xroot_partition_trigger_evt(struct xroot *xr, struct xroot_event_cb *cb,
+ struct platform_device *part, enum xrt_events evt)
+{
+ xrt_subdev_match_t match = cb->cb.xevt_match_cb;
+ xrt_event_cb_t evtcb = cb->cb.xevt_cb;
+ void *arg = cb->cb.xevt_match_arg;
+ struct xrt_partition_ioctl_event e = { evt, &cb->cb };
+ struct xrt_event_arg_subdev esd = { XRT_SUBDEV_PART, part->id };
+ int rc;
+
+ if (match(XRT_SUBDEV_PART, part, arg)) {
+ rc = evtcb(cb->cb.xevt_pdev, evt, &esd);
+ if (rc)
+ return rc;
+ }
+
+ return xrt_subdev_ioctl(part, XRT_PARTITION_EVENT, &e);
+}
+
+static void
+xroot_event_partition(struct xroot *xr, int instance, enum xrt_events evt)
+{
+ int ret;
+ struct platform_device *pdev = NULL;
+ const struct list_head *ptr, *next;
+ struct xroot_event_cb *tmp;
+
+ BUG_ON(instance < 0);
+ ret = xroot_get_partition(xr, instance, &pdev);
+ if (ret)
+ return;
+
+ mutex_lock(&xr->events.cb_lock);
+ list_for_each_safe(ptr, next, &xr->events.cb_list) {
+ int rc;
+
+ tmp = list_entry(ptr, struct xroot_event_cb, list);
+ if (!tmp->initialized)
+ continue;
+
+ rc = xroot_partition_trigger_evt(xr, tmp, pdev, evt);
+ if (rc) {
+ list_del(&tmp->list);
+ vfree(tmp);
+ }
+ }
+ mutex_unlock(&xr->events.cb_lock);
+
+ (void) xroot_put_partition(xr, pdev);
+}
+
+int xroot_create_partition(void *root, char *dtb)
+{
+ struct xroot *xr = (struct xroot *)root;
+ int ret;
+
+ atomic_inc(&xr->parts.bringup_pending);
+ ret = xrt_subdev_pool_add(&xr->parts.pool,
+ XRT_SUBDEV_PART, xroot_parent_cb, xr, dtb);
+ if (ret >= 0) {
+ schedule_work(&xr->parts.bringup_work);
+ } else {
+ atomic_dec(&xr->parts.bringup_pending);
+ atomic_inc(&xr->parts.bringup_failed);
+ xroot_err(xr, "failed to create partition: %d", ret);
+ }
+ return ret;
+}
+
+static int xroot_destroy_single_partition(struct xroot *xr, int instance)
+{
+ struct platform_device *pdev = NULL;
+ int ret;
+
+ BUG_ON(instance < 0);
+ ret = xroot_get_partition(xr, instance, &pdev);
+ if (ret)
+ return ret;
+
+ xroot_event_partition(xr, instance, XRT_EVENT_PRE_REMOVAL);
+
+ /* Now tear down all children in this partition. */
+ ret = xrt_subdev_ioctl(pdev, XRT_PARTITION_FINI_CHILDREN, NULL);
+ (void) xroot_put_partition(xr, pdev);
+ if (!ret) {
+ ret = xrt_subdev_pool_del(&xr->parts.pool,
+ XRT_SUBDEV_PART, instance);
+ }
+
+ return ret;
+}
+
+static int xroot_destroy_partition(struct xroot *xr, int instance)
+{
+ struct platform_device *target = NULL;
+ struct platform_device *deps = NULL;
+ int ret;
+
+ BUG_ON(instance < 0);
+ /*
+ * Make sure target partition exists and can't go away before
+ * we remove it's dependents
+ */
+ ret = xroot_get_partition(xr, instance, &target);
+ if (ret)
+ return ret;
+
+ /*
+ * Remove all partitions depend on target one.
+ * Assuming subdevs in higher partition ID can depend on ones in
+ * lower ID partitions, we remove them in the reservse order.
+ */
+ while (xroot_get_partition(xr, XROOT_PART_LAST, &deps) != -ENOENT) {
+ int inst = deps->id;
+
+ xroot_put_partition(xr, deps);
+ if (instance == inst)
+ break;
+ (void) xroot_destroy_single_partition(xr, inst);
+ deps = NULL;
+ }
+
+ /* Now we can remove the target partition. */
+ xroot_put_partition(xr, target);
+ return xroot_destroy_single_partition(xr, instance);
+}
+
+static int xroot_lookup_partition(struct xroot *xr,
+ struct xrt_parent_ioctl_lookup_partition *arg)
+{
+ int rc = -ENOENT;
+ struct platform_device *part = NULL;
+
+ while (rc < 0 && xroot_get_partition(xr, XROOT_PART_LAST,
+ &part) != -ENOENT) {
+ if (arg->xpilp_match_cb(XRT_SUBDEV_PART, part,
+ arg->xpilp_match_arg)) {
+ rc = part->id;
+ }
+ xroot_put_partition(xr, part);
+ }
+ return rc;
+}
+
+static void xroot_evt_cb_init_work(struct work_struct *work)
+{
+ const struct list_head *ptr, *next;
+ struct xroot_event_cb *tmp;
+ struct platform_device *part = NULL;
+ struct xroot *xr =
+ container_of(work, struct xroot, events.cb_init_work);
+
+ mutex_lock(&xr->events.cb_lock);
+
+ list_for_each_safe(ptr, next, &xr->events.cb_list) {
+ tmp = list_entry(ptr, struct xroot_event_cb, list);
+ if (tmp->initialized)
+ continue;
+
+ while (xroot_get_partition(xr, XROOT_PART_LAST,
+ &part) != -ENOENT) {
+ int rc = xroot_partition_trigger_evt(xr, tmp, part,
+ XRT_EVENT_POST_CREATION);
+
+ (void) xroot_put_partition(xr, part);
+ if (rc & XRT_EVENT_CB_STOP) {
+ list_del(&tmp->list);
+ vfree(tmp);
+ tmp = NULL;
+ break;
+ }
+ }
+
+ if (tmp)
+ tmp->initialized = true;
+ }
+
+ mutex_unlock(&xr->events.cb_lock);
+}
+
+static bool xroot_evt(struct xroot *xr, enum xrt_events evt)
+{
+ const struct list_head *ptr, *next;
+ struct xroot_event_cb *tmp;
+ int rc;
+ bool success = true;
+
+ mutex_lock(&xr->events.cb_lock);
+ list_for_each_safe(ptr, next, &xr->events.cb_list) {
+ tmp = list_entry(ptr, struct xroot_event_cb, list);
+ rc = tmp->cb.xevt_cb(tmp->cb.xevt_pdev, evt, NULL);
+ if (rc & XRT_EVENT_CB_ERR)
+ success = false;
+ if (rc & XRT_EVENT_CB_STOP) {
+ list_del(&tmp->list);
+ vfree(tmp);
+ }
+ }
+ mutex_unlock(&xr->events.cb_lock);
+
+ return success;
+}
+
+static void xroot_evt_async_evt_work(struct work_struct *work)
+{
+ struct xroot_async_evt *tmp;
+ struct xroot *xr =
+ container_of(work, struct xroot, events.async_evt_work);
+ bool success;
+
+ mutex_lock(&xr->events.async_evt_lock);
+ while (!list_empty(&xr->events.async_evt_list)) {
+ tmp = list_first_entry(&xr->events.async_evt_list,
+ struct xroot_async_evt, list);
+ list_del(&tmp->list);
+ mutex_unlock(&xr->events.async_evt_lock);
+
+ success = xroot_evt(xr, tmp->evt.xaevt_event);
+ if (tmp->evt.xaevt_cb) {
+ tmp->evt.xaevt_cb(tmp->evt.xaevt_pdev,
+ tmp->evt.xaevt_event, tmp->evt.xaevt_arg,
+ success);
+ }
+ vfree(tmp);
+
+ mutex_lock(&xr->events.async_evt_lock);
+ }
+ mutex_unlock(&xr->events.async_evt_lock);
+}
+
+static void xroot_evt_init(struct xroot *xr)
+{
+ INIT_LIST_HEAD(&xr->events.cb_list);
+ INIT_LIST_HEAD(&xr->events.async_evt_list);
+ mutex_init(&xr->events.async_evt_lock);
+ mutex_init(&xr->events.cb_lock);
+ INIT_WORK(&xr->events.cb_init_work, xroot_evt_cb_init_work);
+ INIT_WORK(&xr->events.async_evt_work, xroot_evt_async_evt_work);
+}
+
+static void xroot_evt_fini(struct xroot *xr)
+{
+ const struct list_head *ptr, *next;
+ struct xroot_event_cb *tmp;
+
+ flush_scheduled_work();
+
+ BUG_ON(!list_empty(&xr->events.async_evt_list));
+
+ mutex_lock(&xr->events.cb_lock);
+ list_for_each_safe(ptr, next, &xr->events.cb_list) {
+ tmp = list_entry(ptr, struct xroot_event_cb, list);
+ list_del(&tmp->list);
+ vfree(tmp);
+ }
+ mutex_unlock(&xr->events.cb_lock);
+}
+
+static int xroot_evt_cb_add(struct xroot *xr,
+ struct xrt_parent_ioctl_evt_cb *cb)
+{
+ struct xroot_event_cb *new = vzalloc(sizeof(*new));
+
+ if (!new)
+ return -ENOMEM;
+
+ cb->xevt_hdl = new;
+ new->cb = *cb;
+ new->initialized = false;
+
+ mutex_lock(&xr->events.cb_lock);
+ list_add(&new->list, &xr->events.cb_list);
+ mutex_unlock(&xr->events.cb_lock);
+
+ schedule_work(&xr->events.cb_init_work);
+ return 0;
+}
+
+static int xroot_async_evt_add(struct xroot *xr,
+ struct xrt_parent_ioctl_async_broadcast_evt *arg)
+{
+ struct xroot_async_evt *new = vzalloc(sizeof(*new));
+
+ if (!new)
+ return -ENOMEM;
+
+ new->evt = *arg;
+
+ mutex_lock(&xr->events.async_evt_lock);
+ list_add(&new->list, &xr->events.async_evt_list);
+ mutex_unlock(&xr->events.async_evt_lock);
+
+ schedule_work(&xr->events.async_evt_work);
+ return 0;
+}
+
+static void xroot_evt_cb_del(struct xroot *xr, void *hdl)
+{
+ struct xroot_event_cb *cb = (struct xroot_event_cb *)hdl;
+ const struct list_head *ptr;
+ struct xroot_event_cb *tmp;
+
+ mutex_lock(&xr->events.cb_lock);
+ list_for_each(ptr, &xr->events.cb_list) {
+ tmp = list_entry(ptr, struct xroot_event_cb, list);
+ if (tmp == cb)
+ break;
+ }
+ list_del(&cb->list);
+ mutex_unlock(&xr->events.cb_lock);
+ vfree(cb);
+}
+
+static int xroot_get_leaf(struct xroot *xr,
+ struct xrt_parent_ioctl_get_leaf *arg)
+{
+ int rc = -ENOENT;
+ struct platform_device *part = NULL;
+
+ while (rc && xroot_get_partition(xr, XROOT_PART_LAST,
+ &part) != -ENOENT) {
+ rc = xrt_subdev_ioctl(part, XRT_PARTITION_GET_LEAF, arg);
+ xroot_put_partition(xr, part);
+ }
+ return rc;
+}
+
+static int xroot_put_leaf(struct xroot *xr,
+ struct xrt_parent_ioctl_put_leaf *arg)
+{
+ int rc = -ENOENT;
+ struct platform_device *part = NULL;
+
+ while (rc && xroot_get_partition(xr, XROOT_PART_LAST,
+ &part) != -ENOENT) {
+ rc = xrt_subdev_ioctl(part, XRT_PARTITION_PUT_LEAF, arg);
+ xroot_put_partition(xr, part);
+ }
+ return rc;
+}
+
+static int xroot_parent_cb(struct device *dev, void *parg, u32 cmd, void *arg)
+{
+ struct xroot *xr = (struct xroot *)parg;
+ int rc = 0;
+
+ switch (cmd) {
+ /* Leaf actions. */
+ case XRT_PARENT_GET_LEAF: {
+ struct xrt_parent_ioctl_get_leaf *getleaf =
+ (struct xrt_parent_ioctl_get_leaf *)arg;
+ rc = xroot_get_leaf(xr, getleaf);
+ break;
+ }
+ case XRT_PARENT_PUT_LEAF: {
+ struct xrt_parent_ioctl_put_leaf *putleaf =
+ (struct xrt_parent_ioctl_put_leaf *)arg;
+ rc = xroot_put_leaf(xr, putleaf);
+ break;
+ }
+ case XRT_PARENT_GET_LEAF_HOLDERS: {
+ struct xrt_parent_ioctl_get_holders *holders =
+ (struct xrt_parent_ioctl_get_holders *)arg;
+ rc = xrt_subdev_pool_get_holders(&xr->parts.pool,
+ holders->xpigh_pdev, holders->xpigh_holder_buf,
+ holders->xpigh_holder_buf_len);
+ break;
+ }
+
+
+ /* Partition actions. */
+ case XRT_PARENT_CREATE_PARTITION:
+ rc = xroot_create_partition(xr, (char *)arg);
+ break;
+ case XRT_PARENT_REMOVE_PARTITION:
+ rc = xroot_destroy_partition(xr, (int)(uintptr_t)arg);
+ break;
+ case XRT_PARENT_LOOKUP_PARTITION: {
+ struct xrt_parent_ioctl_lookup_partition *getpart =
+ (struct xrt_parent_ioctl_lookup_partition *)arg;
+ rc = xroot_lookup_partition(xr, getpart);
+ break;
+ }
+ case XRT_PARENT_WAIT_PARTITION_BRINGUP:
+ rc = xroot_wait_for_bringup(xr) ? 0 : -EINVAL;
+ break;
+
+
+ /* Event actions. */
+ case XRT_PARENT_ADD_EVENT_CB: {
+ struct xrt_parent_ioctl_evt_cb *cb =
+ (struct xrt_parent_ioctl_evt_cb *)arg;
+ rc = xroot_evt_cb_add(xr, cb);
+ break;
+ }
+ case XRT_PARENT_REMOVE_EVENT_CB:
+ xroot_evt_cb_del(xr, arg);
+ rc = 0;
+ break;
+ case XRT_PARENT_ASYNC_BOARDCAST_EVENT:
+ rc = xroot_async_evt_add(xr,
+ (struct xrt_parent_ioctl_async_broadcast_evt *)arg);
+ break;
+
+
+ /* Device info. */
+ case XRT_PARENT_GET_RESOURCE: {
+ struct xrt_parent_ioctl_get_res *res =
+ (struct xrt_parent_ioctl_get_res *)arg;
+ res->xpigr_res = xr->pdev->resource;
+ break;
+ }
+ case XRT_PARENT_GET_ID: {
+ struct xrt_parent_ioctl_get_id *id =
+ (struct xrt_parent_ioctl_get_id *)arg;
+
+ id->xpigi_vendor_id = xr->pdev->vendor;
+ id->xpigi_device_id = xr->pdev->device;
+ id->xpigi_sub_vendor_id = xr->pdev->subsystem_vendor;
+ id->xpigi_sub_device_id = xr->pdev->subsystem_device;
+ break;
+ }
+
+
+ case XRT_PARENT_HOT_RESET: {
+ xroot_hot_reset(xr->pdev);
+ break;
+ }
+
+ case XRT_PARENT_HWMON: {
+ struct xrt_parent_ioctl_hwmon *hwmon =
+ (struct xrt_parent_ioctl_hwmon *)arg;
+
+ if (hwmon->xpih_register) {
+ hwmon->xpih_hwmon_dev =
+ hwmon_device_register_with_info(DEV(xr->pdev),
+ hwmon->xpih_name, hwmon->xpih_drvdata, NULL,
+ hwmon->xpih_groups);
+ } else {
+ (void) hwmon_device_unregister(hwmon->xpih_hwmon_dev);
+ }
+ break;
+ }
+
+ default:
+ xroot_err(xr, "unknown IOCTL cmd %d", cmd);
+ rc = -EINVAL;
+ break;
+ }
+
+ return rc;
+}
+
+static void xroot_bringup_partition_work(struct work_struct *work)
+{
+ struct platform_device *pdev = NULL;
+ struct xroot *xr = container_of(work, struct xroot, parts.bringup_work);
+
+ while (xroot_get_partition(xr, XROOT_PART_LAST, &pdev) != -ENOENT) {
+ int r, i;
+
+ i = pdev->id;
+ r = xrt_subdev_ioctl(pdev, XRT_PARTITION_INIT_CHILDREN, NULL);
+ (void) xroot_put_partition(xr, pdev);
+ if (r == -EEXIST)
+ continue; /* Already brough up, nothing to do. */
+ if (r)
+ atomic_inc(&xr->parts.bringup_failed);
+
+ xroot_event_partition(xr, i, XRT_EVENT_POST_CREATION);
+
+ if (atomic_dec_and_test(&xr->parts.bringup_pending))
+ complete(&xr->parts.bringup_comp);
+ }
+}
+
+static void xroot_parts_init(struct xroot *xr)
+{
+ xrt_subdev_pool_init(DEV(xr->pdev), &xr->parts.pool);
+ INIT_WORK(&xr->parts.bringup_work, xroot_bringup_partition_work);
+ atomic_set(&xr->parts.bringup_pending, 0);
+ atomic_set(&xr->parts.bringup_failed, 0);
+ init_completion(&xr->parts.bringup_comp);
+}
+
+static void xroot_parts_fini(struct xroot *xr)
+{
+ flush_scheduled_work();
+ (void) xrt_subdev_pool_fini(&xr->parts.pool);
+}
+
+int xroot_add_vsec_node(void *root, char *dtb)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct device *dev = DEV(xr->pdev);
+ struct xrt_md_endpoint ep = { 0 };
+ int cap = 0, ret = 0;
+ u32 off_low, off_high, vsec_bar, header;
+ u64 vsec_off;
+
+ while ((cap = pci_find_next_ext_capability(xr->pdev, cap,
+ PCI_EXT_CAP_ID_VNDR))) {
+ pci_read_config_dword(xr->pdev, cap + PCI_VNDR_HEADER, &header);
+ if (PCI_VNDR_HEADER_ID(header) == XRT_VSEC_ID)
+ break;
+ }
+ if (!cap) {
+ xroot_info(xr, "No Vendor Specific Capability.");
+ return -ENOENT;
+ }
+
+ if (pci_read_config_dword(xr->pdev, cap+8, &off_low) ||
+ pci_read_config_dword(xr->pdev, cap+12, &off_high)) {
+ xroot_err(xr, "pci_read vendor specific failed.");
+ return -EINVAL;
+ }
+
+ ep.ep_name = NODE_VSEC;
+ ret = xrt_md_add_endpoint(dev, dtb, &ep);
+ if (ret) {
+ xroot_err(xr, "add vsec metadata failed, ret %d", ret);
+ goto failed;
+ }
+
+ vsec_bar = cpu_to_be32(off_low & 0xf);
+ ret = xrt_md_set_prop(dev, dtb, NODE_VSEC,
+ NULL, PROP_BAR_IDX, &vsec_bar, sizeof(vsec_bar));
+ if (ret) {
+ xroot_err(xr, "add vsec bar idx failed, ret %d", ret);
+ goto failed;
+ }
+
+ vsec_off = cpu_to_be64(((u64)off_high << 32) | (off_low & ~0xfU));
+ ret = xrt_md_set_prop(dev, dtb, NODE_VSEC,
+ NULL, PROP_OFFSET, &vsec_off, sizeof(vsec_off));
+ if (ret) {
+ xroot_err(xr, "add vsec offset failed, ret %d", ret);
+ goto failed;
+ }
+
+failed:
+ return ret;
+}
+
+int xroot_add_simple_node(void *root, char *dtb, const char *endpoint)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct device *dev = DEV(xr->pdev);
+ struct xrt_md_endpoint ep = { 0 };
+ int ret = 0;
+
+ ep.ep_name = endpoint;
+ ret = xrt_md_add_endpoint(dev, dtb, &ep);
+ if (ret)
+ xroot_err(xr, "add %s failed, ret %d", endpoint, ret);
+
+ return ret;
+}
+
+bool xroot_wait_for_bringup(void *root)
+{
+ struct xroot *xr = (struct xroot *)root;
+
+ wait_for_completion(&xr->parts.bringup_comp);
+ return atomic_xchg(&xr->parts.bringup_failed, 0) == 0;
+}
+
+int xroot_probe(struct pci_dev *pdev, void **root)
+{
+ struct device *dev = DEV(pdev);
+ struct xroot *xr = NULL;
+
+ dev_info(dev, "%s: probing...", __func__);
+
+ xr = devm_kzalloc(dev, sizeof(*xr), GFP_KERNEL);
+ if (!xr)
+ return -ENOMEM;
+
+ xr->pdev = pdev;
+ xroot_parts_init(xr);
+ xroot_evt_init(xr);
+
+ *root = xr;
+ return 0;
+}
+
+void xroot_remove(void *root)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct platform_device *part = NULL;
+
+ xroot_info(xr, "leaving...");
+
+ if (xroot_get_partition(xr, XROOT_PART_FIRST, &part) == 0) {
+ int instance = part->id;
+
+ xroot_put_partition(xr, part);
+ (void) xroot_destroy_partition(xr, instance);
+ }
+
+ xroot_evt_fini(xr);
+ xroot_parts_fini(xr);
+}
+
+static void xroot_broadcast_event_cb(struct platform_device *pdev,
+ enum xrt_events evt, void *arg, bool success)
+{
+ struct completion *comp = (struct completion *)arg;
+
+ complete(comp);
+}
+
+void xroot_broadcast(void *root, enum xrt_events evt)
+{
+ int rc;
+ struct completion comp;
+ struct xroot *xr = (struct xroot *)root;
+ struct xrt_parent_ioctl_async_broadcast_evt e = {
+ NULL, evt, xroot_broadcast_event_cb, &comp
+ };
+
+ init_completion(&comp);
+ rc = xroot_async_evt_add(xr, &e);
+ if (rc == 0)
+ wait_for_completion(&comp);
+ else
+ xroot_err(xr, "can't broadcast event (%d): %d", evt, rc);
+}
diff --git a/drivers/fpga/alveo/common/xrt-root.h b/drivers/fpga/alveo/common/xrt-root.h
new file mode 100644
index 000000000000..e2a1c4554feb
--- /dev/null
+++ b/drivers/fpga/alveo/common/xrt-root.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_ROOT_H_
+#define _XRT_ROOT_H_
+
+#include <linux/pci.h>
+#include "xrt-subdev.h"
+
+int xroot_probe(struct pci_dev *pdev, void **root);
+void xroot_remove(void *root);
+bool xroot_wait_for_bringup(void *root);
+int xroot_add_vsec_node(void *root, char *dtb);
+int xroot_create_partition(void *root, char *dtb);
+int xroot_add_simple_node(void *root, char *dtb, const char *endpoint);
+void xroot_hot_reset(struct pci_dev *pdev);
+void xroot_broadcast(void *root, enum xrt_events evt);
+
+#endif /* _XRT_ROOT_H_ */
diff --git a/drivers/fpga/alveo/common/xrt-xclbin.c b/drivers/fpga/alveo/common/xrt-xclbin.c
new file mode 100644
index 000000000000..b7db1b52a086
--- /dev/null
+++ b/drivers/fpga/alveo/common/xrt-xclbin.c
@@ -0,0 +1,387 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Kernel Driver XCLBIN parser
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors: David Zhang <davidzha@xilinx.com>
+ */
+
+#include <asm/errno.h>
+#include <linux/vmalloc.h>
+#include <linux/device.h>
+#include "xrt-xclbin.h"
+#include "xrt-metadata.h"
+
+/* Used for parsing bitstream header */
+#define XHI_EVEN_MAGIC_BYTE 0x0f
+#define XHI_ODD_MAGIC_BYTE 0xf0
+
+/* Extra mode for IDLE */
+#define XHI_OP_IDLE -1
+#define XHI_BIT_HEADER_FAILURE -1
+
+/* The imaginary module length register */
+#define XHI_MLR 15
+
+const char *xrt_xclbin_kind_to_string(enum axlf_section_kind kind)
+{
+ switch (kind) {
+ case BITSTREAM: return "BITSTREAM";
+ case CLEARING_BITSTREAM: return "CLEARING_BITSTREAM";
+ case EMBEDDED_METADATA: return "EMBEDDED_METADATA";
+ case FIRMWARE: return "FIRMWARE";
+ case DEBUG_DATA: return "DEBUG_DATA";
+ case SCHED_FIRMWARE: return "SCHED_FIRMWARE";
+ case MEM_TOPOLOGY: return "MEM_TOPOLOGY";
+ case CONNECTIVITY: return "CONNECTIVITY";
+ case IP_LAYOUT: return "IP_LAYOUT";
+ case DEBUG_IP_LAYOUT: return "DEBUG_IP_LAYOUT";
+ case DESIGN_CHECK_POINT: return "DESIGN_CHECK_POINT";
+ case CLOCK_FREQ_TOPOLOGY: return "CLOCK_FREQ_TOPOLOGY";
+ case MCS: return "MCS";
+ case BMC: return "BMC";
+ case BUILD_METADATA: return "BUILD_METADATA";
+ case KEYVALUE_METADATA: return "KEYVALUE_METADATA";
+ case USER_METADATA: return "USER_METADATA";
+ case DNA_CERTIFICATE: return "DNA_CERTIFICATE";
+ case PDI: return "PDI";
+ case BITSTREAM_PARTIAL_PDI: return "BITSTREAM_PARTIAL_PDI";
+ case PARTITION_METADATA: return "PARTITION_METADATA";
+ case EMULATION_DATA: return "EMULATION_DATA";
+ case SYSTEM_METADATA: return "SYSTEM_METADATA";
+ case SOFT_KERNEL: return "SOFT_KERNEL";
+ case ASK_FLASH: return "ASK_FLASH";
+ case AIE_METADATA: return "AIE_METADATA";
+ case ASK_GROUP_TOPOLOGY: return "ASK_GROUP_TOPOLOGY";
+ case ASK_GROUP_CONNECTIVITY: return "ASK_GROUP_CONNECTIVITY";
+ default: return "UNKNOWN";
+ }
+}
+
+static const struct axlf_section_header *
+xrt_xclbin_get_section_hdr(const struct axlf *xclbin,
+ enum axlf_section_kind kind)
+{
+ int i = 0;
+
+ for (i = 0; i < xclbin->m_header.m_numSections; i++) {
+ if (xclbin->m_sections[i].m_sectionKind == kind)
+ return &xclbin->m_sections[i];
+ }
+
+ return NULL;
+}
+
+static int
+xrt_xclbin_check_section_hdr(const struct axlf_section_header *header,
+ uint64_t xclbin_len)
+{
+ return (header->m_sectionOffset + header->m_sectionSize) > xclbin_len ?
+ -EINVAL : 0;
+}
+
+static int xrt_xclbin_section_info(const struct axlf *xclbin,
+ enum axlf_section_kind kind,
+ uint64_t *offset, uint64_t *size)
+{
+ const struct axlf_section_header *memHeader = NULL;
+ uint64_t xclbin_len;
+ int err = 0;
+
+ memHeader = xrt_xclbin_get_section_hdr(xclbin, kind);
+ if (!memHeader)
+ return -EINVAL;
+
+ xclbin_len = xclbin->m_header.m_length;
+ err = xrt_xclbin_check_section_hdr(memHeader, xclbin_len);
+ if (err)
+ return err;
+
+ *offset = memHeader->m_sectionOffset;
+ *size = memHeader->m_sectionSize;
+
+ return 0;
+}
+
+/* caller should free the allocated memory for **data */
+int xrt_xclbin_get_section(const char *buf,
+ enum axlf_section_kind kind, void **data, uint64_t *len)
+{
+ const struct axlf *xclbin = (const struct axlf *)buf;
+ void *section = NULL;
+ int err = 0;
+ uint64_t offset = 0;
+ uint64_t size = 0;
+
+ err = xrt_xclbin_section_info(xclbin, kind, &offset, &size);
+ if (err)
+ return err;
+
+ section = vmalloc(size);
+ if (section == NULL)
+ return -ENOMEM;
+
+ memcpy(section, ((const char *)xclbin) + offset, size);
+
+ *data = section;
+ if (len)
+ *len = size;
+
+ return 0;
+}
+
+/* parse bitstream header */
+int xrt_xclbin_parse_header(const unsigned char *data,
+ unsigned int size, struct XHwIcap_Bit_Header *header)
+{
+ unsigned int i;
+ unsigned int len;
+ unsigned int tmp;
+ unsigned int index;
+
+ /* Start Index at start of bitstream */
+ index = 0;
+
+ /* Initialize HeaderLength. If header returned early inidicates
+ * failure.
+ */
+ header->HeaderLength = XHI_BIT_HEADER_FAILURE;
+
+ /* Get "Magic" length */
+ header->MagicLength = data[index++];
+ header->MagicLength = (header->MagicLength << 8) | data[index++];
+
+ /* Read in "magic" */
+ for (i = 0; i < header->MagicLength - 1; i++) {
+ tmp = data[index++];
+ if (i % 2 == 0 && tmp != XHI_EVEN_MAGIC_BYTE)
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ if (i % 2 == 1 && tmp != XHI_ODD_MAGIC_BYTE)
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+ }
+
+ /* Read null end of magic data. */
+ tmp = data[index++];
+
+ /* Read 0x01 (short) */
+ tmp = data[index++];
+ tmp = (tmp << 8) | data[index++];
+
+ /* Check the "0x01" half word */
+ if (tmp != 0x01)
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Read 'a' */
+ tmp = data[index++];
+ if (tmp != 'a')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get Design Name length */
+ len = data[index++];
+ len = (len << 8) | data[index++];
+
+ /* allocate space for design name and final null character. */
+ header->DesignName = vmalloc(len);
+
+ /* Read in Design Name */
+ for (i = 0; i < len; i++)
+ header->DesignName[i] = data[index++];
+
+ if (header->DesignName[len-1] != '\0')
+ return -1;
+
+ /* Read 'b' */
+ tmp = data[index++];
+ if (tmp != 'b')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get Part Name length */
+ len = data[index++];
+ len = (len << 8) | data[index++];
+
+ /* allocate space for part name and final null character. */
+ header->PartName = vmalloc(len);
+
+ /* Read in part name */
+ for (i = 0; i < len; i++)
+ header->PartName[i] = data[index++];
+
+ if (header->PartName[len-1] != '\0')
+ return -1;
+
+ /* Read 'c' */
+ tmp = data[index++];
+ if (tmp != 'c')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get date length */
+ len = data[index++];
+ len = (len << 8) | data[index++];
+
+ /* allocate space for date and final null character. */
+ header->Date = vmalloc(len);
+
+ /* Read in date name */
+ for (i = 0; i < len; i++)
+ header->Date[i] = data[index++];
+
+ if (header->Date[len - 1] != '\0')
+ return -1;
+
+ /* Read 'd' */
+ tmp = data[index++];
+ if (tmp != 'd')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get time length */
+ len = data[index++];
+ len = (len << 8) | data[index++];
+
+ /* allocate space for time and final null character. */
+ header->Time = vmalloc(len);
+
+ /* Read in time name */
+ for (i = 0; i < len; i++)
+ header->Time[i] = data[index++];
+
+ if (header->Time[len - 1] != '\0')
+ return -1;
+
+ /* Read 'e' */
+ tmp = data[index++];
+ if (tmp != 'e')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get byte length of bitstream */
+ header->BitstreamLength = data[index++];
+ header->BitstreamLength = (header->BitstreamLength << 8) | data[index++];
+ header->BitstreamLength = (header->BitstreamLength << 8) | data[index++];
+ header->BitstreamLength = (header->BitstreamLength << 8) | data[index++];
+ header->HeaderLength = index;
+
+ return 0;
+}
+
+void xrt_xclbin_free_header(struct XHwIcap_Bit_Header *header)
+{
+ vfree(header->DesignName);
+ vfree(header->PartName);
+ vfree(header->Date);
+ vfree(header->Time);
+}
+
+struct xrt_clock_desc {
+ char *clock_ep_name;
+ u32 clock_xclbin_type;
+ char *clkfreq_ep_name;
+} clock_desc[] = {
+ {
+ .clock_ep_name = NODE_CLK_KERNEL1,
+ .clock_xclbin_type = CT_DATA,
+ .clkfreq_ep_name = NODE_CLKFREQ_K1,
+ },
+ {
+ .clock_ep_name = NODE_CLK_KERNEL2,
+ .clock_xclbin_type = CT_KERNEL,
+ .clkfreq_ep_name = NODE_CLKFREQ_K2,
+ },
+ {
+ .clock_ep_name = NODE_CLK_KERNEL3,
+ .clock_xclbin_type = CT_SYSTEM,
+ .clkfreq_ep_name = NODE_CLKFREQ_HBM,
+ },
+};
+
+const char *clock_type2epname(enum CLOCK_TYPE type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
+ if (clock_desc[i].clock_xclbin_type == type)
+ return clock_desc[i].clock_ep_name;
+ }
+ return NULL;
+}
+
+static const char *clock_type2clkfreq_name(u32 type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
+ if (clock_desc[i].clock_xclbin_type == type)
+ return clock_desc[i].clkfreq_ep_name;
+ }
+ return NULL;
+}
+
+static int xrt_xclbin_add_clock_metadata(struct device *dev,
+ const char *xclbin, char *dtb)
+{
+ int i;
+ u16 freq;
+ struct clock_freq_topology *clock_topo;
+ int rc = xrt_xclbin_get_section(xclbin,
+ CLOCK_FREQ_TOPOLOGY, (void **)&clock_topo, NULL);
+
+ if (rc)
+ return 0;
+
+ for (i = 0; i < clock_topo->m_count; i++) {
+ u8 type = clock_topo->m_clock_freq[i].m_type;
+ const char *ep_name = clock_type2epname(type);
+ const char *counter_name = clock_type2clkfreq_name(type);
+
+ if (!ep_name || !counter_name)
+ continue;
+
+ freq = cpu_to_be16(clock_topo->m_clock_freq[i].m_freq_Mhz);
+ rc = xrt_md_set_prop(dev, dtb, ep_name,
+ NULL, PROP_CLK_FREQ, &freq, sizeof(freq));
+ if (rc)
+ break;
+
+ rc = xrt_md_set_prop(dev, dtb, ep_name,
+ NULL, PROP_CLK_CNT, counter_name, strlen(counter_name) + 1);
+ if (rc)
+ break;
+ }
+
+ vfree(clock_topo);
+
+ return rc;
+}
+
+int xrt_xclbin_get_metadata(struct device *dev, const char *xclbin, char **dtb)
+{
+ char *md = NULL, *newmd = NULL;
+ u64 len;
+ int rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA,
+ (void **)&md, &len);
+
+ if (rc)
+ goto done;
+
+ /* Sanity check the dtb section. */
+ if (xrt_md_size(dev, md) > len) {
+ rc = -EINVAL;
+ goto done;
+ }
+
+ newmd = xrt_md_dup(dev, md);
+ if (!newmd) {
+ rc = -EFAULT;
+ goto done;
+ }
+ /* Convert various needed xclbin sections into dtb. */
+ rc = xrt_xclbin_add_clock_metadata(dev, xclbin, newmd);
+
+done:
+ if (rc == 0)
+ *dtb = newmd;
+ else
+ vfree(newmd);
+ vfree(md);
+ return rc;
+}
diff --git a/drivers/fpga/alveo/common/xrt-xclbin.h b/drivers/fpga/alveo/common/xrt-xclbin.h
new file mode 100644
index 000000000000..05214d824790
--- /dev/null
+++ b/drivers/fpga/alveo/common/xrt-xclbin.h
@@ -0,0 +1,46 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Kernel Driver XCLBIN parser
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors: David Zhang <davidzha@xilinx.com>
+ */
+
+#ifndef _XRT_XCLBIN_H
+#define _XRT_XCLBIN_H
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/xrt/xclbin.h>
+
+#define ICAP_XCLBIN_V2 "xclbin2"
+#define DMA_HWICAP_BITFILE_BUFFER_SIZE 1024
+#define MAX_XCLBIN_SIZE (1024 * 1024 * 1024) /* Assuming xclbin <= 1G, always */
+
+enum axlf_section_kind;
+struct axlf;
+
+/**
+ * Bitstream header information.
+ */
+struct XHwIcap_Bit_Header {
+ unsigned int HeaderLength; /* Length of header in 32 bit words */
+ unsigned int BitstreamLength; /* Length of bitstream to read in bytes*/
+ unsigned char *DesignName; /* Design name get from bitstream */
+ unsigned char *PartName; /* Part name read from bitstream */
+ unsigned char *Date; /* Date read from bitstream header */
+ unsigned char *Time; /* Bitstream creation time*/
+ unsigned int MagicLength; /* Length of the magic numbers*/
+};
+
+const char *xrt_xclbin_kind_to_string(enum axlf_section_kind kind);
+int xrt_xclbin_get_section(const char *xclbin,
+ enum axlf_section_kind kind, void **data, uint64_t *len);
+int xrt_xclbin_get_metadata(struct device *dev, const char *xclbin, char **dtb);
+int xrt_xclbin_parse_header(const unsigned char *data,
+ unsigned int size, struct XHwIcap_Bit_Header *header);
+void xrt_xclbin_free_header(struct XHwIcap_Bit_Header *header);
+const char *clock_type2epname(enum CLOCK_TYPE type);
+
+#endif /* _XRT_XCLBIN_H */
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH Xilinx Alveo 4/8] fpga: xrt: core infrastructure for xrt-lib module
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
` (2 preceding siblings ...)
2020-11-29 0:00 ` [PATCH Xilinx Alveo 3/8] fpga: xrt: infrastructure support for xmgmt driver Sonal Santan
@ 2020-11-29 0:00 ` Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 6/8] fpga: xrt: header file for platform and parent drivers Sonal Santan
` (5 subsequent siblings)
9 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
From: Sonal Santan <sonal.santan@xilinx.com>
Add xrt-lib kernel module infrastructrure code which defines APIs
for working with device nodes, iteration and lookup of platform
devices, common interfaces for platform devices, plumbing of
function call and ioctls between platform devices and parent
partitions.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
---
drivers/fpga/alveo/lib/xrt-cdev.c | 234 +++++++
drivers/fpga/alveo/lib/xrt-main.c | 275 ++++++++
drivers/fpga/alveo/lib/xrt-main.h | 46 ++
drivers/fpga/alveo/lib/xrt-subdev.c | 1007 +++++++++++++++++++++++++++
4 files changed, 1562 insertions(+)
create mode 100644 drivers/fpga/alveo/lib/xrt-cdev.c
create mode 100644 drivers/fpga/alveo/lib/xrt-main.c
create mode 100644 drivers/fpga/alveo/lib/xrt-main.h
create mode 100644 drivers/fpga/alveo/lib/xrt-subdev.c
diff --git a/drivers/fpga/alveo/lib/xrt-cdev.c b/drivers/fpga/alveo/lib/xrt-cdev.c
new file mode 100644
index 000000000000..b7bef9c8e9ce
--- /dev/null
+++ b/drivers/fpga/alveo/lib/xrt-cdev.c
@@ -0,0 +1,234 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA device node helper functions.
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include "xrt-subdev.h"
+
+extern struct class *xrt_class;
+
+#define XRT_CDEV_DIR "xfpga"
+#define INODE2PDATA(inode) \
+ container_of((inode)->i_cdev, struct xrt_subdev_platdata, xsp_cdev)
+#define INODE2PDEV(inode) \
+ to_platform_device(kobj_to_dev((inode)->i_cdev->kobj.parent))
+#define CDEV_NAME(sysdev) (strchr((sysdev)->kobj.name, '!') + 1)
+
+/* Allow it to be accessed from cdev. */
+static void xrt_devnode_allowed(struct platform_device *pdev)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+
+ /* Allow new opens. */
+ mutex_lock(&pdata->xsp_devnode_lock);
+ pdata->xsp_devnode_online = true;
+ mutex_unlock(&pdata->xsp_devnode_lock);
+}
+
+/* Turn off access from cdev and wait for all existing user to go away. */
+static int xrt_devnode_disallowed(struct platform_device *pdev)
+{
+ int ret = 0;
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ /* Prevent new opens. */
+ pdata->xsp_devnode_online = false;
+ /* Wait for existing user to close. */
+ while (!ret && pdata->xsp_devnode_ref) {
+ int rc;
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+ rc = wait_for_completion_killable(&pdata->xsp_devnode_comp);
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ if (rc == -ERESTARTSYS) {
+ /* Restore online state. */
+ pdata->xsp_devnode_online = true;
+ xrt_err(pdev, "%s is in use, ref=%d",
+ CDEV_NAME(pdata->xsp_sysdev),
+ pdata->xsp_devnode_ref);
+ ret = -EBUSY;
+ }
+ }
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+
+ return ret;
+}
+
+static struct platform_device *
+__xrt_devnode_open(struct inode *inode, bool excl)
+{
+ struct xrt_subdev_platdata *pdata = INODE2PDATA(inode);
+ struct platform_device *pdev = INODE2PDEV(inode);
+ bool opened = false;
+
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ if (pdata->xsp_devnode_online) {
+ if (excl && pdata->xsp_devnode_ref) {
+ xrt_err(pdev, "%s has already been opened exclusively",
+ CDEV_NAME(pdata->xsp_sysdev));
+ } else if (!excl && pdata->xsp_devnode_excl) {
+ xrt_err(pdev, "%s has been opened exclusively",
+ CDEV_NAME(pdata->xsp_sysdev));
+ } else {
+ pdata->xsp_devnode_ref++;
+ pdata->xsp_devnode_excl = excl;
+ opened = true;
+ xrt_info(pdev, "opened %s, ref=%d",
+ CDEV_NAME(pdata->xsp_sysdev),
+ pdata->xsp_devnode_ref);
+ }
+ } else {
+ xrt_err(pdev, "%s is offline", CDEV_NAME(pdata->xsp_sysdev));
+ }
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+
+ return opened ? pdev : NULL;
+}
+
+struct platform_device *
+xrt_devnode_open_excl(struct inode *inode)
+{
+ return __xrt_devnode_open(inode, true);
+}
+
+struct platform_device *
+xrt_devnode_open(struct inode *inode)
+{
+ return __xrt_devnode_open(inode, false);
+}
+EXPORT_SYMBOL_GPL(xrt_devnode_open);
+
+void xrt_devnode_close(struct inode *inode)
+{
+ struct xrt_subdev_platdata *pdata = INODE2PDATA(inode);
+ struct platform_device *pdev = INODE2PDEV(inode);
+ bool notify = false;
+
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ pdata->xsp_devnode_ref--;
+ if (pdata->xsp_devnode_ref == 0) {
+ pdata->xsp_devnode_excl = false;
+ notify = true;
+ }
+ if (notify) {
+ xrt_info(pdev, "closed %s, ref=%d",
+ CDEV_NAME(pdata->xsp_sysdev), pdata->xsp_devnode_ref);
+ } else {
+ xrt_info(pdev, "closed %s, notifying waiter",
+ CDEV_NAME(pdata->xsp_sysdev));
+ }
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+
+ if (notify)
+ complete(&pdata->xsp_devnode_comp);
+}
+EXPORT_SYMBOL_GPL(xrt_devnode_close);
+
+static inline enum xrt_subdev_file_mode
+devnode_mode(struct xrt_subdev_drvdata *drvdata)
+{
+ return drvdata->xsd_file_ops.xsf_mode;
+}
+
+int xrt_devnode_create(struct platform_device *pdev, const char *file_name,
+ const char *inst_name)
+{
+ struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(pdev);
+ struct xrt_subdev_file_ops *fops = &drvdata->xsd_file_ops;
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ struct cdev *cdevp;
+ struct device *sysdev;
+ int ret = 0;
+ char fname[256];
+
+ BUG_ON(fops->xsf_dev_t == (dev_t)-1);
+
+ mutex_init(&pdata->xsp_devnode_lock);
+ init_completion(&pdata->xsp_devnode_comp);
+
+ cdevp = &DEV_PDATA(pdev)->xsp_cdev;
+ cdev_init(cdevp, &fops->xsf_ops);
+ cdevp->owner = fops->xsf_ops.owner;
+ cdevp->dev = MKDEV(MAJOR(fops->xsf_dev_t), pdev->id);
+
+ /*
+ * Set pdev as parent of cdev so that when pdev (and its platform
+ * data) will not be freed when cdev is not freed.
+ */
+ cdev_set_parent(cdevp, &DEV(pdev)->kobj);
+
+ ret = cdev_add(cdevp, cdevp->dev, 1);
+ if (ret) {
+ xrt_err(pdev, "failed to add cdev: %d", ret);
+ goto failed;
+ }
+ if (!file_name)
+ file_name = pdev->name;
+ if (!inst_name) {
+ if (devnode_mode(drvdata) == XRT_SUBDEV_FILE_MULTI_INST) {
+ snprintf(fname, sizeof(fname), "%s/%s/%s.%u",
+ XRT_CDEV_DIR, DEV_PDATA(pdev)->xsp_root_name,
+ file_name, pdev->id);
+ } else {
+ snprintf(fname, sizeof(fname), "%s/%s/%s",
+ XRT_CDEV_DIR, DEV_PDATA(pdev)->xsp_root_name,
+ file_name);
+ }
+ } else {
+ snprintf(fname, sizeof(fname), "%s/%s/%s.%s", XRT_CDEV_DIR,
+ DEV_PDATA(pdev)->xsp_root_name, file_name, inst_name);
+ }
+ sysdev = device_create(xrt_class, NULL, cdevp->dev, NULL, "%s", fname);
+ if (IS_ERR(sysdev)) {
+ ret = PTR_ERR(sysdev);
+ xrt_err(pdev, "failed to create device node: %d", ret);
+ goto failed;
+ }
+ pdata->xsp_sysdev = sysdev;
+
+ xrt_devnode_allowed(pdev);
+
+ xrt_info(pdev, "created (%d, %d): /dev/%s",
+ MAJOR(cdevp->dev), pdev->id, fname);
+ return 0;
+
+failed:
+ device_destroy(xrt_class, cdevp->dev);
+ cdev_del(cdevp);
+ cdevp->owner = NULL;
+ return ret;
+}
+
+int xrt_devnode_destroy(struct platform_device *pdev)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ struct cdev *cdevp = &pdata->xsp_cdev;
+ dev_t dev = cdevp->dev;
+ int rc;
+
+ BUG_ON(!cdevp->owner);
+
+ rc = xrt_devnode_disallowed(pdev);
+ if (rc)
+ return rc;
+
+ xrt_info(pdev, "removed (%d, %d): /dev/%s/%s", MAJOR(dev), MINOR(dev),
+ XRT_CDEV_DIR, CDEV_NAME(pdata->xsp_sysdev));
+ device_destroy(xrt_class, cdevp->dev);
+ pdata->xsp_sysdev = NULL;
+ cdev_del(cdevp);
+ return 0;
+}
diff --git a/drivers/fpga/alveo/lib/xrt-main.c b/drivers/fpga/alveo/lib/xrt-main.c
new file mode 100644
index 000000000000..b962410a628c
--- /dev/null
+++ b/drivers/fpga/alveo/lib/xrt-main.c
@@ -0,0 +1,275 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include "xrt-subdev.h"
+#include "xrt-main.h"
+
+#define XRT_IPLIB_MODULE_NAME "xrt-lib"
+#define XRT_IPLIB_MODULE_VERSION "4.0.0"
+#define XRT_DRVNAME(drv) ((drv)->driver.name)
+#define XRT_MAX_DEVICE_NODES 128
+
+struct mutex xrt_class_lock;
+struct class *xrt_class;
+
+/*
+ * Subdev driver is known by ID to others. We map the ID to it's
+ * struct platform_driver, which contains it's binding name and driver/file ops.
+ * We also map it to the endpoint name in DTB as well, if it's different
+ * than the driver's binding name.
+ */
+static struct xrt_drv_map {
+ enum xrt_subdev_id id;
+ struct platform_driver *drv;
+ struct xrt_subdev_endpoints *eps;
+ struct ida ida; /* manage driver instance and char dev minor */
+} xrt_drv_maps[] = {
+ { XRT_SUBDEV_PART, &xrt_partition_driver, },
+ { XRT_SUBDEV_VSEC, &xrt_vsec_driver, xrt_vsec_endpoints, },
+ { XRT_SUBDEV_VSEC_GOLDEN, &xrt_vsec_golden_driver, xrt_vsec_golden_endpoints, },
+ { XRT_SUBDEV_GPIO, &xrt_gpio_driver, xrt_gpio_endpoints,},
+ { XRT_SUBDEV_AXIGATE, &xrt_axigate_driver, xrt_axigate_endpoints, },
+ { XRT_SUBDEV_ICAP, &xrt_icap_driver, xrt_icap_endpoints, },
+ { XRT_SUBDEV_CALIB, &xrt_calib_driver, xrt_calib_endpoints, },
+ { XRT_SUBDEV_TEST, &xrt_test_driver, xrt_test_endpoints, },
+ { XRT_SUBDEV_MGMT_MAIN, NULL, },
+ { XRT_SUBDEV_QSPI, &xrt_qspi_driver, xrt_qspi_endpoints, },
+ { XRT_SUBDEV_MAILBOX, &xrt_mailbox_driver, xrt_mailbox_endpoints, },
+ { XRT_SUBDEV_CMC, &xrt_cmc_driver, xrt_cmc_endpoints, },
+ { XRT_SUBDEV_CLKFREQ, &xrt_clkfreq_driver, xrt_clkfreq_endpoints, },
+ { XRT_SUBDEV_CLOCK, &xrt_clock_driver, xrt_clock_endpoints, },
+ { XRT_SUBDEV_UCS, &xrt_ucs_driver, xrt_ucs_endpoints, },
+};
+
+static inline struct xrt_subdev_drvdata *
+xrt_drv_map2drvdata(struct xrt_drv_map *map)
+{
+ return (struct xrt_subdev_drvdata *)map->drv->id_table[0].driver_data;
+}
+
+static struct xrt_drv_map *
+xrt_drv_find_map_by_id(enum xrt_subdev_id id)
+{
+ int i;
+ struct xrt_drv_map *map = NULL;
+
+ for (i = 0; i < ARRAY_SIZE(xrt_drv_maps); i++) {
+ struct xrt_drv_map *tmap = &xrt_drv_maps[i];
+
+ if (tmap->id != id)
+ continue;
+ map = tmap;
+ break;
+ }
+ return map;
+}
+
+static int xrt_drv_register_driver(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+ struct xrt_subdev_drvdata *drvdata;
+ int rc = 0;
+ const char *drvname;
+
+ BUG_ON(!map);
+
+ if (!map->drv) {
+ pr_info("skip registration of subdev driver for id %d\n", id);
+ return rc;
+ }
+ drvname = XRT_DRVNAME(map->drv);
+
+ rc = platform_driver_register(map->drv);
+ if (rc) {
+ pr_err("register %s subdev driver failed\n", drvname);
+ return rc;
+ }
+
+ drvdata = xrt_drv_map2drvdata(map);
+ if (drvdata && drvdata->xsd_dev_ops.xsd_post_init) {
+ rc = drvdata->xsd_dev_ops.xsd_post_init();
+ if (rc) {
+ platform_driver_unregister(map->drv);
+ pr_err("%s's post-init, ret %d\n", drvname, rc);
+ return rc;
+ }
+ }
+
+ if (drvdata) {
+ /* Initialize dev_t for char dev node. */
+ if (xrt_devnode_enabled(drvdata)) {
+ rc = alloc_chrdev_region(
+ &drvdata->xsd_file_ops.xsf_dev_t, 0,
+ XRT_MAX_DEVICE_NODES, drvname);
+ if (rc) {
+ if (drvdata->xsd_dev_ops.xsd_pre_exit)
+ drvdata->xsd_dev_ops.xsd_pre_exit();
+ platform_driver_unregister(map->drv);
+ pr_err("failed to alloc dev minor for %s: %d\n",
+ drvname, rc);
+ return rc;
+ }
+ } else {
+ drvdata->xsd_file_ops.xsf_dev_t = (dev_t)-1;
+ }
+ }
+
+ ida_init(&map->ida);
+
+ pr_info("registered %s subdev driver\n", drvname);
+ return 0;
+}
+
+static void xrt_drv_unregister_driver(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+ struct xrt_subdev_drvdata *drvdata;
+ const char *drvname;
+
+ BUG_ON(!map);
+ if (!map->drv) {
+ pr_info("skip unregistration of subdev driver for id %d\n", id);
+ return;
+ }
+
+ drvname = XRT_DRVNAME(map->drv);
+
+ ida_destroy(&map->ida);
+
+ drvdata = xrt_drv_map2drvdata(map);
+ if (drvdata && drvdata->xsd_file_ops.xsf_dev_t != (dev_t)-1) {
+ unregister_chrdev_region(drvdata->xsd_file_ops.xsf_dev_t,
+ XRT_MAX_DEVICE_NODES);
+ }
+
+ if (drvdata && drvdata->xsd_dev_ops.xsd_pre_exit)
+ drvdata->xsd_dev_ops.xsd_pre_exit();
+
+ platform_driver_unregister(map->drv);
+
+ pr_info("unregistered %s subdev driver\n", drvname);
+}
+
+int xrt_subdev_register_external_driver(enum xrt_subdev_id id,
+ struct platform_driver *drv, struct xrt_subdev_endpoints *eps)
+{
+ int i;
+ int result = 0;
+
+ mutex_lock(&xrt_class_lock);
+ for (i = 0; i < ARRAY_SIZE(xrt_drv_maps); i++) {
+ struct xrt_drv_map *map = &xrt_drv_maps[i];
+
+ if (map->id != id)
+ continue;
+ if (map->drv) {
+ result = -EEXIST;
+ pr_err("Id %d already has a registered driver, 0x%p\n",
+ id, map->drv);
+ break;
+ }
+ map->drv = drv;
+ BUG_ON(map->eps);
+ map->eps = eps;
+ xrt_drv_register_driver(id);
+ }
+ mutex_unlock(&xrt_class_lock);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_register_external_driver);
+
+void xrt_subdev_unregister_external_driver(enum xrt_subdev_id id)
+{
+ int i;
+
+ mutex_lock(&xrt_class_lock);
+ for (i = 0; i < ARRAY_SIZE(xrt_drv_maps); i++) {
+ struct xrt_drv_map *map = &xrt_drv_maps[i];
+
+ if (map->id != id)
+ continue;
+ xrt_drv_unregister_driver(id);
+ map->drv = NULL;
+ map->eps = NULL;
+ break;
+ }
+ mutex_unlock(&xrt_class_lock);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_unregister_external_driver);
+
+static __init int xrt_drv_register_drivers(void)
+{
+ int i;
+ int rc = 0;
+
+ mutex_init(&xrt_class_lock);
+ xrt_class = class_create(THIS_MODULE, XRT_IPLIB_MODULE_NAME);
+ if (IS_ERR(xrt_class))
+ return PTR_ERR(xrt_class);
+
+ for (i = 0; i < ARRAY_SIZE(xrt_drv_maps); i++) {
+ rc = xrt_drv_register_driver(xrt_drv_maps[i].id);
+ if (rc)
+ break;
+ }
+ if (!rc)
+ return 0;
+
+ while (i-- > 0)
+ xrt_drv_unregister_driver(xrt_drv_maps[i].id);
+ class_destroy(xrt_class);
+ return rc;
+}
+
+static __exit void xrt_drv_unregister_drivers(void)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(xrt_drv_maps); i++)
+ xrt_drv_unregister_driver(xrt_drv_maps[i].id);
+ class_destroy(xrt_class);
+}
+
+const char *xrt_drv_name(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+
+ if (map)
+ return XRT_DRVNAME(map->drv);
+ return NULL;
+}
+
+int xrt_drv_get_instance(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+
+ return ida_alloc_range(&map->ida, 0, XRT_MAX_DEVICE_NODES, GFP_KERNEL);
+}
+
+void xrt_drv_put_instance(enum xrt_subdev_id id, int instance)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+
+ ida_free(&map->ida, instance);
+}
+
+struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+
+ return map ? map->eps : NULL;
+}
+
+module_init(xrt_drv_register_drivers);
+module_exit(xrt_drv_unregister_drivers);
+
+MODULE_VERSION(XRT_IPLIB_MODULE_VERSION);
+MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
+MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/fpga/alveo/lib/xrt-main.h b/drivers/fpga/alveo/lib/xrt-main.h
new file mode 100644
index 000000000000..f46f90d9e882
--- /dev/null
+++ b/drivers/fpga/alveo/lib/xrt-main.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_MAIN_H_
+#define _XRT_MAIN_H_
+
+extern struct platform_driver xrt_partition_driver;
+extern struct platform_driver xrt_test_driver;
+extern struct platform_driver xrt_vsec_driver;
+extern struct platform_driver xrt_vsec_golden_driver;
+extern struct platform_driver xrt_axigate_driver;
+extern struct platform_driver xrt_qspi_driver;
+extern struct platform_driver xrt_gpio_driver;
+extern struct platform_driver xrt_mailbox_driver;
+extern struct platform_driver xrt_icap_driver;
+extern struct platform_driver xrt_cmc_driver;
+extern struct platform_driver xrt_clkfreq_driver;
+extern struct platform_driver xrt_clock_driver;
+extern struct platform_driver xrt_ucs_driver;
+extern struct platform_driver xrt_calib_driver;
+
+extern struct xrt_subdev_endpoints xrt_vsec_endpoints[];
+extern struct xrt_subdev_endpoints xrt_vsec_golden_endpoints[];
+extern struct xrt_subdev_endpoints xrt_axigate_endpoints[];
+extern struct xrt_subdev_endpoints xrt_test_endpoints[];
+extern struct xrt_subdev_endpoints xrt_qspi_endpoints[];
+extern struct xrt_subdev_endpoints xrt_gpio_endpoints[];
+extern struct xrt_subdev_endpoints xrt_mailbox_endpoints[];
+extern struct xrt_subdev_endpoints xrt_icap_endpoints[];
+extern struct xrt_subdev_endpoints xrt_cmc_endpoints[];
+extern struct xrt_subdev_endpoints xrt_clkfreq_endpoints[];
+extern struct xrt_subdev_endpoints xrt_clock_endpoints[];
+extern struct xrt_subdev_endpoints xrt_ucs_endpoints[];
+extern struct xrt_subdev_endpoints xrt_calib_endpoints[];
+
+extern const char *xrt_drv_name(enum xrt_subdev_id id);
+extern int xrt_drv_get_instance(enum xrt_subdev_id id);
+extern void xrt_drv_put_instance(enum xrt_subdev_id id, int instance);
+extern struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id);
+
+#endif /* _XRT_MAIN_H_ */
diff --git a/drivers/fpga/alveo/lib/xrt-subdev.c b/drivers/fpga/alveo/lib/xrt-subdev.c
new file mode 100644
index 000000000000..644c4ec19429
--- /dev/null
+++ b/drivers/fpga/alveo/lib/xrt-subdev.c
@@ -0,0 +1,1007 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/platform_device.h>
+#include <linux/pci.h>
+#include <linux/vmalloc.h>
+#include "xrt-subdev.h"
+#include "xrt-parent.h"
+#include "xrt-main.h"
+#include "xrt-metadata.h"
+
+#define DEV_IS_PCI(dev) ((dev)->bus == &pci_bus_type)
+static inline struct device *find_root(struct platform_device *pdev)
+{
+ struct device *d = DEV(pdev);
+
+ while (!DEV_IS_PCI(d))
+ d = d->parent;
+ return d;
+}
+
+/*
+ * It represents a holder of a subdev. One holder can repeatedly hold a subdev
+ * as long as there is a unhold corresponding to a hold.
+ */
+struct xrt_subdev_holder {
+ struct list_head xsh_holder_list;
+ struct device *xsh_holder;
+ int xsh_count;
+};
+
+/*
+ * It represents a specific instance of platform driver for a subdev, which
+ * provides services to its clients (another subdev driver or root driver).
+ */
+struct xrt_subdev {
+ struct list_head xs_dev_list;
+ struct list_head xs_holder_list;
+ enum xrt_subdev_id xs_id; /* type of subdev */
+ struct platform_device *xs_pdev; /* a particular subdev inst */
+ struct completion xs_holder_comp;
+};
+
+static struct xrt_subdev *xrt_subdev_alloc(void)
+{
+ struct xrt_subdev *sdev = vzalloc(sizeof(struct xrt_subdev));
+
+ if (!sdev)
+ return NULL;
+
+ INIT_LIST_HEAD(&sdev->xs_dev_list);
+ INIT_LIST_HEAD(&sdev->xs_holder_list);
+ init_completion(&sdev->xs_holder_comp);
+ return sdev;
+}
+
+static void xrt_subdev_free(struct xrt_subdev *sdev)
+{
+ vfree(sdev);
+}
+
+/*
+ * Subdev common sysfs nodes.
+ */
+static ssize_t holders_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ ssize_t len;
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xrt_parent_ioctl_get_holders holders = { pdev, buf, 1024 };
+
+ len = xrt_subdev_parent_ioctl(pdev,
+ XRT_PARENT_GET_LEAF_HOLDERS, &holders);
+ if (len >= holders.xpigh_holder_buf_len)
+ return len;
+ buf[len] = '\n';
+ return len + 1;
+}
+static DEVICE_ATTR_RO(holders);
+
+static struct attribute *xrt_subdev_attrs[] = {
+ &dev_attr_holders.attr,
+ NULL,
+};
+
+static ssize_t metadata_output(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *attr, char *buf, loff_t off, size_t count)
+{
+ struct device *dev = kobj_to_dev(kobj);
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ unsigned char *blob;
+ long size;
+ ssize_t ret = 0;
+
+ blob = pdata->xsp_dtb;
+ size = xrt_md_size(dev, blob);
+ if (size <= 0) {
+ ret = -EINVAL;
+ goto failed;
+ }
+
+ if (off >= size)
+ goto failed;
+
+ if (off + count > size)
+ count = size - off;
+ memcpy(buf, blob + off, count);
+
+ ret = count;
+failed:
+ return ret;
+}
+
+static struct bin_attribute meta_data_attr = {
+ .attr = {
+ .name = "metadata",
+ .mode = 0400
+ },
+ .read = metadata_output,
+ .size = 0
+};
+
+static struct bin_attribute *xrt_subdev_bin_attrs[] = {
+ &meta_data_attr,
+ NULL,
+};
+
+static const struct attribute_group xrt_subdev_attrgroup = {
+ .attrs = xrt_subdev_attrs,
+ .bin_attrs = xrt_subdev_bin_attrs,
+};
+
+static int
+xrt_subdev_getres(struct device *parent, enum xrt_subdev_id id,
+ char *dtb, struct resource **res, int *res_num)
+{
+ struct xrt_subdev_platdata *pdata;
+ struct resource *pci_res = NULL;
+ const u64 *bar_range;
+ const u32 *bar_idx;
+ char *ep_name = NULL, *regmap = NULL;
+ uint bar;
+ int count1 = 0, count2 = 0, ret;
+
+ if (!dtb)
+ return -EINVAL;
+
+ pdata = DEV_PDATA(to_platform_device(parent));
+
+ for (xrt_md_get_next_endpoint(parent, dtb, NULL, NULL,
+ &ep_name, ®map);
+ ep_name != NULL;
+ xrt_md_get_next_endpoint(parent, dtb, ep_name, regmap,
+ &ep_name, ®map)) {
+ ret = xrt_md_get_prop(parent, dtb, ep_name, regmap,
+ PROP_IO_OFFSET, (const void **)&bar_range, NULL);
+ if (!ret)
+ count1++;
+ }
+ if (!count1)
+ return 0;
+
+ *res = vzalloc(sizeof(struct resource) * count1);
+
+ for (xrt_md_get_next_endpoint(parent, dtb, NULL, NULL,
+ &ep_name, ®map);
+ ep_name != NULL;
+ xrt_md_get_next_endpoint(parent, dtb, ep_name, regmap,
+ &ep_name, ®map)) {
+ ret = xrt_md_get_prop(parent, dtb, ep_name, regmap,
+ PROP_IO_OFFSET, (const void **)&bar_range, NULL);
+ if (ret)
+ continue;
+ xrt_md_get_prop(parent, dtb, ep_name, regmap,
+ PROP_BAR_IDX, (const void **)&bar_idx, NULL);
+ bar = bar_idx ? be32_to_cpu(*bar_idx) : 0;
+ xrt_subdev_get_barres(to_platform_device(parent), &pci_res,
+ bar);
+ (*res)[count2].start = pci_res->start +
+ be64_to_cpu(bar_range[0]);
+ (*res)[count2].end = pci_res->start +
+ be64_to_cpu(bar_range[0]) +
+ be64_to_cpu(bar_range[1]) - 1;
+ (*res)[count2].flags = IORESOURCE_MEM;
+ /* check if there is conflicted resource */
+ ret = request_resource(pci_res, *res + count2);
+ if (ret) {
+ dev_err(parent, "Conflict resource %pR\n",
+ *res + count2);
+ vfree(*res);
+ *res_num = 0;
+ *res = NULL;
+ return ret;
+ }
+ release_resource(*res + count2);
+
+ (*res)[count2].parent = pci_res;
+
+ xrt_md_get_epname_pointer(parent, pdata->xsp_dtb, ep_name,
+ regmap, &(*res)[count2].name);
+
+ count2++;
+ }
+
+ BUG_ON(count1 != count2);
+ *res_num = count2;
+
+ return 0;
+}
+
+static inline enum xrt_subdev_file_mode
+xrt_devnode_mode(struct xrt_subdev_drvdata *drvdata)
+{
+ return drvdata->xsd_file_ops.xsf_mode;
+}
+
+static bool xrt_subdev_cdev_auto_creation(struct platform_device *pdev)
+{
+ struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(pdev);
+
+ if (!drvdata)
+ return false;
+
+ return xrt_devnode_enabled(drvdata) &&
+ (xrt_devnode_mode(drvdata) == XRT_SUBDEV_FILE_DEFAULT ||
+ (xrt_devnode_mode(drvdata) == XRT_SUBDEV_FILE_MULTI_INST));
+}
+
+static struct xrt_subdev *
+xrt_subdev_create(struct device *parent, enum xrt_subdev_id id,
+ xrt_subdev_parent_cb_t pcb, void *pcb_arg, char *dtb)
+{
+ struct xrt_subdev *sdev = NULL;
+ struct platform_device *pdev = NULL;
+ struct xrt_subdev_platdata *pdata = NULL;
+ long dtb_len = 0;
+ size_t pdata_sz;
+ int inst = PLATFORM_DEVID_NONE;
+ struct resource *res = NULL;
+ int res_num = 0;
+
+ sdev = xrt_subdev_alloc();
+ if (!sdev) {
+ dev_err(parent, "failed to alloc subdev for ID %d", id);
+ goto fail;
+ }
+ sdev->xs_id = id;
+
+ if (dtb) {
+ xrt_md_pack(parent, dtb);
+ dtb_len = xrt_md_size(parent, dtb);
+ if (dtb_len <= 0) {
+ dev_err(parent, "invalid metadata len %ld", dtb_len);
+ goto fail;
+ }
+ }
+ pdata_sz = sizeof(struct xrt_subdev_platdata) + dtb_len - 1;
+
+ /* Prepare platform data passed to subdev. */
+ pdata = vzalloc(pdata_sz);
+ if (!pdata)
+ goto fail;
+
+ pdata->xsp_parent_cb = pcb;
+ pdata->xsp_parent_cb_arg = pcb_arg;
+ (void) memcpy(pdata->xsp_dtb, dtb, dtb_len);
+ if (id == XRT_SUBDEV_PART) {
+ /* Partition can only be created by root driver. */
+ BUG_ON(parent->bus != &pci_bus_type);
+ pdata->xsp_root_name = dev_name(parent);
+ } else {
+ struct platform_device *part = to_platform_device(parent);
+ /* Leaf can only be created by partition driver. */
+ BUG_ON(parent->bus != &platform_bus_type);
+ BUG_ON(strcmp(xrt_drv_name(XRT_SUBDEV_PART),
+ platform_get_device_id(part)->name));
+ pdata->xsp_root_name = DEV_PDATA(part)->xsp_root_name;
+ }
+
+ /* Obtain dev instance number. */
+ inst = xrt_drv_get_instance(id);
+ if (inst < 0) {
+ dev_err(parent, "failed to obtain instance: %d", inst);
+ goto fail;
+ }
+
+ /* Create subdev. */
+ if (id == XRT_SUBDEV_PART) {
+ pdev = platform_device_register_data(parent,
+ xrt_drv_name(XRT_SUBDEV_PART), inst, pdata, pdata_sz);
+ } else {
+ int rc = xrt_subdev_getres(parent, id, dtb, &res, &res_num);
+
+ if (rc) {
+ dev_err(parent, "failed to get resource for %s.%d: %d",
+ xrt_drv_name(id), inst, rc);
+ goto fail;
+ }
+ pdev = platform_device_register_resndata(parent,
+ xrt_drv_name(id), inst, res, res_num, pdata, pdata_sz);
+ vfree(res);
+ }
+ if (IS_ERR(pdev)) {
+ dev_err(parent, "failed to create subdev for %s inst %d: %ld",
+ xrt_drv_name(id), inst, PTR_ERR(pdev));
+ goto fail;
+ }
+ sdev->xs_pdev = pdev;
+
+ if (device_attach(DEV(pdev)) != 1) {
+ xrt_err(pdev, "failed to attach");
+ goto fail;
+ }
+
+ if (sysfs_create_group(&DEV(pdev)->kobj, &xrt_subdev_attrgroup))
+ xrt_err(pdev, "failed to create sysfs group");
+
+ /*
+ * Create sysfs sym link under root for leaves
+ * under random partitions for easy access to them.
+ */
+ if (id != XRT_SUBDEV_PART) {
+ if (sysfs_create_link(&find_root(pdev)->kobj,
+ &DEV(pdev)->kobj, dev_name(DEV(pdev)))) {
+ xrt_err(pdev, "failed to create sysfs link");
+ }
+ }
+
+ /* All done, ready to handle req thru cdev. */
+ if (xrt_subdev_cdev_auto_creation(pdev)) {
+ (void) xrt_devnode_create(pdev,
+ DEV_DRVDATA(pdev)->xsd_file_ops.xsf_dev_name, NULL);
+ }
+
+ vfree(pdata);
+ return sdev;
+
+fail:
+ vfree(pdata);
+ if (sdev && !IS_ERR_OR_NULL(sdev->xs_pdev))
+ platform_device_unregister(sdev->xs_pdev);
+ if (inst >= 0)
+ xrt_drv_put_instance(id, inst);
+ xrt_subdev_free(sdev);
+ return NULL;
+}
+
+static void xrt_subdev_destroy(struct xrt_subdev *sdev)
+{
+ struct platform_device *pdev = sdev->xs_pdev;
+ int inst = pdev->id;
+ struct device *dev = DEV(pdev);
+
+ /* Take down the device node */
+ if (xrt_subdev_cdev_auto_creation(pdev))
+ (void) xrt_devnode_destroy(pdev);
+ if (sdev->xs_id != XRT_SUBDEV_PART)
+ (void) sysfs_remove_link(&find_root(pdev)->kobj, dev_name(dev));
+ (void) sysfs_remove_group(&dev->kobj, &xrt_subdev_attrgroup);
+ platform_device_unregister(pdev);
+ xrt_drv_put_instance(sdev->xs_id, inst);
+ xrt_subdev_free(sdev);
+}
+
+int xrt_subdev_parent_ioctl(struct platform_device *self, u32 cmd, void *arg)
+{
+ struct device *dev = DEV(self);
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(self);
+
+ return (*pdata->xsp_parent_cb)(dev->parent, pdata->xsp_parent_cb_arg,
+ cmd, arg);
+}
+
+int xrt_subdev_ioctl(struct platform_device *tgt, u32 cmd, void *arg)
+{
+ struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(tgt);
+
+ return (*drvdata->xsd_dev_ops.xsd_ioctl)(tgt, cmd, arg);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_ioctl);
+
+struct platform_device *
+xrt_subdev_get_leaf(struct platform_device *pdev,
+ xrt_subdev_match_t match_cb, void *match_arg)
+{
+ int rc;
+ struct xrt_parent_ioctl_get_leaf get_leaf = {
+ pdev, match_cb, match_arg, };
+
+ rc = xrt_subdev_parent_ioctl(pdev, XRT_PARENT_GET_LEAF, &get_leaf);
+ if (rc)
+ return NULL;
+ return get_leaf.xpigl_leaf;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_get_leaf);
+
+struct subdev_match_arg {
+ enum xrt_subdev_id id;
+ int instance;
+};
+
+static bool subdev_match(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ struct subdev_match_arg *a = (struct subdev_match_arg *)arg;
+ return id == a->id &&
+ (pdev->id == a->instance || PLATFORM_DEVID_NONE == a->instance);
+}
+
+struct platform_device *
+xrt_subdev_get_leaf_by_id(struct platform_device *pdev,
+ enum xrt_subdev_id id, int instance)
+{
+ struct subdev_match_arg arg = { id, instance };
+
+ return xrt_subdev_get_leaf(pdev, subdev_match, &arg);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_get_leaf_by_id);
+
+bool xrt_subdev_has_epname(struct platform_device *pdev, const char *ep_name)
+{
+ struct resource *res;
+ int i;
+
+ for (i = 0, res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ res;
+ res = platform_get_resource(pdev, IORESOURCE_MEM, ++i)) {
+ if (!strncmp(res->name, ep_name, strlen(res->name) + 1))
+ return true;
+ }
+
+ return false;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_has_epname);
+
+static bool xrt_subdev_match_epname(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ return xrt_subdev_has_epname(pdev, arg);
+}
+
+struct platform_device *
+xrt_subdev_get_leaf_by_epname(struct platform_device *pdev, const char *name)
+{
+ return xrt_subdev_get_leaf(pdev, xrt_subdev_match_epname, (void *)name);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_get_leaf_by_epname);
+
+int xrt_subdev_put_leaf(struct platform_device *pdev,
+ struct platform_device *leaf)
+{
+ struct xrt_parent_ioctl_put_leaf put_leaf = { pdev, leaf };
+
+ return xrt_subdev_parent_ioctl(pdev, XRT_PARENT_PUT_LEAF, &put_leaf);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_put_leaf);
+
+int xrt_subdev_create_partition(struct platform_device *pdev, char *dtb)
+{
+ return xrt_subdev_parent_ioctl(pdev,
+ XRT_PARENT_CREATE_PARTITION, dtb);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_create_partition);
+
+int xrt_subdev_destroy_partition(struct platform_device *pdev, int instance)
+{
+ return xrt_subdev_parent_ioctl(pdev,
+ XRT_PARENT_REMOVE_PARTITION, (void *)(uintptr_t)instance);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_destroy_partition);
+
+int xrt_subdev_lookup_partition(struct platform_device *pdev,
+ xrt_subdev_match_t match_cb, void *match_arg)
+{
+ int rc;
+ struct xrt_parent_ioctl_lookup_partition lkp = {
+ pdev, match_cb, match_arg, };
+
+ rc = xrt_subdev_parent_ioctl(pdev, XRT_PARENT_LOOKUP_PARTITION, &lkp);
+ if (rc)
+ return rc;
+ return lkp.xpilp_part_inst;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_lookup_partition);
+
+int xrt_subdev_wait_for_partition_bringup(struct platform_device *pdev)
+{
+ return xrt_subdev_parent_ioctl(pdev,
+ XRT_PARENT_WAIT_PARTITION_BRINGUP, NULL);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_wait_for_partition_bringup);
+
+void *xrt_subdev_add_event_cb(struct platform_device *pdev,
+ xrt_subdev_match_t match, void *match_arg, xrt_event_cb_t cb)
+{
+ struct xrt_parent_ioctl_evt_cb c = { pdev, match, match_arg, cb };
+
+ (void) xrt_subdev_parent_ioctl(pdev, XRT_PARENT_ADD_EVENT_CB, &c);
+ return c.xevt_hdl;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_add_event_cb);
+
+void xrt_subdev_remove_event_cb(struct platform_device *pdev, void *hdl)
+{
+ (void) xrt_subdev_parent_ioctl(pdev, XRT_PARENT_REMOVE_EVENT_CB, hdl);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_remove_event_cb);
+
+static ssize_t
+xrt_subdev_get_holders(struct xrt_subdev *sdev, char *buf, size_t len)
+{
+ const struct list_head *ptr;
+ struct xrt_subdev_holder *h;
+ ssize_t n = 0;
+
+ list_for_each(ptr, &sdev->xs_holder_list) {
+ h = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list);
+ n += snprintf(buf + n, len - n, "%s:%d ",
+ dev_name(h->xsh_holder), h->xsh_count);
+ if (n >= len)
+ break;
+ }
+ return n;
+}
+
+void xrt_subdev_pool_init(struct device *dev, struct xrt_subdev_pool *spool)
+{
+ INIT_LIST_HEAD(&spool->xpool_dev_list);
+ spool->xpool_owner = dev;
+ mutex_init(&spool->xpool_lock);
+ spool->xpool_closing = false;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_init);
+
+static void xrt_subdev_pool_wait_for_holders(struct xrt_subdev_pool *spool,
+ struct xrt_subdev *sdev)
+{
+ const struct list_head *ptr, *next;
+ char holders[128];
+ struct xrt_subdev_holder *holder;
+ struct mutex *lk = &spool->xpool_lock;
+
+ BUG_ON(!mutex_is_locked(lk));
+
+ while (!list_empty(&sdev->xs_holder_list)) {
+ int rc;
+
+ /* It's most likely a bug if we ever enters this loop. */
+ (void) xrt_subdev_get_holders(sdev, holders, sizeof(holders));
+ xrt_err(sdev->xs_pdev, "awaits holders: %s", holders);
+ mutex_unlock(lk);
+ rc = wait_for_completion_killable(&sdev->xs_holder_comp);
+ mutex_lock(lk);
+ if (rc == -ERESTARTSYS) {
+ xrt_err(sdev->xs_pdev,
+ "give up on waiting for holders, clean up now");
+ list_for_each_safe(ptr, next, &sdev->xs_holder_list) {
+ holder = list_entry(ptr,
+ struct xrt_subdev_holder,
+ xsh_holder_list);
+ list_del(&holder->xsh_holder_list);
+ vfree(holder);
+ }
+ }
+ }
+}
+
+int xrt_subdev_pool_fini(struct xrt_subdev_pool *spool)
+{
+ int ret = 0;
+ struct list_head *dl = &spool->xpool_dev_list;
+ struct mutex *lk = &spool->xpool_lock;
+
+ mutex_lock(lk);
+
+ if (spool->xpool_closing) {
+ mutex_unlock(lk);
+ return 0;
+ }
+
+ spool->xpool_closing = true;
+ /* Remove subdev in the reverse order of added. */
+ while (!ret && !list_empty(dl)) {
+ struct xrt_subdev *sdev = list_first_entry(dl,
+ struct xrt_subdev, xs_dev_list);
+ xrt_subdev_pool_wait_for_holders(spool, sdev);
+ list_del(&sdev->xs_dev_list);
+ mutex_unlock(lk);
+ xrt_subdev_destroy(sdev);
+ mutex_lock(lk);
+ }
+
+ mutex_unlock(lk);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_fini);
+
+static int xrt_subdev_hold(struct xrt_subdev *sdev, struct device *holder_dev)
+{
+ const struct list_head *ptr;
+ struct list_head *hl = &sdev->xs_holder_list;
+ struct xrt_subdev_holder *holder;
+ bool found = false;
+
+ list_for_each(ptr, hl) {
+ holder = list_entry(ptr, struct xrt_subdev_holder,
+ xsh_holder_list);
+ if (holder->xsh_holder == holder_dev) {
+ holder->xsh_count++;
+ found = true;
+ break;
+ }
+ }
+
+ if (!found) {
+ holder = vzalloc(sizeof(*holder));
+ if (!holder)
+ return -ENOMEM;
+ holder->xsh_holder = holder_dev;
+ holder->xsh_count = 1;
+ list_add_tail(&holder->xsh_holder_list, hl);
+ }
+
+ return holder->xsh_count;
+}
+
+static int
+xrt_subdev_release(struct xrt_subdev *sdev, struct device *holder_dev)
+{
+ const struct list_head *ptr, *next;
+ struct list_head *hl = &sdev->xs_holder_list;
+ struct xrt_subdev_holder *holder;
+ int count;
+ bool found = false;
+
+ list_for_each_safe(ptr, next, hl) {
+ holder = list_entry(ptr, struct xrt_subdev_holder,
+ xsh_holder_list);
+ if (holder->xsh_holder == holder_dev) {
+ found = true;
+ holder->xsh_count--;
+
+ count = holder->xsh_count;
+ if (count == 0) {
+ list_del(&holder->xsh_holder_list);
+ vfree(holder);
+ if (list_empty(hl))
+ complete(&sdev->xs_holder_comp);
+ }
+ break;
+ }
+ }
+ if (!found) {
+ dev_err(holder_dev, "can't release, %s did not hold %s",
+ dev_name(holder_dev),
+ dev_name(DEV(sdev->xs_pdev)));
+ }
+ return found ? count : -EINVAL;
+}
+
+int xrt_subdev_pool_add(struct xrt_subdev_pool *spool, enum xrt_subdev_id id,
+ xrt_subdev_parent_cb_t pcb, void *pcb_arg, char *dtb)
+{
+ struct mutex *lk = &spool->xpool_lock;
+ struct list_head *dl = &spool->xpool_dev_list;
+ struct xrt_subdev *sdev;
+ int ret = 0;
+
+ sdev = xrt_subdev_create(spool->xpool_owner, id, pcb, pcb_arg, dtb);
+ if (sdev) {
+ mutex_lock(lk);
+ if (spool->xpool_closing) {
+ /* No new subdev when pool is going away. */
+ xrt_err(sdev->xs_pdev, "pool is closing");
+ ret = -ENODEV;
+ } else {
+ list_add(&sdev->xs_dev_list, dl);
+ }
+ mutex_unlock(lk);
+ if (ret)
+ xrt_subdev_destroy(sdev);
+ } else {
+ ret = -EINVAL;
+ }
+
+ return ret ? ret : sdev->xs_pdev->id;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_add);
+
+int xrt_subdev_pool_del(struct xrt_subdev_pool *spool, enum xrt_subdev_id id,
+ int instance)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xpool_lock;
+ struct list_head *dl = &spool->xpool_dev_list;
+ struct xrt_subdev *sdev;
+ int ret = -ENOENT;
+
+ mutex_lock(lk);
+ list_for_each(ptr, dl) {
+ sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (sdev->xs_id != id || sdev->xs_pdev->id != instance)
+ continue;
+ xrt_subdev_pool_wait_for_holders(spool, sdev);
+ list_del(&sdev->xs_dev_list);
+ ret = 0;
+ break;
+ }
+ mutex_unlock(lk);
+ if (ret)
+ return ret;
+
+ xrt_subdev_destroy(sdev);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_del);
+
+static int xrt_subdev_pool_get_impl(struct xrt_subdev_pool *spool,
+ xrt_subdev_match_t match, void *arg, struct device *holder_dev,
+ struct xrt_subdev **sdevp)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xpool_lock;
+ struct list_head *dl = &spool->xpool_dev_list;
+ struct xrt_subdev *sdev = NULL;
+ int ret = -ENOENT;
+
+ mutex_lock(lk);
+
+ if (match == XRT_SUBDEV_MATCH_PREV) {
+ struct platform_device *pdev = (struct platform_device *)arg;
+ struct xrt_subdev *d = NULL;
+
+ if (!pdev) {
+ sdev = list_empty(dl) ? NULL : list_last_entry(dl,
+ struct xrt_subdev, xs_dev_list);
+ } else {
+ list_for_each(ptr, dl) {
+ d = list_entry(ptr, struct xrt_subdev,
+ xs_dev_list);
+ if (d->xs_pdev != pdev)
+ continue;
+ if (!list_is_first(ptr, dl))
+ sdev = list_prev_entry(d, xs_dev_list);
+ break;
+ }
+ }
+ } else if (match == XRT_SUBDEV_MATCH_NEXT) {
+ struct platform_device *pdev = (struct platform_device *)arg;
+ struct xrt_subdev *d = NULL;
+
+ if (!pdev) {
+ sdev = list_first_entry_or_null(dl,
+ struct xrt_subdev, xs_dev_list);
+ } else {
+ list_for_each(ptr, dl) {
+ d = list_entry(ptr, struct xrt_subdev,
+ xs_dev_list);
+ if (d->xs_pdev != pdev)
+ continue;
+ if (!list_is_last(ptr, dl))
+ sdev = list_next_entry(d, xs_dev_list);
+ break;
+ }
+ }
+ } else {
+ list_for_each(ptr, dl) {
+ struct xrt_subdev *d = NULL;
+
+ d = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (d && !match(d->xs_id, d->xs_pdev, arg))
+ continue;
+ sdev = d;
+ break;
+ }
+ }
+
+ if (sdev)
+ ret = xrt_subdev_hold(sdev, holder_dev);
+
+ mutex_unlock(lk);
+
+ if (ret >= 0)
+ *sdevp = sdev;
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_get);
+
+int xrt_subdev_pool_get(struct xrt_subdev_pool *spool,
+ xrt_subdev_match_t match, void *arg, struct device *holder_dev,
+ struct platform_device **pdevp)
+{
+ int rc;
+ struct xrt_subdev *sdev;
+
+ rc = xrt_subdev_pool_get_impl(spool, match, arg, holder_dev, &sdev);
+ if (rc < 0) {
+ if (rc != -ENOENT)
+ dev_err(holder_dev, "failed to hold device: %d", rc);
+ return rc;
+ }
+
+ if (DEV_IS_PCI(holder_dev)) {
+#ifdef SUBDEV_DEBUG
+ dev_info(holder_dev, "%s: %s <<==== %s, ref=%d", __func__,
+ dev_name(holder_dev), dev_name(DEV(sdev->xs_pdev)), rc);
+#endif
+ } else {
+ xrt_info(to_platform_device(holder_dev), "%s <<==== %s",
+ dev_name(holder_dev), dev_name(DEV(sdev->xs_pdev)));
+ }
+
+ *pdevp = sdev->xs_pdev;
+ return 0;
+}
+
+static int xrt_subdev_pool_put_impl(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, struct device *holder_dev)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xpool_lock;
+ struct list_head *dl = &spool->xpool_dev_list;
+ struct xrt_subdev *sdev;
+ int ret = -ENOENT;
+
+ mutex_lock(lk);
+ list_for_each(ptr, dl) {
+ sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (sdev->xs_pdev != pdev)
+ continue;
+ ret = xrt_subdev_release(sdev, holder_dev);
+ break;
+ }
+ mutex_unlock(lk);
+
+ if (ret < 0 && ret != -ENOENT)
+ dev_err(holder_dev, "failed to release device: %d", ret);
+ return ret;
+}
+
+int xrt_subdev_pool_put(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, struct device *holder_dev)
+{
+ int ret = xrt_subdev_pool_put_impl(spool, pdev, holder_dev);
+
+ if (ret < 0)
+ return ret;
+
+ if (DEV_IS_PCI(holder_dev)) {
+#ifdef SUBDEV_DEBUG
+ dev_info(holder_dev, "%s: %s <<==X== %s, ref=%d", __func__,
+ dev_name(holder_dev), dev_name(DEV(spdev)), ret);
+#endif
+ } else {
+ struct platform_device *d = to_platform_device(holder_dev);
+
+ xrt_info(d, "%s <<==X== %s",
+ dev_name(holder_dev), dev_name(DEV(pdev)));
+ }
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_put);
+
+int xrt_subdev_pool_event(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, xrt_subdev_match_t match, void *arg,
+ xrt_event_cb_t xevt_cb, enum xrt_events evt)
+{
+ int rc = 0;
+ struct platform_device *tgt = NULL;
+ struct xrt_subdev *sdev = NULL;
+ struct xrt_event_arg_subdev esd;
+
+ while (!rc && xrt_subdev_pool_get_impl(spool, XRT_SUBDEV_MATCH_NEXT,
+ tgt, DEV(pdev), &sdev) != -ENOENT) {
+ tgt = sdev->xs_pdev;
+ esd.xevt_subdev_id = sdev->xs_id;
+ esd.xevt_subdev_instance = tgt->id;
+ if (match(sdev->xs_id, sdev->xs_pdev, arg))
+ rc = xevt_cb(pdev, evt, &esd);
+ (void) xrt_subdev_pool_put_impl(spool, tgt, DEV(pdev));
+ }
+ return rc;
+}
+
+ssize_t xrt_subdev_pool_get_holders(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, char *buf, size_t len)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xpool_lock;
+ struct list_head *dl = &spool->xpool_dev_list;
+ struct xrt_subdev *sdev;
+ ssize_t ret = 0;
+
+ mutex_lock(lk);
+ list_for_each(ptr, dl) {
+ sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (sdev->xs_pdev != pdev)
+ continue;
+ ret = xrt_subdev_get_holders(sdev, buf, len);
+ break;
+ }
+ mutex_unlock(lk);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_get_holders);
+
+int xrt_subdev_broadcast_event_async(struct platform_device *pdev,
+ enum xrt_events evt, xrt_async_broadcast_event_cb_t cb, void *arg)
+{
+ struct xrt_parent_ioctl_async_broadcast_evt e = { pdev, evt, cb, arg };
+
+ return xrt_subdev_parent_ioctl(pdev,
+ XRT_PARENT_ASYNC_BOARDCAST_EVENT, &e);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_broadcast_event_async);
+
+struct xrt_broadcast_event_arg {
+ struct completion comp;
+ bool success;
+};
+
+static void xrt_broadcast_event_cb(struct platform_device *pdev,
+ enum xrt_events evt, void *arg, bool success)
+{
+ struct xrt_broadcast_event_arg *e =
+ (struct xrt_broadcast_event_arg *)arg;
+
+ e->success = success;
+ complete(&e->comp);
+}
+
+int xrt_subdev_broadcast_event(struct platform_device *pdev,
+ enum xrt_events evt)
+{
+ int ret;
+ struct xrt_broadcast_event_arg e;
+
+ init_completion(&e.comp);
+ e.success = false;
+ ret = xrt_subdev_broadcast_event_async(pdev, evt,
+ xrt_broadcast_event_cb, &e);
+ if (ret == 0)
+ wait_for_completion(&e.comp);
+ return e.success ? 0 : -EINVAL;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_broadcast_event);
+
+void xrt_subdev_hot_reset(struct platform_device *pdev)
+{
+ (void) xrt_subdev_parent_ioctl(pdev, XRT_PARENT_HOT_RESET, NULL);
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_hot_reset);
+
+void xrt_subdev_get_barres(struct platform_device *pdev,
+ struct resource **res, uint bar_idx)
+{
+ struct xrt_parent_ioctl_get_res arg = { 0 };
+
+ BUG_ON(bar_idx > PCI_STD_RESOURCE_END);
+
+ (void) xrt_subdev_parent_ioctl(pdev, XRT_PARENT_GET_RESOURCE, &arg);
+
+ *res = &arg.xpigr_res[bar_idx];
+}
+
+void xrt_subdev_get_parent_id(struct platform_device *pdev,
+ unsigned short *vendor, unsigned short *device,
+ unsigned short *subvendor, unsigned short *subdevice)
+{
+ struct xrt_parent_ioctl_get_id id = { 0 };
+
+ (void) xrt_subdev_parent_ioctl(pdev, XRT_PARENT_GET_ID, (void *)&id);
+ if (vendor)
+ *vendor = id.xpigi_vendor_id;
+ if (device)
+ *device = id.xpigi_device_id;
+ if (subvendor)
+ *subvendor = id.xpigi_sub_vendor_id;
+ if (subdevice)
+ *subdevice = id.xpigi_sub_device_id;
+}
+
+struct device *xrt_subdev_register_hwmon(struct platform_device *pdev,
+ const char *name, void *drvdata, const struct attribute_group **grps)
+{
+ struct xrt_parent_ioctl_hwmon hm = { true, name, drvdata, grps, };
+
+ (void) xrt_subdev_parent_ioctl(pdev, XRT_PARENT_HWMON, (void *)&hm);
+ return hm.xpih_hwmon_dev;
+}
+
+void xrt_subdev_unregister_hwmon(struct platform_device *pdev,
+ struct device *hwmon)
+{
+ struct xrt_parent_ioctl_hwmon hm = { false, };
+
+ hm.xpih_hwmon_dev = hwmon;
+ (void) xrt_subdev_parent_ioctl(pdev, XRT_PARENT_HWMON, (void *)&hm);
+}
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH Xilinx Alveo 6/8] fpga: xrt: header file for platform and parent drivers
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
` (3 preceding siblings ...)
2020-11-29 0:00 ` [PATCH Xilinx Alveo 4/8] fpga: xrt: core infrastructure for xrt-lib module Sonal Santan
@ 2020-11-29 0:00 ` Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver Sonal Santan
` (4 subsequent siblings)
9 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
From: Sonal Santan <sonal.santan@xilinx.com>
Add private header files for platform and parent drivers.
Each header file defines ioctls supported by the platform
or parent driver. The header files also define core data
structures for sending and receiving events by platform
and parent drivers.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
---
drivers/fpga/alveo/include/xmgmt-main.h | 34 +++
drivers/fpga/alveo/include/xrt-axigate.h | 31 ++
drivers/fpga/alveo/include/xrt-calib.h | 28 ++
drivers/fpga/alveo/include/xrt-clkfreq.h | 21 ++
drivers/fpga/alveo/include/xrt-clock.h | 29 ++
drivers/fpga/alveo/include/xrt-cmc.h | 23 ++
drivers/fpga/alveo/include/xrt-ddr-srsr.h | 29 ++
drivers/fpga/alveo/include/xrt-flash.h | 28 ++
drivers/fpga/alveo/include/xrt-gpio.h | 41 +++
drivers/fpga/alveo/include/xrt-icap.h | 27 ++
drivers/fpga/alveo/include/xrt-mailbox.h | 44 +++
drivers/fpga/alveo/include/xrt-metadata.h | 184 ++++++++++++
drivers/fpga/alveo/include/xrt-parent.h | 103 +++++++
drivers/fpga/alveo/include/xrt-partition.h | 33 ++
drivers/fpga/alveo/include/xrt-subdev.h | 333 +++++++++++++++++++++
drivers/fpga/alveo/include/xrt-ucs.h | 22 ++
16 files changed, 1010 insertions(+)
create mode 100644 drivers/fpga/alveo/include/xmgmt-main.h
create mode 100644 drivers/fpga/alveo/include/xrt-axigate.h
create mode 100644 drivers/fpga/alveo/include/xrt-calib.h
create mode 100644 drivers/fpga/alveo/include/xrt-clkfreq.h
create mode 100644 drivers/fpga/alveo/include/xrt-clock.h
create mode 100644 drivers/fpga/alveo/include/xrt-cmc.h
create mode 100644 drivers/fpga/alveo/include/xrt-ddr-srsr.h
create mode 100644 drivers/fpga/alveo/include/xrt-flash.h
create mode 100644 drivers/fpga/alveo/include/xrt-gpio.h
create mode 100644 drivers/fpga/alveo/include/xrt-icap.h
create mode 100644 drivers/fpga/alveo/include/xrt-mailbox.h
create mode 100644 drivers/fpga/alveo/include/xrt-metadata.h
create mode 100644 drivers/fpga/alveo/include/xrt-parent.h
create mode 100644 drivers/fpga/alveo/include/xrt-partition.h
create mode 100644 drivers/fpga/alveo/include/xrt-subdev.h
create mode 100644 drivers/fpga/alveo/include/xrt-ucs.h
diff --git a/drivers/fpga/alveo/include/xmgmt-main.h b/drivers/fpga/alveo/include/xmgmt-main.h
new file mode 100644
index 000000000000..3f26c480ce27
--- /dev/null
+++ b/drivers/fpga/alveo/include/xmgmt-main.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XMGMT_MAIN_H_
+#define _XMGMT_MAIN_H_
+
+#include <linux/xrt/xclbin.h>
+
+enum xrt_mgmt_main_ioctl_cmd {
+ // section needs to be vfree'd by caller
+ XRT_MGMT_MAIN_GET_AXLF_SECTION = 0,
+ // vbnv needs to be kfree'd by caller
+ XRT_MGMT_MAIN_GET_VBNV,
+};
+
+enum provider_kind {
+ XMGMT_BLP,
+ XMGMT_PLP,
+ XMGMT_ULP,
+};
+
+struct xrt_mgmt_main_ioctl_get_axlf_section {
+ enum provider_kind xmmigas_axlf_kind;
+ enum axlf_section_kind xmmigas_section_kind;
+ void *xmmigas_section;
+ u64 xmmigas_section_size;
+};
+
+#endif /* _XMGMT_MAIN_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-axigate.h b/drivers/fpga/alveo/include/xrt-axigate.h
new file mode 100644
index 000000000000..b1dd70546040
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-axigate.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_AXIGATE_H_
+#define _XRT_AXIGATE_H_
+
+
+#include "xrt-subdev.h"
+#include "xrt-metadata.h"
+
+/*
+ * AXIGATE driver IOCTL calls.
+ */
+enum xrt_axigate_ioctl_cmd {
+ XRT_AXIGATE_FREEZE = 0,
+ XRT_AXIGATE_FREE,
+};
+
+/* the ep names are in the order of hardware layers */
+static const char * const xrt_axigate_epnames[] = {
+ NODE_GATE_PLP,
+ NODE_GATE_ULP,
+ NULL
+};
+
+#endif /* _XRT_AXIGATE_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-calib.h b/drivers/fpga/alveo/include/xrt-calib.h
new file mode 100644
index 000000000000..5e5bb5cec285
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-calib.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_CALIB_H_
+#define _XRT_CALIB_H_
+
+#include "xrt-subdev.h"
+#include <linux/xrt/xclbin.h>
+
+/*
+ * Memory calibration driver IOCTL calls.
+ */
+enum xrt_calib_results {
+ XRT_CALIB_UNKNOWN,
+ XRT_CALIB_SUCCEEDED,
+ XRT_CALIB_FAILED,
+};
+
+enum xrt_calib_ioctl_cmd {
+ XRT_CALIB_RESULT = 0,
+};
+
+#endif /* _XRT_CALIB_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-clkfreq.h b/drivers/fpga/alveo/include/xrt-clkfreq.h
new file mode 100644
index 000000000000..60e4109cc05a
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-clkfreq.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_CLKFREQ_H_
+#define _XRT_CLKFREQ_H_
+
+#include "xrt-subdev.h"
+
+/*
+ * CLKFREQ driver IOCTL calls.
+ */
+enum xrt_clkfreq_ioctl_cmd {
+ XRT_CLKFREQ_READ = 0,
+};
+
+#endif /* _XRT_CLKFREQ_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-clock.h b/drivers/fpga/alveo/include/xrt-clock.h
new file mode 100644
index 000000000000..d98d9d619bb2
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-clock.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_CLOCK_H_
+#define _XRT_CLOCK_H_
+
+#include "xrt-subdev.h"
+#include <linux/xrt/xclbin.h>
+
+/*
+ * CLOCK driver IOCTL calls.
+ */
+enum xrt_clock_ioctl_cmd {
+ XRT_CLOCK_SET = 0,
+ XRT_CLOCK_GET,
+ XRT_CLOCK_VERIFY,
+};
+
+struct xrt_clock_ioctl_get {
+ u16 freq;
+ u32 freq_cnter;
+};
+
+#endif /* _XRT_CLOCK_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-cmc.h b/drivers/fpga/alveo/include/xrt-cmc.h
new file mode 100644
index 000000000000..f2bb61a2ab23
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-cmc.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_CMC_H_
+#define _XRT_CMC_H_
+
+#include "xrt-subdev.h"
+#include <linux/xrt/xclbin.h>
+
+/*
+ * CMC driver IOCTL calls.
+ */
+enum xrt_cmc_ioctl_cmd {
+ XRT_CMC_READ_BOARD_INFO = 0,
+ XRT_CMC_READ_SENSORS,
+};
+
+#endif /* _XRT_CMC_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-ddr-srsr.h b/drivers/fpga/alveo/include/xrt-ddr-srsr.h
new file mode 100644
index 000000000000..56dc2ff8ea7c
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-ddr-srsr.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_DDR_SRSR_H_
+#define _XRT_DDR_SRSR_H_
+
+#include "xrt-subdev.h"
+
+/*
+ * ddr-srsr driver IOCTL calls.
+ */
+enum xrt_ddr_srsr_ioctl_cmd {
+ XRT_SRSR_FAST_CALIB,
+ XRT_SRSR_CALIB,
+ XRT_SRSR_EP_NAME,
+};
+
+struct xrt_srsr_ioctl_calib {
+ void *xsic_buf;
+ u32 xsic_size;
+ bool xsic_retention;
+};
+
+#endif
diff --git a/drivers/fpga/alveo/include/xrt-flash.h b/drivers/fpga/alveo/include/xrt-flash.h
new file mode 100644
index 000000000000..949f490a3154
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-flash.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_FLASH_H_
+#define _XRT_FLASH_H_
+
+#include "xrt-subdev.h"
+
+/*
+ * Flash controller driver IOCTL calls.
+ */
+enum xrt_flash_ioctl_cmd {
+ XRT_FLASH_GET_SIZE = 0,
+ XRT_FLASH_READ,
+};
+
+struct xrt_flash_ioctl_read {
+ char *xfir_buf;
+ size_t xfir_size;
+ loff_t xfir_offset;
+};
+
+#endif /* _XRT_FLASH_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-gpio.h b/drivers/fpga/alveo/include/xrt-gpio.h
new file mode 100644
index 000000000000..d85a356f5281
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-gpio.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_GPIO_H_
+#define _XRT_GPIO_H_
+
+#include "xrt-subdev.h"
+
+/*
+ * GPIO driver IOCTL calls.
+ */
+enum xrt_gpio_ioctl_cmd {
+ XRT_GPIO_READ = 0,
+ XRT_GPIO_WRITE,
+};
+
+enum xrt_gpio_id {
+ XRT_GPIO_ROM_UUID,
+ XRT_GPIO_DDR_CALIB,
+ XRT_GPIO_GOLDEN_VER,
+ XRT_GPIO_MAX
+};
+
+struct xrt_gpio_ioctl_rw {
+ u32 xgir_id;
+ void *xgir_buf;
+ u32 xgir_len;
+ u32 xgir_offset;
+};
+
+struct xrt_gpio_ioctl_intf_uuid {
+ u32 xgir_uuid_num;
+ uuid_t *xgir_uuids;
+};
+
+#endif /* _XRT_GPIO_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-icap.h b/drivers/fpga/alveo/include/xrt-icap.h
new file mode 100644
index 000000000000..ea9e688e07eb
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-icap.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_ICAP_H_
+#define _XRT_ICAP_H_
+
+#include "xrt-subdev.h"
+
+/*
+ * ICAP driver IOCTL calls.
+ */
+enum xrt_icap_ioctl_cmd {
+ XRT_ICAP_WRITE = 0,
+ XRT_ICAP_IDCODE,
+};
+
+struct xrt_icap_ioctl_wr {
+ void *xiiw_bit_data;
+ u32 xiiw_data_len;
+};
+
+#endif /* _XRT_ICAP_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-mailbox.h b/drivers/fpga/alveo/include/xrt-mailbox.h
new file mode 100644
index 000000000000..cfa7e112c51b
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-mailbox.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_MAILBOX_H_
+#define _XRT_MAILBOX_H_
+
+/*
+ * Mailbox IP driver IOCTL calls.
+ */
+enum xrt_mailbox_ioctl_cmd {
+ XRT_MAILBOX_POST = 0,
+ XRT_MAILBOX_REQUEST,
+ XRT_MAILBOX_LISTEN,
+};
+
+struct xrt_mailbox_ioctl_post {
+ u64 xmip_req_id; /* 0 means response */
+ bool xmip_sw_ch;
+ void *xmip_data;
+ size_t xmip_data_size;
+};
+
+struct xrt_mailbox_ioctl_request {
+ bool xmir_sw_ch;
+ u32 xmir_resp_ttl;
+ void *xmir_req;
+ size_t xmir_req_size;
+ void *xmir_resp;
+ size_t xmir_resp_size;
+};
+
+typedef void (*mailbox_msg_cb_t)(void *arg, void *data, size_t len,
+ u64 msgid, int err, bool sw_ch);
+struct xrt_mailbox_ioctl_listen {
+ mailbox_msg_cb_t xmil_cb;
+ void *xmil_cb_arg;
+};
+
+#endif /* _XRT_MAILBOX_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-metadata.h b/drivers/fpga/alveo/include/xrt-metadata.h
new file mode 100644
index 000000000000..f445bfc279d2
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-metadata.h
@@ -0,0 +1,184 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Xilinx Alveo FPGA Test Leaf Driver
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_METADATA_H
+#define _XRT_METADATA_H
+
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uuid.h>
+
+#define PROP_COMPATIBLE "compatible"
+#define PROP_PF_NUM "pcie_physical_function"
+#define PROP_BAR_IDX "pcie_bar_mapping"
+#define PROP_IO_OFFSET "reg"
+#define PROP_INTERRUPTS "interrupts"
+#define PROP_INTERFACE_UUID "interface_uuid"
+#define PROP_LOGIC_UUID "logic_uuid"
+#define PROP_VERSION_MAJOR "firmware_version_major"
+
+#define PROP_HWICAP "axi_hwicap"
+#define PROP_PDI_CONFIG "pdi_config_mem"
+
+#define NODE_ENDPOINTS "addressable_endpoints"
+#define INTERFACES_PATH "/interfaces"
+
+#define NODE_FIRMWARE "firmware"
+#define NODE_INTERFACES "interfaces"
+#define NODE_PARTITION_INFO "partition_info"
+
+#define NODE_FLASH "ep_card_flash_program_00"
+#define NODE_XVC_PUB "ep_debug_bscan_user_00"
+#define NODE_XVC_PRI "ep_debug_bscan_mgmt_00"
+#define NODE_SYSMON "ep_cmp_sysmon_00"
+#define NODE_AF_BLP_CTRL_MGMT "ep_firewall_blp_ctrl_mgmt_00"
+#define NODE_AF_BLP_CTRL_USER "ep_firewall_blp_ctrl_user_00"
+#define NODE_AF_CTRL_MGMT "ep_firewall_ctrl_mgmt_00"
+#define NODE_AF_CTRL_USER "ep_firewall_ctrl_user_00"
+#define NODE_AF_CTRL_DEBUG "ep_firewall_ctrl_debug_00"
+#define NODE_AF_DATA_H2C "ep_firewall_data_h2c_00"
+#define NODE_AF_DATA_C2H "ep_firewall_data_c2h_00"
+#define NODE_AF_DATA_P2P "ep_firewall_data_p2p_00"
+#define NODE_AF_DATA_M2M "ep_firewall_data_m2m_00"
+#define NODE_CMC_REG "ep_cmc_regmap_00"
+#define NODE_CMC_RESET "ep_cmc_reset_00"
+#define NODE_CMC_MUTEX "ep_cmc_mutex_00"
+#define NODE_CMC_FW_MEM "ep_cmc_firmware_mem_00"
+#define NODE_ERT_FW_MEM "ep_ert_firmware_mem_00"
+#define NODE_ERT_CQ_MGMT "ep_ert_command_queue_mgmt_00"
+#define NODE_ERT_CQ_USER "ep_ert_command_queue_user_00"
+#define NODE_MAILBOX_MGMT "ep_mailbox_mgmt_00"
+#define NODE_MAILBOX_USER "ep_mailbox_user_00"
+#define NODE_GATE_PLP "ep_pr_isolate_plp_00"
+#define NODE_GATE_ULP "ep_pr_isolate_ulp_00"
+#define NODE_PCIE_MON "ep_pcie_link_mon_00"
+#define NODE_DDR_CALIB "ep_ddr_mem_calib_00"
+#define NODE_CLK_KERNEL1 "ep_aclk_kernel_00"
+#define NODE_CLK_KERNEL2 "ep_aclk_kernel_01"
+#define NODE_CLK_KERNEL3 "ep_aclk_hbm_00"
+#define NODE_KDMA_CTRL "ep_kdma_ctrl_00"
+#define NODE_FPGA_CONFIG "ep_fpga_configuration_00"
+#define NODE_ERT_SCHED "ep_ert_sched_00"
+#define NODE_XDMA "ep_xdma_00"
+#define NODE_MSIX "ep_msix_00"
+#define NODE_QDMA "ep_qdma_00"
+#define NODE_QDMA4 "ep_qdma4_00"
+#define NODE_STM "ep_stream_traffic_manager_00"
+#define NODE_STM4 "ep_stream_traffic_manager4_00"
+#define NODE_CLK_SHUTDOWN "ep_aclk_shutdown_00"
+#define NODE_ERT_BASE "ep_ert_base_address_00"
+#define NODE_ERT_RESET "ep_ert_reset_00"
+#define NODE_CLKFREQ_K1 "ep_freq_cnt_aclk_kernel_00"
+#define NODE_CLKFREQ_K2 "ep_freq_cnt_aclk_kernel_01"
+#define NODE_CLKFREQ_HBM "ep_freq_cnt_aclk_hbm_00"
+#define NODE_GAPPING "ep_gapping_demand_00"
+#define NODE_UCS_CONTROL_STATUS "ep_ucs_control_status_00"
+#define NODE_P2P "ep_p2p_00"
+#define NODE_REMAP_P2P "ep_remap_p2p_00"
+#define NODE_DDR4_RESET_GATE "ep_ddr_mem_srsr_gate_00"
+#define NODE_ADDR_TRANSLATOR "ep_remap_data_c2h_00"
+#define NODE_MAILBOX_XRT "ep_mailbox_user_to_ert_00"
+#define NODE_PMC_INTR "ep_pmc_intr_00"
+#define NODE_PMC_MUX "ep_pmc_mux_00"
+
+/* driver defined endpoints */
+#define NODE_VSEC "drv_ep_vsec_00"
+#define NODE_VSEC_GOLDEN "drv_ep_vsec_golden_00"
+#define NODE_BLP_ROM "drv_ep_blp_rom_00"
+#define NODE_MAILBOX_VSEC "ep_mailbox_vsec_00"
+#define NODE_PLAT_INFO "drv_ep_platform_info_mgmt_00"
+#define NODE_TEST "drv_ep_test_00"
+#define NODE_MGMT_MAIN "drv_ep_mgmt_main_00"
+#define NODE_FLASH_VSEC "drv_ep_card_flash_program_00"
+#define NODE_GOLDEN_VER "drv_ep_golden_ver_00"
+#define NODE_PARTITION_INFO_BLP "partition_info_0"
+#define NODE_PARTITION_INFO_PLP "partition_info_1"
+
+#define NODE_DDR_SRSR "drv_ep_ddr_srsr"
+#define REGMAP_DDR_SRSR "drv_ddr_srsr"
+
+#define PROP_OFFSET "drv_offset"
+#define PROP_CLK_FREQ "drv_clock_frequency"
+#define PROP_CLK_CNT "drv_clock_frequency_counter"
+#define PROP_VBNV "vbnv"
+#define PROP_VROM "vrom"
+#define PROP_PARTITION_LEVEL "partition_level"
+
+struct xrt_md_endpoint {
+ const char *ep_name;
+ u32 bar;
+ long bar_off;
+ ulong size;
+ char *regmap;
+ char *regmap_ver;
+};
+
+/* Note: res_id is defined by leaf driver and must start with 0. */
+struct xrt_iores_map {
+ char *res_name;
+ int res_id;
+};
+
+static inline int xrt_md_res_name2id(const struct xrt_iores_map *res_map,
+ int entry_num, const char *res_name)
+{
+ int i;
+
+ BUG_ON(res_name == NULL);
+ for (i = 0; i < entry_num; i++) {
+ if (!strcmp(res_name, res_map->res_name))
+ return res_map->res_id;
+ res_map++;
+ }
+ return -1;
+}
+
+static inline const char *
+xrt_md_res_id2name(const struct xrt_iores_map *res_map, int entry_num, int id)
+{
+ int i;
+
+ BUG_ON(id > entry_num);
+ for (i = 0; i < entry_num; i++) {
+ if (res_map->res_id == id)
+ return res_map->res_name;
+ res_map++;
+ }
+ return NULL;
+}
+
+long xrt_md_size(struct device *dev, const char *blob);
+int xrt_md_create(struct device *dev, char **blob);
+int xrt_md_add_endpoint(struct device *dev, char *blob,
+ struct xrt_md_endpoint *ep);
+int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
+ char *regmap_name);
+int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
+ const char *regmap_name, const char *prop, const void **val, int *size);
+int xrt_md_set_prop(struct device *dev, char *blob, const char *ep_name,
+ const char *regmap_name, const char *prop, const void *val, int size);
+int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
+ const char *ep_name, const char *regmap_name, const char *new_ep_name);
+int xrt_md_copy_all_eps(struct device *dev, char *blob, const char *src_blob);
+int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ char **next_ep, char **next_regmap);
+int xrt_md_get_compatible_epname(struct device *dev, const char *blob,
+ const char *regmap_name, const char **ep_name);
+int xrt_md_get_epname_pointer(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name, const char **epname);
+void xrt_md_pack(struct device *dev, char *blob);
+char *xrt_md_dup(struct device *dev, const char *blob);
+int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
+ u32 *num_uuids, uuid_t *intf_uuids);
+int xrt_md_check_uuids(struct device *dev, const char *blob, char *subset_blob);
+int xrt_md_uuid_strtoid(struct device *dev, const char *uuidstr, uuid_t *uuid);
+
+#endif
diff --git a/drivers/fpga/alveo/include/xrt-parent.h b/drivers/fpga/alveo/include/xrt-parent.h
new file mode 100644
index 000000000000..28de117fbf91
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-parent.h
@@ -0,0 +1,103 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_PARENT_H_
+#define _XRT_PARENT_H_
+
+#include "xrt-subdev.h"
+#include "xrt-partition.h"
+
+/*
+ * Parent IOCTL calls.
+ */
+enum xrt_parent_ioctl_cmd {
+ /* Leaf actions. */
+ XRT_PARENT_GET_LEAF = 0,
+ XRT_PARENT_PUT_LEAF,
+ XRT_PARENT_GET_LEAF_HOLDERS,
+
+ /* Partition actions. */
+ XRT_PARENT_CREATE_PARTITION,
+ XRT_PARENT_REMOVE_PARTITION,
+ XRT_PARENT_LOOKUP_PARTITION,
+ XRT_PARENT_WAIT_PARTITION_BRINGUP,
+
+ /* Event actions. */
+ XRT_PARENT_ADD_EVENT_CB,
+ XRT_PARENT_REMOVE_EVENT_CB,
+ XRT_PARENT_ASYNC_BOARDCAST_EVENT,
+
+ /* Device info. */
+ XRT_PARENT_GET_RESOURCE,
+ XRT_PARENT_GET_ID,
+
+ /* Misc. */
+ XRT_PARENT_HOT_RESET,
+ XRT_PARENT_HWMON,
+};
+
+struct xrt_parent_ioctl_get_leaf {
+ struct platform_device *xpigl_pdev; /* caller's pdev */
+ xrt_subdev_match_t xpigl_match_cb;
+ void *xpigl_match_arg;
+ struct platform_device *xpigl_leaf; /* target leaf pdev */
+};
+
+struct xrt_parent_ioctl_put_leaf {
+ struct platform_device *xpipl_pdev; /* caller's pdev */
+ struct platform_device *xpipl_leaf; /* target's pdev */
+};
+
+struct xrt_parent_ioctl_lookup_partition {
+ struct platform_device *xpilp_pdev; /* caller's pdev */
+ xrt_subdev_match_t xpilp_match_cb;
+ void *xpilp_match_arg;
+ int xpilp_part_inst;
+};
+
+struct xrt_parent_ioctl_evt_cb {
+ struct platform_device *xevt_pdev; /* caller's pdev */
+ xrt_subdev_match_t xevt_match_cb;
+ void *xevt_match_arg;
+ xrt_event_cb_t xevt_cb;
+ void *xevt_hdl;
+};
+
+struct xrt_parent_ioctl_async_broadcast_evt {
+ struct platform_device *xaevt_pdev; /* caller's pdev */
+ enum xrt_events xaevt_event;
+ xrt_async_broadcast_event_cb_t xaevt_cb;
+ void *xaevt_arg;
+};
+
+struct xrt_parent_ioctl_get_holders {
+ struct platform_device *xpigh_pdev; /* caller's pdev */
+ char *xpigh_holder_buf;
+ size_t xpigh_holder_buf_len;
+};
+
+struct xrt_parent_ioctl_get_res {
+ struct resource *xpigr_res;
+};
+
+struct xrt_parent_ioctl_get_id {
+ unsigned short xpigi_vendor_id;
+ unsigned short xpigi_device_id;
+ unsigned short xpigi_sub_vendor_id;
+ unsigned short xpigi_sub_device_id;
+};
+
+struct xrt_parent_ioctl_hwmon {
+ bool xpih_register;
+ const char *xpih_name;
+ void *xpih_drvdata;
+ const struct attribute_group **xpih_groups;
+ struct device *xpih_hwmon_dev;
+};
+
+#endif /* _XRT_PARENT_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-partition.h b/drivers/fpga/alveo/include/xrt-partition.h
new file mode 100644
index 000000000000..e0048f2a146f
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-partition.h
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_PARTITION_H_
+#define _XRT_PARTITION_H_
+
+#include "xrt-subdev.h"
+
+/*
+ * Partition driver IOCTL calls.
+ */
+enum xrt_partition_ioctl_cmd {
+ XRT_PARTITION_GET_LEAF = 0,
+ XRT_PARTITION_PUT_LEAF,
+ XRT_PARTITION_INIT_CHILDREN,
+ XRT_PARTITION_FINI_CHILDREN,
+ XRT_PARTITION_EVENT,
+};
+
+struct xrt_partition_ioctl_event {
+ enum xrt_events xpie_evt;
+ struct xrt_parent_ioctl_evt_cb *xpie_cb;
+};
+
+extern int xrt_subdev_parent_ioctl(struct platform_device *pdev,
+ u32 cmd, void *arg);
+
+#endif /* _XRT_PARTITION_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-subdev.h b/drivers/fpga/alveo/include/xrt-subdev.h
new file mode 100644
index 000000000000..65ecbd9c596b
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-subdev.h
@@ -0,0 +1,333 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_SUBDEV_H_
+#define _XRT_SUBDEV_H_
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/fs.h>
+#include <linux/cdev.h>
+#include <linux/pci.h>
+#include <linux/libfdt_env.h>
+#include "libfdt.h"
+
+/*
+ * Every subdev driver should have an ID for others to refer to it.
+ * There can be unlimited number of instances of a subdev driver. A
+ * <subdev_id, subdev_instance> tuple should be a unique identification of
+ * a specific instance of a subdev driver.
+ * NOTE: PLEASE do not change the order of IDs. Sub devices in the same
+ * partition are initialized by this order.
+ */
+enum xrt_subdev_id {
+ XRT_SUBDEV_PART = 0,
+ XRT_SUBDEV_VSEC,
+ XRT_SUBDEV_VSEC_GOLDEN,
+ XRT_SUBDEV_GPIO,
+ XRT_SUBDEV_AXIGATE,
+ XRT_SUBDEV_ICAP,
+ XRT_SUBDEV_TEST,
+ XRT_SUBDEV_MGMT_MAIN,
+ XRT_SUBDEV_QSPI,
+ XRT_SUBDEV_MAILBOX,
+ XRT_SUBDEV_CMC,
+ XRT_SUBDEV_CALIB,
+ XRT_SUBDEV_CLKFREQ,
+ XRT_SUBDEV_CLOCK,
+ XRT_SUBDEV_SRSR,
+ XRT_SUBDEV_UCS,
+ XRT_SUBDEV_NUM,
+};
+
+/*
+ * If populated by subdev driver, parent will handle the mechanics of
+ * char device (un)registration.
+ */
+enum xrt_subdev_file_mode {
+ // Infra create cdev, default file name
+ XRT_SUBDEV_FILE_DEFAULT = 0,
+ // Infra create cdev, need to encode inst num in file name
+ XRT_SUBDEV_FILE_MULTI_INST,
+ // No auto creation of cdev by infra, leaf handles it by itself
+ XRT_SUBDEV_FILE_NO_AUTO,
+};
+struct xrt_subdev_file_ops {
+ const struct file_operations xsf_ops;
+ dev_t xsf_dev_t;
+ const char *xsf_dev_name;
+ enum xrt_subdev_file_mode xsf_mode;
+};
+
+/*
+ * Subdev driver callbacks populated by subdev driver.
+ */
+struct xrt_subdev_drv_ops {
+ /*
+ * Per driver module callback. Don't take any arguments.
+ * If defined these are called as part of driver (un)registration.
+ */
+ int (*xsd_post_init)(void);
+ void (*xsd_pre_exit)(void);
+
+ /*
+ * Per driver instance callback. The pdev points to the instance.
+ * If defined these are called by other leaf drivers.
+ * Note that root driver may call into xsd_ioctl of a partition driver.
+ */
+ int (*xsd_ioctl)(struct platform_device *pdev, u32 cmd, void *arg);
+};
+
+/*
+ * Defined and populated by subdev driver, exported as driver_data in
+ * struct platform_device_id.
+ */
+struct xrt_subdev_drvdata {
+ struct xrt_subdev_file_ops xsd_file_ops;
+ struct xrt_subdev_drv_ops xsd_dev_ops;
+};
+
+/*
+ * Partially initialized by parent driver, then, passed in as subdev driver's
+ * platform data when creating subdev driver instance by calling platform
+ * device register API (platform_device_register_data() or the likes).
+ *
+ * Once device register API returns, platform driver framework makes a copy of
+ * this buffer and maintains its life cycle. The content of the buffer is
+ * completely owned by subdev driver.
+ *
+ * Thus, parent driver should be very careful when it touches this buffer
+ * again once it's handed over to subdev driver. And the data structure
+ * should not contain pointers pointing to buffers that is managed by
+ * other or parent drivers since it could have been freed before platform
+ * data buffer is freed by platform driver framework.
+ */
+typedef int (*xrt_subdev_parent_cb_t)(struct device *, void *, u32, void *);
+struct xrt_subdev_platdata {
+ /*
+ * Per driver instance callback. The pdev points to the instance.
+ * Should always be defined for subdev driver to call into its parent.
+ */
+ xrt_subdev_parent_cb_t xsp_parent_cb;
+ void *xsp_parent_cb_arg;
+
+ /* Something to associate w/ root for msg printing. */
+ const char *xsp_root_name;
+
+ /*
+ * Char dev support for this subdev instance.
+ * Initialized by subdev driver.
+ */
+ struct cdev xsp_cdev;
+ struct device *xsp_sysdev;
+ struct mutex xsp_devnode_lock;
+ struct completion xsp_devnode_comp;
+ int xsp_devnode_ref;
+ bool xsp_devnode_online;
+ bool xsp_devnode_excl;
+
+ /*
+ * Subdev driver specific init data. The buffer should be embedded
+ * in this data structure buffer after dtb, so that it can be freed
+ * together with platform data.
+ */
+ loff_t xsp_priv_off; /* Offset into this platform data buffer. */
+ size_t xsp_priv_len;
+
+ /*
+ * Populated by parent driver to describe the device tree for
+ * the subdev driver to handle. Should always be last one since it's
+ * of variable length.
+ */
+ char xsp_dtb[sizeof(struct fdt_header)];
+};
+
+/*
+ * this struct define the endpoints belong to the same subdevice
+ */
+struct xrt_subdev_ep_names {
+ const char *ep_name;
+ const char *regmap_name;
+};
+
+struct xrt_subdev_endpoints {
+ struct xrt_subdev_ep_names *xse_names;
+ /* minimum number of endpoints to support the subdevice */
+ u32 xse_min_ep;
+};
+
+/*
+ * It manages a list of xrt_subdevs for root and partition drivers.
+ */
+struct xrt_subdev_pool {
+ struct list_head xpool_dev_list;
+ struct device *xpool_owner;
+ struct mutex xpool_lock;
+ bool xpool_closing;
+};
+
+typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id,
+ struct platform_device *, void *);
+#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1)
+#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2)
+
+/* All subdev drivers should use below common routines to print out msg. */
+#define DEV(pdev) (&(pdev)->dev)
+#define DEV_PDATA(pdev) \
+ ((struct xrt_subdev_platdata *)dev_get_platdata(DEV(pdev)))
+#define DEV_DRVDATA(pdev) \
+ ((struct xrt_subdev_drvdata *) \
+ platform_get_device_id(pdev)->driver_data)
+#define FMT_PRT(prt_fn, pdev, fmt, args...) \
+ prt_fn(DEV(pdev), "%s %s: "fmt, \
+ DEV_PDATA(pdev)->xsp_root_name, __func__, ##args)
+#define xrt_err(pdev, fmt, args...) FMT_PRT(dev_err, pdev, fmt, ##args)
+#define xrt_warn(pdev, fmt, args...) FMT_PRT(dev_warn, pdev, fmt, ##args)
+#define xrt_info(pdev, fmt, args...) FMT_PRT(dev_info, pdev, fmt, ##args)
+#define xrt_dbg(pdev, fmt, args...) FMT_PRT(dev_dbg, pdev, fmt, ##args)
+
+/*
+ * Event notification.
+ */
+enum xrt_events {
+ XRT_EVENT_TEST = 0, // for testing
+ /*
+ * Events related to specific subdev
+ * Callback arg: struct xrt_event_arg_subdev
+ */
+ XRT_EVENT_POST_CREATION,
+ XRT_EVENT_PRE_REMOVAL,
+ /*
+ * Events related to change of the whole board
+ * Callback arg: <none>
+ */
+ XRT_EVENT_PRE_HOT_RESET,
+ XRT_EVENT_POST_HOT_RESET,
+ XRT_EVENT_PRE_GATE_CLOSE,
+ XRT_EVENT_POST_GATE_OPEN,
+ XRT_EVENT_POST_ATTACH,
+ XRT_EVENT_PRE_DETACH,
+};
+
+typedef int (*xrt_event_cb_t)(struct platform_device *pdev,
+ enum xrt_events evt, void *arg);
+typedef void (*xrt_async_broadcast_event_cb_t)(struct platform_device *pdev,
+ enum xrt_events evt, void *arg, bool success);
+
+struct xrt_event_arg_subdev {
+ enum xrt_subdev_id xevt_subdev_id;
+ int xevt_subdev_instance;
+};
+
+/*
+ * Flags in return value from event callback.
+ */
+/* Done with event handling, continue waiting for the next one */
+#define XRT_EVENT_CB_CONTINUE 0x0
+/* Done with event handling, stop waiting for the next one */
+#define XRT_EVENT_CB_STOP 0x1
+/* Error processing event */
+#define XRT_EVENT_CB_ERR 0x2
+
+/*
+ * Subdev pool API for root and partition drivers only.
+ */
+extern void xrt_subdev_pool_init(struct device *dev,
+ struct xrt_subdev_pool *spool);
+extern int xrt_subdev_pool_fini(struct xrt_subdev_pool *spool);
+extern int xrt_subdev_pool_get(struct xrt_subdev_pool *spool,
+ xrt_subdev_match_t match, void *arg, struct device *holder_dev,
+ struct platform_device **pdevp);
+extern int xrt_subdev_pool_put(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, struct device *holder_dev);
+extern int xrt_subdev_pool_add(struct xrt_subdev_pool *spool,
+ enum xrt_subdev_id id, xrt_subdev_parent_cb_t pcb,
+ void *pcb_arg, char *dtb);
+extern int xrt_subdev_pool_del(struct xrt_subdev_pool *spool,
+ enum xrt_subdev_id id, int instance);
+extern int xrt_subdev_pool_event(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, xrt_subdev_match_t match, void *arg,
+ xrt_event_cb_t xevt_cb, enum xrt_events evt);
+extern ssize_t xrt_subdev_pool_get_holders(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, char *buf, size_t len);
+/*
+ * For leaf drivers.
+ */
+extern bool xrt_subdev_has_epname(struct platform_device *pdev, const char *nm);
+extern struct platform_device *xrt_subdev_get_leaf(
+ struct platform_device *pdev, xrt_subdev_match_t cb, void *arg);
+extern struct platform_device *xrt_subdev_get_leaf_by_id(
+ struct platform_device *pdev, enum xrt_subdev_id id, int instance);
+extern struct platform_device *xrt_subdev_get_leaf_by_epname(
+ struct platform_device *pdev, const char *name);
+extern int xrt_subdev_put_leaf(struct platform_device *pdev,
+ struct platform_device *leaf);
+extern int xrt_subdev_create_partition(struct platform_device *pdev,
+ char *dtb);
+extern int xrt_subdev_destroy_partition(struct platform_device *pdev,
+ int instance);
+extern int xrt_subdev_lookup_partition(
+ struct platform_device *pdev, xrt_subdev_match_t cb, void *arg);
+extern int xrt_subdev_wait_for_partition_bringup(struct platform_device *pdev);
+extern void *xrt_subdev_add_event_cb(struct platform_device *pdev,
+ xrt_subdev_match_t match, void *match_arg, xrt_event_cb_t cb);
+extern void xrt_subdev_remove_event_cb(
+ struct platform_device *pdev, void *hdl);
+extern int xrt_subdev_ioctl(struct platform_device *tgt, u32 cmd, void *arg);
+extern int xrt_subdev_broadcast_event(struct platform_device *pdev,
+ enum xrt_events evt);
+extern int xrt_subdev_broadcast_event_async(struct platform_device *pdev,
+ enum xrt_events evt, xrt_async_broadcast_event_cb_t cb, void *arg);
+extern void xrt_subdev_hot_reset(struct platform_device *pdev);
+extern void xrt_subdev_get_barres(struct platform_device *pdev,
+ struct resource **res, uint bar_idx);
+extern void xrt_subdev_get_parent_id(struct platform_device *pdev,
+ unsigned short *vendor, unsigned short *device,
+ unsigned short *subvendor, unsigned short *subdevice);
+extern struct device *xrt_subdev_register_hwmon(struct platform_device *pdev,
+ const char *name, void *drvdata, const struct attribute_group **grps);
+extern void xrt_subdev_unregister_hwmon(struct platform_device *pdev,
+ struct device *hwmon);
+
+extern int xrt_subdev_register_external_driver(enum xrt_subdev_id id,
+ struct platform_driver *drv, struct xrt_subdev_endpoints *eps);
+extern void xrt_subdev_unregister_external_driver(enum xrt_subdev_id id);
+
+/*
+ * Char dev APIs.
+ */
+static inline bool xrt_devnode_enabled(struct xrt_subdev_drvdata *drvdata)
+{
+ return drvdata && drvdata->xsd_file_ops.xsf_ops.open != NULL;
+}
+extern int xrt_devnode_create(struct platform_device *pdev,
+ const char *file_name, const char *inst_name);
+extern int xrt_devnode_destroy(struct platform_device *pdev);
+extern struct platform_device *xrt_devnode_open_excl(struct inode *inode);
+extern struct platform_device *xrt_devnode_open(struct inode *inode);
+extern void xrt_devnode_close(struct inode *inode);
+
+/* Helpers. */
+static inline void xrt_memcpy_fromio(void *buf, void __iomem *iomem, u32 size)
+{
+ int i;
+
+ BUG_ON(size & 0x3);
+ for (i = 0; i < size / 4; i++)
+ ((u32 *)buf)[i] = ioread32((char *)(iomem) + sizeof(u32) * i);
+}
+static inline void xrt_memcpy_toio(void __iomem *iomem, void *buf, u32 size)
+{
+ int i;
+
+ BUG_ON(size & 0x3);
+ for (i = 0; i < size / 4; i++)
+ iowrite32(((u32 *)buf)[i], ((char *)(iomem) + sizeof(u32) * i));
+}
+
+#endif /* _XRT_SUBDEV_H_ */
diff --git a/drivers/fpga/alveo/include/xrt-ucs.h b/drivers/fpga/alveo/include/xrt-ucs.h
new file mode 100644
index 000000000000..a64b15bda865
--- /dev/null
+++ b/drivers/fpga/alveo/include/xrt-ucs.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_UCS_H_
+#define _XRT_UCS_H_
+
+#include "xrt-subdev.h"
+
+/*
+ * UCS driver IOCTL calls.
+ */
+enum xrt_ucs_ioctl_cmd {
+ XRT_UCS_CHECK = 0,
+ XRT_UCS_ENABLE,
+};
+
+#endif /* _XRT_UCS_H_ */
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
` (4 preceding siblings ...)
2020-11-29 0:00 ` [PATCH Xilinx Alveo 6/8] fpga: xrt: header file for platform and parent drivers Sonal Santan
@ 2020-11-29 0:00 ` Sonal Santan
2020-12-01 20:51 ` Moritz Fischer
2020-12-02 3:00 ` Xu Yilun
2020-11-29 0:00 ` [PATCH Xilinx Alveo 8/8] fpga: xrt: Kconfig and Makefile updates for XRT drivers Sonal Santan
` (3 subsequent siblings)
9 siblings, 2 replies; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
From: Sonal Santan <sonal.santan@xilinx.com>
Add management physical function driver core. The driver attaches
to management physical function of Alveo devices. It instantiates
the root driver and one or more partition drivers which in turn
instantiate platform drivers. The instantiation of partition and
platform drivers is completely data driven. The driver integrates
with FPGA manager and provides xclbin download service.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
---
drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c | 194 ++++
drivers/fpga/alveo/mgmt/xmgmt-fmgr.h | 29 +
drivers/fpga/alveo/mgmt/xmgmt-main-impl.h | 36 +
drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c | 930 +++++++++++++++++++
drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c | 190 ++++
drivers/fpga/alveo/mgmt/xmgmt-main.c | 843 +++++++++++++++++
drivers/fpga/alveo/mgmt/xmgmt-root.c | 375 ++++++++
7 files changed, 2597 insertions(+)
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main.c
create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-root.c
diff --git a/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c b/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
new file mode 100644
index 000000000000..d451b5a2c291
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
@@ -0,0 +1,194 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2019-2020 Xilinx, Inc.
+ * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
+ *
+ * Authors: Sonal.Santan@xilinx.com
+ */
+
+#include <linux/cred.h>
+#include <linux/efi.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/platform_device.h>
+#include <linux/module.h>
+#include <linux/vmalloc.h>
+
+#include "xrt-subdev.h"
+#include "xmgmt-fmgr.h"
+#include "xrt-axigate.h"
+#include "xmgmt-main-impl.h"
+
+/*
+ * Container to capture and cache full xclbin as it is passed in blocks by FPGA
+ * Manager. Driver needs access to full xclbin to walk through xclbin sections.
+ * FPGA Manager's .write() backend sends incremental blocks without any
+ * knowledge of xclbin format forcing us to collect the blocks and stitch them
+ * together here.
+ */
+
+struct xfpga_klass {
+ const struct platform_device *pdev;
+ struct axlf *blob;
+ char name[64];
+ size_t count;
+ size_t total_count;
+ struct mutex axlf_lock;
+ int reader_ref;
+ enum fpga_mgr_states state;
+ enum xfpga_sec_level sec_level;
+};
+
+struct key *xfpga_keys;
+
+static int xmgmt_pr_write_init(struct fpga_manager *mgr,
+ struct fpga_image_info *info, const char *buf, size_t count)
+{
+ struct xfpga_klass *obj = mgr->priv;
+ const struct axlf *bin = (const struct axlf *)buf;
+
+ if (count < sizeof(struct axlf)) {
+ obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
+ return -EINVAL;
+ }
+
+ if (count > bin->m_header.m_length) {
+ obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
+ return -EINVAL;
+ }
+
+ /* Free up the previous blob */
+ vfree(obj->blob);
+ obj->blob = vmalloc(bin->m_header.m_length);
+ if (!obj->blob) {
+ obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
+ return -ENOMEM;
+ }
+
+ xrt_info(obj->pdev, "Begin download of xclbin %pUb of length %lld B",
+ &bin->m_header.uuid, bin->m_header.m_length);
+
+ obj->count = 0;
+ obj->total_count = bin->m_header.m_length;
+ obj->state = FPGA_MGR_STATE_WRITE_INIT;
+ return 0;
+}
+
+static int xmgmt_pr_write(struct fpga_manager *mgr,
+ const char *buf, size_t count)
+{
+ struct xfpga_klass *obj = mgr->priv;
+ char *curr = (char *)obj->blob;
+
+ if ((obj->state != FPGA_MGR_STATE_WRITE_INIT) &&
+ (obj->state != FPGA_MGR_STATE_WRITE)) {
+ obj->state = FPGA_MGR_STATE_WRITE_ERR;
+ return -EINVAL;
+ }
+
+ curr += obj->count;
+ obj->count += count;
+
+ /*
+ * The xclbin buffer should not be longer than advertised in the header
+ */
+ if (obj->total_count < obj->count) {
+ obj->state = FPGA_MGR_STATE_WRITE_ERR;
+ return -EINVAL;
+ }
+
+ xrt_info(obj->pdev, "Copying block of %zu B of xclbin", count);
+ memcpy(curr, buf, count);
+ obj->state = FPGA_MGR_STATE_WRITE;
+ return 0;
+}
+
+
+static int xmgmt_pr_write_complete(struct fpga_manager *mgr,
+ struct fpga_image_info *info)
+{
+ int result = 0;
+ struct xfpga_klass *obj = mgr->priv;
+
+ if (obj->state != FPGA_MGR_STATE_WRITE) {
+ obj->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
+ return -EINVAL;
+ }
+
+ /* Check if we got the complete xclbin */
+ if (obj->blob->m_header.m_length != obj->count) {
+ obj->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
+ return -EINVAL;
+ }
+
+ result = xmgmt_ulp_download((void *)obj->pdev, obj->blob);
+
+ obj->state = result ? FPGA_MGR_STATE_WRITE_COMPLETE_ERR :
+ FPGA_MGR_STATE_WRITE_COMPLETE;
+ xrt_info(obj->pdev, "Finish downloading of xclbin %pUb: %d",
+ &obj->blob->m_header.uuid, result);
+ vfree(obj->blob);
+ obj->blob = NULL;
+ obj->count = 0;
+ return result;
+}
+
+static enum fpga_mgr_states xmgmt_pr_state(struct fpga_manager *mgr)
+{
+ struct xfpga_klass *obj = mgr->priv;
+
+ return obj->state;
+}
+
+static const struct fpga_manager_ops xmgmt_pr_ops = {
+ .initial_header_size = sizeof(struct axlf),
+ .write_init = xmgmt_pr_write_init,
+ .write = xmgmt_pr_write,
+ .write_complete = xmgmt_pr_write_complete,
+ .state = xmgmt_pr_state,
+};
+
+
+struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev)
+{
+ struct fpga_manager *fmgr;
+ int ret = 0;
+ struct xfpga_klass *obj = vzalloc(sizeof(struct xfpga_klass));
+
+ xrt_info(pdev, "probing...");
+ if (!obj)
+ return ERR_PTR(-ENOMEM);
+
+ snprintf(obj->name, sizeof(obj->name), "Xilinx Alveo FPGA Manager");
+ obj->state = FPGA_MGR_STATE_UNKNOWN;
+ obj->pdev = pdev;
+ fmgr = fpga_mgr_create(&pdev->dev,
+ obj->name,
+ &xmgmt_pr_ops,
+ obj);
+ if (!fmgr)
+ return ERR_PTR(-ENOMEM);
+
+ obj->sec_level = XFPGA_SEC_NONE;
+ ret = fpga_mgr_register(fmgr);
+ if (ret) {
+ fpga_mgr_free(fmgr);
+ kfree(obj);
+ return ERR_PTR(ret);
+ }
+ mutex_init(&obj->axlf_lock);
+ return fmgr;
+}
+
+int xmgmt_fmgr_remove(struct fpga_manager *fmgr)
+{
+ struct xfpga_klass *obj = fmgr->priv;
+
+ mutex_destroy(&obj->axlf_lock);
+ obj->state = FPGA_MGR_STATE_UNKNOWN;
+ fpga_mgr_unregister(fmgr);
+ vfree(obj->blob);
+ vfree(obj);
+ return 0;
+}
diff --git a/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h b/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
new file mode 100644
index 000000000000..2beba649609f
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2019-2020 Xilinx, Inc.
+ * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
+ *
+ * Authors: Sonal.Santan@xilinx.com
+ */
+
+#ifndef _XMGMT_FMGR_H_
+#define _XMGMT_FMGR_H_
+
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/mutex.h>
+
+#include <linux/xrt/xclbin.h>
+
+enum xfpga_sec_level {
+ XFPGA_SEC_NONE = 0,
+ XFPGA_SEC_DEDICATE,
+ XFPGA_SEC_SYSTEM,
+ XFPGA_SEC_MAX = XFPGA_SEC_SYSTEM,
+};
+
+struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev);
+int xmgmt_fmgr_remove(struct fpga_manager *fmgr);
+
+#endif
diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h b/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
new file mode 100644
index 000000000000..c89024cb8d46
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XMGMT_MAIN_IMPL_H_
+#define _XMGMT_MAIN_IMPL_H_
+
+#include "xrt-subdev.h"
+#include "xmgmt-main.h"
+
+extern struct platform_driver xmgmt_main_driver;
+extern struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[];
+
+extern int xmgmt_ulp_download(struct platform_device *pdev, const void *xclbin);
+extern int bitstream_axlf_mailbox(struct platform_device *pdev,
+ const void *xclbin);
+extern int xmgmt_hot_reset(struct platform_device *pdev);
+
+/* Getting dtb for specified partition. Caller should vfree returned dtb .*/
+extern char *xmgmt_get_dtb(struct platform_device *pdev,
+ enum provider_kind kind);
+extern char *xmgmt_get_vbnv(struct platform_device *pdev);
+extern int xmgmt_get_provider_uuid(struct platform_device *pdev,
+ enum provider_kind kind, uuid_t *uuid);
+
+extern void *xmgmt_pdev2mailbox(struct platform_device *pdev);
+extern void *xmgmt_mailbox_probe(struct platform_device *pdev);
+extern void xmgmt_mailbox_remove(void *handle);
+extern void xmgmt_peer_notify_state(void *handle, bool online);
+
+#endif /* _XMGMT_MAIN_IMPL_H_ */
diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c b/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
new file mode 100644
index 000000000000..b3d82fc3618b
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
@@ -0,0 +1,930 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA MGMT PF entry point driver
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Peer communication via mailbox
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/crc32c.h>
+#include <linux/xrt/mailbox_proto.h>
+#include "xmgmt-main-impl.h"
+#include "xrt-mailbox.h"
+#include "xrt-cmc.h"
+#include "xrt-metadata.h"
+#include "xrt-xclbin.h"
+#include "xrt-clock.h"
+#include "xrt-calib.h"
+#include "xrt-icap.h"
+
+struct xmgmt_mailbox {
+ struct platform_device *pdev;
+ struct platform_device *mailbox;
+ struct mutex lock;
+ void *evt_hdl;
+ char *test_msg;
+ bool peer_in_same_domain;
+};
+
+#define XMGMT_MAILBOX_PRT_REQ(xmbx, send, request, sw_ch) do { \
+ const char *dir = (send) ? ">>>>>" : "<<<<<"; \
+ \
+ if ((request)->req == XCL_MAILBOX_REQ_PEER_DATA) { \
+ struct xcl_mailbox_peer_data *p = \
+ (struct xcl_mailbox_peer_data *)(request)->data;\
+ \
+ xrt_info((xmbx)->pdev, "%s(%s) %s%s", \
+ mailbox_req2name((request)->req), \
+ mailbox_group_kind2name(p->kind), \
+ dir, mailbox_chan2name(sw_ch)); \
+ } else { \
+ xrt_info((xmbx)->pdev, "%s %s%s", \
+ mailbox_req2name((request)->req), \
+ dir, mailbox_chan2name(sw_ch)); \
+ } \
+} while (0)
+#define XMGMT_MAILBOX_PRT_REQ_SEND(xmbx, req, sw_ch) \
+ XMGMT_MAILBOX_PRT_REQ(xmbx, true, req, sw_ch)
+#define XMGMT_MAILBOX_PRT_REQ_RECV(xmbx, req, sw_ch) \
+ XMGMT_MAILBOX_PRT_REQ(xmbx, false, req, sw_ch)
+#define XMGMT_MAILBOX_PRT_RESP(xmbx, resp) \
+ xrt_info((xmbx)->pdev, "respond %ld bytes >>>>>%s", \
+ (resp)->xmip_data_size, mailbox_chan2name((resp)->xmip_sw_ch))
+
+static inline struct xmgmt_mailbox *pdev2mbx(struct platform_device *pdev)
+{
+ return (struct xmgmt_mailbox *)xmgmt_pdev2mailbox(pdev);
+}
+
+static void xmgmt_mailbox_post(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch, void *buf, size_t len)
+{
+ int rc;
+ struct xrt_mailbox_ioctl_post post = {
+ .xmip_req_id = msgid,
+ .xmip_sw_ch = sw_ch,
+ .xmip_data = buf,
+ .xmip_data_size = len
+ };
+
+ BUG_ON(!mutex_is_locked(&xmbx->lock));
+
+ if (!xmbx->mailbox) {
+ xrt_err(xmbx->pdev, "mailbox not available");
+ return;
+ }
+
+ if (msgid == 0) {
+ XMGMT_MAILBOX_PRT_REQ_SEND(xmbx,
+ (struct xcl_mailbox_req *)buf, sw_ch);
+ } else {
+ XMGMT_MAILBOX_PRT_RESP(xmbx, &post);
+ }
+
+ rc = xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_POST, &post);
+ if (rc)
+ xrt_err(xmbx->pdev, "failed to post msg: %d", rc);
+}
+
+static void xmgmt_mailbox_notify(struct xmgmt_mailbox *xmbx, bool sw_ch,
+ struct xcl_mailbox_req *req, size_t len)
+{
+ xmgmt_mailbox_post(xmbx, 0, sw_ch, req, len);
+}
+
+static void xmgmt_mailbox_respond(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch, void *buf, size_t len)
+{
+ mutex_lock(&xmbx->lock);
+ xmgmt_mailbox_post(xmbx, msgid, sw_ch, buf, len);
+ mutex_unlock(&xmbx->lock);
+}
+
+static void xmgmt_mailbox_resp_test_msg(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ char *msg;
+
+ mutex_lock(&xmbx->lock);
+
+ if (xmbx->test_msg == NULL) {
+ mutex_unlock(&xmbx->lock);
+ xrt_err(pdev, "test msg is not set, drop request");
+ return;
+ }
+ msg = xmbx->test_msg;
+ xmbx->test_msg = NULL;
+
+ mutex_unlock(&xmbx->lock);
+
+ xmgmt_mailbox_respond(xmbx, msgid, sw_ch, msg, strlen(msg) + 1);
+ vfree(msg);
+}
+
+static int xmgmt_mailbox_dtb_add_prop(struct platform_device *pdev,
+ char *dst_dtb, const char *ep_name, const char *regmap_name,
+ const char *prop, const void *val, int size)
+{
+ int rc = xrt_md_set_prop(DEV(pdev), dst_dtb, ep_name, regmap_name,
+ prop, val, size);
+
+ if (rc) {
+ xrt_err(pdev, "failed to set %s@(%s, %s): %d",
+ ep_name, regmap_name, prop, rc);
+ }
+ return rc;
+}
+
+static int xmgmt_mailbox_dtb_add_vbnv(struct platform_device *pdev, char *dtb)
+{
+ int rc = 0;
+ char *vbnv = xmgmt_get_vbnv(pdev);
+
+ if (vbnv == NULL) {
+ xrt_err(pdev, "failed to get VBNV");
+ return -ENOENT;
+ }
+ rc = xmgmt_mailbox_dtb_add_prop(pdev, dtb, NULL, NULL,
+ PROP_VBNV, vbnv, strlen(vbnv) + 1);
+ kfree(vbnv);
+ return rc;
+}
+
+static int xmgmt_mailbox_dtb_copy_logic_uuid(struct platform_device *pdev,
+ const char *src_dtb, char *dst_dtb)
+{
+ const void *val;
+ int sz;
+ int rc = xrt_md_get_prop(DEV(pdev), src_dtb, NULL, NULL,
+ PROP_LOGIC_UUID, &val, &sz);
+
+ if (rc) {
+ xrt_err(pdev, "failed to get %s: %d", PROP_LOGIC_UUID, rc);
+ return rc;
+ }
+ return xmgmt_mailbox_dtb_add_prop(pdev, dst_dtb, NULL, NULL,
+ PROP_LOGIC_UUID, val, sz);
+}
+
+static int xmgmt_mailbox_dtb_add_vrom(struct platform_device *pdev,
+ const char *src_dtb, char *dst_dtb)
+{
+ /* For compatibility for legacy xrt driver. */
+ enum FeatureBitMask {
+ UNIFIED_PLATFORM = 0x0000000000000001
+ , XARE_ENBLD = 0x0000000000000002
+ , BOARD_MGMT_ENBLD = 0x0000000000000004
+ , MB_SCHEDULER = 0x0000000000000008
+ , PROM_MASK = 0x0000000000000070
+ , DEBUG_MASK = 0x000000000000FF00
+ , PEER_TO_PEER = 0x0000000000010000
+ , FBM_UUID = 0x0000000000020000
+ , HBM = 0x0000000000040000
+ , CDMA = 0x0000000000080000
+ , QDMA = 0x0000000000100000
+ , RUNTIME_CLK_SCALE = 0x0000000000200000
+ , PASSTHROUGH_VIRTUALIZATION = 0x0000000000400000
+ };
+ struct FeatureRomHeader {
+ unsigned char EntryPointString[4];
+ uint8_t MajorVersion;
+ uint8_t MinorVersion;
+ uint32_t VivadoBuildID;
+ uint32_t IPBuildID;
+ uint64_t TimeSinceEpoch;
+ unsigned char FPGAPartName[64];
+ unsigned char VBNVName[64];
+ uint8_t DDRChannelCount;
+ uint8_t DDRChannelSize;
+ uint64_t DRBaseAddress;
+ uint64_t FeatureBitMap;
+ unsigned char uuid[16];
+ uint8_t HBMCount;
+ uint8_t HBMSize;
+ uint32_t CDMABaseAddress[4];
+ } header = { 0 };
+ char *vbnv = xmgmt_get_vbnv(pdev);
+ int rc;
+
+ *(u32 *)header.EntryPointString = 0x786e6c78;
+
+ if (vbnv)
+ strncpy(header.VBNVName, vbnv, sizeof(header.VBNVName) - 1);
+ kfree(vbnv);
+
+ header.FeatureBitMap = UNIFIED_PLATFORM;
+ rc = xrt_md_get_prop(DEV(pdev), src_dtb,
+ NODE_CMC_FW_MEM, NULL, PROP_IO_OFFSET, NULL, NULL);
+ if (rc == 0)
+ header.FeatureBitMap |= BOARD_MGMT_ENBLD;
+ rc = xrt_md_get_prop(DEV(pdev), src_dtb,
+ NODE_ERT_FW_MEM, NULL, PROP_IO_OFFSET, NULL, NULL);
+ if (rc == 0)
+ header.FeatureBitMap |= MB_SCHEDULER;
+
+ return xmgmt_mailbox_dtb_add_prop(pdev, dst_dtb, NULL, NULL,
+ PROP_VROM, &header, sizeof(header));
+}
+
+static u32 xmgmt_mailbox_dtb_user_pf(struct platform_device *pdev,
+ const char *dtb, const char *epname, const char *regmap)
+{
+ const u32 *pfnump;
+ int rc = xrt_md_get_prop(DEV(pdev), dtb, epname, regmap,
+ PROP_PF_NUM, (const void **)&pfnump, NULL);
+
+ if (rc)
+ return -1;
+ return be32_to_cpu(*pfnump);
+}
+
+static int xmgmt_mailbox_dtb_copy_user_endpoints(struct platform_device *pdev,
+ const char *src, char *dst)
+{
+ int rc = 0;
+ char *epname = NULL, *regmap = NULL;
+ u32 pfnum = xmgmt_mailbox_dtb_user_pf(pdev, src,
+ NODE_MAILBOX_USER, NULL);
+ const u32 level = cpu_to_be32(1);
+ struct device *dev = DEV(pdev);
+
+ if (pfnum == (u32)-1) {
+ xrt_err(pdev, "failed to get user pf num");
+ rc = -EINVAL;
+ }
+
+ for (xrt_md_get_next_endpoint(dev, src, NULL, NULL, &epname, ®map);
+ rc == 0 && epname != NULL;
+ xrt_md_get_next_endpoint(dev, src, epname, regmap,
+ &epname, ®map)) {
+ if (pfnum !=
+ xmgmt_mailbox_dtb_user_pf(pdev, src, epname, regmap))
+ continue;
+ rc = xrt_md_copy_endpoint(dev, dst, src, epname, regmap, NULL);
+ if (rc) {
+ xrt_err(pdev, "failed to copy (%s, %s): %d",
+ epname, regmap, rc);
+ } else {
+ rc = xrt_md_set_prop(dev, dst, epname, regmap,
+ PROP_PARTITION_LEVEL, &level, sizeof(level));
+ if (rc) {
+ xrt_err(pdev,
+ "can't set level for (%s, %s): %d",
+ epname, regmap, rc);
+ }
+ }
+ }
+ return rc;
+}
+
+static char *xmgmt_mailbox_user_dtb(struct platform_device *pdev)
+{
+ /* TODO: add support for PLP. */
+ const char *src = NULL;
+ char *dst = NULL;
+ struct device *dev = DEV(pdev);
+ int rc = xrt_md_create(dev, &dst);
+
+ if (rc || dst == NULL)
+ return NULL;
+
+ rc = xmgmt_mailbox_dtb_add_vbnv(pdev, dst);
+ if (rc)
+ goto fail;
+
+ src = xmgmt_get_dtb(pdev, XMGMT_BLP);
+ if (src == NULL) {
+ xrt_err(pdev, "failed to get BLP dtb");
+ goto fail;
+ }
+
+ rc = xmgmt_mailbox_dtb_copy_logic_uuid(pdev, src, dst);
+ if (rc)
+ goto fail;
+
+ rc = xmgmt_mailbox_dtb_add_vrom(pdev, src, dst);
+ if (rc)
+ goto fail;
+
+ rc = xrt_md_copy_endpoint(dev, dst, src, NODE_PARTITION_INFO,
+ NULL, NODE_PARTITION_INFO_BLP);
+ if (rc)
+ goto fail;
+
+ rc = xrt_md_copy_endpoint(dev, dst, src, NODE_INTERFACES, NULL, NULL);
+ if (rc)
+ goto fail;
+
+ rc = xmgmt_mailbox_dtb_copy_user_endpoints(pdev, src, dst);
+ if (rc)
+ goto fail;
+
+ xrt_md_pack(dev, dst);
+ vfree(src);
+ return dst;
+
+fail:
+ vfree(src);
+ vfree(dst);
+ return NULL;
+}
+
+static void xmgmt_mailbox_resp_subdev(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch, u64 offset, u64 size)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ char *dtb = xmgmt_mailbox_user_dtb(pdev);
+ long dtbsz;
+ struct xcl_subdev *hdr;
+ u64 totalsz;
+
+ if (dtb == NULL)
+ return;
+
+ dtbsz = xrt_md_size(DEV(pdev), dtb);
+ totalsz = dtbsz + sizeof(*hdr) - sizeof(hdr->data);
+ if (offset != 0 || totalsz > size) {
+ /* Only support fetching dtb in one shot. */
+ vfree(dtb);
+ xrt_err(pdev, "need %lldB, user buffer size is %lldB, dropped",
+ totalsz, size);
+ return;
+ }
+
+ hdr = vzalloc(totalsz);
+ if (hdr == NULL) {
+ vfree(dtb);
+ return;
+ }
+
+ hdr->ver = 1;
+ hdr->size = dtbsz;
+ hdr->rtncode = XRT_MSG_SUBDEV_RTN_COMPLETE;
+ (void) memcpy(hdr->data, dtb, dtbsz);
+
+ xmgmt_mailbox_respond(xmbx, msgid, sw_ch, hdr, totalsz);
+
+ vfree(dtb);
+ vfree(hdr);
+}
+
+static void xmgmt_mailbox_resp_sensor(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch, u64 offset, u64 size)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ struct xcl_sensor sensors = { 0 };
+ struct platform_device *cmcpdev = xrt_subdev_get_leaf_by_id(pdev,
+ XRT_SUBDEV_CMC, PLATFORM_DEVID_NONE);
+ int rc;
+
+ if (cmcpdev) {
+ rc = xrt_subdev_ioctl(cmcpdev, XRT_CMC_READ_SENSORS, &sensors);
+ (void) xrt_subdev_put_leaf(pdev, cmcpdev);
+ if (rc)
+ xrt_err(pdev, "can't read sensors: %d", rc);
+ }
+
+ xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &sensors,
+ min((u64)sizeof(sensors), size));
+}
+
+static int xmgmt_mailbox_get_freq(struct xmgmt_mailbox *xmbx,
+ enum CLOCK_TYPE type, u64 *freq, u64 *freq_cnter)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ const char *clkname =
+ clock_type2epname(type) ? clock_type2epname(type) : "UNKNOWN";
+ struct platform_device *clkpdev =
+ xrt_subdev_get_leaf_by_epname(pdev, clkname);
+ int rc;
+ struct xrt_clock_ioctl_get getfreq = { 0 };
+
+ if (clkpdev == NULL) {
+ xrt_info(pdev, "%s clock is not available", clkname);
+ return -ENOENT;
+ }
+
+ rc = xrt_subdev_ioctl(clkpdev, XRT_CLOCK_GET, &getfreq);
+ (void) xrt_subdev_put_leaf(pdev, clkpdev);
+ if (rc) {
+ xrt_err(pdev, "can't get %s clock frequency: %d", clkname, rc);
+ return rc;
+ }
+
+ if (freq)
+ *freq = getfreq.freq;
+ if (freq_cnter)
+ *freq_cnter = getfreq.freq_cnter;
+ return 0;
+}
+
+static int xmgmt_mailbox_get_icap_idcode(struct xmgmt_mailbox *xmbx, u64 *id)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ struct platform_device *icappdev = xrt_subdev_get_leaf_by_id(pdev,
+ XRT_SUBDEV_ICAP, PLATFORM_DEVID_NONE);
+ int rc;
+
+ if (icappdev == NULL) {
+ xrt_err(pdev, "can't find icap");
+ return -ENOENT;
+ }
+
+ rc = xrt_subdev_ioctl(icappdev, XRT_ICAP_IDCODE, id);
+ (void) xrt_subdev_put_leaf(pdev, icappdev);
+ if (rc)
+ xrt_err(pdev, "can't get icap idcode: %d", rc);
+ return rc;
+}
+
+static int xmgmt_mailbox_get_mig_calib(struct xmgmt_mailbox *xmbx, u64 *calib)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ struct platform_device *calibpdev = xrt_subdev_get_leaf_by_id(pdev,
+ XRT_SUBDEV_CALIB, PLATFORM_DEVID_NONE);
+ int rc;
+ enum xrt_calib_results res;
+
+ if (calibpdev == NULL) {
+ xrt_err(pdev, "can't find mig calibration subdev");
+ return -ENOENT;
+ }
+
+ rc = xrt_subdev_ioctl(calibpdev, XRT_CALIB_RESULT, &res);
+ (void) xrt_subdev_put_leaf(pdev, calibpdev);
+ if (rc) {
+ xrt_err(pdev, "can't get mig calibration result: %d", rc);
+ } else {
+ if (res == XRT_CALIB_SUCCEEDED)
+ *calib = 1;
+ else
+ *calib = 0;
+ }
+ return rc;
+}
+
+static void xmgmt_mailbox_resp_icap(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch, u64 offset, u64 size)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ struct xcl_pr_region icap = { 0 };
+
+ (void) xmgmt_mailbox_get_freq(xmbx,
+ CT_DATA, &icap.freq_data, &icap.freq_cntr_data);
+ (void) xmgmt_mailbox_get_freq(xmbx,
+ CT_KERNEL, &icap.freq_kernel, &icap.freq_cntr_kernel);
+ (void) xmgmt_mailbox_get_freq(xmbx,
+ CT_SYSTEM, &icap.freq_system, &icap.freq_cntr_system);
+ (void) xmgmt_mailbox_get_icap_idcode(xmbx, &icap.idcode);
+ (void) xmgmt_mailbox_get_mig_calib(xmbx, &icap.mig_calib);
+ BUG_ON(sizeof(icap.uuid) != sizeof(uuid_t));
+ (void) xmgmt_get_provider_uuid(pdev, XMGMT_ULP, (uuid_t *)&icap.uuid);
+
+ xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &icap,
+ min((u64)sizeof(icap), size));
+}
+
+static void xmgmt_mailbox_resp_bdinfo(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch, u64 offset, u64 size)
+{
+ struct platform_device *pdev = xmbx->pdev;
+ struct xcl_board_info *info = vzalloc(sizeof(*info));
+ struct platform_device *cmcpdev;
+ int rc;
+
+ if (info == NULL)
+ return;
+
+ cmcpdev = xrt_subdev_get_leaf_by_id(pdev,
+ XRT_SUBDEV_CMC, PLATFORM_DEVID_NONE);
+ if (cmcpdev) {
+ rc = xrt_subdev_ioctl(cmcpdev, XRT_CMC_READ_BOARD_INFO, info);
+ (void) xrt_subdev_put_leaf(pdev, cmcpdev);
+ if (rc)
+ xrt_err(pdev, "can't read board info: %d", rc);
+ }
+
+ xmgmt_mailbox_respond(xmbx, msgid, sw_ch, info,
+ min((u64)sizeof(*info), size));
+
+ vfree(info);
+}
+
+static void xmgmt_mailbox_simple_respond(struct xmgmt_mailbox *xmbx,
+ u64 msgid, bool sw_ch, int rc)
+{
+ xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &rc, sizeof(rc));
+}
+
+static void xmgmt_mailbox_resp_peer_data(struct xmgmt_mailbox *xmbx,
+ struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
+{
+ struct xcl_mailbox_peer_data *pdata =
+ (struct xcl_mailbox_peer_data *)req->data;
+
+ if (len < (sizeof(*req) + sizeof(*pdata) - 1)) {
+ xrt_err(xmbx->pdev, "received corrupted %s, dropped",
+ mailbox_req2name(req->req));
+ return;
+ }
+
+ switch (pdata->kind) {
+ case XCL_SENSOR:
+ xmgmt_mailbox_resp_sensor(xmbx, msgid, sw_ch,
+ pdata->offset, pdata->size);
+ break;
+ case XCL_ICAP:
+ xmgmt_mailbox_resp_icap(xmbx, msgid, sw_ch,
+ pdata->offset, pdata->size);
+ break;
+ case XCL_BDINFO:
+ xmgmt_mailbox_resp_bdinfo(xmbx, msgid, sw_ch,
+ pdata->offset, pdata->size);
+ break;
+ case XCL_SUBDEV:
+ xmgmt_mailbox_resp_subdev(xmbx, msgid, sw_ch,
+ pdata->offset, pdata->size);
+ break;
+ case XCL_MIG_ECC:
+ case XCL_FIREWALL:
+ case XCL_DNA: /* TODO **/
+ xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, 0);
+ break;
+ default:
+ xrt_err(xmbx->pdev, "%s(%s) request not handled",
+ mailbox_req2name(req->req),
+ mailbox_group_kind2name(pdata->kind));
+ break;
+ }
+}
+
+static bool xmgmt_mailbox_is_same_domain(struct xmgmt_mailbox *xmbx,
+ struct xcl_mailbox_conn *mb_conn)
+{
+ uint32_t crc_chk;
+ phys_addr_t paddr;
+ struct platform_device *pdev = xmbx->pdev;
+
+ paddr = virt_to_phys((void *)mb_conn->kaddr);
+ if (paddr != (phys_addr_t)mb_conn->paddr) {
+ xrt_info(pdev, "paddrs differ, user 0x%llx, mgmt 0x%llx",
+ mb_conn->paddr, paddr);
+ return false;
+ }
+
+ crc_chk = crc32c_le(~0, (void *)mb_conn->kaddr, PAGE_SIZE);
+ if (crc_chk != mb_conn->crc32) {
+ xrt_info(pdev, "CRCs differ, user 0x%x, mgmt 0x%x",
+ mb_conn->crc32, crc_chk);
+ return false;
+ }
+
+ return true;
+}
+
+static void xmgmt_mailbox_resp_user_probe(struct xmgmt_mailbox *xmbx,
+ struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
+{
+ struct xcl_mailbox_conn_resp *resp = vzalloc(sizeof(*resp));
+ struct xcl_mailbox_conn *conn = (struct xcl_mailbox_conn *)req->data;
+
+ if (resp == NULL)
+ return;
+
+ if (len < (sizeof(*req) + sizeof(*conn) - 1)) {
+ xrt_err(xmbx->pdev, "received corrupted %s, dropped",
+ mailbox_req2name(req->req));
+ vfree(resp);
+ return;
+ }
+
+ resp->conn_flags |= XCL_MB_PEER_READY;
+ if (xmgmt_mailbox_is_same_domain(xmbx, conn)) {
+ xmbx->peer_in_same_domain = true;
+ resp->conn_flags |= XCL_MB_PEER_SAME_DOMAIN;
+ }
+
+ xmgmt_mailbox_respond(xmbx, msgid, sw_ch, resp, sizeof(*resp));
+ vfree(resp);
+}
+
+static void xmgmt_mailbox_resp_hot_reset(struct xmgmt_mailbox *xmbx,
+ struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
+{
+ int ret;
+ struct platform_device *pdev = xmbx->pdev;
+
+ xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, 0);
+
+ ret = xmgmt_hot_reset(pdev);
+ if (ret)
+ xrt_err(pdev, "failed to hot reset: %d", ret);
+ else
+ xmgmt_peer_notify_state(xmbx, true);
+}
+
+static void xmgmt_mailbox_resp_load_xclbin(struct xmgmt_mailbox *xmbx,
+ struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
+{
+ struct xcl_mailbox_bitstream_kaddr *kaddr =
+ (struct xcl_mailbox_bitstream_kaddr *)req->data;
+ void *xclbin = (void *)(uintptr_t)kaddr->addr;
+ int ret = bitstream_axlf_mailbox(xmbx->pdev, xclbin);
+
+ xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, ret);
+}
+
+static void xmgmt_mailbox_listener(void *arg, void *data, size_t len,
+ u64 msgid, int err, bool sw_ch)
+{
+ struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)arg;
+ struct platform_device *pdev = xmbx->pdev;
+ struct xcl_mailbox_req *req = (struct xcl_mailbox_req *)data;
+
+ if (err) {
+ xrt_err(pdev, "failed to receive request: %d", err);
+ return;
+ }
+ if (len < sizeof(*req)) {
+ xrt_err(pdev, "received corrupted request");
+ return;
+ }
+
+ XMGMT_MAILBOX_PRT_REQ_RECV(xmbx, req, sw_ch);
+ switch (req->req) {
+ case XCL_MAILBOX_REQ_TEST_READ:
+ xmgmt_mailbox_resp_test_msg(xmbx, msgid, sw_ch);
+ break;
+ case XCL_MAILBOX_REQ_PEER_DATA:
+ xmgmt_mailbox_resp_peer_data(xmbx, req, len, msgid, sw_ch);
+ break;
+ case XCL_MAILBOX_REQ_READ_P2P_BAR_ADDR: /* TODO */
+ xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, -ENOTSUPP);
+ break;
+ case XCL_MAILBOX_REQ_USER_PROBE:
+ xmgmt_mailbox_resp_user_probe(xmbx, req, len, msgid, sw_ch);
+ break;
+ case XCL_MAILBOX_REQ_HOT_RESET:
+ xmgmt_mailbox_resp_hot_reset(xmbx, req, len, msgid, sw_ch);
+ break;
+ case XCL_MAILBOX_REQ_LOAD_XCLBIN_KADDR:
+ if (xmbx->peer_in_same_domain) {
+ xmgmt_mailbox_resp_load_xclbin(xmbx,
+ req, len, msgid, sw_ch);
+ } else {
+ xrt_err(pdev, "%s not handled, not in same domain",
+ mailbox_req2name(req->req));
+ }
+ break;
+ default:
+ xrt_err(pdev, "%s(%d) request not handled",
+ mailbox_req2name(req->req), req->req);
+ break;
+ }
+}
+
+static void xmgmt_mailbox_reg_listener(struct xmgmt_mailbox *xmbx)
+{
+ struct xrt_mailbox_ioctl_listen listen = {
+ xmgmt_mailbox_listener, xmbx };
+
+ BUG_ON(!mutex_is_locked(&xmbx->lock));
+ if (!xmbx->mailbox)
+ return;
+ (void) xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_LISTEN, &listen);
+}
+
+static void xmgmt_mailbox_unreg_listener(struct xmgmt_mailbox *xmbx)
+{
+ struct xrt_mailbox_ioctl_listen listen = { 0 };
+
+ BUG_ON(!mutex_is_locked(&xmbx->lock));
+ BUG_ON(!xmbx->mailbox);
+ (void) xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_LISTEN, &listen);
+}
+
+static bool xmgmt_mailbox_leaf_match(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ return (id == XRT_SUBDEV_MAILBOX);
+}
+
+static int xmgmt_mailbox_event_cb(struct platform_device *pdev,
+ enum xrt_events evt, void *arg)
+{
+ struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
+ struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev *)arg;
+
+ switch (evt) {
+ case XRT_EVENT_POST_CREATION:
+ BUG_ON(esd->xevt_subdev_id != XRT_SUBDEV_MAILBOX);
+ BUG_ON(xmbx->mailbox);
+ mutex_lock(&xmbx->lock);
+ xmbx->mailbox = xrt_subdev_get_leaf_by_id(pdev,
+ XRT_SUBDEV_MAILBOX, PLATFORM_DEVID_NONE);
+ xmgmt_mailbox_reg_listener(xmbx);
+ mutex_unlock(&xmbx->lock);
+ break;
+ case XRT_EVENT_PRE_REMOVAL:
+ BUG_ON(esd->xevt_subdev_id != XRT_SUBDEV_MAILBOX);
+ BUG_ON(!xmbx->mailbox);
+ mutex_lock(&xmbx->lock);
+ xmgmt_mailbox_unreg_listener(xmbx);
+ (void) xrt_subdev_put_leaf(pdev, xmbx->mailbox);
+ xmbx->mailbox = NULL;
+ mutex_unlock(&xmbx->lock);
+ break;
+ default:
+ break;
+ }
+
+ return XRT_EVENT_CB_CONTINUE;
+}
+
+static ssize_t xmgmt_mailbox_user_dtb_show(struct file *filp,
+ struct kobject *kobj, struct bin_attribute *attr,
+ char *buf, loff_t off, size_t count)
+{
+ struct device *dev = kobj_to_dev(kobj);
+ struct platform_device *pdev = to_platform_device(dev);
+ char *blob = NULL;
+ long size;
+ ssize_t ret = 0;
+
+ blob = xmgmt_mailbox_user_dtb(pdev);
+ if (!blob) {
+ ret = -ENOENT;
+ goto failed;
+ }
+
+ size = xrt_md_size(dev, blob);
+ if (size <= 0) {
+ ret = -EINVAL;
+ goto failed;
+ }
+
+ if (off >= size)
+ goto failed;
+ if (off + count > size)
+ count = size - off;
+ memcpy(buf, blob + off, count);
+
+ ret = count;
+failed:
+ vfree(blob);
+ return ret;
+}
+
+static struct bin_attribute meta_data_attr = {
+ .attr = {
+ .name = "metadata_for_user",
+ .mode = 0400
+ },
+ .read = xmgmt_mailbox_user_dtb_show,
+ .size = 0
+};
+
+static struct bin_attribute *xmgmt_mailbox_bin_attrs[] = {
+ &meta_data_attr,
+ NULL,
+};
+
+int xmgmt_mailbox_get_test_msg(struct xmgmt_mailbox *xmbx, bool sw_ch,
+ char *buf, size_t *len)
+{
+ int rc;
+ struct platform_device *pdev = xmbx->pdev;
+ struct xcl_mailbox_req req = { 0, XCL_MAILBOX_REQ_TEST_READ, };
+ struct xrt_mailbox_ioctl_request leaf_req = {
+ .xmir_sw_ch = sw_ch,
+ .xmir_resp_ttl = 1,
+ .xmir_req = &req,
+ .xmir_req_size = sizeof(req),
+ .xmir_resp = buf,
+ .xmir_resp_size = *len
+ };
+
+ mutex_lock(&xmbx->lock);
+ if (xmbx->mailbox) {
+ XMGMT_MAILBOX_PRT_REQ_SEND(xmbx, &req, leaf_req.xmir_sw_ch);
+ /*
+ * mgmt should never send request to peer. it should send
+ * either notification or response. here is the only exception
+ * for debugging purpose.
+ */
+ rc = xrt_subdev_ioctl(xmbx->mailbox,
+ XRT_MAILBOX_REQUEST, &leaf_req);
+ } else {
+ rc = -ENODEV;
+ xrt_err(pdev, "mailbox not available");
+ }
+ mutex_unlock(&xmbx->lock);
+
+ if (rc == 0)
+ *len = leaf_req.xmir_resp_size;
+ return rc;
+}
+
+int xmgmt_mailbox_set_test_msg(struct xmgmt_mailbox *xmbx,
+ char *buf, size_t len)
+{
+ mutex_lock(&xmbx->lock);
+
+ if (xmbx->test_msg)
+ vfree(xmbx->test_msg);
+ xmbx->test_msg = vmalloc(len);
+ if (xmbx->test_msg == NULL) {
+ mutex_unlock(&xmbx->lock);
+ return -ENOMEM;
+ }
+ (void) memcpy(xmbx->test_msg, buf, len);
+
+ mutex_unlock(&xmbx->lock);
+ return 0;
+}
+
+static ssize_t peer_msg_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ size_t len = 4096;
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
+ int ret = xmgmt_mailbox_get_test_msg(xmbx, false, buf, &len);
+
+ return ret == 0 ? len : ret;
+}
+static ssize_t peer_msg_store(struct device *dev,
+ struct device_attribute *da, const char *buf, size_t count)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
+ int ret = xmgmt_mailbox_set_test_msg(xmbx, (char *)buf, count);
+
+ return ret == 0 ? count : ret;
+}
+/* Message test i/f. */
+static DEVICE_ATTR_RW(peer_msg);
+
+static struct attribute *xmgmt_mailbox_attrs[] = {
+ &dev_attr_peer_msg.attr,
+ NULL,
+};
+
+static const struct attribute_group xmgmt_mailbox_attrgroup = {
+ .bin_attrs = xmgmt_mailbox_bin_attrs,
+ .attrs = xmgmt_mailbox_attrs,
+};
+
+void *xmgmt_mailbox_probe(struct platform_device *pdev)
+{
+ struct xmgmt_mailbox *xmbx =
+ devm_kzalloc(DEV(pdev), sizeof(*xmbx), GFP_KERNEL);
+
+ if (!xmbx)
+ return NULL;
+ xmbx->pdev = pdev;
+ mutex_init(&xmbx->lock);
+
+ xmbx->evt_hdl = xrt_subdev_add_event_cb(pdev,
+ xmgmt_mailbox_leaf_match, NULL, xmgmt_mailbox_event_cb);
+ (void) sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_mailbox_attrgroup);
+ return xmbx;
+}
+
+void xmgmt_mailbox_remove(void *handle)
+{
+ struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)handle;
+ struct platform_device *pdev = xmbx->pdev;
+
+ (void) sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_mailbox_attrgroup);
+ if (xmbx->evt_hdl)
+ (void) xrt_subdev_remove_event_cb(pdev, xmbx->evt_hdl);
+ if (xmbx->mailbox)
+ (void) xrt_subdev_put_leaf(pdev, xmbx->mailbox);
+ if (xmbx->test_msg)
+ vfree(xmbx->test_msg);
+}
+
+void xmgmt_peer_notify_state(void *handle, bool online)
+{
+ struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)handle;
+ struct xcl_mailbox_peer_state *st;
+ struct xcl_mailbox_req *req;
+ size_t reqlen = sizeof(*req) + sizeof(*st) - 1;
+
+ req = vzalloc(reqlen);
+ if (req == NULL)
+ return;
+
+ req->req = XCL_MAILBOX_REQ_MGMT_STATE;
+ st = (struct xcl_mailbox_peer_state *)req->data;
+ st->state_flags = online ? XCL_MB_STATE_ONLINE : XCL_MB_STATE_OFFLINE;
+ mutex_lock(&xmbx->lock);
+ xmgmt_mailbox_notify(xmbx, false, req, reqlen);
+ mutex_unlock(&xmbx->lock);
+}
diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c b/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
new file mode 100644
index 000000000000..042d86fcef41
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
@@ -0,0 +1,190 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA MGMT PF entry point driver
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * xclbin download
+ *
+ * Authors:
+ * Lizhi Hou <lizhi.hou@xilinx.com>
+ */
+
+#include <linux/firmware.h>
+#include <linux/uaccess.h>
+#include "xrt-xclbin.h"
+#include "xrt-metadata.h"
+#include "xrt-subdev.h"
+#include "xrt-gpio.h"
+#include "xmgmt-main.h"
+#include "xrt-icap.h"
+#include "xrt-axigate.h"
+
+static int xmgmt_download_bitstream(struct platform_device *pdev,
+ const void *xclbin)
+{
+ struct platform_device *icap_leaf = NULL;
+ struct XHwIcap_Bit_Header bit_header = { 0 };
+ struct xrt_icap_ioctl_wr arg;
+ char *bitstream = NULL;
+ int ret;
+
+ ret = xrt_xclbin_get_section(xclbin, BITSTREAM, (void **)&bitstream,
+ NULL);
+ if (ret || !bitstream) {
+ xrt_err(pdev, "bitstream not found");
+ return -ENOENT;
+ }
+ ret = xrt_xclbin_parse_header(bitstream,
+ DMA_HWICAP_BITFILE_BUFFER_SIZE, &bit_header);
+ if (ret) {
+ ret = -EINVAL;
+ xrt_err(pdev, "invalid bitstream header");
+ goto done;
+ }
+ icap_leaf = xrt_subdev_get_leaf_by_id(pdev, XRT_SUBDEV_ICAP,
+ PLATFORM_DEVID_NONE);
+ if (!icap_leaf) {
+ ret = -ENODEV;
+ xrt_err(pdev, "icap does not exist");
+ goto done;
+ }
+ arg.xiiw_bit_data = bitstream + bit_header.HeaderLength;
+ arg.xiiw_data_len = bit_header.BitstreamLength;
+ ret = xrt_subdev_ioctl(icap_leaf, XRT_ICAP_WRITE, &arg);
+ if (ret)
+ xrt_err(pdev, "write bitstream failed, ret = %d", ret);
+
+done:
+ if (icap_leaf)
+ xrt_subdev_put_leaf(pdev, icap_leaf);
+ vfree(bitstream);
+
+ return ret;
+}
+
+static bool match_shell(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ const char *ulp_gate;
+ int ret;
+
+ if (!pdata || xrt_md_size(&pdev->dev, pdata->xsp_dtb) <= 0)
+ return false;
+
+ ret = xrt_md_get_epname_pointer(&pdev->dev, pdata->xsp_dtb,
+ NODE_GATE_ULP, NULL, &ulp_gate);
+ if (ret)
+ return false;
+
+ ret = xrt_md_check_uuids(&pdev->dev, pdata->xsp_dtb, arg);
+ if (ret)
+ return false;
+
+ return true;
+}
+
+static bool match_ulp(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ const char *ulp_gate;
+ int ret;
+
+ if (!pdata || xrt_md_size(&pdev->dev, pdata->xsp_dtb) <= 0)
+ return false;
+
+ ret = xrt_md_check_uuids(&pdev->dev, pdata->xsp_dtb, arg);
+ if (ret)
+ return false;
+
+ ret = xrt_md_get_epname_pointer(&pdev->dev, pdata->xsp_dtb,
+ NODE_GATE_ULP, NULL, &ulp_gate);
+ if (!ret)
+ return false;
+
+ return true;
+}
+
+int xmgmt_ulp_download(struct platform_device *pdev, const void *xclbin)
+{
+ struct platform_device *axigate_leaf;
+ char *dtb = NULL;
+ int ret = 0, part_inst;
+
+ ret = xrt_xclbin_get_metadata(DEV(pdev), xclbin, &dtb);
+ if (ret) {
+ xrt_err(pdev, "can not get partition metadata, ret %d", ret);
+ goto failed;
+ }
+
+ part_inst = xrt_subdev_lookup_partition(pdev, match_shell, dtb);
+ if (part_inst < 0) {
+ xrt_err(pdev, "not found matching plp.");
+ ret = -ENODEV;
+ goto failed;
+ }
+
+ /*
+ * Find ulp partition with interface uuid from incoming xclbin, which
+ * is verified before with matching plp partition.
+ */
+ part_inst = xrt_subdev_lookup_partition(pdev, match_ulp, dtb);
+ if (part_inst >= 0) {
+ ret = xrt_subdev_destroy_partition(pdev, part_inst);
+ if (ret) {
+ xrt_err(pdev, "failed to destroy existing ulp, %d",
+ ret);
+ goto failed;
+ }
+ }
+
+ axigate_leaf = xrt_subdev_get_leaf_by_epname(pdev, NODE_GATE_ULP);
+
+ /* gate may not be exist for 0rp */
+ if (axigate_leaf) {
+ ret = xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE,
+ NULL);
+ if (ret) {
+ xrt_err(pdev, "can not freeze gate %s, %d",
+ NODE_GATE_ULP, ret);
+ xrt_subdev_put_leaf(pdev, axigate_leaf);
+ goto failed;
+ }
+ }
+ ret = xmgmt_download_bitstream(pdev, xclbin);
+ if (axigate_leaf) {
+ xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
+
+ /* Do we really need this extra toggling gate before setting
+ * clocks?
+ * xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE, NULL);
+ * xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
+ */
+
+ xrt_subdev_put_leaf(pdev, axigate_leaf);
+ }
+ if (ret) {
+ xrt_err(pdev, "bitstream download failed, ret %d", ret);
+ goto failed;
+ }
+ ret = xrt_subdev_create_partition(pdev, dtb);
+ if (ret < 0) {
+ xrt_err(pdev, "failed creating partition, ret %d", ret);
+ goto failed;
+ }
+
+ ret = xrt_subdev_wait_for_partition_bringup(pdev);
+ if (ret)
+ xrt_err(pdev, "partiton bringup failed, ret %d", ret);
+
+ /*
+ * TODO: needs to check individual subdevs to see if there
+ * is any error, such as clock setting, memory bank calibration.
+ */
+
+failed:
+ vfree(dtb);
+ return ret;
+}
diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main.c b/drivers/fpga/alveo/mgmt/xmgmt-main.c
new file mode 100644
index 000000000000..23e68e3a4ae1
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/xmgmt-main.c
@@ -0,0 +1,843 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA MGMT PF entry point driver
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Sonal Santan <sonals@xilinx.com>
+ */
+
+#include <linux/firmware.h>
+#include <linux/uaccess.h>
+#include "xrt-xclbin.h"
+#include "xrt-metadata.h"
+#include "xrt-flash.h"
+#include "xrt-subdev.h"
+#include <linux/xrt/flash_xrt_data.h>
+#include <linux/xrt/xmgmt-ioctl.h>
+#include "xrt-gpio.h"
+#include "xmgmt-main.h"
+#include "xmgmt-fmgr.h"
+#include "xrt-icap.h"
+#include "xrt-axigate.h"
+#include "xmgmt-main-impl.h"
+
+#define XMGMT_MAIN "xmgmt_main"
+
+struct xmgmt_main {
+ struct platform_device *pdev;
+ void *evt_hdl;
+ char *firmware_blp;
+ char *firmware_plp;
+ char *firmware_ulp;
+ bool flash_ready;
+ bool gpio_ready;
+ struct fpga_manager *fmgr;
+ void *mailbox_hdl;
+ struct mutex busy_mutex;
+
+ uuid_t *blp_intf_uuids;
+ u32 blp_intf_uuid_num;
+};
+
+char *xmgmt_get_vbnv(struct platform_device *pdev)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ const char *vbnv;
+ char *ret;
+ int i;
+
+ if (xmm->firmware_plp) {
+ vbnv = ((struct axlf *)xmm->firmware_plp)->
+ m_header.m_platformVBNV;
+ } else if (xmm->firmware_blp) {
+ vbnv = ((struct axlf *)xmm->firmware_blp)->
+ m_header.m_platformVBNV;
+ } else {
+ return NULL;
+ }
+
+ ret = kstrdup(vbnv, GFP_KERNEL);
+ for (i = 0; i < strlen(ret); i++) {
+ if (ret[i] == ':' || ret[i] == '.')
+ ret[i] = '_';
+ }
+ return ret;
+}
+
+static bool xmgmt_main_leaf_match(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ if (id == XRT_SUBDEV_GPIO)
+ return xrt_subdev_has_epname(pdev, arg);
+ else if (id == XRT_SUBDEV_QSPI)
+ return true;
+
+ return false;
+}
+
+static int get_dev_uuid(struct platform_device *pdev, char *uuidstr, size_t len)
+{
+ char uuid[16];
+ struct platform_device *gpio_leaf;
+ struct xrt_gpio_ioctl_rw gpio_arg = { 0 };
+ int err, i, count;
+
+ gpio_leaf = xrt_subdev_get_leaf_by_epname(pdev, NODE_BLP_ROM);
+ if (!gpio_leaf) {
+ xrt_err(pdev, "can not get %s", NODE_BLP_ROM);
+ return -EINVAL;
+ }
+
+ gpio_arg.xgir_id = XRT_GPIO_ROM_UUID;
+ gpio_arg.xgir_buf = uuid;
+ gpio_arg.xgir_len = sizeof(uuid);
+ gpio_arg.xgir_offset = 0;
+ err = xrt_subdev_ioctl(gpio_leaf, XRT_GPIO_READ, &gpio_arg);
+ xrt_subdev_put_leaf(pdev, gpio_leaf);
+ if (err) {
+ xrt_err(pdev, "can not get uuid: %d", err);
+ return err;
+ }
+
+ for (count = 0, i = sizeof(uuid) - sizeof(u32);
+ i >= 0 && len > count; i -= sizeof(u32)) {
+ count += snprintf(uuidstr + count, len - count,
+ "%08x", *(u32 *)&uuid[i]);
+ }
+ return 0;
+}
+
+int xmgmt_hot_reset(struct platform_device *pdev)
+{
+ int ret = xrt_subdev_broadcast_event(pdev, XRT_EVENT_PRE_HOT_RESET);
+
+ if (ret) {
+ xrt_err(pdev, "offline failed, hot reset is canceled");
+ return ret;
+ }
+
+ (void) xrt_subdev_hot_reset(pdev);
+ xrt_subdev_broadcast_event(pdev, XRT_EVENT_POST_HOT_RESET);
+ return 0;
+}
+
+static ssize_t reset_store(struct device *dev,
+ struct device_attribute *da, const char *buf, size_t count)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+
+ (void) xmgmt_hot_reset(pdev);
+ return count;
+}
+static DEVICE_ATTR_WO(reset);
+
+static ssize_t VBNV_show(struct device *dev,
+ struct device_attribute *da, char *buf)
+{
+ ssize_t ret;
+ char *vbnv;
+ struct platform_device *pdev = to_platform_device(dev);
+
+ vbnv = xmgmt_get_vbnv(pdev);
+ ret = sprintf(buf, "%s\n", vbnv);
+ kfree(vbnv);
+ return ret;
+}
+static DEVICE_ATTR_RO(VBNV);
+
+static ssize_t logic_uuids_show(struct device *dev,
+ struct device_attribute *da, char *buf)
+{
+ ssize_t ret;
+ char uuid[80];
+ struct platform_device *pdev = to_platform_device(dev);
+
+ /*
+ * Getting UUID pointed to by VSEC,
+ * should be the same as logic UUID of BLP.
+ * TODO: add PLP logic UUID
+ */
+ ret = get_dev_uuid(pdev, uuid, sizeof(uuid));
+ if (ret)
+ return ret;
+ ret = sprintf(buf, "%s\n", uuid);
+ return ret;
+}
+static DEVICE_ATTR_RO(logic_uuids);
+
+static inline void uuid2str(const uuid_t *uuid, char *uuidstr, size_t len)
+{
+ int i, p;
+ u8 *u = (u8 *)uuid;
+
+ BUG_ON(sizeof(uuid_t) * 2 + 1 > len);
+ for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
+ (void) snprintf(&uuidstr[p*2], 3, "%02x", u[i]);
+}
+
+static ssize_t interface_uuids_show(struct device *dev,
+ struct device_attribute *da, char *buf)
+{
+ ssize_t ret = 0;
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ u32 i;
+
+ /*
+ * TODO: add PLP interface UUID
+ */
+ for (i = 0; i < xmm->blp_intf_uuid_num; i++) {
+ char uuidstr[80];
+
+ uuid2str(&xmm->blp_intf_uuids[i], uuidstr, sizeof(uuidstr));
+ ret += sprintf(buf + ret, "%s\n", uuidstr);
+ }
+ return ret;
+}
+static DEVICE_ATTR_RO(interface_uuids);
+
+static struct attribute *xmgmt_main_attrs[] = {
+ &dev_attr_reset.attr,
+ &dev_attr_VBNV.attr,
+ &dev_attr_logic_uuids.attr,
+ &dev_attr_interface_uuids.attr,
+ NULL,
+};
+
+static ssize_t ulp_image_write(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *attr, char *buffer, loff_t off, size_t count)
+{
+ struct xmgmt_main *xmm =
+ dev_get_drvdata(container_of(kobj, struct device, kobj));
+ struct axlf *xclbin;
+ ulong len;
+
+ if (off == 0) {
+ if (count < sizeof(*xclbin)) {
+ xrt_err(xmm->pdev, "count is too small %ld", count);
+ return -EINVAL;
+ }
+
+ if (xmm->firmware_ulp) {
+ vfree(xmm->firmware_ulp);
+ xmm->firmware_ulp = NULL;
+ }
+ xclbin = (struct axlf *)buffer;
+ xmm->firmware_ulp = vmalloc(xclbin->m_header.m_length);
+ if (!xmm->firmware_ulp)
+ return -ENOMEM;
+ } else
+ xclbin = (struct axlf *)xmm->firmware_ulp;
+
+ len = xclbin->m_header.m_length;
+ if (off + count >= len && off < len) {
+ memcpy(xmm->firmware_ulp + off, buffer, len - off);
+ xmgmt_ulp_download(xmm->pdev, xmm->firmware_ulp);
+ } else if (off + count < len) {
+ memcpy(xmm->firmware_ulp + off, buffer, count);
+ }
+
+ return count;
+}
+
+static struct bin_attribute ulp_image_attr = {
+ .attr = {
+ .name = "ulp_image",
+ .mode = 0200
+ },
+ .write = ulp_image_write,
+ .size = 0
+};
+
+static struct bin_attribute *xmgmt_main_bin_attrs[] = {
+ &ulp_image_attr,
+ NULL,
+};
+
+static const struct attribute_group xmgmt_main_attrgroup = {
+ .attrs = xmgmt_main_attrs,
+ .bin_attrs = xmgmt_main_bin_attrs,
+};
+
+static int load_firmware_from_flash(struct platform_device *pdev,
+ char **fw_buf, size_t *len)
+{
+ struct platform_device *flash_leaf = NULL;
+ struct flash_data_header header = { 0 };
+ const size_t magiclen = sizeof(header.fdh_id_begin.fdi_magic);
+ size_t flash_size = 0;
+ int ret = 0;
+ char *buf = NULL;
+ struct flash_data_ident id = { 0 };
+ struct xrt_flash_ioctl_read frd = { 0 };
+
+ xrt_info(pdev, "try loading fw from flash");
+
+ flash_leaf = xrt_subdev_get_leaf_by_id(pdev, XRT_SUBDEV_QSPI,
+ PLATFORM_DEVID_NONE);
+ if (flash_leaf == NULL) {
+ xrt_err(pdev, "failed to hold flash leaf");
+ return -ENODEV;
+ }
+
+ (void) xrt_subdev_ioctl(flash_leaf, XRT_FLASH_GET_SIZE, &flash_size);
+ if (flash_size == 0) {
+ xrt_err(pdev, "failed to get flash size");
+ ret = -EINVAL;
+ goto done;
+ }
+
+ frd.xfir_buf = (char *)&header;
+ frd.xfir_size = sizeof(header);
+ frd.xfir_offset = flash_size - sizeof(header);
+ ret = xrt_subdev_ioctl(flash_leaf, XRT_FLASH_READ, &frd);
+ if (ret) {
+ xrt_err(pdev, "failed to read header from flash: %d", ret);
+ goto done;
+ }
+
+ /* Pick the end ident since header is aligned in the end of flash. */
+ id = header.fdh_id_end;
+ if (strncmp(id.fdi_magic, XRT_DATA_MAGIC, magiclen)) {
+ char tmp[sizeof(id.fdi_magic) + 1] = { 0 };
+
+ memcpy(tmp, id.fdi_magic, magiclen);
+ xrt_info(pdev, "ignore meta data, bad magic: %s", tmp);
+ ret = -ENOENT;
+ goto done;
+ }
+ if (id.fdi_version != 0) {
+ xrt_info(pdev, "flash meta data version is not supported: %d",
+ id.fdi_version);
+ ret = -EOPNOTSUPP;
+ goto done;
+ }
+
+ buf = vmalloc(header.fdh_data_len);
+ if (buf == NULL) {
+ ret = -ENOMEM;
+ goto done;
+ }
+
+ frd.xfir_buf = buf;
+ frd.xfir_size = header.fdh_data_len;
+ frd.xfir_offset = header.fdh_data_offset;
+ ret = xrt_subdev_ioctl(flash_leaf, XRT_FLASH_READ, &frd);
+ if (ret) {
+ xrt_err(pdev, "failed to read meta data from flash: %d", ret);
+ goto done;
+ } else if (flash_xrt_data_get_parity32(buf, header.fdh_data_len) ^
+ header.fdh_data_parity) {
+ xrt_err(pdev, "meta data is corrupted");
+ ret = -EINVAL;
+ goto done;
+ }
+
+ xrt_info(pdev, "found meta data of %d bytes @0x%x",
+ header.fdh_data_len, header.fdh_data_offset);
+ *fw_buf = buf;
+ *len = header.fdh_data_len;
+
+done:
+ (void) xrt_subdev_put_leaf(pdev, flash_leaf);
+ return ret;
+}
+
+static int load_firmware_from_disk(struct platform_device *pdev, char **fw_buf,
+ size_t *len)
+{
+ char uuid[80];
+ int err = 0;
+ char fw_name[256];
+ const struct firmware *fw;
+
+ err = get_dev_uuid(pdev, uuid, sizeof(uuid));
+ if (err)
+ return err;
+
+ (void) snprintf(fw_name,
+ sizeof(fw_name), "xilinx/%s/partition.xsabin", uuid);
+ xrt_info(pdev, "try loading fw: %s", fw_name);
+
+ err = request_firmware(&fw, fw_name, DEV(pdev));
+ if (err)
+ return err;
+
+ *fw_buf = vmalloc(fw->size);
+ *len = fw->size;
+ if (*fw_buf != NULL)
+ memcpy(*fw_buf, fw->data, fw->size);
+ else
+ err = -ENOMEM;
+
+ release_firmware(fw);
+ return 0;
+}
+
+static const char *xmgmt_get_axlf_firmware(struct xmgmt_main *xmm,
+ enum provider_kind kind)
+{
+ switch (kind) {
+ case XMGMT_BLP:
+ return xmm->firmware_blp;
+ case XMGMT_PLP:
+ return xmm->firmware_plp;
+ case XMGMT_ULP:
+ return xmm->firmware_ulp;
+ default:
+ xrt_err(xmm->pdev, "unknown axlf kind: %d", kind);
+ return NULL;
+ }
+}
+
+char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ char *dtb = NULL;
+ const char *provider = xmgmt_get_axlf_firmware(xmm, kind);
+ int rc;
+
+ if (provider == NULL)
+ return dtb;
+
+ rc = xrt_xclbin_get_metadata(DEV(pdev), provider, &dtb);
+ if (rc)
+ xrt_err(pdev, "failed to find dtb: %d", rc);
+ return dtb;
+}
+
+static const char *get_uuid_from_firmware(struct platform_device *pdev,
+ const char *axlf)
+{
+ const void *uuid = NULL;
+ const void *uuiddup = NULL;
+ void *dtb = NULL;
+ int rc;
+
+ rc = xrt_xclbin_get_section(axlf, PARTITION_METADATA, &dtb, NULL);
+ if (rc)
+ return NULL;
+
+ rc = xrt_md_get_prop(DEV(pdev), dtb, NULL, NULL,
+ PROP_LOGIC_UUID, &uuid, NULL);
+ if (!rc)
+ uuiddup = kstrdup(uuid, GFP_KERNEL);
+ vfree(dtb);
+ return uuiddup;
+}
+
+static bool is_valid_firmware(struct platform_device *pdev,
+ char *fw_buf, size_t fw_len)
+{
+ struct axlf *axlf = (struct axlf *)fw_buf;
+ size_t axlflen = axlf->m_header.m_length;
+ const char *fw_uuid;
+ char dev_uuid[80];
+ int err;
+
+ err = get_dev_uuid(pdev, dev_uuid, sizeof(dev_uuid));
+ if (err)
+ return false;
+
+ if (memcmp(fw_buf, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)) != 0) {
+ xrt_err(pdev, "unknown fw format");
+ return false;
+ }
+
+ if (axlflen > fw_len) {
+ xrt_err(pdev, "truncated fw, length: %ld, expect: %ld",
+ fw_len, axlflen);
+ return false;
+ }
+
+ fw_uuid = get_uuid_from_firmware(pdev, fw_buf);
+ if (fw_uuid == NULL || strcmp(fw_uuid, dev_uuid) != 0) {
+ xrt_err(pdev, "bad fw UUID: %s, expect: %s",
+ fw_uuid ? fw_uuid : "<none>", dev_uuid);
+ kfree(fw_uuid);
+ return false;
+ }
+
+ kfree(fw_uuid);
+ return true;
+}
+
+int xmgmt_get_provider_uuid(struct platform_device *pdev,
+ enum provider_kind kind, uuid_t *uuid)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ const char *fwbuf;
+ const char *fw_uuid;
+ int rc = -ENOENT;
+
+ mutex_lock(&xmm->busy_mutex);
+
+ fwbuf = xmgmt_get_axlf_firmware(xmm, kind);
+ if (fwbuf == NULL)
+ goto done;
+
+ fw_uuid = get_uuid_from_firmware(pdev, fwbuf);
+ if (fw_uuid == NULL)
+ goto done;
+
+ rc = xrt_md_uuid_strtoid(DEV(pdev), fw_uuid, uuid);
+ kfree(fw_uuid);
+
+done:
+ mutex_unlock(&xmm->busy_mutex);
+ return rc;
+}
+
+static int xmgmt_create_blp(struct xmgmt_main *xmm)
+{
+ struct platform_device *pdev = xmm->pdev;
+ int rc = 0;
+ char *dtb = NULL;
+
+ dtb = xmgmt_get_dtb(pdev, XMGMT_BLP);
+ if (dtb) {
+ rc = xrt_subdev_create_partition(pdev, dtb);
+ if (rc < 0)
+ xrt_err(pdev, "failed to create BLP: %d", rc);
+ else
+ rc = 0;
+
+ BUG_ON(xmm->blp_intf_uuids);
+ xrt_md_get_intf_uuids(&pdev->dev, dtb,
+ &xmm->blp_intf_uuid_num, NULL);
+ if (xmm->blp_intf_uuid_num > 0) {
+ xmm->blp_intf_uuids = vzalloc(sizeof(uuid_t) *
+ xmm->blp_intf_uuid_num);
+ xrt_md_get_intf_uuids(&pdev->dev, dtb,
+ &xmm->blp_intf_uuid_num, xmm->blp_intf_uuids);
+ }
+ }
+
+ vfree(dtb);
+ return rc;
+}
+
+static int xmgmt_main_event_cb(struct platform_device *pdev,
+ enum xrt_events evt, void *arg)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev *)arg;
+ enum xrt_subdev_id id;
+ int instance;
+ size_t fwlen;
+
+ switch (evt) {
+ case XRT_EVENT_POST_CREATION: {
+ id = esd->xevt_subdev_id;
+ instance = esd->xevt_subdev_instance;
+ xrt_info(pdev, "processing event %d for (%d, %d)",
+ evt, id, instance);
+
+ if (id == XRT_SUBDEV_GPIO)
+ xmm->gpio_ready = true;
+ else if (id == XRT_SUBDEV_QSPI)
+ xmm->flash_ready = true;
+ else
+ BUG_ON(1);
+
+ if (xmm->gpio_ready && xmm->flash_ready) {
+ int rc;
+
+ rc = load_firmware_from_disk(pdev, &xmm->firmware_blp,
+ &fwlen);
+ if (rc != 0) {
+ rc = load_firmware_from_flash(pdev,
+ &xmm->firmware_blp, &fwlen);
+ }
+ if (rc == 0 && is_valid_firmware(pdev,
+ xmm->firmware_blp, fwlen))
+ (void) xmgmt_create_blp(xmm);
+ else
+ xrt_err(pdev,
+ "failed to find firmware, giving up");
+ xmm->evt_hdl = NULL;
+ }
+ break;
+ }
+ case XRT_EVENT_POST_ATTACH:
+ xmgmt_peer_notify_state(xmm->mailbox_hdl, true);
+ break;
+ case XRT_EVENT_PRE_DETACH:
+ xmgmt_peer_notify_state(xmm->mailbox_hdl, false);
+ break;
+ default:
+ xrt_info(pdev, "ignored event %d", evt);
+ break;
+ }
+
+ return XRT_EVENT_CB_CONTINUE;
+}
+
+static int xmgmt_main_probe(struct platform_device *pdev)
+{
+ struct xmgmt_main *xmm;
+
+ xrt_info(pdev, "probing...");
+
+ xmm = devm_kzalloc(DEV(pdev), sizeof(*xmm), GFP_KERNEL);
+ if (!xmm)
+ return -ENOMEM;
+
+ xmm->pdev = pdev;
+ platform_set_drvdata(pdev, xmm);
+ xmm->fmgr = xmgmt_fmgr_probe(pdev);
+ xmm->mailbox_hdl = xmgmt_mailbox_probe(pdev);
+ mutex_init(&xmm->busy_mutex);
+
+ xmm->evt_hdl = xrt_subdev_add_event_cb(pdev,
+ xmgmt_main_leaf_match, NODE_BLP_ROM, xmgmt_main_event_cb);
+
+ /* Ready to handle req thru sysfs nodes. */
+ if (sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup))
+ xrt_err(pdev, "failed to create sysfs group");
+ return 0;
+}
+
+static int xmgmt_main_remove(struct platform_device *pdev)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+
+ /* By now, partition driver should prevent any inter-leaf call. */
+
+ xrt_info(pdev, "leaving...");
+
+ if (xmm->evt_hdl)
+ (void) xrt_subdev_remove_event_cb(pdev, xmm->evt_hdl);
+ vfree(xmm->blp_intf_uuids);
+ vfree(xmm->firmware_blp);
+ vfree(xmm->firmware_plp);
+ vfree(xmm->firmware_ulp);
+ (void) xmgmt_fmgr_remove(xmm->fmgr);
+ xmgmt_mailbox_remove(xmm->mailbox_hdl);
+ (void) sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup);
+ return 0;
+}
+
+static int
+xmgmt_main_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ int ret = 0;
+
+ xrt_info(pdev, "handling IOCTL cmd: %d", cmd);
+
+ switch (cmd) {
+ case XRT_MGMT_MAIN_GET_AXLF_SECTION: {
+ struct xrt_mgmt_main_ioctl_get_axlf_section *get =
+ (struct xrt_mgmt_main_ioctl_get_axlf_section *)arg;
+ const char *firmware =
+ xmgmt_get_axlf_firmware(xmm, get->xmmigas_axlf_kind);
+
+ if (firmware == NULL) {
+ ret = -ENOENT;
+ } else {
+ ret = xrt_xclbin_get_section(firmware,
+ get->xmmigas_section_kind,
+ &get->xmmigas_section,
+ &get->xmmigas_section_size);
+ }
+ break;
+ }
+ case XRT_MGMT_MAIN_GET_VBNV: {
+ char **vbnv_p = (char **)arg;
+
+ *vbnv_p = xmgmt_get_vbnv(pdev);
+ break;
+ }
+ default:
+ xrt_err(pdev, "unknown cmd: %d", cmd);
+ ret = -EINVAL;
+ break;
+ }
+ return ret;
+}
+
+static int xmgmt_main_open(struct inode *inode, struct file *file)
+{
+ struct platform_device *pdev = xrt_devnode_open(inode);
+
+ /* Device may have gone already when we get here. */
+ if (!pdev)
+ return -ENODEV;
+
+ xrt_info(pdev, "opened");
+ file->private_data = platform_get_drvdata(pdev);
+ return 0;
+}
+
+static int xmgmt_main_close(struct inode *inode, struct file *file)
+{
+ struct xmgmt_main *xmm = file->private_data;
+
+ xrt_devnode_close(inode);
+
+ xrt_info(xmm->pdev, "closed");
+ return 0;
+}
+
+static int xmgmt_bitstream_axlf_fpga_mgr(struct xmgmt_main *xmm,
+ void *axlf, size_t size)
+{
+ int ret;
+ struct fpga_image_info info = { 0 };
+
+ BUG_ON(!mutex_is_locked(&xmm->busy_mutex));
+
+ /*
+ * Should any error happens during download, we can't trust
+ * the cached xclbin any more.
+ */
+ vfree(xmm->firmware_ulp);
+ xmm->firmware_ulp = NULL;
+
+ info.buf = (char *)axlf;
+ info.count = size;
+ ret = fpga_mgr_load(xmm->fmgr, &info);
+ if (ret == 0)
+ xmm->firmware_ulp = axlf;
+
+ return ret;
+}
+
+int bitstream_axlf_mailbox(struct platform_device *pdev, const void *axlf)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ void *copy_buffer = NULL;
+ size_t copy_buffer_size = 0;
+ const struct axlf *xclbin_obj = axlf;
+ int ret = 0;
+
+ if (memcmp(xclbin_obj->m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
+ return -EINVAL;
+
+ copy_buffer_size = xclbin_obj->m_header.m_length;
+ if (copy_buffer_size > MAX_XCLBIN_SIZE)
+ return -EINVAL;
+ copy_buffer = vmalloc(copy_buffer_size);
+ if (copy_buffer == NULL)
+ return -ENOMEM;
+ (void) memcpy(copy_buffer, axlf, copy_buffer_size);
+
+ mutex_lock(&xmm->busy_mutex);
+ ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
+ mutex_unlock(&xmm->busy_mutex);
+ if (ret)
+ vfree(copy_buffer);
+ return ret;
+}
+
+static int bitstream_axlf_ioctl(struct xmgmt_main *xmm, const void __user *arg)
+{
+ void *copy_buffer = NULL;
+ size_t copy_buffer_size = 0;
+ struct xmgmt_ioc_bitstream_axlf ioc_obj = { 0 };
+ struct axlf xclbin_obj = { {0} };
+ int ret = 0;
+
+ if (copy_from_user((void *)&ioc_obj, arg, sizeof(ioc_obj)))
+ return -EFAULT;
+ if (copy_from_user((void *)&xclbin_obj, ioc_obj.xclbin,
+ sizeof(xclbin_obj)))
+ return -EFAULT;
+ if (memcmp(xclbin_obj.m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
+ return -EINVAL;
+
+ copy_buffer_size = xclbin_obj.m_header.m_length;
+ if (copy_buffer_size > MAX_XCLBIN_SIZE)
+ return -EINVAL;
+ copy_buffer = vmalloc(copy_buffer_size);
+ if (copy_buffer == NULL)
+ return -ENOMEM;
+
+ if (copy_from_user(copy_buffer, ioc_obj.xclbin, copy_buffer_size)) {
+ vfree(copy_buffer);
+ return -EFAULT;
+ }
+
+ ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
+ if (ret)
+ vfree(copy_buffer);
+
+ return ret;
+}
+
+static long xmgmt_main_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ long result = 0;
+ struct xmgmt_main *xmm = filp->private_data;
+
+ BUG_ON(!xmm);
+
+ if (_IOC_TYPE(cmd) != XMGMT_IOC_MAGIC)
+ return -ENOTTY;
+
+ mutex_lock(&xmm->busy_mutex);
+
+ xrt_info(xmm->pdev, "ioctl cmd %d, arg %ld", cmd, arg);
+ switch (cmd) {
+ case XMGMT_IOCICAPDOWNLOAD_AXLF:
+ result = bitstream_axlf_ioctl(xmm, (const void __user *)arg);
+ break;
+ default:
+ result = -ENOTTY;
+ break;
+ }
+
+ mutex_unlock(&xmm->busy_mutex);
+ return result;
+}
+
+void *xmgmt_pdev2mailbox(struct platform_device *pdev)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+
+ return xmm->mailbox_hdl;
+}
+
+struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names []){
+ { .ep_name = NODE_MGMT_MAIN },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+struct xrt_subdev_drvdata xmgmt_main_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xmgmt_main_leaf_ioctl,
+ },
+ .xsd_file_ops = {
+ .xsf_ops = {
+ .owner = THIS_MODULE,
+ .open = xmgmt_main_open,
+ .release = xmgmt_main_close,
+ .unlocked_ioctl = xmgmt_main_ioctl,
+ },
+ .xsf_dev_name = "xmgmt",
+ },
+};
+
+static const struct platform_device_id xmgmt_main_id_table[] = {
+ { XMGMT_MAIN, (kernel_ulong_t)&xmgmt_main_data },
+ { },
+};
+
+struct platform_driver xmgmt_main_driver = {
+ .driver = {
+ .name = XMGMT_MAIN,
+ },
+ .probe = xmgmt_main_probe,
+ .remove = xmgmt_main_remove,
+ .id_table = xmgmt_main_id_table,
+};
diff --git a/drivers/fpga/alveo/mgmt/xmgmt-root.c b/drivers/fpga/alveo/mgmt/xmgmt-root.c
new file mode 100644
index 000000000000..005fd5e42651
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/xmgmt-root.c
@@ -0,0 +1,375 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/aer.h>
+#include <linux/vmalloc.h>
+#include <linux/delay.h>
+
+#include "xrt-root.h"
+#include "xrt-subdev.h"
+#include "xmgmt-main-impl.h"
+#include "xrt-metadata.h"
+
+#define XMGMT_MODULE_NAME "xmgmt"
+#define XMGMT_DRIVER_VERSION "4.0.0"
+
+#define XMGMT_PDEV(xm) ((xm)->pdev)
+#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
+#define xmgmt_err(xm, fmt, args...) \
+ dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_warn(xm, fmt, args...) \
+ dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_info(xm, fmt, args...) \
+ dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_dbg(xm, fmt, args...) \
+ dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define XMGMT_DEV_ID(pdev) \
+ ((pci_domain_nr(pdev->bus) << 16) | \
+ PCI_DEVID(pdev->bus->number, 0))
+
+static struct class *xmgmt_class;
+static const struct pci_device_id xmgmt_pci_ids[] = {
+ { PCI_DEVICE(0x10EE, 0xd020), },
+ { PCI_DEVICE(0x10EE, 0x5020), },
+ { 0, }
+};
+
+struct xmgmt {
+ struct pci_dev *pdev;
+ void *root;
+
+ /* save config for pci reset */
+ u32 saved_config[8][16];
+ bool ready;
+};
+
+static int xmgmt_config_pci(struct xmgmt *xm)
+{
+ struct pci_dev *pdev = XMGMT_PDEV(xm);
+ int rc;
+
+ rc = pcim_enable_device(pdev);
+ if (rc < 0) {
+ xmgmt_err(xm, "failed to enable device: %d", rc);
+ return rc;
+ }
+
+ rc = pci_enable_pcie_error_reporting(pdev);
+ if (rc)
+ xmgmt_warn(xm, "failed to enable AER: %d", rc);
+
+ pci_set_master(pdev);
+
+ rc = pcie_get_readrq(pdev);
+ if (rc < 0) {
+ xmgmt_err(xm, "failed to read mrrs %d", rc);
+ return rc;
+ }
+ if (rc > 512) {
+ rc = pcie_set_readrq(pdev, 512);
+ if (rc) {
+ xmgmt_err(xm, "failed to force mrrs %d", rc);
+ return rc;
+ }
+ }
+
+ return 0;
+}
+
+static void xmgmt_save_config_space(struct pci_dev *pdev, u32 *saved_config)
+{
+ int i;
+
+ for (i = 0; i < 16; i++)
+ pci_read_config_dword(pdev, i * 4, &saved_config[i]);
+}
+
+static int xmgmt_match_slot_and_save(struct device *dev, void *data)
+{
+ struct xmgmt *xm = data;
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
+ pci_cfg_access_lock(pdev);
+ pci_save_state(pdev);
+ xmgmt_save_config_space(pdev,
+ xm->saved_config[PCI_FUNC(pdev->devfn)]);
+ }
+
+ return 0;
+}
+
+static void xmgmt_pci_save_config_all(struct xmgmt *xm)
+{
+ bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
+}
+
+static void xmgmt_restore_config_space(struct pci_dev *pdev, u32 *config_saved)
+{
+ int i;
+ u32 val;
+
+ for (i = 0; i < 16; i++) {
+ pci_read_config_dword(pdev, i * 4, &val);
+ if (val == config_saved[i])
+ continue;
+
+ pci_write_config_dword(pdev, i * 4, config_saved[i]);
+ pci_read_config_dword(pdev, i * 4, &val);
+ if (val != config_saved[i]) {
+ dev_err(&pdev->dev,
+ "restore config at %d failed", i * 4);
+ }
+ }
+}
+
+static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
+{
+ struct xmgmt *xm = data;
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
+ xmgmt_restore_config_space(pdev,
+ xm->saved_config[PCI_FUNC(pdev->devfn)]);
+
+ pci_restore_state(pdev);
+ pci_cfg_access_unlock(pdev);
+ }
+
+ return 0;
+}
+
+static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
+{
+ bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
+}
+
+void xroot_hot_reset(struct pci_dev *pdev)
+{
+ struct xmgmt *xm = pci_get_drvdata(pdev);
+ struct pci_bus *bus;
+ u8 pci_bctl;
+ u16 pci_cmd, devctl;
+ int i;
+
+ xmgmt_info(xm, "hot reset start");
+
+ xmgmt_pci_save_config_all(xm);
+
+ pci_disable_device(pdev);
+
+ bus = pdev->bus;
+
+ /*
+ * When flipping the SBR bit, device can fall off the bus. This is
+ * usually no problem at all so long as drivers are working properly
+ * after SBR. However, some systems complain bitterly when the device
+ * falls off the bus.
+ * The quick solution is to temporarily disable the SERR reporting of
+ * switch port during SBR.
+ */
+
+ pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
+ pci_write_config_word(bus->self, PCI_COMMAND,
+ (pci_cmd & ~PCI_COMMAND_SERR));
+ pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
+ pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
+ (devctl & ~PCI_EXP_DEVCTL_FERE));
+ pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
+ pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
+ pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
+
+ msleep(100);
+ pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
+ pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
+ ssleep(1);
+
+ pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
+ pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
+
+ pci_enable_device(pdev);
+
+ for (i = 0; i < 300; i++) {
+ pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
+ if (pci_cmd != 0xffff)
+ break;
+ msleep(20);
+ }
+
+ xmgmt_info(xm, "waiting for %d ms", i * 20);
+
+ xmgmt_pci_restore_config_all(xm);
+
+ xmgmt_config_pci(xm);
+}
+
+static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
+{
+ char *dtb = NULL;
+ int ret;
+
+ ret = xrt_md_create(DEV(xm->pdev), &dtb);
+ if (ret) {
+ xmgmt_err(xm, "create metadata failed, ret %d", ret);
+ goto failed;
+ }
+
+ ret = xroot_add_simple_node(xm->root, dtb, NODE_TEST);
+ if (ret)
+ goto failed;
+
+ ret = xroot_add_vsec_node(xm->root, dtb);
+ if (ret == -ENOENT) {
+ /*
+ * We may be dealing with a MFG board.
+ * Try vsec-golden which will bring up all hard-coded leaves
+ * at hard-coded offsets.
+ */
+ ret = xroot_add_simple_node(xm, dtb, NODE_VSEC_GOLDEN);
+ } else if (ret == 0) {
+ ret = xroot_add_simple_node(xm->root, dtb, NODE_MGMT_MAIN);
+ }
+ if (ret)
+ goto failed;
+
+ *root_dtb = dtb;
+ return 0;
+
+failed:
+ vfree(dtb);
+ return ret;
+}
+
+static ssize_t ready_show(struct device *dev,
+ struct device_attribute *da, char *buf)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct xmgmt *xm = pci_get_drvdata(pdev);
+
+ return sprintf(buf, "%d\n", xm->ready);
+}
+static DEVICE_ATTR_RO(ready);
+
+static struct attribute *xmgmt_root_attrs[] = {
+ &dev_attr_ready.attr,
+ NULL
+};
+
+static struct attribute_group xmgmt_root_attr_group = {
+ .attrs = xmgmt_root_attrs,
+};
+
+static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ int ret;
+ struct device *dev = DEV(pdev);
+ struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
+ char *dtb = NULL;
+
+ if (!xm)
+ return -ENOMEM;
+ xm->pdev = pdev;
+ pci_set_drvdata(pdev, xm);
+
+ ret = xmgmt_config_pci(xm);
+ if (ret)
+ goto failed;
+
+ ret = xroot_probe(pdev, &xm->root);
+ if (ret)
+ goto failed;
+
+ ret = xmgmt_create_root_metadata(xm, &dtb);
+ if (ret)
+ goto failed_metadata;
+
+ ret = xroot_create_partition(xm->root, dtb);
+ vfree(dtb);
+ if (ret)
+ xmgmt_err(xm, "failed to create root partition: %d", ret);
+
+ if (!xroot_wait_for_bringup(xm->root))
+ xmgmt_err(xm, "failed to bringup all partitions");
+ else
+ xm->ready = true;
+
+ ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
+ if (ret) {
+ /* Warning instead of failing the probe. */
+ xrt_warn(pdev, "create xmgmt root attrs failed: %d", ret);
+ }
+
+ xroot_broadcast(xm->root, XRT_EVENT_POST_ATTACH);
+ xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
+ return 0;
+
+failed_metadata:
+ (void) xroot_remove(xm->root);
+failed:
+ pci_set_drvdata(pdev, NULL);
+ return ret;
+}
+
+static void xmgmt_remove(struct pci_dev *pdev)
+{
+ struct xmgmt *xm = pci_get_drvdata(pdev);
+
+ xroot_broadcast(xm->root, XRT_EVENT_PRE_DETACH);
+ sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
+ (void) xroot_remove(xm->root);
+ pci_disable_pcie_error_reporting(xm->pdev);
+ xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
+}
+
+static struct pci_driver xmgmt_driver = {
+ .name = XMGMT_MODULE_NAME,
+ .id_table = xmgmt_pci_ids,
+ .probe = xmgmt_probe,
+ .remove = xmgmt_remove,
+};
+
+static int __init xmgmt_init(void)
+{
+ int res = xrt_subdev_register_external_driver(XRT_SUBDEV_MGMT_MAIN,
+ &xmgmt_main_driver, xrt_mgmt_main_endpoints);
+
+ if (res)
+ return res;
+
+ xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
+ if (IS_ERR(xmgmt_class))
+ return PTR_ERR(xmgmt_class);
+
+ res = pci_register_driver(&xmgmt_driver);
+ if (res) {
+ class_destroy(xmgmt_class);
+ return res;
+ }
+
+ return 0;
+}
+
+static __exit void xmgmt_exit(void)
+{
+ pci_unregister_driver(&xmgmt_driver);
+ class_destroy(xmgmt_class);
+ xrt_subdev_unregister_external_driver(XRT_SUBDEV_MGMT_MAIN);
+}
+
+module_init(xmgmt_init);
+module_exit(xmgmt_exit);
+
+MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
+MODULE_VERSION(XMGMT_DRIVER_VERSION);
+MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
+MODULE_DESCRIPTION("Xilinx Alveo management function driver");
+MODULE_LICENSE("GPL v2");
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH Xilinx Alveo 8/8] fpga: xrt: Kconfig and Makefile updates for XRT drivers
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
` (5 preceding siblings ...)
2020-11-29 0:00 ` [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver Sonal Santan
@ 2020-11-29 0:00 ` Sonal Santan
2020-11-30 18:08 ` [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Rob Herring
` (2 subsequent siblings)
9 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-11-29 0:00 UTC (permalink / raw)
To: linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
From: Sonal Santan <sonal.santan@xilinx.com>
Update fpga Kconfig/Makefile and add Kconfig/Makefile for
new drivers.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
---
drivers/fpga/Kconfig | 2 ++
drivers/fpga/Makefile | 3 +++
drivers/fpga/alveo/Kconfig | 7 ++++++
drivers/fpga/alveo/lib/Kconfig | 11 +++++++++
drivers/fpga/alveo/lib/Makefile | 42 ++++++++++++++++++++++++++++++++
drivers/fpga/alveo/mgmt/Kconfig | 11 +++++++++
drivers/fpga/alveo/mgmt/Makefile | 28 +++++++++++++++++++++
7 files changed, 104 insertions(+)
create mode 100644 drivers/fpga/alveo/Kconfig
create mode 100644 drivers/fpga/alveo/lib/Kconfig
create mode 100644 drivers/fpga/alveo/lib/Makefile
create mode 100644 drivers/fpga/alveo/mgmt/Kconfig
create mode 100644 drivers/fpga/alveo/mgmt/Makefile
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 7cd5a29fc437..8687ef231308 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -215,4 +215,6 @@ config FPGA_MGR_ZYNQMP_FPGA
to configure the programmable logic(PL) through PS
on ZynqMP SoC.
+source "drivers/fpga/alveo/Kconfig"
+
endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index d8e21dfc6778..59943dccf405 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -46,3 +46,6 @@ dfl-afu-objs += dfl-afu-error.o
# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
+
+obj-$(CONFIG_FPGA_ALVEO_LIB) += alveo/lib/
+obj-$(CONFIG_FPGA_ALVEO_XMGMT) += alveo/mgmt/
diff --git a/drivers/fpga/alveo/Kconfig b/drivers/fpga/alveo/Kconfig
new file mode 100644
index 000000000000..a583c3543945
--- /dev/null
+++ b/drivers/fpga/alveo/Kconfig
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Xilinx Alveo FPGA device configuration
+#
+
+source "drivers/fpga/alveo/lib/Kconfig"
+source "drivers/fpga/alveo/mgmt/Kconfig"
diff --git a/drivers/fpga/alveo/lib/Kconfig b/drivers/fpga/alveo/lib/Kconfig
new file mode 100644
index 000000000000..62175af2108e
--- /dev/null
+++ b/drivers/fpga/alveo/lib/Kconfig
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Xilinx Alveo FPGA device configuration
+#
+
+config FPGA_ALVEO_LIB
+ tristate "Xilinx Alveo Driver Library"
+ depends on HWMON && PCI
+ select LIBFDT
+ help
+ Xilinx Alveo FPGA PCIe device driver common library.
diff --git a/drivers/fpga/alveo/lib/Makefile b/drivers/fpga/alveo/lib/Makefile
new file mode 100644
index 000000000000..a14204dc489d
--- /dev/null
+++ b/drivers/fpga/alveo/lib/Makefile
@@ -0,0 +1,42 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020 Xilinx, Inc. All rights reserved.
+#
+# Authors: Sonal.Santan@xilinx.com
+#
+
+FULL_ALVEO_PATH=$(srctree)/$(src)/..
+FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
+
+obj-$(CONFIG_FPGA_ALVEO_LIB) := xrt-lib.o
+
+xrt-lib-objs := \
+ xrt-main.o \
+ xrt-subdev.o \
+ xrt-cdev.o \
+ ../common/xrt-metadata.o \
+ subdevs/xrt-partition.o \
+ subdevs/xrt-test.o \
+ subdevs/xrt-vsec.o \
+ subdevs/xrt-vsec-golden.o \
+ subdevs/xrt-axigate.o \
+ subdevs/xrt-qspi.o \
+ subdevs/xrt-gpio.o \
+ subdevs/xrt-mailbox.o \
+ subdevs/xrt-icap.o \
+ subdevs/xrt-cmc.o \
+ subdevs/xrt-cmc-ctrl.o \
+ subdevs/xrt-cmc-sensors.o \
+ subdevs/xrt-cmc-mailbox.o \
+ subdevs/xrt-cmc-bdinfo.o \
+ subdevs/xrt-cmc-sc.o \
+ subdevs/xrt-srsr.o \
+ subdevs/xrt-clock.o \
+ subdevs/xrt-clkfreq.o \
+ subdevs/xrt-ucs.o \
+ subdevs/xrt-calib.o
+
+
+ccflags-y := -I$(FULL_ALVEO_PATH)/include \
+ -I$(FULL_ALVEO_PATH)/common \
+ -I$(FULL_DTC_PATH)
diff --git a/drivers/fpga/alveo/mgmt/Kconfig b/drivers/fpga/alveo/mgmt/Kconfig
new file mode 100644
index 000000000000..8a5590842dad
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/Kconfig
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Xilinx Alveo FPGA device configuration
+#
+
+config FPGA_ALVEO_XMGMT
+ tristate "Xilinx Alveo Management Driver"
+ depends on HWMON && PCI && FPGA_ALVEO_LIB
+ select LIBFDT
+ help
+ Xilinx Alveo FPGA PCIe device driver for Management Physical Function.
diff --git a/drivers/fpga/alveo/mgmt/Makefile b/drivers/fpga/alveo/mgmt/Makefile
new file mode 100644
index 000000000000..08be7952a832
--- /dev/null
+++ b/drivers/fpga/alveo/mgmt/Makefile
@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2019-2020 Xilinx, Inc. All rights reserved.
+#
+# Authors: Sonal.Santan@xilinx.com
+#
+
+FULL_ALVEO_PATH=$(srctree)/$(src)/..
+FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
+
+obj-$(CONFIG_FPGA_ALVEO_XMGMT) += xmgmt.o
+
+commondir := ../common
+
+xmgmt-objs := xmgmt-root.o \
+ xmgmt-main.o \
+ xmgmt-fmgr-drv.o \
+ xmgmt-main-ulp.o \
+ xmgmt-main-mailbox.o \
+ $(commondir)/xrt-root.o \
+ $(commondir)/xrt-metadata.o \
+ $(commondir)/xrt-xclbin.o
+
+
+
+ccflags-y := -I$(FULL_ALVEO_PATH)/include \
+ -I$(FULL_ALVEO_PATH)/common \
+ -I$(FULL_DTC_PATH)
--
2.17.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
` (6 preceding siblings ...)
2020-11-29 0:00 ` [PATCH Xilinx Alveo 8/8] fpga: xrt: Kconfig and Makefile updates for XRT drivers Sonal Santan
@ 2020-11-30 18:08 ` Rob Herring
2020-12-01 19:39 ` Sonal Santan
2020-12-02 2:14 ` Xu Yilun
2020-12-06 16:31 ` Tom Rix
9 siblings, 1 reply; 28+ messages in thread
From: Rob Herring @ 2020-11-30 18:08 UTC (permalink / raw)
To: Sonal Santan
Cc: linux-kernel, Sonal Santan, linux-fpga, maxz, lizhih,
Michal Simek, Stefano Stabellini, devicetree
On Sat, Nov 28, 2020 at 5:02 PM Sonal Santan <sonal.santan@xilinx.com> wrote:
>
> Hello,
>
> This patch series adds management physical function driver for Xilinx Alveo PCIe
> accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> This driver is part of Xilinx Runtime (XRT) open source stack.
>
> The patch depends on the "PATCH Xilinx Alveo libfdt prep" which was posted
> before.
>
> ALVEO PLATFORM ARCHITECTURE
>
> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
> re-configurable *user* partition. The shell partition is automatically loaded from
> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
> till the next cold reboot. The shell exposes two PCIe physical functions:
>
> 1. management physical function
> 2. user physical function
>
> The patch series includes Documentation/xrt.rst which describes Alveo
> platform, xmgmt driver architecture and deployment model in more more detail.
>
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image
> using Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> tools. The image is packaged as xclbin and contains partial bitstream for the
> user partition and necessary metadata. Users can dynamically swap the image
> running on the user partition in order to switch between different workloads.
>
> ALVEO DRIVERS
>
> Alveo Linux kernel driver *xmgmt* binds to management physical function of
> Alveo platform. The modular driver framework is organized into several
> platform drivers which primarily handle the following functionality:
>
> 1. Loading firmware container also called xsabin at driver attach time
> 2. Loading of user compiled xclbin with FPGA Manager integration
> 3. Clock scaling of image running on user partition
> 4. In-band sensors: temp, voltage, power, etc.
> 5. Device reset and rescan
> 6. Flashing static *shell* partition
>
> The platform drivers are packaged into *xrt-lib* helper module with a well
> defined interfaces the details of which can be found in Documentation/xrt.rst.
>
> xmgmt driver is second generation Alveo management driver and evolution of
> the first generation (out of tree) Alveo management driver, xclmgmt. The
> sources of the first generation drivers were posted on LKML last year--
> https://lore.kernel.org/lkml/20190319215401.6562-1-sonal.santan@xilinx.com/
>
> Changes since the first generation driver include the following: the driver
> has been re-architected as data driven modular driver; the driver has been
> split into xmgmt and xrt-lib; user physical function driver has been removed
> from the patch series.
>
> Alveo/XRT security and platform architecture is documented on the following
> GitHub pages:
> https://xilinx.github.io/XRT/master/html/security.html
> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>
> User physical function driver is not included in this patch series.
>
> TESTING AND VALIDATION
>
> xmgmt driver can be tested with full XRT open source stack which includes
> user space libraries, board utilities and (out of tree) first generation
> user physical function driver xocl. XRT open source runtime stack is
> available at https://github.com/Xilinx/XRT. This patch series has been
> validated on Alveo U50 platform.
>
> Complete documentation for XRT open source stack can be found here--
> https://xilinx.github.io/XRT/master/html/index.html
I've not gotten into the patch details, but I'm not clear on what the
lifecycle of the DT looks like here. What's the starting point and
what manipulations to the DT are being done? I'm trying to understand
if using libfdt is the right way versus operating on an unflattened
tree.
Rob
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files
2020-11-29 0:00 ` [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files Sonal Santan
@ 2020-12-01 4:27 ` Moritz Fischer
2020-12-02 18:57 ` Sonal Santan
0 siblings, 1 reply; 28+ messages in thread
From: Moritz Fischer @ 2020-12-01 4:27 UTC (permalink / raw)
To: Sonal Santan
Cc: linux-kernel, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
Hi Sonal,
On Sat, Nov 28, 2020 at 04:00:34PM -0800, Sonal Santan wrote:
> From: Sonal Santan <sonal.santan@xilinx.com>
>
> Add XRT UAPI header files which describe flash layout, XRT
> mailbox protocol, xclBin/axlf FPGA image container format and
> XRT management physical function driver ioctl interfaces.
>
> flash_xrt_data.h:
> Layout used by XRT to store private data on flash.
>
> mailbox_proto.h:
> Mailbox opcodes and high level data structures representing
> various kinds of information like sensors, clock, etc.
>
> mailbox_transport.h:
> Transport protocol used by mailbox.
>
> xclbin.h:
> Container format used to store compiled FPGA image which includes
> bitstream and metadata.
Can these headers be introduced together with the code that uses them as
logical change?
I haven't looked too closely, but it helps reviewing if you can break it
into smaller pieces that can stand by themselves.
Thanks,
Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-11-29 0:00 ` [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers Sonal Santan
@ 2020-12-01 4:54 ` Moritz Fischer
2020-12-02 21:24 ` Max Zhen
0 siblings, 1 reply; 28+ messages in thread
From: Moritz Fischer @ 2020-12-01 4:54 UTC (permalink / raw)
To: Sonal Santan
Cc: linux-kernel, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
On Sat, Nov 28, 2020 at 04:00:33PM -0800, Sonal Santan wrote:
> From: Sonal Santan <sonal.santan@xilinx.com>
>
> Describe Alveo XRT driver architecture and provide basic overview
> of Xilinx Alveo platform.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> ---
> Documentation/fpga/index.rst | 1 +
> Documentation/fpga/xrt.rst | 588 +++++++++++++++++++++++++++++++++++
> 2 files changed, 589 insertions(+)
> create mode 100644 Documentation/fpga/xrt.rst
>
> diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
> index f80f95667ca2..30134357b70d 100644
> --- a/Documentation/fpga/index.rst
> +++ b/Documentation/fpga/index.rst
> @@ -8,6 +8,7 @@ fpga
> :maxdepth: 1
>
> dfl
> + xrt
>
> .. only:: subproject and html
>
> diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
> new file mode 100644
> index 000000000000..9f37d46459b0
> --- /dev/null
> +++ b/Documentation/fpga/xrt.rst
> @@ -0,1 +1,588 @@
> +==================================
> +XRTV2 Linux Kernel Driver Overview
> +==================================
> +
> +XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_ drivers which
> +support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_ PCIe platforms
> +from Xilinx.
> +
> +XRTV2 drivers support *subsystem* style data driven platforms where driver's configuration
> +and behavior is determined by meta data provided by platform (in *device tree* format).
> +Primary management physical function (MPF) driver is called **xmgmt**. Primary user physical
> +function (UPF) driver is called **xuser** and HW subsystem drivers are packaged into a library
> +module called **xrt-lib**, which is shared by **xmgmt** and **xuser** (WIP).
WIP?
> +
> +Alveo Platform Overview
> +=======================
> +
> +Alveo platforms are architected as two physical FPGA partitions: *Shell* and *User*. Shell
Nit: The Shell provides ...
> +provides basic infrastructure for the Alveo platform like PCIe connectivity, board management,
> +Dynamic Function Exchange (DFX), sensors, clocking, reset, and security. User partition contains
> +user compiled binary which is loaded by a process called DFX also known as partial reconfiguration.
> +
> +Physical partitions require strict HW compatibility with each other for DFX to work properly.
> +Every physical partition has two interface UUIDs: *parent* UUID and *child* UUID. For simple
> +single stage platforms Shell → User forms parent child relationship. For complex two stage
> +platforms Base → Shell → User forms the parent child relationship chain.
> +
> +.. note::
> + Partition compatibility matching is key design component of Alveo platforms and XRT. Partitions
> + have child and parent relationship. A loaded partition exposes child partition UUID to advertise
> + its compatibility requirement for child partition. When loading a child partition the xmgmt
> + management driver matches parent UUID of the child partition against child UUID exported by the
> + parent. Parent and child partition UUIDs are stored in the *xclbin* (for user) or *xsabin* (for
> + base and shell). Except for root UUID, VSEC, hardware itself does not know about UUIDs. UUIDs are
> + stored in xsabin and xclbin.
> +
> +
> +The physical partitions and their loading is illustrated below::
> +
> + SHELL USER
> + +-----------+ +-------------------+
> + | | | |
> + | VSEC UUID | CHILD PARENT | LOGIC UUID |
> + | o------->|<--------o |
> + | | UUID UUID | |
> + +-----+-----+ +--------+----------+
> + | |
> + . .
> + | |
> + +---+---+ +------+--------+
> + | POR | | USER COMPILED |
> + | FLASH | | XCLBIN |
> + +-------+ +---------------+
> +
> +
> +Loading Sequence
> +----------------
> +
> +Shell partition is loaded from flash at system boot time. It establishes the PCIe link and exposes
Nit: The Shell
> +two physical functions to the BIOS. After OS boot, xmgmt driver attaches to PCIe physical function
> +0 exposed by the Shell and then looks for VSEC in PCIe extended configuration space. Using VSEC it
> +determines the logic UUID of Shell and uses the UUID to load matching *xsabin* file from Linux
> +firmware directory. The xsabin file contains metadata to discover peripherals that are part of Shell
> +and firmware(s) for any embedded soft processors in Shell.
Neat.
> +
> +Shell exports child interface UUID which is used for compatibility check when loading user compiled
Nit: The Shell
> +xclbin over the User partition as part of DFX. When a user requests loading of a specific xclbin the
> +xmgmt management driver reads the parent interface UUID specified in the xclbin and matches it with
> +child interface UUID exported by Shell to determine if xclbin is compatible with the Shell. If match
> +fails loading of xclbin is denied.
> +
> +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command. When loading xclbin xmgmt driver
> +performs the following operations:
> +
> +1. Sanity check the xclbin contents
> +2. Isolate the User partition
> +3. Download the bitstream using the FPGA config engine (ICAP)
> +4. De-isolate the User partition
Is this modelled as bridges and regions?
> +5. Program the clocks (ClockWiz) driving the User partition
> +6. Wait for memory controller (MIG) calibration
> +
> +`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
> +provides more detailed information on platform loading.
> +
> +xsabin
> +------
> +
> +Each Alveo platform comes packaged with its own xsabin. The xsabin is trusted component of the
> +platform. For format details refer to :ref:`xsabin/xclbin Container Format`. xsabin contains
> +basic information like UUIDs, platform name and metadata in the form of device tree. See
> +:ref:`Device Tree Usage` for details and example.
> +
> +xclbin
> +------
> +
> +xclbin is compiled by end user using
> +`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_ tool set from
> +Xilinx. The xclbin contains sections describing user compiled acceleration engines/kernels, memory
> +subsystems, clocking information etc. It also contains bitstream for the user partition, UUIDs,
> +platform name, etc. xclbin uses the same container format as xsabin which is described below.
> +
> +
> +xsabin/xclbin Container Format
> +------------------------------
> +
> +xclbin/xsabin is ELF-like binary container format. It is structured as series of sections.
> +There is a file header followed by several section headers which is followed by sections.
> +A section header points to an actual section. There is an optional signature at the end.
> +The format is defined by header file ``xclbin.h``. The following figure illustrates a
> +typical xclbin::
> +
> +
> + +---------------------+
> + | |
> + | HEADER |
> + +---------------------+
> + | SECTION HEADER |
> + | |
> + +---------------------+
> + | ... |
> + | |
> + +---------------------+
> + | SECTION HEADER |
> + | |
> + +---------------------+
> + | SECTION |
> + | |
> + +---------------------+
> + | ... |
> + | |
> + +---------------------+
> + | SECTION |
> + | |
> + +---------------------+
> + | SIGNATURE |
> + | (OPTIONAL) |
> + +---------------------+
> +
> +
> +xclbin/xsabin files can be packaged, un-packaged and inspected using XRT utility called
> +**xclbinutil**. xclbinutil is part of XRT open source software stack. The source code for
> +xclbinutil can be found at https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
> +
> +For example to enumerate the contents of a xclbin/xsabin use the *--info* switch as shown
> +below::
> +
> + xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> + xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
> +
> +
> +Device Tree Usage
> +-----------------
> +
> +As mentioned previously xsabin stores metadata which advertise HW subsystems present in a partition.
> +The metadata is stored in device tree format with well defined schema. Subsystem instantiations are
> +captured as children of ``addressable_endpoints`` node. Subsystem nodes have standard attributes like
> +``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe specific attributes:
> +``pcie_physical_function`` and ``pcie_bar_mapping``. These identify which PCIe physical function and
> +which BAR space in that physical function the subsystem resides. XRT management driver uses this
> +information to bind *platform drivers* to the subsystem instantiations. The platform drivers are
> +found in **xrt-lib.ko** kernel module defined later. Below is an example of device tree for Alveo U50
> +platform::
I might be missing something, but couldn't you structure the addressable
endpoints in a way that encode the physical function as a parent / child
relation?
What are the regs relative to?
> +
> + /dts-v1/;
> +
> + /{
> + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> +
> + schema_version {
> + major = <0x01>;
> + minor = <0x00>;
> + };
> +
> + interfaces {
> +
> + @0 {
> + interface_uuid = "862c7020a250293e32036f19956669e5";
> + };
> + };
> +
> + addressable_endpoints {
> +
> + ep_blp_rom_00 {
> + reg = <0x00 0x1f04000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> + };
> +
> + ep_card_flash_program_00 {
> + reg = <0x00 0x1f06000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
> + interrupts = <0x03 0x03>;
> + };
> +
> + ep_cmc_firmware_mem_00 {
> + reg = <0x00 0x1e20000 0x00 0x20000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> + firmware {
> + firmware_product_name = "cmc";
> + firmware_branch_name = "u50";
> + firmware_version_major = <0x01>;
> + firmware_version_minor = <0x00>;
> + };
> + };
> +
> + ep_cmc_intc_00 {
> + reg = <0x00 0x1e03000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> + interrupts = <0x04 0x04>;
> + };
> +
> + ep_cmc_mutex_00 {
> + reg = <0x00 0x1e02000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_cmc_regmap_00 {
> + reg = <0x00 0x1e08000 0x00 0x2000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> + firmware {
> + firmware_product_name = "sc-fw";
> + firmware_branch_name = "u50";
> + firmware_version_major = <0x05>;
> + };
> + };
> +
> + ep_cmc_reset_00 {
> + reg = <0x00 0x1e01000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_ddr_mem_calib_00 {
> + reg = <0x00 0x63000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_debug_bscan_mgmt_00 {
> + reg = <0x00 0x1e90000 0x00 0x10000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
> + };
> +
> + ep_ert_base_address_00 {
> + reg = <0x00 0x21000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_ert_command_queue_mgmt_00 {
> + reg = <0x00 0x40000 0x00 0x10000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> + };
> +
> + ep_ert_command_queue_user_00 {
> + reg = <0x00 0x40000 0x00 0x10000>;
> + pcie_physical_function = <0x01>;
> + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> + };
> +
> + ep_ert_firmware_mem_00 {
> + reg = <0x00 0x30000 0x00 0x8000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> + firmware {
> + firmware_product_name = "ert";
> + firmware_branch_name = "v20";
> + firmware_version_major = <0x01>;
> + };
> + };
> +
> + ep_ert_intc_00 {
> + reg = <0x00 0x23000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> + interrupts = <0x05 0x05>;
> + };
> +
> + ep_ert_reset_00 {
> + reg = <0x00 0x22000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_ert_sched_00 {
> + reg = <0x00 0x50000 0x00 0x1000>;
> + pcie_physical_function = <0x01>;
> + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> + interrupts = <0x09 0x0c>;
> + };
> +
> + ep_fpga_configuration_00 {
> + reg = <0x00 0x1e88000 0x00 0x8000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
> + interrupts = <0x02 0x02>;
> + };
> +
> + ep_icap_reset_00 {
> + reg = <0x00 0x1f07000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_mailbox_mgmt_00 {
> + reg = <0x00 0x1f10000 0x00 0x10000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
> + interrupts = <0x00 0x00>;
> + };
> +
> + ep_mailbox_user_00 {
> + reg = <0x00 0x1f00000 0x00 0x10000>;
> + pcie_physical_function = <0x01>;
> + compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
> + interrupts = <0x08 0x08>;
> + };
> +
> + ep_msix_00 {
> + reg = <0x00 0x00 0x00 0x20000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> + pcie_bar_mapping = <0x02>;
> + };
> +
> + ep_pcie_link_mon_00 {
> + reg = <0x00 0x1f05000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_pr_isolate_plp_00 {
> + reg = <0x00 0x1f01000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_pr_isolate_ulp_00 {
> + reg = <0x00 0x1000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_uuid_rom_00 {
> + reg = <0x00 0x64000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> + };
> +
> + ep_xdma_00 {
> + reg = <0x00 0x00 0x00 0x10000>;
> + pcie_physical_function = <0x01>;
> + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> + pcie_bar_mapping = <0x02>;
> + };
> + };
> +
> + }
> +
> +
> +
> +Deployment Models
> +=================
> +
> +Baremetal
> +---------
> +
> +In bare-metal deployments both MPF and UPF are visible and accessible. xmgmt driver binds to
> +MPF. xmgmt driver operations are privileged and available to system administrator. The full
> +stack is illustrated below::
> +
> +
> + HOST
> +
> + [XMGMT] [XUSER]
> + | |
> + | |
> + +-----+ +-----+
> + | MPF | | UPF |
> + | | | |
> + | PF0 | | PF1 |
> + +--+--+ +--+--+
> + ......... ^................. ^..........
> + | |
> + | PCIe DEVICE |
> + | |
> + +--+------------------+--+
> + | SHELL |
> + | |
> + +------------------------+
> + | USER |
> + | |
> + | |
> + | |
> + | |
> + +------------------------+
> +
> +
> +
> +Virtualized
> +-----------
> +
> +In virtualized deployments privileged MPF is assigned to host but unprivileged UPF
> +is assigned to guest VM via PCIe pass-through. xmgmt driver in host binds to MPF.
> +xmgmt driver operations are privileged and only accessible by hosting service provider.
> +The full stack is illustrated below::
> +
> +
> + .............
> + HOST . VM .
> + . .
> + [XMGMT] . [XUSER] .
> + | . | .
> + | . | .
> + +-----+ . +-----+ .
> + | MPF | . | UPF | .
> + | | . | | .
> + | PF0 | . | PF1 | .
> + +--+--+ . +--+--+ .
> + ......... ^................. ^..........
> + | |
> + | PCIe DEVICE |
> + | |
> + +--+------------------+--+
> + | SHELL |
> + | |
> + +------------------------+
> + | USER |
> + | |
> + | |
> + | |
> + | |
> + +------------------------+
> +
> +
> +
> +Driver Modules
> +==============
> +
> +xrt-lib.ko
> +----------
> +
> +Repository of all subsystem drivers and pure software modules that can potentially
> +be shared between xmgmt and xuser. All these drivers are structured as Linux
> +*platform driver* and are instantiated by xmgmt (or xuser in future) based on meta
> +data associated with hardware. The metadata is in the form of device tree as
> +explained before.
> +
> +xmgmt.ko
> +--------
> +
> +The xmgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
> +PCIE device. It consists of one *root* driver, one or more *partition* drivers
> +and one or more *leaf* drivers. The root and MPF specific leaf drivers are in
> +xmgmt.ko. The partition driver and other leaf drivers are in xrt-lib.ko.
> +
> +The instantiation of specific partition driver or leaf driver is completely data
> +driven based on meta data (mostly in device tree format) found through VSEC
> +capability and inside firmware files, such as xsabin or xclbin file. The root
> +driver manages life cycle of multiple partition drivers, which, in turn, manages
> +multiple leaf drivers. This allows a single set of driver code to support all
> +kinds of subsystems exposed by different shells. The difference among all
> +these subsystems will be handled in leaf drivers with root and partition drivers
> +being part of the infrastructure and provide common services for all leaves found
> +on all platforms.
> +
> +
> +xmgmt-root
> +^^^^^^^^^^
> +
> +The xmgmt-root driver is a PCIe device driver attaches to MPF. It's part of the
Nit: s/attaches/attached ?
> +infrastructure of the MPF driver and resides in xmgmt.ko. This driver
> +
> +* manages one or more partition drivers
> +* provides access to functionalities that requires pci_dev, such as PCIE config
> + space access, to other leaf drivers through parent calls
> +* together with partition driver, facilities event callbacks for other leaf drivers
> +* together with partition driver, facilities inter-leaf driver calls for other leaf
> + drivers
> +
> +When root driver starts, it will explicitly create an initial partition instance,
> +which contains leaf drivers that will trigger the creation of other partition
> +instances. The root driver will wait for all partitions and leaves to be created
> +before it returns from it's probe routine and claim success of the initialization
> +of the entire xmgmt driver.
> +
> +partition
> +^^^^^^^^^
> +
> +The partition driver is a platform device driver whose life cycle is managed by
> +root and does not have real IO mem or IRQ resources. It's part of the
> +infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
> +
> +* manages one or more leaf drivers so that multiple leaves can be managed as a group
> +* provides access to root from leaves, so that parent calls, event notifications
> + and inter-leaf calls can happen
> +
> +In xmgmt, an initial partition driver instance will be created by root, which
> +contains leaves that will trigger partition instances to be created to manage
> +groups of leaves found on different partitions on hardware, such as VSEC, Shell,
> +and User.
> +
> +leaves
> +^^^^^^
> +
> +The leaf driver is a platform device driver whose life cycle is managed by
> +a partition driver and may or may not have real IO mem or IRQ resources. They
> +are the real meat of xmgmt and contains platform specific code to Shell and User
> +found on a MPF.
> +
> +A leaf driver may not have real hardware resources when it merely acts as a driver
> +that manages certain in-memory states for xmgmt. These in-memory states could be
> +shared by multiple other leaves.
> +
> +Leaf drivers assigned to specific hardware resources drive specific subsystem in
> +the device. To manipulate the subsystem or carry out a task, a leaf driver may ask
> +help from root via parent calls and/or from other leaves via inter-leaf calls.
> +
> +A leaf can also broadcast events through infrastructure code for other leaves
> +to process. It can also receive event notification from infrastructure about certain
> +events, such as post-creation or pre-exit of a particular leaf.
> +
> +
> +Driver Interfaces
> +=================
> +
> +xmgmt Driver Ioctls
> +-------------------
> +
> +Ioctls exposed by xmgmt driver to user space are enumerated in the following table:
> +
> +== ===================== ============================= ===========================
> +# Functionality ioctl request code data format
> +== ===================== ============================= ===========================
> +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
> +2 CL frequency scaling XMGMT_IOCFREQSCALE xmgmt_ioc_freqscaling
> +== ===================== ============================= ===========================
> +
> +xmgmt Driver Sysfs
> +------------------
> +
> +xmgmt driver exposes a rich set of sysfs interfaces. Subsystem platform drivers
> +export sysfs node for every platform instance.
> +
> +Every partition also exports its UUIDs. See below for examples::
> +
> + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> +
> +
> +hwmon
> +-----
> +
> +xmgmt driver exposes standard hwmon interface to report voltage, current, temperature,
> +power, etc. These can easily be viewed using *sensors* command line utility.
> +
> +
> +mailbox
> +-------
> +
> +xmgmt communicates with user physical function driver via HW mailbox. Mailbox opcodes
> +are defined in ``mailbox_proto.h``. `Mailbox Inter-domain Communication Protocol
> +<https://xilinx.github.io/XRT/master/html/mailbox.proto.html>`_ defines the full
> +specification. xmgmt implements subset of the specification. It provides the following
> +services to the UPF driver:
> +
> +1. Responding to *are you there* request including determining if the two drivers are
> + running in the same OS domain
> +2. Provide sensor readings, loaded xclbin UUID, clock frequency, shell information, etc.
> +3. Perform PCIe hot reset
> +4. Download user compiled xclbin
Is this gonna use the mailbox framework?
> +
> +
> +Platform Security Considerations
> +================================
> +
> +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
> +discusses the deployment options and security implications in great detail.
> --
> 2.17.1
That's a lot of text, I'll have to read it again most likely,
- Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
2020-11-30 18:08 ` [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Rob Herring
@ 2020-12-01 19:39 ` Sonal Santan
0 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-12-01 19:39 UTC (permalink / raw)
To: Rob Herring
Cc: linux-kernel, linux-fpga, Max Zhen, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
Hi,
> -----Original Message-----
> From: Rob Herring <robh@kernel.org>
> Sent: Monday, November 30, 2020 10:09 AM
> To: Sonal Santan <sonals@xilinx.com>
> Cc: linux-kernel@vger.kernel.org; Sonal Santan <sonals@xilinx.com>; linux-
> fpga@vger.kernel.org; Max Zhen <maxz@xilinx.com>; Lizhi Hou
> <lizhih@xilinx.com>; Michal Simek <michals@xilinx.com>; Stefano Stabellini
> <stefanos@xilinx.com>; devicetree@vger.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
>
> On Sat, Nov 28, 2020 at 5:02 PM Sonal Santan <sonal.santan@xilinx.com>
> wrote:
> >
> > Hello,
> >
> > This patch series adds management physical function driver for Xilinx
> > Alveo PCIe accelerator cards,
> > https://www.xilinx.com/products/boards-and-kits/alveo.html
> > This driver is part of Xilinx Runtime (XRT) open source stack.
> >
> > The patch depends on the "PATCH Xilinx Alveo libfdt prep" which was
> > posted before.
> >
> > ALVEO PLATFORM ARCHITECTURE
> >
> > Alveo PCIe FPGA based platforms have a static *shell* partition and a
> > partial re-configurable *user* partition. The shell partition is
> > automatically loaded from flash when host is booted and PCIe is
> > enumerated by BIOS. Shell cannot be changed till the next cold reboot. The
> shell exposes two PCIe physical functions:
> >
> > 1. management physical function
> > 2. user physical function
> >
> > The patch series includes Documentation/xrt.rst which describes Alveo
> > platform, xmgmt driver architecture and deployment model in more more
> detail.
> >
> > Users compile their high level design in C/C++/OpenCL or RTL into FPGA
> > image using Vitis
> > https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> > tools. The image is packaged as xclbin and contains partial bitstream
> > for the user partition and necessary metadata. Users can dynamically
> > swap the image running on the user partition in order to switch between
> different workloads.
> >
> > ALVEO DRIVERS
> >
> > Alveo Linux kernel driver *xmgmt* binds to management physical
> > function of Alveo platform. The modular driver framework is organized
> > into several platform drivers which primarily handle the following
> functionality:
> >
> > 1. Loading firmware container also called xsabin at driver attach
> > time 2. Loading of user compiled xclbin with FPGA Manager integration
> > 3. Clock scaling of image running on user partition 4. In-band
> > sensors: temp, voltage, power, etc.
> > 5. Device reset and rescan
> > 6. Flashing static *shell* partition
> >
> > The platform drivers are packaged into *xrt-lib* helper module with a
> > well defined interfaces the details of which can be found in
> Documentation/xrt.rst.
> >
> > xmgmt driver is second generation Alveo management driver and
> > evolution of the first generation (out of tree) Alveo management
> > driver, xclmgmt. The sources of the first generation drivers were
> > posted on LKML last year--
> > https://lore.kernel.org/lkml/20190319215401.6562-1-sonal.santan@xilinx
> > .com/
> >
> > Changes since the first generation driver include the following: the
> > driver has been re-architected as data driven modular driver; the
> > driver has been split into xmgmt and xrt-lib; user physical function
> > driver has been removed from the patch series.
> >
> > Alveo/XRT security and platform architecture is documented on the
> > following GitHub pages:
> > https://xilinx.github.io/XRT/master/html/security.html
> > https://xilinx.github.io/XRT/master/html/platforms_partitions.html
> >
> > User physical function driver is not included in this patch series.
> >
> > TESTING AND VALIDATION
> >
> > xmgmt driver can be tested with full XRT open source stack which
> > includes user space libraries, board utilities and (out of tree) first
> > generation user physical function driver xocl. XRT open source runtime
> > stack is available at https://github.com/Xilinx/XRT. This patch series
> > has been validated on Alveo U50 platform.
> >
> > Complete documentation for XRT open source stack can be found here--
> > https://xilinx.github.io/XRT/master/html/index.html
>
> I've not gotten into the patch details, but I'm not clear on what the lifecycle of
> the DT looks like here. What's the starting point and what manipulations to the
> DT are being done? I'm trying to understand if using libfdt is the right way
> versus operating on an unflattened tree.
The DT is created when *xmgmt* driver attaches to the device and reads the
xsabin. The xsabin defines the shell and HW subsystems contained in the shell.
Since the shell is live for the lifetime of the driver the DT is captured in the
partition subdev. The partition then looks for "addressable_endpoints" node and
walks the list of child end point nodes each of which is then copied into its own
instance of subdev. The life cycle of the copied nodes is same as the owning subdev.
All the DT nodes are released when the partition together with its child subdevs
goes away which happens when the driver is unloaded.
xmgmt driver also collects all end points which advertise "pcie_physical_function
= <0x01>" and then constructs a DT on the fly which is then sent to the user
physical function driver via mailbox. This requires support for manipulating device
tree nodes.
In the next revision of the driver we would like to add support for a variation of
platform which has three partitions: base, shell and user--
https://xilinx.github.io/XRT/master/html/platforms_partitions.html#two-stage-platforms
In this model, *base* is initialized like described above. However *shell* can be
changed dynamically by service provider. This means xmgmt would load DT
corresponding to shell partition when shell is loaded and tear it down when shell
is unloaded. The DT corresponding to *base* remains unaffected.
Thanks,
-Sonal
>
> Rob
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver
2020-11-29 0:00 ` [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver Sonal Santan
@ 2020-12-01 20:51 ` Moritz Fischer
[not found] ` <BY5PR02MB60683E3470179E6AD10FEE26B9F20@BY5PR02MB6068.namprd02.prod.outlook.com>
2020-12-02 3:00 ` Xu Yilun
1 sibling, 1 reply; 28+ messages in thread
From: Moritz Fischer @ 2020-12-01 20:51 UTC (permalink / raw)
To: Sonal Santan
Cc: linux-kernel, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
Hi Sonal,
On Sat, Nov 28, 2020 at 04:00:39PM -0800, Sonal Santan wrote:
> From: Sonal Santan <sonal.santan@xilinx.com>
>
> Add management physical function driver core. The driver attaches
> to management physical function of Alveo devices. It instantiates
> the root driver and one or more partition drivers which in turn
> instantiate platform drivers. The instantiation of partition and
> platform drivers is completely data driven. The driver integrates
> with FPGA manager and provides xclbin download service.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> ---
> drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c | 194 ++++
> drivers/fpga/alveo/mgmt/xmgmt-fmgr.h | 29 +
> drivers/fpga/alveo/mgmt/xmgmt-main-impl.h | 36 +
> drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c | 930 +++++++++++++++++++
> drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c | 190 ++++
> drivers/fpga/alveo/mgmt/xmgmt-main.c | 843 +++++++++++++++++
> drivers/fpga/alveo/mgmt/xmgmt-root.c | 375 ++++++++
> 7 files changed, 2597 insertions(+)
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-root.c
>
> diff --git a/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c b/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> new file mode 100644
> index 000000000000..d451b5a2c291
> --- /dev/null
> +++ b/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> @@ -0,0 +1,194 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2019-2020 Xilinx, Inc.
> + * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
> + *
> + * Authors: Sonal.Santan@xilinx.com
> + */
> +
> +#include <linux/cred.h>
> +#include <linux/efi.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/platform_device.h>
> +#include <linux/module.h>
> +#include <linux/vmalloc.h>
> +
> +#include "xrt-subdev.h"
> +#include "xmgmt-fmgr.h"
> +#include "xrt-axigate.h"
> +#include "xmgmt-main-impl.h"
> +
> +/*
> + * Container to capture and cache full xclbin as it is passed in blocks by FPGA
> + * Manager. Driver needs access to full xclbin to walk through xclbin sections.
> + * FPGA Manager's .write() backend sends incremental blocks without any
> + * knowledge of xclbin format forcing us to collect the blocks and stitch them
> + * together here.
> + */
> +
> +struct xfpga_klass {
Nit: xfpga_priv or xfpga_drvdata?
> + const struct platform_device *pdev;
> + struct axlf *blob;
> + char name[64];
Nit: 64 could be a named constant ?
> + size_t count;
> + size_t total_count;
> + struct mutex axlf_lock;
> + int reader_ref;
> + enum fpga_mgr_states state;
> + enum xfpga_sec_level sec_level;
This appears unused, do you want to add this with the code that uses it?
> +};
Maybe add some kerneldoc markup?
> +
> +struct key *xfpga_keys;
Appears unused, can you introduce this together with the code using it?
> +
> +static int xmgmt_pr_write_init(struct fpga_manager *mgr,
> + struct fpga_image_info *info, const char *buf, size_t count)
> +{
> + struct xfpga_klass *obj = mgr->priv;
> + const struct axlf *bin = (const struct axlf *)buf;
Nit: Reverse x-mas tree please.
xxxxxx
xxxx
xxx
x
> +
> + if (count < sizeof(struct axlf)) {
> + obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> + return -EINVAL;
> + }
> +
> + if (count > bin->m_header.m_length) {
> + obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> + return -EINVAL;
> + }
> +
> + /* Free up the previous blob */
> + vfree(obj->blob);
> + obj->blob = vmalloc(bin->m_header.m_length);
> + if (!obj->blob) {
> + obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> + return -ENOMEM;
> + }
> +
> + xrt_info(obj->pdev, "Begin download of xclbin %pUb of length %lld B",
> + &bin->m_header.uuid, bin->m_header.m_length);
We already have framework level prints for that (admittedly somewhat
less verbose). Please remove.
> +
> + obj->count = 0;
> + obj->total_count = bin->m_header.m_length;
> + obj->state = FPGA_MGR_STATE_WRITE_INIT;
Does the framework state tracking not work for you?
> + return 0;
> +}
> +
> +static int xmgmt_pr_write(struct fpga_manager *mgr,
> + const char *buf, size_t count)
> +{
> + struct xfpga_klass *obj = mgr->priv;
> + char *curr = (char *)obj->blob;
> +
> + if ((obj->state != FPGA_MGR_STATE_WRITE_INIT) &&
> + (obj->state != FPGA_MGR_STATE_WRITE)) {
> + obj->state = FPGA_MGR_STATE_WRITE_ERR;
> + return -EINVAL;
> + }
> +
> + curr += obj->count;
> + obj->count += count;
> +
> + /*
> + * The xclbin buffer should not be longer than advertised in the header
> + */
> + if (obj->total_count < obj->count) {
> + obj->state = FPGA_MGR_STATE_WRITE_ERR;
> + return -EINVAL;
> + }
> +
> + xrt_info(obj->pdev, "Copying block of %zu B of xclbin", count);
Please drop those.
> + memcpy(curr, buf, count);
I'm confused. Why are we just copying things around here. What picks
this up afterwards?
> + obj->state = FPGA_MGR_STATE_WRITE;
> + return 0;
> +}
> +
> +
> +static int xmgmt_pr_write_complete(struct fpga_manager *mgr,
> + struct fpga_image_info *info)
> +{
> + int result = 0;
> + struct xfpga_klass *obj = mgr->priv;
> +
> + if (obj->state != FPGA_MGR_STATE_WRITE) {
> + obj->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> + return -EINVAL;
> + }
> +
> + /* Check if we got the complete xclbin */
> + if (obj->blob->m_header.m_length != obj->count) {
> + obj->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> + return -EINVAL;
> + }
> +
> + result = xmgmt_ulp_download((void *)obj->pdev, obj->blob);
> +
> + obj->state = result ? FPGA_MGR_STATE_WRITE_COMPLETE_ERR :
> + FPGA_MGR_STATE_WRITE_COMPLETE;
Why the separate state tracking?
> + xrt_info(obj->pdev, "Finish downloading of xclbin %pUb: %d",
> + &obj->blob->m_header.uuid, result);
> + vfree(obj->blob);
> + obj->blob = NULL;
> + obj->count = 0;
> + return result;
> +}
> +
> +static enum fpga_mgr_states xmgmt_pr_state(struct fpga_manager *mgr)
> +{
> + struct xfpga_klass *obj = mgr->priv;
> +
> + return obj->state;
> +}
> +
> +static const struct fpga_manager_ops xmgmt_pr_ops = {
> + .initial_header_size = sizeof(struct axlf),
> + .write_init = xmgmt_pr_write_init,
> + .write = xmgmt_pr_write,
> + .write_complete = xmgmt_pr_write_complete,
> + .state = xmgmt_pr_state,
> +};
> +
> +
> +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev)
> +{
> + struct fpga_manager *fmgr;
> + int ret = 0;
> + struct xfpga_klass *obj = vzalloc(sizeof(struct xfpga_klass));
> +
> + xrt_info(pdev, "probing...");
Drop this, please.
> + if (!obj)
> + return ERR_PTR(-ENOMEM);
> +
> + snprintf(obj->name, sizeof(obj->name), "Xilinx Alveo FPGA Manager");
> + obj->state = FPGA_MGR_STATE_UNKNOWN;
> + obj->pdev = pdev;
> + fmgr = fpga_mgr_create(&pdev->dev,
> + obj->name,
> + &xmgmt_pr_ops,
> + obj);
I think (eyeballed) this fits on two lines?
> + if (!fmgr)
> + return ERR_PTR(-ENOMEM);
> +
> + obj->sec_level = XFPGA_SEC_NONE;
Seems unused so far, please drop until it's used.
> + ret = fpga_mgr_register(fmgr);
> + if (ret) {
> + fpga_mgr_free(fmgr);
> + kfree(obj);
> + return ERR_PTR(ret);
> + }
> + mutex_init(&obj->axlf_lock);
> + return fmgr;
Since this patchset will wait at least till next cycle, you might want
to look into the devm_* functions for registering and creating FPGA
Managers.
> +}
> +
> +int xmgmt_fmgr_remove(struct fpga_manager *fmgr)
> +{
> + struct xfpga_klass *obj = fmgr->priv;
> +
> + mutex_destroy(&obj->axlf_lock);
> + obj->state = FPGA_MGR_STATE_UNKNOWN;
> + fpga_mgr_unregister(fmgr);
> + vfree(obj->blob);
> + vfree(obj);
> + return 0;
> +}
> diff --git a/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h b/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> new file mode 100644
> index 000000000000..2beba649609f
> --- /dev/null
> +++ b/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> @@ -0,0 +1,29 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2019-2020 Xilinx, Inc.
> + * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
> + *
> + * Authors: Sonal.Santan@xilinx.com
> + */
> +
> +#ifndef _XMGMT_FMGR_H_
> +#define _XMGMT_FMGR_H_
> +
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/mutex.h>
> +
> +#include <linux/xrt/xclbin.h>
> +
> +enum xfpga_sec_level {
> + XFPGA_SEC_NONE = 0,
> + XFPGA_SEC_DEDICATE,
> + XFPGA_SEC_SYSTEM,
> + XFPGA_SEC_MAX = XFPGA_SEC_SYSTEM,
> +};
> +
> +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev);
> +int xmgmt_fmgr_remove(struct fpga_manager *fmgr);
> +
> +#endif
> diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h b/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> new file mode 100644
> index 000000000000..c89024cb8d46
> --- /dev/null
> +++ b/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> @@ -0,0 +1,36 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2020 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XMGMT_MAIN_IMPL_H_
> +#define _XMGMT_MAIN_IMPL_H_
> +
> +#include "xrt-subdev.h"
> +#include "xmgmt-main.h"
> +
> +extern struct platform_driver xmgmt_main_driver;
> +extern struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[];
> +
> +extern int xmgmt_ulp_download(struct platform_device *pdev, const void *xclbin);
> +extern int bitstream_axlf_mailbox(struct platform_device *pdev,
> + const void *xclbin);
> +extern int xmgmt_hot_reset(struct platform_device *pdev);
> +
> +/* Getting dtb for specified partition. Caller should vfree returned dtb .*/
> +extern char *xmgmt_get_dtb(struct platform_device *pdev,
> + enum provider_kind kind);
> +extern char *xmgmt_get_vbnv(struct platform_device *pdev);
> +extern int xmgmt_get_provider_uuid(struct platform_device *pdev,
> + enum provider_kind kind, uuid_t *uuid);
> +
> +extern void *xmgmt_pdev2mailbox(struct platform_device *pdev);
> +extern void *xmgmt_mailbox_probe(struct platform_device *pdev);
> +extern void xmgmt_mailbox_remove(void *handle);
> +extern void xmgmt_peer_notify_state(void *handle, bool online);
> +
> +#endif /* _XMGMT_MAIN_IMPL_H_ */
> diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c b/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> new file mode 100644
> index 000000000000..b3d82fc3618b
> --- /dev/null
> +++ b/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> @@ -0,0 +1,930 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA MGMT PF entry point driver
> + *
> + * Copyright (C) 2020 Xilinx, Inc.
> + *
> + * Peer communication via mailbox
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/crc32c.h>
> +#include <linux/xrt/mailbox_proto.h>
> +#include "xmgmt-main-impl.h"
> +#include "xrt-mailbox.h"
> +#include "xrt-cmc.h"
> +#include "xrt-metadata.h"
> +#include "xrt-xclbin.h"
> +#include "xrt-clock.h"
> +#include "xrt-calib.h"
> +#include "xrt-icap.h"
> +
> +struct xmgmt_mailbox {
> + struct platform_device *pdev;
> + struct platform_device *mailbox;
> + struct mutex lock;
> + void *evt_hdl;
> + char *test_msg;
> + bool peer_in_same_domain;
> +};
> +
> +#define XMGMT_MAILBOX_PRT_REQ(xmbx, send, request, sw_ch) do { \
> + const char *dir = (send) ? ">>>>>" : "<<<<<"; \
> + \
> + if ((request)->req == XCL_MAILBOX_REQ_PEER_DATA) { \
> + struct xcl_mailbox_peer_data *p = \
> + (struct xcl_mailbox_peer_data *)(request)->data;\
> + \
> + xrt_info((xmbx)->pdev, "%s(%s) %s%s", \
> + mailbox_req2name((request)->req), \
> + mailbox_group_kind2name(p->kind), \
> + dir, mailbox_chan2name(sw_ch)); \
> + } else { \
> + xrt_info((xmbx)->pdev, "%s %s%s", \
> + mailbox_req2name((request)->req), \
> + dir, mailbox_chan2name(sw_ch)); \
> + } \
> +} while (0)
> +#define XMGMT_MAILBOX_PRT_REQ_SEND(xmbx, req, sw_ch) \
> + XMGMT_MAILBOX_PRT_REQ(xmbx, true, req, sw_ch)
> +#define XMGMT_MAILBOX_PRT_REQ_RECV(xmbx, req, sw_ch) \
> + XMGMT_MAILBOX_PRT_REQ(xmbx, false, req, sw_ch)
> +#define XMGMT_MAILBOX_PRT_RESP(xmbx, resp) \
> + xrt_info((xmbx)->pdev, "respond %ld bytes >>>>>%s", \
> + (resp)->xmip_data_size, mailbox_chan2name((resp)->xmip_sw_ch))
> +
> +static inline struct xmgmt_mailbox *pdev2mbx(struct platform_device *pdev)
> +{
> + return (struct xmgmt_mailbox *)xmgmt_pdev2mailbox(pdev);
> +}
> +
> +static void xmgmt_mailbox_post(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch, void *buf, size_t len)
> +{
> + int rc;
> + struct xrt_mailbox_ioctl_post post = {
> + .xmip_req_id = msgid,
> + .xmip_sw_ch = sw_ch,
> + .xmip_data = buf,
> + .xmip_data_size = len
> + };
> +
> + BUG_ON(!mutex_is_locked(&xmbx->lock));
> +
> + if (!xmbx->mailbox) {
> + xrt_err(xmbx->pdev, "mailbox not available");
> + return;
> + }
> +
> + if (msgid == 0) {
> + XMGMT_MAILBOX_PRT_REQ_SEND(xmbx,
> + (struct xcl_mailbox_req *)buf, sw_ch);
> + } else {
> + XMGMT_MAILBOX_PRT_RESP(xmbx, &post);
> + }
> +
> + rc = xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_POST, &post);
> + if (rc)
> + xrt_err(xmbx->pdev, "failed to post msg: %d", rc);
> +}
> +
> +static void xmgmt_mailbox_notify(struct xmgmt_mailbox *xmbx, bool sw_ch,
> + struct xcl_mailbox_req *req, size_t len)
> +{
> + xmgmt_mailbox_post(xmbx, 0, sw_ch, req, len);
> +}
> +
> +static void xmgmt_mailbox_respond(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch, void *buf, size_t len)
> +{
> + mutex_lock(&xmbx->lock);
> + xmgmt_mailbox_post(xmbx, msgid, sw_ch, buf, len);
> + mutex_unlock(&xmbx->lock);
> +}
> +
> +static void xmgmt_mailbox_resp_test_msg(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + char *msg;
> +
> + mutex_lock(&xmbx->lock);
> +
> + if (xmbx->test_msg == NULL) {
> + mutex_unlock(&xmbx->lock);
> + xrt_err(pdev, "test msg is not set, drop request");
> + return;
> + }
> + msg = xmbx->test_msg;
> + xmbx->test_msg = NULL;
> +
> + mutex_unlock(&xmbx->lock);
> +
> + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, msg, strlen(msg) + 1);
> + vfree(msg);
> +}
> +
> +static int xmgmt_mailbox_dtb_add_prop(struct platform_device *pdev,
> + char *dst_dtb, const char *ep_name, const char *regmap_name,
> + const char *prop, const void *val, int size)
> +{
> + int rc = xrt_md_set_prop(DEV(pdev), dst_dtb, ep_name, regmap_name,
> + prop, val, size);
> +
> + if (rc) {
> + xrt_err(pdev, "failed to set %s@(%s, %s): %d",
> + ep_name, regmap_name, prop, rc);
> + }
> + return rc;
> +}
> +
> +static int xmgmt_mailbox_dtb_add_vbnv(struct platform_device *pdev, char *dtb)
> +{
> + int rc = 0;
> + char *vbnv = xmgmt_get_vbnv(pdev);
> +
> + if (vbnv == NULL) {
> + xrt_err(pdev, "failed to get VBNV");
> + return -ENOENT;
> + }
> + rc = xmgmt_mailbox_dtb_add_prop(pdev, dtb, NULL, NULL,
> + PROP_VBNV, vbnv, strlen(vbnv) + 1);
> + kfree(vbnv);
> + return rc;
> +}
> +
> +static int xmgmt_mailbox_dtb_copy_logic_uuid(struct platform_device *pdev,
> + const char *src_dtb, char *dst_dtb)
> +{
> + const void *val;
> + int sz;
> + int rc = xrt_md_get_prop(DEV(pdev), src_dtb, NULL, NULL,
> + PROP_LOGIC_UUID, &val, &sz);
> +
> + if (rc) {
> + xrt_err(pdev, "failed to get %s: %d", PROP_LOGIC_UUID, rc);
> + return rc;
> + }
> + return xmgmt_mailbox_dtb_add_prop(pdev, dst_dtb, NULL, NULL,
> + PROP_LOGIC_UUID, val, sz);
> +}
> +
> +static int xmgmt_mailbox_dtb_add_vrom(struct platform_device *pdev,
> + const char *src_dtb, char *dst_dtb)
> +{
> + /* For compatibility for legacy xrt driver. */
> + enum FeatureBitMask {
> + UNIFIED_PLATFORM = 0x0000000000000001
> + , XARE_ENBLD = 0x0000000000000002
> + , BOARD_MGMT_ENBLD = 0x0000000000000004
> + , MB_SCHEDULER = 0x0000000000000008
> + , PROM_MASK = 0x0000000000000070
> + , DEBUG_MASK = 0x000000000000FF00
> + , PEER_TO_PEER = 0x0000000000010000
> + , FBM_UUID = 0x0000000000020000
> + , HBM = 0x0000000000040000
> + , CDMA = 0x0000000000080000
> + , QDMA = 0x0000000000100000
> + , RUNTIME_CLK_SCALE = 0x0000000000200000
> + , PASSTHROUGH_VIRTUALIZATION = 0x0000000000400000
> + };
> + struct FeatureRomHeader {
> + unsigned char EntryPointString[4];
> + uint8_t MajorVersion;
> + uint8_t MinorVersion;
> + uint32_t VivadoBuildID;
> + uint32_t IPBuildID;
> + uint64_t TimeSinceEpoch;
> + unsigned char FPGAPartName[64];
> + unsigned char VBNVName[64];
> + uint8_t DDRChannelCount;
> + uint8_t DDRChannelSize;
> + uint64_t DRBaseAddress;
> + uint64_t FeatureBitMap;
> + unsigned char uuid[16];
> + uint8_t HBMCount;
> + uint8_t HBMSize;
> + uint32_t CDMABaseAddress[4];
> + } header = { 0 };
> + char *vbnv = xmgmt_get_vbnv(pdev);
> + int rc;
> +
> + *(u32 *)header.EntryPointString = 0x786e6c78;
> +
> + if (vbnv)
> + strncpy(header.VBNVName, vbnv, sizeof(header.VBNVName) - 1);
> + kfree(vbnv);
> +
> + header.FeatureBitMap = UNIFIED_PLATFORM;
> + rc = xrt_md_get_prop(DEV(pdev), src_dtb,
> + NODE_CMC_FW_MEM, NULL, PROP_IO_OFFSET, NULL, NULL);
> + if (rc == 0)
> + header.FeatureBitMap |= BOARD_MGMT_ENBLD;
> + rc = xrt_md_get_prop(DEV(pdev), src_dtb,
> + NODE_ERT_FW_MEM, NULL, PROP_IO_OFFSET, NULL, NULL);
> + if (rc == 0)
> + header.FeatureBitMap |= MB_SCHEDULER;
> +
> + return xmgmt_mailbox_dtb_add_prop(pdev, dst_dtb, NULL, NULL,
> + PROP_VROM, &header, sizeof(header));
> +}
> +
> +static u32 xmgmt_mailbox_dtb_user_pf(struct platform_device *pdev,
> + const char *dtb, const char *epname, const char *regmap)
> +{
> + const u32 *pfnump;
> + int rc = xrt_md_get_prop(DEV(pdev), dtb, epname, regmap,
> + PROP_PF_NUM, (const void **)&pfnump, NULL);
> +
> + if (rc)
> + return -1;
> + return be32_to_cpu(*pfnump);
> +}
> +
> +static int xmgmt_mailbox_dtb_copy_user_endpoints(struct platform_device *pdev,
> + const char *src, char *dst)
> +{
> + int rc = 0;
> + char *epname = NULL, *regmap = NULL;
> + u32 pfnum = xmgmt_mailbox_dtb_user_pf(pdev, src,
> + NODE_MAILBOX_USER, NULL);
> + const u32 level = cpu_to_be32(1);
> + struct device *dev = DEV(pdev);
> +
> + if (pfnum == (u32)-1) {
> + xrt_err(pdev, "failed to get user pf num");
> + rc = -EINVAL;
> + }
> +
> + for (xrt_md_get_next_endpoint(dev, src, NULL, NULL, &epname, ®map);
> + rc == 0 && epname != NULL;
> + xrt_md_get_next_endpoint(dev, src, epname, regmap,
> + &epname, ®map)) {
> + if (pfnum !=
> + xmgmt_mailbox_dtb_user_pf(pdev, src, epname, regmap))
> + continue;
> + rc = xrt_md_copy_endpoint(dev, dst, src, epname, regmap, NULL);
> + if (rc) {
> + xrt_err(pdev, "failed to copy (%s, %s): %d",
> + epname, regmap, rc);
> + } else {
> + rc = xrt_md_set_prop(dev, dst, epname, regmap,
> + PROP_PARTITION_LEVEL, &level, sizeof(level));
> + if (rc) {
> + xrt_err(pdev,
> + "can't set level for (%s, %s): %d",
> + epname, regmap, rc);
> + }
> + }
> + }
> + return rc;
> +}
> +
> +static char *xmgmt_mailbox_user_dtb(struct platform_device *pdev)
> +{
> + /* TODO: add support for PLP. */
> + const char *src = NULL;
> + char *dst = NULL;
> + struct device *dev = DEV(pdev);
> + int rc = xrt_md_create(dev, &dst);
> +
> + if (rc || dst == NULL)
> + return NULL;
> +
> + rc = xmgmt_mailbox_dtb_add_vbnv(pdev, dst);
> + if (rc)
> + goto fail;
> +
> + src = xmgmt_get_dtb(pdev, XMGMT_BLP);
> + if (src == NULL) {
> + xrt_err(pdev, "failed to get BLP dtb");
> + goto fail;
> + }
> +
> + rc = xmgmt_mailbox_dtb_copy_logic_uuid(pdev, src, dst);
> + if (rc)
> + goto fail;
> +
> + rc = xmgmt_mailbox_dtb_add_vrom(pdev, src, dst);
> + if (rc)
> + goto fail;
> +
> + rc = xrt_md_copy_endpoint(dev, dst, src, NODE_PARTITION_INFO,
> + NULL, NODE_PARTITION_INFO_BLP);
> + if (rc)
> + goto fail;
> +
> + rc = xrt_md_copy_endpoint(dev, dst, src, NODE_INTERFACES, NULL, NULL);
> + if (rc)
> + goto fail;
> +
> + rc = xmgmt_mailbox_dtb_copy_user_endpoints(pdev, src, dst);
> + if (rc)
> + goto fail;
> +
> + xrt_md_pack(dev, dst);
> + vfree(src);
> + return dst;
> +
> +fail:
> + vfree(src);
> + vfree(dst);
> + return NULL;
> +}
> +
> +static void xmgmt_mailbox_resp_subdev(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch, u64 offset, u64 size)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + char *dtb = xmgmt_mailbox_user_dtb(pdev);
> + long dtbsz;
> + struct xcl_subdev *hdr;
> + u64 totalsz;
> +
> + if (dtb == NULL)
> + return;
> +
> + dtbsz = xrt_md_size(DEV(pdev), dtb);
> + totalsz = dtbsz + sizeof(*hdr) - sizeof(hdr->data);
> + if (offset != 0 || totalsz > size) {
> + /* Only support fetching dtb in one shot. */
> + vfree(dtb);
> + xrt_err(pdev, "need %lldB, user buffer size is %lldB, dropped",
> + totalsz, size);
> + return;
> + }
> +
> + hdr = vzalloc(totalsz);
> + if (hdr == NULL) {
> + vfree(dtb);
> + return;
> + }
> +
> + hdr->ver = 1;
> + hdr->size = dtbsz;
> + hdr->rtncode = XRT_MSG_SUBDEV_RTN_COMPLETE;
> + (void) memcpy(hdr->data, dtb, dtbsz);
> +
> + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, hdr, totalsz);
> +
> + vfree(dtb);
> + vfree(hdr);
> +}
> +
> +static void xmgmt_mailbox_resp_sensor(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch, u64 offset, u64 size)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + struct xcl_sensor sensors = { 0 };
> + struct platform_device *cmcpdev = xrt_subdev_get_leaf_by_id(pdev,
> + XRT_SUBDEV_CMC, PLATFORM_DEVID_NONE);
> + int rc;
> +
> + if (cmcpdev) {
> + rc = xrt_subdev_ioctl(cmcpdev, XRT_CMC_READ_SENSORS, &sensors);
> + (void) xrt_subdev_put_leaf(pdev, cmcpdev);
> + if (rc)
> + xrt_err(pdev, "can't read sensors: %d", rc);
> + }
> +
> + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &sensors,
> + min((u64)sizeof(sensors), size));
> +}
> +
> +static int xmgmt_mailbox_get_freq(struct xmgmt_mailbox *xmbx,
> + enum CLOCK_TYPE type, u64 *freq, u64 *freq_cnter)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + const char *clkname =
> + clock_type2epname(type) ? clock_type2epname(type) : "UNKNOWN";
> + struct platform_device *clkpdev =
> + xrt_subdev_get_leaf_by_epname(pdev, clkname);
> + int rc;
> + struct xrt_clock_ioctl_get getfreq = { 0 };
> +
> + if (clkpdev == NULL) {
> + xrt_info(pdev, "%s clock is not available", clkname);
> + return -ENOENT;
> + }
> +
> + rc = xrt_subdev_ioctl(clkpdev, XRT_CLOCK_GET, &getfreq);
> + (void) xrt_subdev_put_leaf(pdev, clkpdev);
> + if (rc) {
> + xrt_err(pdev, "can't get %s clock frequency: %d", clkname, rc);
> + return rc;
> + }
> +
> + if (freq)
> + *freq = getfreq.freq;
> + if (freq_cnter)
> + *freq_cnter = getfreq.freq_cnter;
> + return 0;
> +}
> +
> +static int xmgmt_mailbox_get_icap_idcode(struct xmgmt_mailbox *xmbx, u64 *id)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + struct platform_device *icappdev = xrt_subdev_get_leaf_by_id(pdev,
> + XRT_SUBDEV_ICAP, PLATFORM_DEVID_NONE);
> + int rc;
> +
> + if (icappdev == NULL) {
> + xrt_err(pdev, "can't find icap");
> + return -ENOENT;
> + }
> +
> + rc = xrt_subdev_ioctl(icappdev, XRT_ICAP_IDCODE, id);
> + (void) xrt_subdev_put_leaf(pdev, icappdev);
> + if (rc)
> + xrt_err(pdev, "can't get icap idcode: %d", rc);
> + return rc;
> +}
> +
> +static int xmgmt_mailbox_get_mig_calib(struct xmgmt_mailbox *xmbx, u64 *calib)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + struct platform_device *calibpdev = xrt_subdev_get_leaf_by_id(pdev,
> + XRT_SUBDEV_CALIB, PLATFORM_DEVID_NONE);
> + int rc;
> + enum xrt_calib_results res;
> +
> + if (calibpdev == NULL) {
> + xrt_err(pdev, "can't find mig calibration subdev");
> + return -ENOENT;
> + }
> +
> + rc = xrt_subdev_ioctl(calibpdev, XRT_CALIB_RESULT, &res);
> + (void) xrt_subdev_put_leaf(pdev, calibpdev);
> + if (rc) {
> + xrt_err(pdev, "can't get mig calibration result: %d", rc);
> + } else {
> + if (res == XRT_CALIB_SUCCEEDED)
> + *calib = 1;
> + else
> + *calib = 0;
> + }
> + return rc;
> +}
> +
> +static void xmgmt_mailbox_resp_icap(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch, u64 offset, u64 size)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + struct xcl_pr_region icap = { 0 };
> +
> + (void) xmgmt_mailbox_get_freq(xmbx,
> + CT_DATA, &icap.freq_data, &icap.freq_cntr_data);
> + (void) xmgmt_mailbox_get_freq(xmbx,
> + CT_KERNEL, &icap.freq_kernel, &icap.freq_cntr_kernel);
> + (void) xmgmt_mailbox_get_freq(xmbx,
> + CT_SYSTEM, &icap.freq_system, &icap.freq_cntr_system);
> + (void) xmgmt_mailbox_get_icap_idcode(xmbx, &icap.idcode);
> + (void) xmgmt_mailbox_get_mig_calib(xmbx, &icap.mig_calib);
> + BUG_ON(sizeof(icap.uuid) != sizeof(uuid_t));
> + (void) xmgmt_get_provider_uuid(pdev, XMGMT_ULP, (uuid_t *)&icap.uuid);
> +
> + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &icap,
> + min((u64)sizeof(icap), size));
> +}
> +
> +static void xmgmt_mailbox_resp_bdinfo(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch, u64 offset, u64 size)
> +{
> + struct platform_device *pdev = xmbx->pdev;
> + struct xcl_board_info *info = vzalloc(sizeof(*info));
> + struct platform_device *cmcpdev;
> + int rc;
> +
> + if (info == NULL)
> + return;
> +
> + cmcpdev = xrt_subdev_get_leaf_by_id(pdev,
> + XRT_SUBDEV_CMC, PLATFORM_DEVID_NONE);
> + if (cmcpdev) {
> + rc = xrt_subdev_ioctl(cmcpdev, XRT_CMC_READ_BOARD_INFO, info);
> + (void) xrt_subdev_put_leaf(pdev, cmcpdev);
> + if (rc)
> + xrt_err(pdev, "can't read board info: %d", rc);
> + }
> +
> + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, info,
> + min((u64)sizeof(*info), size));
> +
> + vfree(info);
> +}
> +
> +static void xmgmt_mailbox_simple_respond(struct xmgmt_mailbox *xmbx,
> + u64 msgid, bool sw_ch, int rc)
> +{
> + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &rc, sizeof(rc));
> +}
> +
> +static void xmgmt_mailbox_resp_peer_data(struct xmgmt_mailbox *xmbx,
> + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> +{
> + struct xcl_mailbox_peer_data *pdata =
> + (struct xcl_mailbox_peer_data *)req->data;
> +
> + if (len < (sizeof(*req) + sizeof(*pdata) - 1)) {
> + xrt_err(xmbx->pdev, "received corrupted %s, dropped",
> + mailbox_req2name(req->req));
> + return;
> + }
> +
> + switch (pdata->kind) {
> + case XCL_SENSOR:
> + xmgmt_mailbox_resp_sensor(xmbx, msgid, sw_ch,
> + pdata->offset, pdata->size);
> + break;
> + case XCL_ICAP:
> + xmgmt_mailbox_resp_icap(xmbx, msgid, sw_ch,
> + pdata->offset, pdata->size);
> + break;
> + case XCL_BDINFO:
> + xmgmt_mailbox_resp_bdinfo(xmbx, msgid, sw_ch,
> + pdata->offset, pdata->size);
> + break;
> + case XCL_SUBDEV:
> + xmgmt_mailbox_resp_subdev(xmbx, msgid, sw_ch,
> + pdata->offset, pdata->size);
> + break;
> + case XCL_MIG_ECC:
> + case XCL_FIREWALL:
> + case XCL_DNA: /* TODO **/
> + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, 0);
> + break;
> + default:
> + xrt_err(xmbx->pdev, "%s(%s) request not handled",
> + mailbox_req2name(req->req),
> + mailbox_group_kind2name(pdata->kind));
> + break;
> + }
> +}
> +
> +static bool xmgmt_mailbox_is_same_domain(struct xmgmt_mailbox *xmbx,
> + struct xcl_mailbox_conn *mb_conn)
> +{
> + uint32_t crc_chk;
> + phys_addr_t paddr;
> + struct platform_device *pdev = xmbx->pdev;
> +
> + paddr = virt_to_phys((void *)mb_conn->kaddr);
> + if (paddr != (phys_addr_t)mb_conn->paddr) {
> + xrt_info(pdev, "paddrs differ, user 0x%llx, mgmt 0x%llx",
> + mb_conn->paddr, paddr);
> + return false;
> + }
> +
> + crc_chk = crc32c_le(~0, (void *)mb_conn->kaddr, PAGE_SIZE);
> + if (crc_chk != mb_conn->crc32) {
> + xrt_info(pdev, "CRCs differ, user 0x%x, mgmt 0x%x",
> + mb_conn->crc32, crc_chk);
> + return false;
> + }
> +
> + return true;
> +}
> +
> +static void xmgmt_mailbox_resp_user_probe(struct xmgmt_mailbox *xmbx,
> + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> +{
> + struct xcl_mailbox_conn_resp *resp = vzalloc(sizeof(*resp));
> + struct xcl_mailbox_conn *conn = (struct xcl_mailbox_conn *)req->data;
> +
> + if (resp == NULL)
> + return;
> +
> + if (len < (sizeof(*req) + sizeof(*conn) - 1)) {
> + xrt_err(xmbx->pdev, "received corrupted %s, dropped",
> + mailbox_req2name(req->req));
> + vfree(resp);
> + return;
> + }
> +
> + resp->conn_flags |= XCL_MB_PEER_READY;
> + if (xmgmt_mailbox_is_same_domain(xmbx, conn)) {
> + xmbx->peer_in_same_domain = true;
> + resp->conn_flags |= XCL_MB_PEER_SAME_DOMAIN;
> + }
> +
> + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, resp, sizeof(*resp));
> + vfree(resp);
> +}
> +
> +static void xmgmt_mailbox_resp_hot_reset(struct xmgmt_mailbox *xmbx,
> + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> +{
> + int ret;
> + struct platform_device *pdev = xmbx->pdev;
> +
> + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, 0);
> +
> + ret = xmgmt_hot_reset(pdev);
> + if (ret)
> + xrt_err(pdev, "failed to hot reset: %d", ret);
> + else
> + xmgmt_peer_notify_state(xmbx, true);
> +}
> +
> +static void xmgmt_mailbox_resp_load_xclbin(struct xmgmt_mailbox *xmbx,
> + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> +{
> + struct xcl_mailbox_bitstream_kaddr *kaddr =
> + (struct xcl_mailbox_bitstream_kaddr *)req->data;
> + void *xclbin = (void *)(uintptr_t)kaddr->addr;
> + int ret = bitstream_axlf_mailbox(xmbx->pdev, xclbin);
> +
> + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, ret);
> +}
> +
> +static void xmgmt_mailbox_listener(void *arg, void *data, size_t len,
> + u64 msgid, int err, bool sw_ch)
> +{
> + struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)arg;
> + struct platform_device *pdev = xmbx->pdev;
> + struct xcl_mailbox_req *req = (struct xcl_mailbox_req *)data;
> +
> + if (err) {
> + xrt_err(pdev, "failed to receive request: %d", err);
> + return;
> + }
> + if (len < sizeof(*req)) {
> + xrt_err(pdev, "received corrupted request");
> + return;
> + }
> +
> + XMGMT_MAILBOX_PRT_REQ_RECV(xmbx, req, sw_ch);
> + switch (req->req) {
> + case XCL_MAILBOX_REQ_TEST_READ:
> + xmgmt_mailbox_resp_test_msg(xmbx, msgid, sw_ch);
> + break;
> + case XCL_MAILBOX_REQ_PEER_DATA:
> + xmgmt_mailbox_resp_peer_data(xmbx, req, len, msgid, sw_ch);
> + break;
> + case XCL_MAILBOX_REQ_READ_P2P_BAR_ADDR: /* TODO */
> + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, -ENOTSUPP);
> + break;
> + case XCL_MAILBOX_REQ_USER_PROBE:
> + xmgmt_mailbox_resp_user_probe(xmbx, req, len, msgid, sw_ch);
> + break;
> + case XCL_MAILBOX_REQ_HOT_RESET:
> + xmgmt_mailbox_resp_hot_reset(xmbx, req, len, msgid, sw_ch);
> + break;
> + case XCL_MAILBOX_REQ_LOAD_XCLBIN_KADDR:
> + if (xmbx->peer_in_same_domain) {
> + xmgmt_mailbox_resp_load_xclbin(xmbx,
> + req, len, msgid, sw_ch);
> + } else {
> + xrt_err(pdev, "%s not handled, not in same domain",
> + mailbox_req2name(req->req));
> + }
> + break;
> + default:
> + xrt_err(pdev, "%s(%d) request not handled",
> + mailbox_req2name(req->req), req->req);
> + break;
> + }
> +}
> +
> +static void xmgmt_mailbox_reg_listener(struct xmgmt_mailbox *xmbx)
> +{
> + struct xrt_mailbox_ioctl_listen listen = {
> + xmgmt_mailbox_listener, xmbx };
> +
> + BUG_ON(!mutex_is_locked(&xmbx->lock));
> + if (!xmbx->mailbox)
> + return;
> + (void) xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_LISTEN, &listen);
> +}
> +
> +static void xmgmt_mailbox_unreg_listener(struct xmgmt_mailbox *xmbx)
> +{
> + struct xrt_mailbox_ioctl_listen listen = { 0 };
> +
> + BUG_ON(!mutex_is_locked(&xmbx->lock));
> + BUG_ON(!xmbx->mailbox);
> + (void) xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_LISTEN, &listen);
> +}
> +
> +static bool xmgmt_mailbox_leaf_match(enum xrt_subdev_id id,
> + struct platform_device *pdev, void *arg)
> +{
> + return (id == XRT_SUBDEV_MAILBOX);
> +}
> +
> +static int xmgmt_mailbox_event_cb(struct platform_device *pdev,
> + enum xrt_events evt, void *arg)
> +{
> + struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
> + struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev *)arg;
> +
> + switch (evt) {
> + case XRT_EVENT_POST_CREATION:
> + BUG_ON(esd->xevt_subdev_id != XRT_SUBDEV_MAILBOX);
> + BUG_ON(xmbx->mailbox);
> + mutex_lock(&xmbx->lock);
> + xmbx->mailbox = xrt_subdev_get_leaf_by_id(pdev,
> + XRT_SUBDEV_MAILBOX, PLATFORM_DEVID_NONE);
> + xmgmt_mailbox_reg_listener(xmbx);
> + mutex_unlock(&xmbx->lock);
> + break;
> + case XRT_EVENT_PRE_REMOVAL:
> + BUG_ON(esd->xevt_subdev_id != XRT_SUBDEV_MAILBOX);
> + BUG_ON(!xmbx->mailbox);
> + mutex_lock(&xmbx->lock);
> + xmgmt_mailbox_unreg_listener(xmbx);
> + (void) xrt_subdev_put_leaf(pdev, xmbx->mailbox);
> + xmbx->mailbox = NULL;
> + mutex_unlock(&xmbx->lock);
> + break;
> + default:
> + break;
> + }
> +
> + return XRT_EVENT_CB_CONTINUE;
> +}
> +
> +static ssize_t xmgmt_mailbox_user_dtb_show(struct file *filp,
> + struct kobject *kobj, struct bin_attribute *attr,
> + char *buf, loff_t off, size_t count)
> +{
> + struct device *dev = kobj_to_dev(kobj);
> + struct platform_device *pdev = to_platform_device(dev);
> + char *blob = NULL;
> + long size;
> + ssize_t ret = 0;
> +
> + blob = xmgmt_mailbox_user_dtb(pdev);
> + if (!blob) {
> + ret = -ENOENT;
> + goto failed;
> + }
> +
> + size = xrt_md_size(dev, blob);
> + if (size <= 0) {
> + ret = -EINVAL;
> + goto failed;
> + }
> +
> + if (off >= size)
> + goto failed;
> + if (off + count > size)
> + count = size - off;
> + memcpy(buf, blob + off, count);
> +
> + ret = count;
> +failed:
> + vfree(blob);
> + return ret;
> +}
> +
> +static struct bin_attribute meta_data_attr = {
> + .attr = {
> + .name = "metadata_for_user",
> + .mode = 0400
> + },
> + .read = xmgmt_mailbox_user_dtb_show,
> + .size = 0
> +};
> +
> +static struct bin_attribute *xmgmt_mailbox_bin_attrs[] = {
> + &meta_data_attr,
> + NULL,
> +};
> +
> +int xmgmt_mailbox_get_test_msg(struct xmgmt_mailbox *xmbx, bool sw_ch,
> + char *buf, size_t *len)
> +{
> + int rc;
> + struct platform_device *pdev = xmbx->pdev;
> + struct xcl_mailbox_req req = { 0, XCL_MAILBOX_REQ_TEST_READ, };
> + struct xrt_mailbox_ioctl_request leaf_req = {
> + .xmir_sw_ch = sw_ch,
> + .xmir_resp_ttl = 1,
> + .xmir_req = &req,
> + .xmir_req_size = sizeof(req),
> + .xmir_resp = buf,
> + .xmir_resp_size = *len
> + };
> +
> + mutex_lock(&xmbx->lock);
> + if (xmbx->mailbox) {
> + XMGMT_MAILBOX_PRT_REQ_SEND(xmbx, &req, leaf_req.xmir_sw_ch);
> + /*
> + * mgmt should never send request to peer. it should send
> + * either notification or response. here is the only exception
> + * for debugging purpose.
> + */
> + rc = xrt_subdev_ioctl(xmbx->mailbox,
> + XRT_MAILBOX_REQUEST, &leaf_req);
> + } else {
> + rc = -ENODEV;
> + xrt_err(pdev, "mailbox not available");
> + }
> + mutex_unlock(&xmbx->lock);
> +
> + if (rc == 0)
> + *len = leaf_req.xmir_resp_size;
> + return rc;
> +}
> +
> +int xmgmt_mailbox_set_test_msg(struct xmgmt_mailbox *xmbx,
> + char *buf, size_t len)
> +{
> + mutex_lock(&xmbx->lock);
> +
> + if (xmbx->test_msg)
> + vfree(xmbx->test_msg);
> + xmbx->test_msg = vmalloc(len);
> + if (xmbx->test_msg == NULL) {
> + mutex_unlock(&xmbx->lock);
> + return -ENOMEM;
> + }
> + (void) memcpy(xmbx->test_msg, buf, len);
> +
> + mutex_unlock(&xmbx->lock);
> + return 0;
> +}
> +
> +static ssize_t peer_msg_show(struct device *dev,
> + struct device_attribute *attr, char *buf)
> +{
> + size_t len = 4096;
> + struct platform_device *pdev = to_platform_device(dev);
> + struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
> + int ret = xmgmt_mailbox_get_test_msg(xmbx, false, buf, &len);
> +
> + return ret == 0 ? len : ret;
> +}
> +static ssize_t peer_msg_store(struct device *dev,
> + struct device_attribute *da, const char *buf, size_t count)
> +{
> + struct platform_device *pdev = to_platform_device(dev);
> + struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
> + int ret = xmgmt_mailbox_set_test_msg(xmbx, (char *)buf, count);
> +
> + return ret == 0 ? count : ret;
> +}
> +/* Message test i/f. */
> +static DEVICE_ATTR_RW(peer_msg);
> +
> +static struct attribute *xmgmt_mailbox_attrs[] = {
> + &dev_attr_peer_msg.attr,
> + NULL,
> +};
> +
> +static const struct attribute_group xmgmt_mailbox_attrgroup = {
> + .bin_attrs = xmgmt_mailbox_bin_attrs,
> + .attrs = xmgmt_mailbox_attrs,
> +};
> +
> +void *xmgmt_mailbox_probe(struct platform_device *pdev)
> +{
> + struct xmgmt_mailbox *xmbx =
> + devm_kzalloc(DEV(pdev), sizeof(*xmbx), GFP_KERNEL);
> +
> + if (!xmbx)
> + return NULL;
> + xmbx->pdev = pdev;
> + mutex_init(&xmbx->lock);
> +
> + xmbx->evt_hdl = xrt_subdev_add_event_cb(pdev,
> + xmgmt_mailbox_leaf_match, NULL, xmgmt_mailbox_event_cb);
> + (void) sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_mailbox_attrgroup);
> + return xmbx;
> +}
> +
> +void xmgmt_mailbox_remove(void *handle)
> +{
> + struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)handle;
> + struct platform_device *pdev = xmbx->pdev;
> +
> + (void) sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_mailbox_attrgroup);
> + if (xmbx->evt_hdl)
> + (void) xrt_subdev_remove_event_cb(pdev, xmbx->evt_hdl);
> + if (xmbx->mailbox)
> + (void) xrt_subdev_put_leaf(pdev, xmbx->mailbox);
> + if (xmbx->test_msg)
> + vfree(xmbx->test_msg);
> +}
> +
> +void xmgmt_peer_notify_state(void *handle, bool online)
> +{
> + struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)handle;
> + struct xcl_mailbox_peer_state *st;
> + struct xcl_mailbox_req *req;
> + size_t reqlen = sizeof(*req) + sizeof(*st) - 1;
> +
> + req = vzalloc(reqlen);
> + if (req == NULL)
> + return;
> +
> + req->req = XCL_MAILBOX_REQ_MGMT_STATE;
> + st = (struct xcl_mailbox_peer_state *)req->data;
> + st->state_flags = online ? XCL_MB_STATE_ONLINE : XCL_MB_STATE_OFFLINE;
> + mutex_lock(&xmbx->lock);
> + xmgmt_mailbox_notify(xmbx, false, req, reqlen);
> + mutex_unlock(&xmbx->lock);
> +}
> diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c b/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> new file mode 100644
> index 000000000000..042d86fcef41
> --- /dev/null
> +++ b/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> @@ -0,0 +1,190 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA MGMT PF entry point driver
> + *
> + * Copyright (C) 2020 Xilinx, Inc.
> + *
> + * xclbin download
> + *
> + * Authors:
> + * Lizhi Hou <lizhi.hou@xilinx.com>
> + */
> +
> +#include <linux/firmware.h>
> +#include <linux/uaccess.h>
> +#include "xrt-xclbin.h"
> +#include "xrt-metadata.h"
> +#include "xrt-subdev.h"
> +#include "xrt-gpio.h"
> +#include "xmgmt-main.h"
> +#include "xrt-icap.h"
> +#include "xrt-axigate.h"
> +
> +static int xmgmt_download_bitstream(struct platform_device *pdev,
> + const void *xclbin)
> +{
> + struct platform_device *icap_leaf = NULL;
> + struct XHwIcap_Bit_Header bit_header = { 0 };
Please fix the style error in struct name ...
> + struct xrt_icap_ioctl_wr arg;
> + char *bitstream = NULL;
> + int ret;
> +
> + ret = xrt_xclbin_get_section(xclbin, BITSTREAM, (void **)&bitstream,
> + NULL);
> + if (ret || !bitstream) {
> + xrt_err(pdev, "bitstream not found");
> + return -ENOENT;
> + }
> + ret = xrt_xclbin_parse_header(bitstream,
> + DMA_HWICAP_BITFILE_BUFFER_SIZE, &bit_header);
> + if (ret) {
> + ret = -EINVAL;
> + xrt_err(pdev, "invalid bitstream header");
> + goto done;
> + }
> + icap_leaf = xrt_subdev_get_leaf_by_id(pdev, XRT_SUBDEV_ICAP,
> + PLATFORM_DEVID_NONE);
> + if (!icap_leaf) {
> + ret = -ENODEV;
> + xrt_err(pdev, "icap does not exist");
> + goto done;
> + }
> + arg.xiiw_bit_data = bitstream + bit_header.HeaderLength;
> + arg.xiiw_data_len = bit_header.BitstreamLength;
> + ret = xrt_subdev_ioctl(icap_leaf, XRT_ICAP_WRITE, &arg);
> + if (ret)
> + xrt_err(pdev, "write bitstream failed, ret = %d", ret);
> +
> +done:
> + if (icap_leaf)
> + xrt_subdev_put_leaf(pdev, icap_leaf);
> + vfree(bitstream);
> +
> + return ret;
> +}
> +
> +static bool match_shell(enum xrt_subdev_id id,
> + struct platform_device *pdev, void *arg)
> +{
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> + const char *ulp_gate;
> + int ret;
> +
> + if (!pdata || xrt_md_size(&pdev->dev, pdata->xsp_dtb) <= 0)
> + return false;
> +
> + ret = xrt_md_get_epname_pointer(&pdev->dev, pdata->xsp_dtb,
> + NODE_GATE_ULP, NULL, &ulp_gate);
> + if (ret)
> + return false;
> +
> + ret = xrt_md_check_uuids(&pdev->dev, pdata->xsp_dtb, arg);
> + if (ret)
> + return false;
> +
> + return true;
> +}
> +
> +static bool match_ulp(enum xrt_subdev_id id,
> + struct platform_device *pdev, void *arg)
> +{
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> + const char *ulp_gate;
> + int ret;
> +
> + if (!pdata || xrt_md_size(&pdev->dev, pdata->xsp_dtb) <= 0)
> + return false;
> +
> + ret = xrt_md_check_uuids(&pdev->dev, pdata->xsp_dtb, arg);
> + if (ret)
> + return false;
> +
> + ret = xrt_md_get_epname_pointer(&pdev->dev, pdata->xsp_dtb,
> + NODE_GATE_ULP, NULL, &ulp_gate);
> + if (!ret)
> + return false;
> +
> + return true;
> +}
> +
> +int xmgmt_ulp_download(struct platform_device *pdev, const void *xclbin)
> +{
> + struct platform_device *axigate_leaf;
> + char *dtb = NULL;
> + int ret = 0, part_inst;
> +
> + ret = xrt_xclbin_get_metadata(DEV(pdev), xclbin, &dtb);
> + if (ret) {
> + xrt_err(pdev, "can not get partition metadata, ret %d", ret);
> + goto failed;
> + }
> +
> + part_inst = xrt_subdev_lookup_partition(pdev, match_shell, dtb);
> + if (part_inst < 0) {
> + xrt_err(pdev, "not found matching plp.");
> + ret = -ENODEV;
> + goto failed;
> + }
> +
> + /*
> + * Find ulp partition with interface uuid from incoming xclbin, which
> + * is verified before with matching plp partition.
> + */
> + part_inst = xrt_subdev_lookup_partition(pdev, match_ulp, dtb);
> + if (part_inst >= 0) {
> + ret = xrt_subdev_destroy_partition(pdev, part_inst);
> + if (ret) {
> + xrt_err(pdev, "failed to destroy existing ulp, %d",
> + ret);
> + goto failed;
> + }
> + }
> +
> + axigate_leaf = xrt_subdev_get_leaf_by_epname(pdev, NODE_GATE_ULP);
> +
> + /* gate may not be exist for 0rp */
> + if (axigate_leaf) {
> + ret = xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE,
> + NULL);
> + if (ret) {
> + xrt_err(pdev, "can not freeze gate %s, %d",
> + NODE_GATE_ULP, ret);
> + xrt_subdev_put_leaf(pdev, axigate_leaf);
> + goto failed;
> + }
> + }
> + ret = xmgmt_download_bitstream(pdev, xclbin);
> + if (axigate_leaf) {
> + xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
> +
> + /* Do we really need this extra toggling gate before setting
> + * clocks?
> + * xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE, NULL);
> + * xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
> + */
> +
> + xrt_subdev_put_leaf(pdev, axigate_leaf);
> + }
> + if (ret) {
> + xrt_err(pdev, "bitstream download failed, ret %d", ret);
> + goto failed;
> + }
> + ret = xrt_subdev_create_partition(pdev, dtb);
> + if (ret < 0) {
> + xrt_err(pdev, "failed creating partition, ret %d", ret);
> + goto failed;
> + }
> +
> + ret = xrt_subdev_wait_for_partition_bringup(pdev);
> + if (ret)
> + xrt_err(pdev, "partiton bringup failed, ret %d", ret);
> +
> + /*
> + * TODO: needs to check individual subdevs to see if there
> + * is any error, such as clock setting, memory bank calibration.
> + */
> +
> +failed:
> + vfree(dtb);
> + return ret;
> +}
> diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main.c b/drivers/fpga/alveo/mgmt/xmgmt-main.c
> new file mode 100644
> index 000000000000..23e68e3a4ae1
> --- /dev/null
> +++ b/drivers/fpga/alveo/mgmt/xmgmt-main.c
> @@ -0,0 +1,843 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA MGMT PF entry point driver
> + *
> + * Copyright (C) 2020 Xilinx, Inc.
> + *
> + * Authors:
> + * Sonal Santan <sonals@xilinx.com>
> + */
> +
> +#include <linux/firmware.h>
> +#include <linux/uaccess.h>
> +#include "xrt-xclbin.h"
> +#include "xrt-metadata.h"
> +#include "xrt-flash.h"
> +#include "xrt-subdev.h"
> +#include <linux/xrt/flash_xrt_data.h>
> +#include <linux/xrt/xmgmt-ioctl.h>
> +#include "xrt-gpio.h"
> +#include "xmgmt-main.h"
> +#include "xmgmt-fmgr.h"
> +#include "xrt-icap.h"
> +#include "xrt-axigate.h"
> +#include "xmgmt-main-impl.h"
> +
> +#define XMGMT_MAIN "xmgmt_main"
> +
> +struct xmgmt_main {
> + struct platform_device *pdev;
> + void *evt_hdl;
> + char *firmware_blp;
> + char *firmware_plp;
> + char *firmware_ulp;
> + bool flash_ready;
> + bool gpio_ready;
> + struct fpga_manager *fmgr;
> + void *mailbox_hdl;
> + struct mutex busy_mutex;
> +
> + uuid_t *blp_intf_uuids;
> + u32 blp_intf_uuid_num;
> +};
> +
> +char *xmgmt_get_vbnv(struct platform_device *pdev)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + const char *vbnv;
> + char *ret;
> + int i;
> +
> + if (xmm->firmware_plp) {
> + vbnv = ((struct axlf *)xmm->firmware_plp)->
> + m_header.m_platformVBNV;
> + } else if (xmm->firmware_blp) {
> + vbnv = ((struct axlf *)xmm->firmware_blp)->
> + m_header.m_platformVBNV;
> + } else {
> + return NULL;
> + }
> +
> + ret = kstrdup(vbnv, GFP_KERNEL);
> + for (i = 0; i < strlen(ret); i++) {
> + if (ret[i] == ':' || ret[i] == '.')
> + ret[i] = '_';
> + }
> + return ret;
> +}
> +
> +static bool xmgmt_main_leaf_match(enum xrt_subdev_id id,
> + struct platform_device *pdev, void *arg)
> +{
> + if (id == XRT_SUBDEV_GPIO)
> + return xrt_subdev_has_epname(pdev, arg);
> + else if (id == XRT_SUBDEV_QSPI)
> + return true;
> +
> + return false;
> +}
> +
> +static int get_dev_uuid(struct platform_device *pdev, char *uuidstr, size_t len)
> +{
> + char uuid[16];
> + struct platform_device *gpio_leaf;
> + struct xrt_gpio_ioctl_rw gpio_arg = { 0 };
> + int err, i, count;
> +
> + gpio_leaf = xrt_subdev_get_leaf_by_epname(pdev, NODE_BLP_ROM);
> + if (!gpio_leaf) {
> + xrt_err(pdev, "can not get %s", NODE_BLP_ROM);
> + return -EINVAL;
> + }
> +
> + gpio_arg.xgir_id = XRT_GPIO_ROM_UUID;
> + gpio_arg.xgir_buf = uuid;
> + gpio_arg.xgir_len = sizeof(uuid);
> + gpio_arg.xgir_offset = 0;
> + err = xrt_subdev_ioctl(gpio_leaf, XRT_GPIO_READ, &gpio_arg);
> + xrt_subdev_put_leaf(pdev, gpio_leaf);
> + if (err) {
> + xrt_err(pdev, "can not get uuid: %d", err);
> + return err;
> + }
> +
> + for (count = 0, i = sizeof(uuid) - sizeof(u32);
> + i >= 0 && len > count; i -= sizeof(u32)) {
> + count += snprintf(uuidstr + count, len - count,
> + "%08x", *(u32 *)&uuid[i]);
> + }
> + return 0;
> +}
> +
> +int xmgmt_hot_reset(struct platform_device *pdev)
> +{
> + int ret = xrt_subdev_broadcast_event(pdev, XRT_EVENT_PRE_HOT_RESET);
> +
> + if (ret) {
> + xrt_err(pdev, "offline failed, hot reset is canceled");
> + return ret;
> + }
> +
> + (void) xrt_subdev_hot_reset(pdev);
> + xrt_subdev_broadcast_event(pdev, XRT_EVENT_POST_HOT_RESET);
> + return 0;
> +}
> +
> +static ssize_t reset_store(struct device *dev,
> + struct device_attribute *da, const char *buf, size_t count)
> +{
> + struct platform_device *pdev = to_platform_device(dev);
> +
> + (void) xmgmt_hot_reset(pdev);
> + return count;
> +}
> +static DEVICE_ATTR_WO(reset);
> +
> +static ssize_t VBNV_show(struct device *dev,
> + struct device_attribute *da, char *buf)
> +{
> + ssize_t ret;
> + char *vbnv;
> + struct platform_device *pdev = to_platform_device(dev);
> +
> + vbnv = xmgmt_get_vbnv(pdev);
> + ret = sprintf(buf, "%s\n", vbnv);
> + kfree(vbnv);
> + return ret;
> +}
> +static DEVICE_ATTR_RO(VBNV);
> +
> +static ssize_t logic_uuids_show(struct device *dev,
> + struct device_attribute *da, char *buf)
> +{
> + ssize_t ret;
> + char uuid[80];
> + struct platform_device *pdev = to_platform_device(dev);
> +
> + /*
> + * Getting UUID pointed to by VSEC,
> + * should be the same as logic UUID of BLP.
> + * TODO: add PLP logic UUID
> + */
> + ret = get_dev_uuid(pdev, uuid, sizeof(uuid));
> + if (ret)
> + return ret;
> + ret = sprintf(buf, "%s\n", uuid);
> + return ret;
> +}
> +static DEVICE_ATTR_RO(logic_uuids);
> +
> +static inline void uuid2str(const uuid_t *uuid, char *uuidstr, size_t len)
> +{
> + int i, p;
> + u8 *u = (u8 *)uuid;
> +
> + BUG_ON(sizeof(uuid_t) * 2 + 1 > len);
> + for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
> + (void) snprintf(&uuidstr[p*2], 3, "%02x", u[i]);
> +}
> +
> +static ssize_t interface_uuids_show(struct device *dev,
> + struct device_attribute *da, char *buf)
> +{
> + ssize_t ret = 0;
> + struct platform_device *pdev = to_platform_device(dev);
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + u32 i;
> +
> + /*
> + * TODO: add PLP interface UUID
> + */
> + for (i = 0; i < xmm->blp_intf_uuid_num; i++) {
> + char uuidstr[80];
> +
> + uuid2str(&xmm->blp_intf_uuids[i], uuidstr, sizeof(uuidstr));
> + ret += sprintf(buf + ret, "%s\n", uuidstr);
> + }
> + return ret;
> +}
> +static DEVICE_ATTR_RO(interface_uuids);
> +
> +static struct attribute *xmgmt_main_attrs[] = {
> + &dev_attr_reset.attr,
> + &dev_attr_VBNV.attr,
> + &dev_attr_logic_uuids.attr,
> + &dev_attr_interface_uuids.attr,
> + NULL,
> +};
> +
> +static ssize_t ulp_image_write(struct file *filp, struct kobject *kobj,
> + struct bin_attribute *attr, char *buffer, loff_t off, size_t count)
> +{
> + struct xmgmt_main *xmm =
> + dev_get_drvdata(container_of(kobj, struct device, kobj));
> + struct axlf *xclbin;
> + ulong len;
> +
> + if (off == 0) {
> + if (count < sizeof(*xclbin)) {
> + xrt_err(xmm->pdev, "count is too small %ld", count);
> + return -EINVAL;
> + }
> +
> + if (xmm->firmware_ulp) {
> + vfree(xmm->firmware_ulp);
> + xmm->firmware_ulp = NULL;
> + }
> + xclbin = (struct axlf *)buffer;
> + xmm->firmware_ulp = vmalloc(xclbin->m_header.m_length);
> + if (!xmm->firmware_ulp)
> + return -ENOMEM;
> + } else
> + xclbin = (struct axlf *)xmm->firmware_ulp;
> +
> + len = xclbin->m_header.m_length;
> + if (off + count >= len && off < len) {
> + memcpy(xmm->firmware_ulp + off, buffer, len - off);
> + xmgmt_ulp_download(xmm->pdev, xmm->firmware_ulp);
> + } else if (off + count < len) {
> + memcpy(xmm->firmware_ulp + off, buffer, count);
> + }
> +
> + return count;
> +}
> +
> +static struct bin_attribute ulp_image_attr = {
> + .attr = {
> + .name = "ulp_image",
> + .mode = 0200
> + },
> + .write = ulp_image_write,
> + .size = 0
> +};
> +
> +static struct bin_attribute *xmgmt_main_bin_attrs[] = {
> + &ulp_image_attr,
> + NULL,
> +};
> +
> +static const struct attribute_group xmgmt_main_attrgroup = {
> + .attrs = xmgmt_main_attrs,
> + .bin_attrs = xmgmt_main_bin_attrs,
> +};
> +
> +static int load_firmware_from_flash(struct platform_device *pdev,
> + char **fw_buf, size_t *len)
> +{
> + struct platform_device *flash_leaf = NULL;
> + struct flash_data_header header = { 0 };
> + const size_t magiclen = sizeof(header.fdh_id_begin.fdi_magic);
> + size_t flash_size = 0;
> + int ret = 0;
> + char *buf = NULL;
> + struct flash_data_ident id = { 0 };
> + struct xrt_flash_ioctl_read frd = { 0 };
> +
> + xrt_info(pdev, "try loading fw from flash");
> +
> + flash_leaf = xrt_subdev_get_leaf_by_id(pdev, XRT_SUBDEV_QSPI,
> + PLATFORM_DEVID_NONE);
> + if (flash_leaf == NULL) {
> + xrt_err(pdev, "failed to hold flash leaf");
> + return -ENODEV;
> + }
> +
> + (void) xrt_subdev_ioctl(flash_leaf, XRT_FLASH_GET_SIZE, &flash_size);
> + if (flash_size == 0) {
> + xrt_err(pdev, "failed to get flash size");
> + ret = -EINVAL;
> + goto done;
> + }
> +
> + frd.xfir_buf = (char *)&header;
> + frd.xfir_size = sizeof(header);
> + frd.xfir_offset = flash_size - sizeof(header);
> + ret = xrt_subdev_ioctl(flash_leaf, XRT_FLASH_READ, &frd);
> + if (ret) {
> + xrt_err(pdev, "failed to read header from flash: %d", ret);
> + goto done;
> + }
> +
> + /* Pick the end ident since header is aligned in the end of flash. */
> + id = header.fdh_id_end;
> + if (strncmp(id.fdi_magic, XRT_DATA_MAGIC, magiclen)) {
> + char tmp[sizeof(id.fdi_magic) + 1] = { 0 };
> +
> + memcpy(tmp, id.fdi_magic, magiclen);
> + xrt_info(pdev, "ignore meta data, bad magic: %s", tmp);
> + ret = -ENOENT;
> + goto done;
> + }
> + if (id.fdi_version != 0) {
> + xrt_info(pdev, "flash meta data version is not supported: %d",
> + id.fdi_version);
> + ret = -EOPNOTSUPP;
> + goto done;
> + }
> +
> + buf = vmalloc(header.fdh_data_len);
> + if (buf == NULL) {
> + ret = -ENOMEM;
> + goto done;
> + }
> +
> + frd.xfir_buf = buf;
> + frd.xfir_size = header.fdh_data_len;
> + frd.xfir_offset = header.fdh_data_offset;
> + ret = xrt_subdev_ioctl(flash_leaf, XRT_FLASH_READ, &frd);
> + if (ret) {
> + xrt_err(pdev, "failed to read meta data from flash: %d", ret);
> + goto done;
> + } else if (flash_xrt_data_get_parity32(buf, header.fdh_data_len) ^
> + header.fdh_data_parity) {
> + xrt_err(pdev, "meta data is corrupted");
> + ret = -EINVAL;
> + goto done;
> + }
> +
> + xrt_info(pdev, "found meta data of %d bytes @0x%x",
> + header.fdh_data_len, header.fdh_data_offset);
> + *fw_buf = buf;
> + *len = header.fdh_data_len;
> +
> +done:
> + (void) xrt_subdev_put_leaf(pdev, flash_leaf);
> + return ret;
> +}
> +
> +static int load_firmware_from_disk(struct platform_device *pdev, char **fw_buf,
> + size_t *len)
> +{
> + char uuid[80];
> + int err = 0;
> + char fw_name[256];
> + const struct firmware *fw;
> +
> + err = get_dev_uuid(pdev, uuid, sizeof(uuid));
> + if (err)
> + return err;
> +
> + (void) snprintf(fw_name,
> + sizeof(fw_name), "xilinx/%s/partition.xsabin", uuid);
> + xrt_info(pdev, "try loading fw: %s", fw_name);
> +
> + err = request_firmware(&fw, fw_name, DEV(pdev));
> + if (err)
> + return err;
> +
> + *fw_buf = vmalloc(fw->size);
> + *len = fw->size;
> + if (*fw_buf != NULL)
> + memcpy(*fw_buf, fw->data, fw->size);
> + else
> + err = -ENOMEM;
> +
> + release_firmware(fw);
> + return 0;
> +}
> +
> +static const char *xmgmt_get_axlf_firmware(struct xmgmt_main *xmm,
> + enum provider_kind kind)
> +{
> + switch (kind) {
> + case XMGMT_BLP:
> + return xmm->firmware_blp;
> + case XMGMT_PLP:
> + return xmm->firmware_plp;
> + case XMGMT_ULP:
> + return xmm->firmware_ulp;
> + default:
> + xrt_err(xmm->pdev, "unknown axlf kind: %d", kind);
> + return NULL;
> + }
> +}
> +
> +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + char *dtb = NULL;
> + const char *provider = xmgmt_get_axlf_firmware(xmm, kind);
> + int rc;
> +
> + if (provider == NULL)
> + return dtb;
> +
> + rc = xrt_xclbin_get_metadata(DEV(pdev), provider, &dtb);
> + if (rc)
> + xrt_err(pdev, "failed to find dtb: %d", rc);
> + return dtb;
> +}
> +
> +static const char *get_uuid_from_firmware(struct platform_device *pdev,
> + const char *axlf)
> +{
> + const void *uuid = NULL;
> + const void *uuiddup = NULL;
> + void *dtb = NULL;
> + int rc;
> +
> + rc = xrt_xclbin_get_section(axlf, PARTITION_METADATA, &dtb, NULL);
> + if (rc)
> + return NULL;
> +
> + rc = xrt_md_get_prop(DEV(pdev), dtb, NULL, NULL,
> + PROP_LOGIC_UUID, &uuid, NULL);
> + if (!rc)
> + uuiddup = kstrdup(uuid, GFP_KERNEL);
> + vfree(dtb);
> + return uuiddup;
> +}
> +
> +static bool is_valid_firmware(struct platform_device *pdev,
> + char *fw_buf, size_t fw_len)
> +{
> + struct axlf *axlf = (struct axlf *)fw_buf;
> + size_t axlflen = axlf->m_header.m_length;
> + const char *fw_uuid;
> + char dev_uuid[80];
> + int err;
> +
> + err = get_dev_uuid(pdev, dev_uuid, sizeof(dev_uuid));
> + if (err)
> + return false;
> +
> + if (memcmp(fw_buf, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)) != 0) {
> + xrt_err(pdev, "unknown fw format");
> + return false;
> + }
> +
> + if (axlflen > fw_len) {
> + xrt_err(pdev, "truncated fw, length: %ld, expect: %ld",
> + fw_len, axlflen);
> + return false;
> + }
> +
> + fw_uuid = get_uuid_from_firmware(pdev, fw_buf);
> + if (fw_uuid == NULL || strcmp(fw_uuid, dev_uuid) != 0) {
> + xrt_err(pdev, "bad fw UUID: %s, expect: %s",
> + fw_uuid ? fw_uuid : "<none>", dev_uuid);
> + kfree(fw_uuid);
> + return false;
> + }
> +
> + kfree(fw_uuid);
> + return true;
> +}
> +
> +int xmgmt_get_provider_uuid(struct platform_device *pdev,
> + enum provider_kind kind, uuid_t *uuid)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + const char *fwbuf;
> + const char *fw_uuid;
> + int rc = -ENOENT;
> +
> + mutex_lock(&xmm->busy_mutex);
> +
> + fwbuf = xmgmt_get_axlf_firmware(xmm, kind);
> + if (fwbuf == NULL)
> + goto done;
> +
> + fw_uuid = get_uuid_from_firmware(pdev, fwbuf);
> + if (fw_uuid == NULL)
> + goto done;
> +
> + rc = xrt_md_uuid_strtoid(DEV(pdev), fw_uuid, uuid);
> + kfree(fw_uuid);
> +
> +done:
> + mutex_unlock(&xmm->busy_mutex);
> + return rc;
> +}
> +
> +static int xmgmt_create_blp(struct xmgmt_main *xmm)
> +{
> + struct platform_device *pdev = xmm->pdev;
> + int rc = 0;
> + char *dtb = NULL;
> +
> + dtb = xmgmt_get_dtb(pdev, XMGMT_BLP);
> + if (dtb) {
> + rc = xrt_subdev_create_partition(pdev, dtb);
> + if (rc < 0)
> + xrt_err(pdev, "failed to create BLP: %d", rc);
> + else
> + rc = 0;
> +
> + BUG_ON(xmm->blp_intf_uuids);
> + xrt_md_get_intf_uuids(&pdev->dev, dtb,
> + &xmm->blp_intf_uuid_num, NULL);
> + if (xmm->blp_intf_uuid_num > 0) {
> + xmm->blp_intf_uuids = vzalloc(sizeof(uuid_t) *
> + xmm->blp_intf_uuid_num);
> + xrt_md_get_intf_uuids(&pdev->dev, dtb,
> + &xmm->blp_intf_uuid_num, xmm->blp_intf_uuids);
> + }
> + }
> +
> + vfree(dtb);
> + return rc;
> +}
> +
> +static int xmgmt_main_event_cb(struct platform_device *pdev,
> + enum xrt_events evt, void *arg)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev *)arg;
> + enum xrt_subdev_id id;
> + int instance;
> + size_t fwlen;
> +
> + switch (evt) {
> + case XRT_EVENT_POST_CREATION: {
> + id = esd->xevt_subdev_id;
> + instance = esd->xevt_subdev_instance;
> + xrt_info(pdev, "processing event %d for (%d, %d)",
> + evt, id, instance);
> +
> + if (id == XRT_SUBDEV_GPIO)
> + xmm->gpio_ready = true;
> + else if (id == XRT_SUBDEV_QSPI)
> + xmm->flash_ready = true;
> + else
> + BUG_ON(1);
> +
> + if (xmm->gpio_ready && xmm->flash_ready) {
> + int rc;
> +
> + rc = load_firmware_from_disk(pdev, &xmm->firmware_blp,
> + &fwlen);
> + if (rc != 0) {
> + rc = load_firmware_from_flash(pdev,
> + &xmm->firmware_blp, &fwlen);
> + }
> + if (rc == 0 && is_valid_firmware(pdev,
> + xmm->firmware_blp, fwlen))
> + (void) xmgmt_create_blp(xmm);
> + else
> + xrt_err(pdev,
> + "failed to find firmware, giving up");
> + xmm->evt_hdl = NULL;
> + }
> + break;
> + }
> + case XRT_EVENT_POST_ATTACH:
> + xmgmt_peer_notify_state(xmm->mailbox_hdl, true);
> + break;
> + case XRT_EVENT_PRE_DETACH:
> + xmgmt_peer_notify_state(xmm->mailbox_hdl, false);
> + break;
> + default:
> + xrt_info(pdev, "ignored event %d", evt);
> + break;
> + }
> +
> + return XRT_EVENT_CB_CONTINUE;
> +}
> +
> +static int xmgmt_main_probe(struct platform_device *pdev)
> +{
> + struct xmgmt_main *xmm;
> +
> + xrt_info(pdev, "probing...");
> +
> + xmm = devm_kzalloc(DEV(pdev), sizeof(*xmm), GFP_KERNEL);
> + if (!xmm)
> + return -ENOMEM;
> +
> + xmm->pdev = pdev;
> + platform_set_drvdata(pdev, xmm);
> + xmm->fmgr = xmgmt_fmgr_probe(pdev);
> + xmm->mailbox_hdl = xmgmt_mailbox_probe(pdev);
> + mutex_init(&xmm->busy_mutex);
> +
> + xmm->evt_hdl = xrt_subdev_add_event_cb(pdev,
> + xmgmt_main_leaf_match, NODE_BLP_ROM, xmgmt_main_event_cb);
> +
> + /* Ready to handle req thru sysfs nodes. */
> + if (sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup))
> + xrt_err(pdev, "failed to create sysfs group");
> + return 0;
> +}
> +
> +static int xmgmt_main_remove(struct platform_device *pdev)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> +
> + /* By now, partition driver should prevent any inter-leaf call. */
> +
> + xrt_info(pdev, "leaving...");
> +
> + if (xmm->evt_hdl)
> + (void) xrt_subdev_remove_event_cb(pdev, xmm->evt_hdl);
> + vfree(xmm->blp_intf_uuids);
> + vfree(xmm->firmware_blp);
> + vfree(xmm->firmware_plp);
> + vfree(xmm->firmware_ulp);
> + (void) xmgmt_fmgr_remove(xmm->fmgr);
> + xmgmt_mailbox_remove(xmm->mailbox_hdl);
> + (void) sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup);
> + return 0;
> +}
> +
> +static int
> +xmgmt_main_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + int ret = 0;
> +
> + xrt_info(pdev, "handling IOCTL cmd: %d", cmd);
> +
> + switch (cmd) {
> + case XRT_MGMT_MAIN_GET_AXLF_SECTION: {
> + struct xrt_mgmt_main_ioctl_get_axlf_section *get =
> + (struct xrt_mgmt_main_ioctl_get_axlf_section *)arg;
> + const char *firmware =
> + xmgmt_get_axlf_firmware(xmm, get->xmmigas_axlf_kind);
> +
> + if (firmware == NULL) {
> + ret = -ENOENT;
> + } else {
> + ret = xrt_xclbin_get_section(firmware,
> + get->xmmigas_section_kind,
> + &get->xmmigas_section,
> + &get->xmmigas_section_size);
> + }
> + break;
> + }
> + case XRT_MGMT_MAIN_GET_VBNV: {
> + char **vbnv_p = (char **)arg;
> +
> + *vbnv_p = xmgmt_get_vbnv(pdev);
> + break;
> + }
> + default:
> + xrt_err(pdev, "unknown cmd: %d", cmd);
> + ret = -EINVAL;
> + break;
> + }
> + return ret;
> +}
> +
> +static int xmgmt_main_open(struct inode *inode, struct file *file)
> +{
> + struct platform_device *pdev = xrt_devnode_open(inode);
> +
> + /* Device may have gone already when we get here. */
> + if (!pdev)
> + return -ENODEV;
> +
> + xrt_info(pdev, "opened");
> + file->private_data = platform_get_drvdata(pdev);
> + return 0;
> +}
> +
> +static int xmgmt_main_close(struct inode *inode, struct file *file)
> +{
> + struct xmgmt_main *xmm = file->private_data;
> +
> + xrt_devnode_close(inode);
> +
> + xrt_info(xmm->pdev, "closed");
> + return 0;
> +}
> +
> +static int xmgmt_bitstream_axlf_fpga_mgr(struct xmgmt_main *xmm,
> + void *axlf, size_t size)
> +{
> + int ret;
> + struct fpga_image_info info = { 0 };
> +
> + BUG_ON(!mutex_is_locked(&xmm->busy_mutex));
> +
> + /*
> + * Should any error happens during download, we can't trust
> + * the cached xclbin any more.
> + */
> + vfree(xmm->firmware_ulp);
> + xmm->firmware_ulp = NULL;
> +
> + info.buf = (char *)axlf;
> + info.count = size;
> + ret = fpga_mgr_load(xmm->fmgr, &info);
> + if (ret == 0)
> + xmm->firmware_ulp = axlf;
> +
> + return ret;
> +}
> +
> +int bitstream_axlf_mailbox(struct platform_device *pdev, const void *axlf)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + void *copy_buffer = NULL;
> + size_t copy_buffer_size = 0;
> + const struct axlf *xclbin_obj = axlf;
> + int ret = 0;
> +
> + if (memcmp(xclbin_obj->m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
> + return -EINVAL;
> +
> + copy_buffer_size = xclbin_obj->m_header.m_length;
> + if (copy_buffer_size > MAX_XCLBIN_SIZE)
> + return -EINVAL;
> + copy_buffer = vmalloc(copy_buffer_size);
> + if (copy_buffer == NULL)
> + return -ENOMEM;
> + (void) memcpy(copy_buffer, axlf, copy_buffer_size);
> +
> + mutex_lock(&xmm->busy_mutex);
> + ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
> + mutex_unlock(&xmm->busy_mutex);
> + if (ret)
> + vfree(copy_buffer);
> + return ret;
> +}
> +
> +static int bitstream_axlf_ioctl(struct xmgmt_main *xmm, const void __user *arg)
> +{
> + void *copy_buffer = NULL;
> + size_t copy_buffer_size = 0;
> + struct xmgmt_ioc_bitstream_axlf ioc_obj = { 0 };
> + struct axlf xclbin_obj = { {0} };
> + int ret = 0;
> +
> + if (copy_from_user((void *)&ioc_obj, arg, sizeof(ioc_obj)))
> + return -EFAULT;
> + if (copy_from_user((void *)&xclbin_obj, ioc_obj.xclbin,
> + sizeof(xclbin_obj)))
> + return -EFAULT;
> + if (memcmp(xclbin_obj.m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
> + return -EINVAL;
> +
> + copy_buffer_size = xclbin_obj.m_header.m_length;
> + if (copy_buffer_size > MAX_XCLBIN_SIZE)
> + return -EINVAL;
> + copy_buffer = vmalloc(copy_buffer_size);
> + if (copy_buffer == NULL)
> + return -ENOMEM;
> +
> + if (copy_from_user(copy_buffer, ioc_obj.xclbin, copy_buffer_size)) {
> + vfree(copy_buffer);
> + return -EFAULT;
> + }
> +
> + ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
> + if (ret)
> + vfree(copy_buffer);
> +
> + return ret;
> +}
> +
> +static long xmgmt_main_ioctl(struct file *filp, unsigned int cmd,
> + unsigned long arg)
> +{
> + long result = 0;
> + struct xmgmt_main *xmm = filp->private_data;
> +
> + BUG_ON(!xmm);
> +
> + if (_IOC_TYPE(cmd) != XMGMT_IOC_MAGIC)
> + return -ENOTTY;
> +
> + mutex_lock(&xmm->busy_mutex);
> +
> + xrt_info(xmm->pdev, "ioctl cmd %d, arg %ld", cmd, arg);
> + switch (cmd) {
> + case XMGMT_IOCICAPDOWNLOAD_AXLF:
> + result = bitstream_axlf_ioctl(xmm, (const void __user *)arg);
> + break;
> + default:
> + result = -ENOTTY;
> + break;
> + }
> +
> + mutex_unlock(&xmm->busy_mutex);
> + return result;
> +}
> +
> +void *xmgmt_pdev2mailbox(struct platform_device *pdev)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> +
> + return xmm->mailbox_hdl;
> +}
> +
> +struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names []){
> + { .ep_name = NODE_MGMT_MAIN },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +struct xrt_subdev_drvdata xmgmt_main_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xmgmt_main_leaf_ioctl,
> + },
> + .xsd_file_ops = {
> + .xsf_ops = {
> + .owner = THIS_MODULE,
> + .open = xmgmt_main_open,
> + .release = xmgmt_main_close,
> + .unlocked_ioctl = xmgmt_main_ioctl,
> + },
> + .xsf_dev_name = "xmgmt",
> + },
> +};
> +
> +static const struct platform_device_id xmgmt_main_id_table[] = {
> + { XMGMT_MAIN, (kernel_ulong_t)&xmgmt_main_data },
> + { },
> +};
> +
> +struct platform_driver xmgmt_main_driver = {
> + .driver = {
> + .name = XMGMT_MAIN,
> + },
> + .probe = xmgmt_main_probe,
> + .remove = xmgmt_main_remove,
> + .id_table = xmgmt_main_id_table,
> +};
> diff --git a/drivers/fpga/alveo/mgmt/xmgmt-root.c b/drivers/fpga/alveo/mgmt/xmgmt-root.c
> new file mode 100644
> index 000000000000..005fd5e42651
> --- /dev/null
> +++ b/drivers/fpga/alveo/mgmt/xmgmt-root.c
> @@ -0,0 +1,375 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2020 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/aer.h>
> +#include <linux/vmalloc.h>
> +#include <linux/delay.h>
> +
> +#include "xrt-root.h"
> +#include "xrt-subdev.h"
> +#include "xmgmt-main-impl.h"
> +#include "xrt-metadata.h"
> +
> +#define XMGMT_MODULE_NAME "xmgmt"
> +#define XMGMT_DRIVER_VERSION "4.0.0"
> +
> +#define XMGMT_PDEV(xm) ((xm)->pdev)
> +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
> +#define xmgmt_err(xm, fmt, args...) \
> + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define xmgmt_warn(xm, fmt, args...) \
> + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define xmgmt_info(xm, fmt, args...) \
> + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define xmgmt_dbg(xm, fmt, args...) \
> + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define XMGMT_DEV_ID(pdev) \
> + ((pci_domain_nr(pdev->bus) << 16) | \
> + PCI_DEVID(pdev->bus->number, 0))
> +
> +static struct class *xmgmt_class;
> +static const struct pci_device_id xmgmt_pci_ids[] = {
> + { PCI_DEVICE(0x10EE, 0xd020), },
> + { PCI_DEVICE(0x10EE, 0x5020), },
> + { 0, }
> +};
> +
> +struct xmgmt {
> + struct pci_dev *pdev;
> + void *root;
> +
> + /* save config for pci reset */
> + u32 saved_config[8][16];
> + bool ready;
> +};
> +
> +static int xmgmt_config_pci(struct xmgmt *xm)
> +{
> + struct pci_dev *pdev = XMGMT_PDEV(xm);
> + int rc;
> +
> + rc = pcim_enable_device(pdev);
> + if (rc < 0) {
> + xmgmt_err(xm, "failed to enable device: %d", rc);
> + return rc;
> + }
> +
> + rc = pci_enable_pcie_error_reporting(pdev);
> + if (rc)
> + xmgmt_warn(xm, "failed to enable AER: %d", rc);
> +
> + pci_set_master(pdev);
> +
> + rc = pcie_get_readrq(pdev);
> + if (rc < 0) {
> + xmgmt_err(xm, "failed to read mrrs %d", rc);
> + return rc;
> + }
> + if (rc > 512) {
> + rc = pcie_set_readrq(pdev, 512);
> + if (rc) {
> + xmgmt_err(xm, "failed to force mrrs %d", rc);
> + return rc;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static void xmgmt_save_config_space(struct pci_dev *pdev, u32 *saved_config)
> +{
> + int i;
> +
> + for (i = 0; i < 16; i++)
> + pci_read_config_dword(pdev, i * 4, &saved_config[i]);
> +}
> +
> +static int xmgmt_match_slot_and_save(struct device *dev, void *data)
> +{
> + struct xmgmt *xm = data;
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> + pci_cfg_access_lock(pdev);
> + pci_save_state(pdev);
> + xmgmt_save_config_space(pdev,
> + xm->saved_config[PCI_FUNC(pdev->devfn)]);
> + }
> +
> + return 0;
> +}
> +
> +static void xmgmt_pci_save_config_all(struct xmgmt *xm)
> +{
> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
> +}
> +
> +static void xmgmt_restore_config_space(struct pci_dev *pdev, u32 *config_saved)
> +{
> + int i;
> + u32 val;
> +
> + for (i = 0; i < 16; i++) {
> + pci_read_config_dword(pdev, i * 4, &val);
> + if (val == config_saved[i])
> + continue;
> +
> + pci_write_config_dword(pdev, i * 4, config_saved[i]);
> + pci_read_config_dword(pdev, i * 4, &val);
> + if (val != config_saved[i]) {
> + dev_err(&pdev->dev,
> + "restore config at %d failed", i * 4);
> + }
> + }
> +}
> +
> +static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
> +{
> + struct xmgmt *xm = data;
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> + xmgmt_restore_config_space(pdev,
> + xm->saved_config[PCI_FUNC(pdev->devfn)]);
> +
> + pci_restore_state(pdev);
> + pci_cfg_access_unlock(pdev);
> + }
> +
> + return 0;
> +}
> +
> +static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
> +{
> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
> +}
> +
> +void xroot_hot_reset(struct pci_dev *pdev)
> +{
> + struct xmgmt *xm = pci_get_drvdata(pdev);
> + struct pci_bus *bus;
> + u8 pci_bctl;
> + u16 pci_cmd, devctl;
> + int i;
> +
> + xmgmt_info(xm, "hot reset start");
> +
> + xmgmt_pci_save_config_all(xm);
> +
> + pci_disable_device(pdev);
> +
> + bus = pdev->bus;
> +
> + /*
> + * When flipping the SBR bit, device can fall off the bus. This is
> + * usually no problem at all so long as drivers are working properly
> + * after SBR. However, some systems complain bitterly when the device
> + * falls off the bus.
> + * The quick solution is to temporarily disable the SERR reporting of
> + * switch port during SBR.
> + */
> +
> + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
> + pci_write_config_word(bus->self, PCI_COMMAND,
> + (pci_cmd & ~PCI_COMMAND_SERR));
> + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
> + (devctl & ~PCI_EXP_DEVCTL_FERE));
> + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
> + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
> +
> + msleep(100);
> + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
> + ssleep(1);
> +
> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
> + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
> +
> + pci_enable_device(pdev);
> +
> + for (i = 0; i < 300; i++) {
> + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
> + if (pci_cmd != 0xffff)
> + break;
> + msleep(20);
> + }
> +
> + xmgmt_info(xm, "waiting for %d ms", i * 20);
> +
> + xmgmt_pci_restore_config_all(xm);
> +
> + xmgmt_config_pci(xm);
> +}
> +
> +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
> +{
> + char *dtb = NULL;
> + int ret;
> +
> + ret = xrt_md_create(DEV(xm->pdev), &dtb);
> + if (ret) {
> + xmgmt_err(xm, "create metadata failed, ret %d", ret);
> + goto failed;
> + }
> +
> + ret = xroot_add_simple_node(xm->root, dtb, NODE_TEST);
> + if (ret)
> + goto failed;
> +
> + ret = xroot_add_vsec_node(xm->root, dtb);
> + if (ret == -ENOENT) {
> + /*
> + * We may be dealing with a MFG board.
> + * Try vsec-golden which will bring up all hard-coded leaves
> + * at hard-coded offsets.
> + */
> + ret = xroot_add_simple_node(xm, dtb, NODE_VSEC_GOLDEN);
> + } else if (ret == 0) {
> + ret = xroot_add_simple_node(xm->root, dtb, NODE_MGMT_MAIN);
> + }
> + if (ret)
> + goto failed;
> +
> + *root_dtb = dtb;
> + return 0;
> +
> +failed:
> + vfree(dtb);
> + return ret;
> +}
> +
> +static ssize_t ready_show(struct device *dev,
> + struct device_attribute *da, char *buf)
> +{
> + struct pci_dev *pdev = to_pci_dev(dev);
> + struct xmgmt *xm = pci_get_drvdata(pdev);
> +
> + return sprintf(buf, "%d\n", xm->ready);
> +}
> +static DEVICE_ATTR_RO(ready);
> +
> +static struct attribute *xmgmt_root_attrs[] = {
> + &dev_attr_ready.attr,
> + NULL
> +};
> +
> +static struct attribute_group xmgmt_root_attr_group = {
> + .attrs = xmgmt_root_attrs,
> +};
> +
> +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> +{
> + int ret;
> + struct device *dev = DEV(pdev);
> + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
> + char *dtb = NULL;
> +
> + if (!xm)
> + return -ENOMEM;
> + xm->pdev = pdev;
> + pci_set_drvdata(pdev, xm);
> +
> + ret = xmgmt_config_pci(xm);
> + if (ret)
> + goto failed;
> +
> + ret = xroot_probe(pdev, &xm->root);
> + if (ret)
> + goto failed;
> +
> + ret = xmgmt_create_root_metadata(xm, &dtb);
> + if (ret)
> + goto failed_metadata;
> +
> + ret = xroot_create_partition(xm->root, dtb);
> + vfree(dtb);
> + if (ret)
> + xmgmt_err(xm, "failed to create root partition: %d", ret);
> +
> + if (!xroot_wait_for_bringup(xm->root))
> + xmgmt_err(xm, "failed to bringup all partitions");
> + else
> + xm->ready = true;
> +
> + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> + if (ret) {
> + /* Warning instead of failing the probe. */
> + xrt_warn(pdev, "create xmgmt root attrs failed: %d", ret);
> + }
> +
> + xroot_broadcast(xm->root, XRT_EVENT_POST_ATTACH);
> + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
> + return 0;
> +
> +failed_metadata:
> + (void) xroot_remove(xm->root);
> +failed:
> + pci_set_drvdata(pdev, NULL);
> + return ret;
> +}
> +
> +static void xmgmt_remove(struct pci_dev *pdev)
> +{
> + struct xmgmt *xm = pci_get_drvdata(pdev);
> +
> + xroot_broadcast(xm->root, XRT_EVENT_PRE_DETACH);
> + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> + (void) xroot_remove(xm->root);
> + pci_disable_pcie_error_reporting(xm->pdev);
> + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
> +}
> +
> +static struct pci_driver xmgmt_driver = {
> + .name = XMGMT_MODULE_NAME,
> + .id_table = xmgmt_pci_ids,
> + .probe = xmgmt_probe,
> + .remove = xmgmt_remove,
> +};
> +
> +static int __init xmgmt_init(void)
> +{
> + int res = xrt_subdev_register_external_driver(XRT_SUBDEV_MGMT_MAIN,
> + &xmgmt_main_driver, xrt_mgmt_main_endpoints);
> +
> + if (res)
> + return res;
> +
> + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
> + if (IS_ERR(xmgmt_class))
> + return PTR_ERR(xmgmt_class);
> +
> + res = pci_register_driver(&xmgmt_driver);
> + if (res) {
> + class_destroy(xmgmt_class);
> + return res;
> + }
> +
> + return 0;
> +}
> +
> +static __exit void xmgmt_exit(void)
> +{
> + pci_unregister_driver(&xmgmt_driver);
> + class_destroy(xmgmt_class);
> + xrt_subdev_unregister_external_driver(XRT_SUBDEV_MGMT_MAIN);
> +}
> +
> +module_init(xmgmt_init);
> +module_exit(xmgmt_exit);
> +
> +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
> +MODULE_VERSION(XMGMT_DRIVER_VERSION);
> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
> +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
> +MODULE_LICENSE("GPL v2");
> --
> 2.17.1
I have not yet looked at the whole thing, but why does the FPGA Manager
only copy things around?
Any reason your partitions cannot be modelled as FPGA regions/Bridges?
It would be helpful to split this up into smaller chunks, that make it
easier to review. The FPGA Manager driver should be a separate patch,
etc.
- Moritz
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
` (7 preceding siblings ...)
2020-11-30 18:08 ` [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Rob Herring
@ 2020-12-02 2:14 ` Xu Yilun
2020-12-02 5:33 ` Sonal Santan
2020-12-06 16:31 ` Tom Rix
9 siblings, 1 reply; 28+ messages in thread
From: Xu Yilun @ 2020-12-02 2:14 UTC (permalink / raw)
To: Sonal Santan
Cc: linux-kernel, Sonal Santan, linux-fpga, maxz, lizhih,
michal.simek, stefanos, devicetree
On Sat, Nov 28, 2020 at 04:00:32PM -0800, Sonal Santan wrote:
> Hello,
>
> This patch series adds management physical function driver for Xilinx Alveo PCIe
> accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> This driver is part of Xilinx Runtime (XRT) open source stack.
>
> The patch depends on the "PATCH Xilinx Alveo libfdt prep" which was posted
> before.
>
> ALVEO PLATFORM ARCHITECTURE
>
> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
> re-configurable *user* partition. The shell partition is automatically loaded from
> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
> till the next cold reboot. The shell exposes two PCIe physical functions:
>
> 1. management physical function
> 2. user physical function
>
> The patch series includes Documentation/xrt.rst which describes Alveo
> platform, xmgmt driver architecture and deployment model in more more detail.
>
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image
> using Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> tools. The image is packaged as xclbin and contains partial bitstream for the
> user partition and necessary metadata. Users can dynamically swap the image
> running on the user partition in order to switch between different workloads.
>
> ALVEO DRIVERS
>
> Alveo Linux kernel driver *xmgmt* binds to management physical function of
> Alveo platform. The modular driver framework is organized into several
> platform drivers which primarily handle the following functionality:
>
> 1. Loading firmware container also called xsabin at driver attach time
> 2. Loading of user compiled xclbin with FPGA Manager integration
> 3. Clock scaling of image running on user partition
> 4. In-band sensors: temp, voltage, power, etc.
> 5. Device reset and rescan
> 6. Flashing static *shell* partition
>
> The platform drivers are packaged into *xrt-lib* helper module with a well
> defined interfaces the details of which can be found in Documentation/xrt.rst.
>
> xmgmt driver is second generation Alveo management driver and evolution of
> the first generation (out of tree) Alveo management driver, xclmgmt. The
> sources of the first generation drivers were posted on LKML last year--
> https://lore.kernel.org/lkml/20190319215401.6562-1-sonal.santan@xilinx.com/
>
> Changes since the first generation driver include the following: the driver
> has been re-architected as data driven modular driver; the driver has been
> split into xmgmt and xrt-lib; user physical function driver has been removed
> from the patch series.
>
> Alveo/XRT security and platform architecture is documented on the following
> GitHub pages:
> https://xilinx.github.io/XRT/master/html/security.html
> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>
> User physical function driver is not included in this patch series.
>
> TESTING AND VALIDATION
>
> xmgmt driver can be tested with full XRT open source stack which includes
> user space libraries, board utilities and (out of tree) first generation
> user physical function driver xocl. XRT open source runtime stack is
> available at https://github.com/Xilinx/XRT. This patch series has been
> validated on Alveo U50 platform.
>
> Complete documentation for XRT open source stack can be found here--
> https://xilinx.github.io/XRT/master/html/index.html
>
> Thanks,
> -Sonal
>
> Sonal Santan (8):
> Documentation: fpga: Add a document describing Alveo XRT drivers
> fpga: xrt: Add UAPI header files
> fpga: xrt: infrastructure support for xmgmt driver
> fpga: xrt: core infrastructure for xrt-lib module
> fpga: xrt: platform drivers for subsystems in shell partition
Seems the Patch #5 is missing in this seriies.
Thanks,
Yilun
> fpga: xrt: header file for platform and parent drivers
> fpga: xrt: Alveo management physical function driver
> fpga: xrt: Kconfig and Makefile updates for XRT drivers
>
> Documentation/fpga/index.rst | 1 +
> Documentation/fpga/xrt.rst | 588 +++++
> drivers/fpga/Kconfig | 2 +
> drivers/fpga/Makefile | 3 +
> drivers/fpga/alveo/Kconfig | 7 +
> drivers/fpga/alveo/common/xrt-metadata.c | 590 +++++
> drivers/fpga/alveo/common/xrt-root.c | 744 +++++++
> drivers/fpga/alveo/common/xrt-root.h | 24 +
> drivers/fpga/alveo/common/xrt-xclbin.c | 387 ++++
> drivers/fpga/alveo/common/xrt-xclbin.h | 46 +
> drivers/fpga/alveo/include/xmgmt-main.h | 34 +
> drivers/fpga/alveo/include/xrt-axigate.h | 31 +
> drivers/fpga/alveo/include/xrt-calib.h | 28 +
> drivers/fpga/alveo/include/xrt-clkfreq.h | 21 +
> drivers/fpga/alveo/include/xrt-clock.h | 29 +
> drivers/fpga/alveo/include/xrt-cmc.h | 23 +
> drivers/fpga/alveo/include/xrt-ddr-srsr.h | 29 +
> drivers/fpga/alveo/include/xrt-flash.h | 28 +
> drivers/fpga/alveo/include/xrt-gpio.h | 41 +
> drivers/fpga/alveo/include/xrt-icap.h | 27 +
> drivers/fpga/alveo/include/xrt-mailbox.h | 44 +
> drivers/fpga/alveo/include/xrt-metadata.h | 184 ++
> drivers/fpga/alveo/include/xrt-parent.h | 103 +
> drivers/fpga/alveo/include/xrt-partition.h | 33 +
> drivers/fpga/alveo/include/xrt-subdev.h | 333 +++
> drivers/fpga/alveo/include/xrt-ucs.h | 22 +
> drivers/fpga/alveo/lib/Kconfig | 11 +
> drivers/fpga/alveo/lib/Makefile | 42 +
> drivers/fpga/alveo/lib/subdevs/xrt-axigate.c | 298 +++
> drivers/fpga/alveo/lib/subdevs/xrt-calib.c | 291 +++
> drivers/fpga/alveo/lib/subdevs/xrt-clkfreq.c | 214 ++
> drivers/fpga/alveo/lib/subdevs/xrt-clock.c | 638 ++++++
> .../fpga/alveo/lib/subdevs/xrt-cmc-bdinfo.c | 343 +++
> drivers/fpga/alveo/lib/subdevs/xrt-cmc-ctrl.c | 322 +++
> drivers/fpga/alveo/lib/subdevs/xrt-cmc-impl.h | 135 ++
> .../fpga/alveo/lib/subdevs/xrt-cmc-mailbox.c | 320 +++
> drivers/fpga/alveo/lib/subdevs/xrt-cmc-sc.c | 361 ++++
> .../fpga/alveo/lib/subdevs/xrt-cmc-sensors.c | 445 ++++
> drivers/fpga/alveo/lib/subdevs/xrt-cmc.c | 239 +++
> drivers/fpga/alveo/lib/subdevs/xrt-gpio.c | 198 ++
> drivers/fpga/alveo/lib/subdevs/xrt-icap.c | 306 +++
> drivers/fpga/alveo/lib/subdevs/xrt-mailbox.c | 1905 +++++++++++++++++
> .../fpga/alveo/lib/subdevs/xrt-partition.c | 261 +++
> drivers/fpga/alveo/lib/subdevs/xrt-qspi.c | 1347 ++++++++++++
> drivers/fpga/alveo/lib/subdevs/xrt-srsr.c | 322 +++
> drivers/fpga/alveo/lib/subdevs/xrt-test.c | 274 +++
> drivers/fpga/alveo/lib/subdevs/xrt-ucs.c | 238 ++
> .../fpga/alveo/lib/subdevs/xrt-vsec-golden.c | 238 ++
> drivers/fpga/alveo/lib/subdevs/xrt-vsec.c | 337 +++
> drivers/fpga/alveo/lib/xrt-cdev.c | 234 ++
> drivers/fpga/alveo/lib/xrt-main.c | 275 +++
> drivers/fpga/alveo/lib/xrt-main.h | 46 +
> drivers/fpga/alveo/lib/xrt-subdev.c | 1007 +++++++++
> drivers/fpga/alveo/mgmt/Kconfig | 11 +
> drivers/fpga/alveo/mgmt/Makefile | 28 +
> drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c | 194 ++
> drivers/fpga/alveo/mgmt/xmgmt-fmgr.h | 29 +
> drivers/fpga/alveo/mgmt/xmgmt-main-impl.h | 36 +
> drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c | 930 ++++++++
> drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c | 190 ++
> drivers/fpga/alveo/mgmt/xmgmt-main.c | 843 ++++++++
> drivers/fpga/alveo/mgmt/xmgmt-root.c | 375 ++++
> include/uapi/linux/xrt/flash_xrt_data.h | 67 +
> include/uapi/linux/xrt/mailbox_proto.h | 394 ++++
> include/uapi/linux/xrt/mailbox_transport.h | 74 +
> include/uapi/linux/xrt/xclbin.h | 418 ++++
> include/uapi/linux/xrt/xmgmt-ioctl.h | 72 +
> 67 files changed, 17710 insertions(+)
> create mode 100644 Documentation/fpga/xrt.rst
> create mode 100644 drivers/fpga/alveo/Kconfig
> create mode 100644 drivers/fpga/alveo/common/xrt-metadata.c
> create mode 100644 drivers/fpga/alveo/common/xrt-root.c
> create mode 100644 drivers/fpga/alveo/common/xrt-root.h
> create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.c
> create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.h
> create mode 100644 drivers/fpga/alveo/include/xmgmt-main.h
> create mode 100644 drivers/fpga/alveo/include/xrt-axigate.h
> create mode 100644 drivers/fpga/alveo/include/xrt-calib.h
> create mode 100644 drivers/fpga/alveo/include/xrt-clkfreq.h
> create mode 100644 drivers/fpga/alveo/include/xrt-clock.h
> create mode 100644 drivers/fpga/alveo/include/xrt-cmc.h
> create mode 100644 drivers/fpga/alveo/include/xrt-ddr-srsr.h
> create mode 100644 drivers/fpga/alveo/include/xrt-flash.h
> create mode 100644 drivers/fpga/alveo/include/xrt-gpio.h
> create mode 100644 drivers/fpga/alveo/include/xrt-icap.h
> create mode 100644 drivers/fpga/alveo/include/xrt-mailbox.h
> create mode 100644 drivers/fpga/alveo/include/xrt-metadata.h
> create mode 100644 drivers/fpga/alveo/include/xrt-parent.h
> create mode 100644 drivers/fpga/alveo/include/xrt-partition.h
> create mode 100644 drivers/fpga/alveo/include/xrt-subdev.h
> create mode 100644 drivers/fpga/alveo/include/xrt-ucs.h
> create mode 100644 drivers/fpga/alveo/lib/Kconfig
> create mode 100644 drivers/fpga/alveo/lib/Makefile
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-axigate.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-calib.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-clkfreq.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-clock.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-bdinfo.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-ctrl.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-impl.h
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-mailbox.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-sc.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-sensors.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-gpio.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-icap.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-mailbox.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-partition.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-qspi.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-srsr.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-test.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-ucs.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-vsec-golden.c
> create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-vsec.c
> create mode 100644 drivers/fpga/alveo/lib/xrt-cdev.c
> create mode 100644 drivers/fpga/alveo/lib/xrt-main.c
> create mode 100644 drivers/fpga/alveo/lib/xrt-main.h
> create mode 100644 drivers/fpga/alveo/lib/xrt-subdev.c
> create mode 100644 drivers/fpga/alveo/mgmt/Kconfig
> create mode 100644 drivers/fpga/alveo/mgmt/Makefile
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main.c
> create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-root.c
> create mode 100644 include/uapi/linux/xrt/flash_xrt_data.h
> create mode 100644 include/uapi/linux/xrt/mailbox_proto.h
> create mode 100644 include/uapi/linux/xrt/mailbox_transport.h
> create mode 100644 include/uapi/linux/xrt/xclbin.h
> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>
> --
> 2.17.1
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver
2020-11-29 0:00 ` [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver Sonal Santan
2020-12-01 20:51 ` Moritz Fischer
@ 2020-12-02 3:00 ` Xu Yilun
2020-12-04 4:40 ` Max Zhen
1 sibling, 1 reply; 28+ messages in thread
From: Xu Yilun @ 2020-12-02 3:00 UTC (permalink / raw)
To: Sonal Santan
Cc: linux-kernel, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
> +static int xmgmt_main_event_cb(struct platform_device *pdev,
> + enum xrt_events evt, void *arg)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev *)arg;
> + enum xrt_subdev_id id;
> + int instance;
> + size_t fwlen;
> +
> + switch (evt) {
> + case XRT_EVENT_POST_CREATION: {
> + id = esd->xevt_subdev_id;
> + instance = esd->xevt_subdev_instance;
> + xrt_info(pdev, "processing event %d for (%d, %d)",
> + evt, id, instance);
> +
> + if (id == XRT_SUBDEV_GPIO)
> + xmm->gpio_ready = true;
> + else if (id == XRT_SUBDEV_QSPI)
> + xmm->flash_ready = true;
> + else
> + BUG_ON(1);
> +
> + if (xmm->gpio_ready && xmm->flash_ready) {
> + int rc;
> +
> + rc = load_firmware_from_disk(pdev, &xmm->firmware_blp,
> + &fwlen);
> + if (rc != 0) {
> + rc = load_firmware_from_flash(pdev,
> + &xmm->firmware_blp, &fwlen);
I'm curious that before the shell metadata is loaded, how the QSPI
subdev is enumerated and get to work? The QSPI DT info itself is
stored in metadata, is it?
I didn't find the creation of leaf platform devices, maybe I can find
the answer in the missing Patch #5?
Thanks,
Yilun
> + }
> + if (rc == 0 && is_valid_firmware(pdev,
> + xmm->firmware_blp, fwlen))
> + (void) xmgmt_create_blp(xmm);
> + else
> + xrt_err(pdev,
> + "failed to find firmware, giving up");
> + xmm->evt_hdl = NULL;
> + }
> + break;
> + }
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
2020-12-02 2:14 ` Xu Yilun
@ 2020-12-02 5:33 ` Sonal Santan
0 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-12-02 5:33 UTC (permalink / raw)
To: Xu Yilun
Cc: linux-kernel, linux-fpga, Max Zhen, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
> -----Original Message-----
> From: Xu Yilun <yilun.xu@intel.com>
> Sent: Tuesday, December 1, 2020 6:14 PM
> To: Sonal Santan <sonals@xilinx.com>
> Cc: linux-kernel@vger.kernel.org; Sonal Santan <sonals@xilinx.com>; linux-
> fpga@vger.kernel.org; Max Zhen <maxz@xilinx.com>; Lizhi Hou
> <lizhih@xilinx.com>; Michal Simek <michals@xilinx.com>; Stefano Stabellini
> <stefanos@xilinx.com>; devicetree@vger.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
>
> On Sat, Nov 28, 2020 at 04:00:32PM -0800, Sonal Santan wrote:
> > Hello,
> >
> > This patch series adds management physical function driver for Xilinx
> > Alveo PCIe accelerator cards,
> > https://www.xilinx.com/products/boards-and-kits/alveo.html
> > This driver is part of Xilinx Runtime (XRT) open source stack.
> >
> > The patch depends on the "PATCH Xilinx Alveo libfdt prep" which was
> > posted before.
> >
> > ALVEO PLATFORM ARCHITECTURE
> >
> > Alveo PCIe FPGA based platforms have a static *shell* partition and a
> > partial re-configurable *user* partition. The shell partition is
> > automatically loaded from flash when host is booted and PCIe is
> > enumerated by BIOS. Shell cannot be changed till the next cold reboot. The
> shell exposes two PCIe physical functions:
> >
> > 1. management physical function
> > 2. user physical function
> >
> > The patch series includes Documentation/xrt.rst which describes Alveo
> > platform, xmgmt driver architecture and deployment model in more more
> detail.
> >
> > Users compile their high level design in C/C++/OpenCL or RTL into FPGA
> > image using Vitis
> > https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> > tools. The image is packaged as xclbin and contains partial bitstream
> > for the user partition and necessary metadata. Users can dynamically
> > swap the image running on the user partition in order to switch between
> different workloads.
> >
> > ALVEO DRIVERS
> >
> > Alveo Linux kernel driver *xmgmt* binds to management physical
> > function of Alveo platform. The modular driver framework is organized
> > into several platform drivers which primarily handle the following
> functionality:
> >
> > 1. Loading firmware container also called xsabin at driver attach
> > time 2. Loading of user compiled xclbin with FPGA Manager integration
> > 3. Clock scaling of image running on user partition 4. In-band
> > sensors: temp, voltage, power, etc.
> > 5. Device reset and rescan
> > 6. Flashing static *shell* partition
> >
> > The platform drivers are packaged into *xrt-lib* helper module with a
> > well defined interfaces the details of which can be found in
> Documentation/xrt.rst.
> >
> > xmgmt driver is second generation Alveo management driver and
> > evolution of the first generation (out of tree) Alveo management
> > driver, xclmgmt. The sources of the first generation drivers were
> > posted on LKML last year--
> > https://lore.kernel.org/lkml/20190319215401.6562-1-sonal.santan@xilinx
> > .com/
> >
> > Changes since the first generation driver include the following: the
> > driver has been re-architected as data driven modular driver; the
> > driver has been split into xmgmt and xrt-lib; user physical function
> > driver has been removed from the patch series.
> >
> > Alveo/XRT security and platform architecture is documented on the
> > following GitHub pages:
> > https://xilinx.github.io/XRT/master/html/security.html
> > https://xilinx.github.io/XRT/master/html/platforms_partitions.html
> >
> > User physical function driver is not included in this patch series.
> >
> > TESTING AND VALIDATION
> >
> > xmgmt driver can be tested with full XRT open source stack which
> > includes user space libraries, board utilities and (out of tree) first
> > generation user physical function driver xocl. XRT open source runtime
> > stack is available at https://github.com/Xilinx/XRT. This patch series
> > has been validated on Alveo U50 platform.
> >
> > Complete documentation for XRT open source stack can be found here--
> > https://xilinx.github.io/XRT/master/html/index.html
> >
> > Thanks,
> > -Sonal
> >
> > Sonal Santan (8):
> > Documentation: fpga: Add a document describing Alveo XRT drivers
> > fpga: xrt: Add UAPI header files
> > fpga: xrt: infrastructure support for xmgmt driver
> > fpga: xrt: core infrastructure for xrt-lib module
> > fpga: xrt: platform drivers for subsystems in shell partition
>
> Seems the Patch #5 is missing in this seriies.
Patch #5 was posted along with the rest. You can find a copy here--
https://lore.kernel.org/lkml/20201129000040.24777-6-sonals@xilinx.com/
Thanks,
Sonal
>
> Thanks,
> Yilun
>
> > fpga: xrt: header file for platform and parent drivers
> > fpga: xrt: Alveo management physical function driver
> > fpga: xrt: Kconfig and Makefile updates for XRT drivers
> >
> > Documentation/fpga/index.rst | 1 +
> > Documentation/fpga/xrt.rst | 588 +++++
> > drivers/fpga/Kconfig | 2 +
> > drivers/fpga/Makefile | 3 +
> > drivers/fpga/alveo/Kconfig | 7 +
> > drivers/fpga/alveo/common/xrt-metadata.c | 590 +++++
> > drivers/fpga/alveo/common/xrt-root.c | 744 +++++++
> > drivers/fpga/alveo/common/xrt-root.h | 24 +
> > drivers/fpga/alveo/common/xrt-xclbin.c | 387 ++++
> > drivers/fpga/alveo/common/xrt-xclbin.h | 46 +
> > drivers/fpga/alveo/include/xmgmt-main.h | 34 +
> > drivers/fpga/alveo/include/xrt-axigate.h | 31 +
> > drivers/fpga/alveo/include/xrt-calib.h | 28 +
> > drivers/fpga/alveo/include/xrt-clkfreq.h | 21 +
> > drivers/fpga/alveo/include/xrt-clock.h | 29 +
> > drivers/fpga/alveo/include/xrt-cmc.h | 23 +
> > drivers/fpga/alveo/include/xrt-ddr-srsr.h | 29 +
> > drivers/fpga/alveo/include/xrt-flash.h | 28 +
> > drivers/fpga/alveo/include/xrt-gpio.h | 41 +
> > drivers/fpga/alveo/include/xrt-icap.h | 27 +
> > drivers/fpga/alveo/include/xrt-mailbox.h | 44 +
> > drivers/fpga/alveo/include/xrt-metadata.h | 184 ++
> > drivers/fpga/alveo/include/xrt-parent.h | 103 +
> > drivers/fpga/alveo/include/xrt-partition.h | 33 +
> > drivers/fpga/alveo/include/xrt-subdev.h | 333 +++
> > drivers/fpga/alveo/include/xrt-ucs.h | 22 +
> > drivers/fpga/alveo/lib/Kconfig | 11 +
> > drivers/fpga/alveo/lib/Makefile | 42 +
> > drivers/fpga/alveo/lib/subdevs/xrt-axigate.c | 298 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-calib.c | 291 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-clkfreq.c | 214 ++
> > drivers/fpga/alveo/lib/subdevs/xrt-clock.c | 638 ++++++
> > .../fpga/alveo/lib/subdevs/xrt-cmc-bdinfo.c | 343 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-cmc-ctrl.c | 322 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-cmc-impl.h | 135 ++
> > .../fpga/alveo/lib/subdevs/xrt-cmc-mailbox.c | 320 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-cmc-sc.c | 361 ++++
> > .../fpga/alveo/lib/subdevs/xrt-cmc-sensors.c | 445 ++++
> > drivers/fpga/alveo/lib/subdevs/xrt-cmc.c | 239 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-gpio.c | 198 ++
> > drivers/fpga/alveo/lib/subdevs/xrt-icap.c | 306 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-mailbox.c | 1905 +++++++++++++++++
> > .../fpga/alveo/lib/subdevs/xrt-partition.c | 261 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-qspi.c | 1347 ++++++++++++
> > drivers/fpga/alveo/lib/subdevs/xrt-srsr.c | 322 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-test.c | 274 +++
> > drivers/fpga/alveo/lib/subdevs/xrt-ucs.c | 238 ++
> > .../fpga/alveo/lib/subdevs/xrt-vsec-golden.c | 238 ++
> > drivers/fpga/alveo/lib/subdevs/xrt-vsec.c | 337 +++
> > drivers/fpga/alveo/lib/xrt-cdev.c | 234 ++
> > drivers/fpga/alveo/lib/xrt-main.c | 275 +++
> > drivers/fpga/alveo/lib/xrt-main.h | 46 +
> > drivers/fpga/alveo/lib/xrt-subdev.c | 1007 +++++++++
> > drivers/fpga/alveo/mgmt/Kconfig | 11 +
> > drivers/fpga/alveo/mgmt/Makefile | 28 +
> > drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c | 194 ++
> > drivers/fpga/alveo/mgmt/xmgmt-fmgr.h | 29 +
> > drivers/fpga/alveo/mgmt/xmgmt-main-impl.h | 36 +
> > drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c | 930 ++++++++
> > drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c | 190 ++
> > drivers/fpga/alveo/mgmt/xmgmt-main.c | 843 ++++++++
> > drivers/fpga/alveo/mgmt/xmgmt-root.c | 375 ++++
> > include/uapi/linux/xrt/flash_xrt_data.h | 67 +
> > include/uapi/linux/xrt/mailbox_proto.h | 394 ++++
> > include/uapi/linux/xrt/mailbox_transport.h | 74 +
> > include/uapi/linux/xrt/xclbin.h | 418 ++++
> > include/uapi/linux/xrt/xmgmt-ioctl.h | 72 +
> > 67 files changed, 17710 insertions(+) create mode 100644
> > Documentation/fpga/xrt.rst create mode 100644
> > drivers/fpga/alveo/Kconfig create mode 100644
> > drivers/fpga/alveo/common/xrt-metadata.c
> > create mode 100644 drivers/fpga/alveo/common/xrt-root.c
> > create mode 100644 drivers/fpga/alveo/common/xrt-root.h
> > create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.c
> > create mode 100644 drivers/fpga/alveo/common/xrt-xclbin.h
> > create mode 100644 drivers/fpga/alveo/include/xmgmt-main.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-axigate.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-calib.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-clkfreq.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-clock.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-cmc.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-ddr-srsr.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-flash.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-gpio.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-icap.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-mailbox.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-metadata.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-parent.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-partition.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-subdev.h
> > create mode 100644 drivers/fpga/alveo/include/xrt-ucs.h
> > create mode 100644 drivers/fpga/alveo/lib/Kconfig create mode 100644
> > drivers/fpga/alveo/lib/Makefile create mode 100644
> > drivers/fpga/alveo/lib/subdevs/xrt-axigate.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-calib.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-clkfreq.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-clock.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-bdinfo.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-ctrl.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-impl.h
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-mailbox.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-sc.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc-sensors.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-cmc.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-gpio.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-icap.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-mailbox.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-partition.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-qspi.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-srsr.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-test.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-ucs.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-vsec-golden.c
> > create mode 100644 drivers/fpga/alveo/lib/subdevs/xrt-vsec.c
> > create mode 100644 drivers/fpga/alveo/lib/xrt-cdev.c create mode
> > 100644 drivers/fpga/alveo/lib/xrt-main.c create mode 100644
> > drivers/fpga/alveo/lib/xrt-main.h create mode 100644
> > drivers/fpga/alveo/lib/xrt-subdev.c
> > create mode 100644 drivers/fpga/alveo/mgmt/Kconfig create mode
> > 100644 drivers/fpga/alveo/mgmt/Makefile create mode 100644
> > drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-root.c
> > create mode 100644 include/uapi/linux/xrt/flash_xrt_data.h
> > create mode 100644 include/uapi/linux/xrt/mailbox_proto.h
> > create mode 100644 include/uapi/linux/xrt/mailbox_transport.h
> > create mode 100644 include/uapi/linux/xrt/xclbin.h create mode
> > 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
> >
> > --
> > 2.17.1
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files
2020-12-01 4:27 ` Moritz Fischer
@ 2020-12-02 18:57 ` Sonal Santan
2020-12-02 23:47 ` Moritz Fischer
0 siblings, 1 reply; 28+ messages in thread
From: Sonal Santan @ 2020-12-02 18:57 UTC (permalink / raw)
To: Moritz Fischer
Cc: linux-kernel, linux-fpga, Max Zhen, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
Hi Moritz,
> -----Original Message-----
> From: Moritz Fischer <mdf@kernel.org>
> Sent: Monday, November 30, 2020 8:27 PM
> To: Sonal Santan <sonals@xilinx.com>
> Cc: linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Max Zhen
> <maxz@xilinx.com>; Lizhi Hou <lizhih@xilinx.com>; Michal Simek
> <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> devicetree@vger.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files
>
> Hi Sonal,
>
> On Sat, Nov 28, 2020 at 04:00:34PM -0800, Sonal Santan wrote:
> > From: Sonal Santan <sonal.santan@xilinx.com>
> >
> > Add XRT UAPI header files which describe flash layout, XRT mailbox
> > protocol, xclBin/axlf FPGA image container format and XRT management
> > physical function driver ioctl interfaces.
> >
> > flash_xrt_data.h:
> > Layout used by XRT to store private data on flash.
> >
> > mailbox_proto.h:
> > Mailbox opcodes and high level data structures representing various
> > kinds of information like sensors, clock, etc.
> >
> > mailbox_transport.h:
> > Transport protocol used by mailbox.
> >
> > xclbin.h:
> > Container format used to store compiled FPGA image which includes
> > bitstream and metadata.
>
> Can these headers be introduced together with the code that uses them as
> logical change?
>
> I haven't looked too closely, but it helps reviewing if you can break it into
> smaller pieces that can stand by themselves.
>
These UAPI header files are used by multiple source files hence I wanted to get
these reviewed separately. However if this is getting in the way, in the next
version of the patch series I would look into arranging the files differently.
You can browse the changes here as well--
https://github.com/Xilinx/linux-xoclv2/tree/xrtv2-A
Thanks,
-Sonal
> Thanks,
> Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-12-01 4:54 ` Moritz Fischer
@ 2020-12-02 21:24 ` Max Zhen
2020-12-02 23:10 ` Moritz Fischer
0 siblings, 1 reply; 28+ messages in thread
From: Max Zhen @ 2020-12-02 21:24 UTC (permalink / raw)
To: Moritz Fischer, Sonal Santan
Cc: linux-kernel, linux-fpga, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
Hi Moritz,
Thanks for your feedback. Please see my reply inline.
Thanks,
-Max
> -----Original Message-----
> From: Moritz Fischer <mdf@kernel.org>
> Sent: Monday, November 30, 2020 20:55
> To: Sonal Santan <sonals@xilinx.com>
> Cc: linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Max Zhen
> <maxz@xilinx.com>; Lizhi Hou <lizhih@xilinx.com>; Michal Simek
> <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> devicetree@vger.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document
> describing Alveo XRT drivers
>
>
> On Sat, Nov 28, 2020 at 04:00:33PM -0800, Sonal Santan wrote:
> > From: Sonal Santan <sonal.santan@xilinx.com>
> >
> > Describe Alveo XRT driver architecture and provide basic overview of
> > Xilinx Alveo platform.
> >
> > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > ---
> > Documentation/fpga/index.rst | 1 +
> > Documentation/fpga/xrt.rst | 588
> +++++++++++++++++++++++++++++++++++
> > 2 files changed, 589 insertions(+)
> > create mode 100644 Documentation/fpga/xrt.rst
> >
> > diff --git a/Documentation/fpga/index.rst
> > b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d
> 100644
> > --- a/Documentation/fpga/index.rst
> > +++ b/Documentation/fpga/index.rst
> > @@ -8,6 +8,7 @@ fpga
> > :maxdepth: 1
> >
> > dfl
> > + xrt
> >
> > .. only:: subproject and html
> >
> > diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
> > new file mode 100644 index 000000000000..9f37d46459b0
> > --- /dev/null
> > +++ b/Documentation/fpga/xrt.rst
> > @@ -0,1 +1,588 @@
> > +==================================
> > +XRTV2 Linux Kernel Driver Overview
> > +==================================
> > +
> > +XRTV2 drivers are second generation `XRT
> > +<https://github.com/Xilinx/XRT>`_ drivers which support `Alveo
> > +<https://www.xilinx.com/products/boards-and-kits/alveo.html>`_ PCIe
> platforms from Xilinx.
> > +
> > +XRTV2 drivers support *subsystem* style data driven platforms where
> > +driver's configuration and behavior is determined by meta data provided
> by platform (in *device tree* format).
> > +Primary management physical function (MPF) driver is called
> > +**xmgmt**. Primary user physical function (UPF) driver is called
> > +**xuser** and HW subsystem drivers are packaged into a library module
> called **xrt-lib**, which is shared by **xmgmt** and **xuser** (WIP).
> WIP?
Working in progress. I'll expand it in the doc.
> > +
> > +Alveo Platform Overview
> > +=======================
> > +
> > +Alveo platforms are architected as two physical FPGA partitions:
> > +*Shell* and *User*. Shell
> Nit: The Shell provides ...
Sure. Will fix.
> > +provides basic infrastructure for the Alveo platform like PCIe
> > +connectivity, board management, Dynamic Function Exchange (DFX),
> > +sensors, clocking, reset, and security. User partition contains user
> compiled binary which is loaded by a process called DFX also known as partial
> reconfiguration.
> > +
> > +Physical partitions require strict HW compatibility with each other for DFX
> to work properly.
> > +Every physical partition has two interface UUIDs: *parent* UUID and
> > +*child* UUID. For simple single stage platforms Shell → User forms
> > +parent child relationship. For complex two stage platforms Base → Shell
> → User forms the parent child relationship chain.
> > +
> > +.. note::
> > + Partition compatibility matching is key design component of Alveo
> platforms and XRT. Partitions
> > + have child and parent relationship. A loaded partition exposes child
> partition UUID to advertise
> > + its compatibility requirement for child partition. When loading a child
> partition the xmgmt
> > + management driver matches parent UUID of the child partition against
> child UUID exported by the
> > + parent. Parent and child partition UUIDs are stored in the *xclbin* (for
> user) or *xsabin* (for
> > + base and shell). Except for root UUID, VSEC, hardware itself does not
> know about UUIDs. UUIDs are
> > + stored in xsabin and xclbin.
> > +
> > +
> > +The physical partitions and their loading is illustrated below::
> > +
> > + SHELL USER
> > + +-----------+ +-------------------+
> > + | | | |
> > + | VSEC UUID | CHILD PARENT | LOGIC UUID |
> > + | o------->|<--------o |
> > + | | UUID UUID | |
> > + +-----+-----+ +--------+----------+
> > + | |
> > + . .
> > + | |
> > + +---+---+ +------+--------+
> > + | POR | | USER COMPILED |
> > + | FLASH | | XCLBIN |
> > + +-------+ +---------------+
> > +
> > +
> > +Loading Sequence
> > +----------------
> > +
> > +Shell partition is loaded from flash at system boot time. It
> > +establishes the PCIe link and exposes
> Nit: The Shell
Will fix.
> > +two physical functions to the BIOS. After OS boot, xmgmt driver
> > +attaches to PCIe physical function
> > +0 exposed by the Shell and then looks for VSEC in PCIe extended
> > +configuration space. Using VSEC it determines the logic UUID of Shell
> > +and uses the UUID to load matching *xsabin* file from Linux firmware
> > +directory. The xsabin file contains metadata to discover peripherals that
> are part of Shell and firmware(s) for any embedded soft processors in Shell.
>
> Neat.
Thanks :-).
> > +
> > +Shell exports child interface UUID which is used for compatibility
> > +check when loading user compiled
> Nit: The Shell
Sure.
> > +xclbin over the User partition as part of DFX. When a user requests
> > +loading of a specific xclbin the xmgmt management driver reads the
> > +parent interface UUID specified in the xclbin and matches it with
> > +child interface UUID exported by Shell to determine if xclbin is compatible
> with the Shell. If match fails loading of xclbin is denied.
> > +
> > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
> > +When loading xclbin xmgmt driver performs the following operations:
> > +
> > +1. Sanity check the xclbin contents
> > +2. Isolate the User partition
> > +3. Download the bitstream using the FPGA config engine (ICAP) 4.
> > +De-isolate the User partition
> Is this modelled as bridges and regions?
Alveo drivers as written today do not use fpga bridge and region framework. It seems that if we add support for that framework, it’s possible to receive PR program request from kernel outside of xmgmt driver? Currently, we can’t support this and PR program can only be initiated using XRT’s runtime API in user space.
Or maybe we have missed some points about the use case for this framework?
>
> > +5. Program the clocks (ClockWiz) driving the User partition 6. Wait
> > +for memory controller (MIG) calibration
> > +
> > +`Platform Loading Overview
> > +<https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`
> > +_ provides more detailed information on platform loading.
> > +
> > +xsabin
> > +------
> > +
> > +Each Alveo platform comes packaged with its own xsabin. The xsabin is
> > +trusted component of the platform. For format details refer to
> > +:ref:`xsabin/xclbin Container Format`. xsabin contains basic
> > +information like UUIDs, platform name and metadata in the form of
> device tree. See :ref:`Device Tree Usage` for details and example.
> > +
> > +xclbin
> > +------
> > +
> > +xclbin is compiled by end user using
> > +`Vitis
> > +<https://www.xilinx.com/products/design-tools/vitis/vitis-platform.ht
> > +ml>`_ tool set from Xilinx. The xclbin contains sections describing
> > +user compiled acceleration engines/kernels, memory subsystems,
> clocking information etc. It also contains bitstream for the user partition,
> UUIDs, platform name, etc. xclbin uses the same container format as xsabin
> which is described below.
> > +
> > +
> > +xsabin/xclbin Container Format
> > +------------------------------
> > +
> > +xclbin/xsabin is ELF-like binary container format. It is structured as series
> of sections.
> > +There is a file header followed by several section headers which is
> followed by sections.
> > +A section header points to an actual section. There is an optional signature
> at the end.
> > +The format is defined by header file ``xclbin.h``. The following
> > +figure illustrates a typical xclbin::
> > +
> > +
> > + +---------------------+
> > + | |
> > + | HEADER |
> > + +---------------------+
> > + | SECTION HEADER |
> > + | |
> > + +---------------------+
> > + | ... |
> > + | |
> > + +---------------------+
> > + | SECTION HEADER |
> > + | |
> > + +---------------------+
> > + | SECTION |
> > + | |
> > + +---------------------+
> > + | ... |
> > + | |
> > + +---------------------+
> > + | SECTION |
> > + | |
> > + +---------------------+
> > + | SIGNATURE |
> > + | (OPTIONAL) |
> > + +---------------------+
> > +
> > +
> > +xclbin/xsabin files can be packaged, un-packaged and inspected using
> > +XRT utility called **xclbinutil**. xclbinutil is part of XRT open
> > +source software stack. The source code for xclbinutil can be found at
> > +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbi
> > +nutil
> > +
> > +For example to enumerate the contents of a xclbin/xsabin use the
> > +*--info* switch as shown
> > +below::
> > +
> > + xclbinutil --info --input
> > + /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> > + xclbinutil --info --input
> > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsab
> > + in
> > +
> > +
> > +Device Tree Usage
> > +-----------------
> > +
> > +As mentioned previously xsabin stores metadata which advertise HW
> subsystems present in a partition.
> > +The metadata is stored in device tree format with well defined
> > +schema. Subsystem instantiations are captured as children of
> > +``addressable_endpoints`` node. Subsystem nodes have standard
> attributes like ``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe
> specific attributes:
> > +``pcie_physical_function`` and ``pcie_bar_mapping``. These identify
> > +which PCIe physical function and which BAR space in that physical
> > +function the subsystem resides. XRT management driver uses this
> > +information to bind *platform drivers* to the subsystem
> > +instantiations. The platform drivers are found in **xrt-lib.ko**
> > +kernel module defined later. Below is an example of device tree for
> > +Alveo U50
> > +platform::
>
> I might be missing something, but couldn't you structure the addressable
> endpoints in a way that encode the physical function as a parent / child
> relation?
Alveo driver does not generate the metadata. The metadata is formatted and generated by HW tools when the Alveo HW platform is built.
>
> What are the regs relative to?
Regs indicates offset of the register on the PCIE BAR of the Alveo device.
> > +
> > + /dts-v1/;
> > +
> > + /{
> > + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> > +
> > + schema_version {
> > + major = <0x01>;
> > + minor = <0x00>;
> > + };
> > +
> > + interfaces {
> > +
> > + @0 {
> > + interface_uuid = "862c7020a250293e32036f19956669e5";
> > + };
> > + };
> > +
> > + addressable_endpoints {
> > +
> > + ep_blp_rom_00 {
> > + reg = <0x00 0x1f04000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> > + };
> > +
> > + ep_card_flash_program_00 {
> > + reg = <0x00 0x1f06000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_quad_spi-
> 1.0\0axi_quad_spi";
> > + interrupts = <0x03 0x03>;
> > + };
> > +
> > + ep_cmc_firmware_mem_00 {
> > + reg = <0x00 0x1e20000 0x00 0x20000>;
> > + pcie_physical_function = <0x00>;
> > + compatible =
> > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > +
> > + firmware {
> > + firmware_product_name = "cmc";
> > + firmware_branch_name = "u50";
> > + firmware_version_major = <0x01>;
> > + firmware_version_minor = <0x00>;
> > + };
> > + };
> > +
> > + ep_cmc_intc_00 {
> > + reg = <0x00 0x1e03000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > + interrupts = <0x04 0x04>;
> > + };
> > +
> > + ep_cmc_mutex_00 {
> > + reg = <0x00 0x1e02000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_cmc_regmap_00 {
> > + reg = <0x00 0x1e08000 0x00 0x2000>;
> > + pcie_physical_function = <0x00>;
> > + compatible =
> > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > +
> > + firmware {
> > + firmware_product_name = "sc-fw";
> > + firmware_branch_name = "u50";
> > + firmware_version_major = <0x05>;
> > + };
> > + };
> > +
> > + ep_cmc_reset_00 {
> > + reg = <0x00 0x1e01000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_ddr_mem_calib_00 {
> > + reg = <0x00 0x63000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_debug_bscan_mgmt_00 {
> > + reg = <0x00 0x1e90000 0x00 0x10000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-debug_bridge-
> 1.0\0debug_bridge";
> > + };
> > +
> > + ep_ert_base_address_00 {
> > + reg = <0x00 0x21000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_ert_command_queue_mgmt_00 {
> > + reg = <0x00 0x40000 0x00 0x10000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> > + };
> > +
> > + ep_ert_command_queue_user_00 {
> > + reg = <0x00 0x40000 0x00 0x10000>;
> > + pcie_physical_function = <0x01>;
> > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> > + };
> > +
> > + ep_ert_firmware_mem_00 {
> > + reg = <0x00 0x30000 0x00 0x8000>;
> > + pcie_physical_function = <0x00>;
> > + compatible =
> > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > +
> > + firmware {
> > + firmware_product_name = "ert";
> > + firmware_branch_name = "v20";
> > + firmware_version_major = <0x01>;
> > + };
> > + };
> > +
> > + ep_ert_intc_00 {
> > + reg = <0x00 0x23000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > + interrupts = <0x05 0x05>;
> > + };
> > +
> > + ep_ert_reset_00 {
> > + reg = <0x00 0x22000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_ert_sched_00 {
> > + reg = <0x00 0x50000 0x00 0x1000>;
> > + pcie_physical_function = <0x01>;
> > + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> > + interrupts = <0x09 0x0c>;
> > + };
> > +
> > + ep_fpga_configuration_00 {
> > + reg = <0x00 0x1e88000 0x00 0x8000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
> > + interrupts = <0x02 0x02>;
> > + };
> > +
> > + ep_icap_reset_00 {
> > + reg = <0x00 0x1f07000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_mailbox_mgmt_00 {
> > + reg = <0x00 0x1f10000 0x00 0x10000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
> > + interrupts = <0x00 0x00>;
> > + };
> > +
> > + ep_mailbox_user_00 {
> > + reg = <0x00 0x1f00000 0x00 0x10000>;
> > + pcie_physical_function = <0x01>;
> > + compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
> > + interrupts = <0x08 0x08>;
> > + };
> > +
> > + ep_msix_00 {
> > + reg = <0x00 0x00 0x00 0x20000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> > + pcie_bar_mapping = <0x02>;
> > + };
> > +
> > + ep_pcie_link_mon_00 {
> > + reg = <0x00 0x1f05000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_pr_isolate_plp_00 {
> > + reg = <0x00 0x1f01000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_pr_isolate_ulp_00 {
> > + reg = <0x00 0x1000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_uuid_rom_00 {
> > + reg = <0x00 0x64000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> > + };
> > +
> > + ep_xdma_00 {
> > + reg = <0x00 0x00 0x00 0x10000>;
> > + pcie_physical_function = <0x01>;
> > + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> > + pcie_bar_mapping = <0x02>;
> > + };
> > + };
> > +
> > + }
> > +
> > +
> > +
> > +Deployment Models
> > +=================
> > +
> > +Baremetal
> > +---------
> > +
> > +In bare-metal deployments both MPF and UPF are visible and
> > +accessible. xmgmt driver binds to MPF. xmgmt driver operations are
> > +privileged and available to system administrator. The full stack is
> illustrated below::
> > +
> > +
> > + HOST
> > +
> > + [XMGMT] [XUSER]
> > + | |
> > + | |
> > + +-----+ +-----+
> > + | MPF | | UPF |
> > + | | | |
> > + | PF0 | | PF1 |
> > + +--+--+ +--+--+
> > + ......... ^................. ^..........
> > + | |
> > + | PCIe DEVICE |
> > + | |
> > + +--+------------------+--+
> > + | SHELL |
> > + | |
> > + +------------------------+
> > + | USER |
> > + | |
> > + | |
> > + | |
> > + | |
> > + +------------------------+
> > +
> > +
> > +
> > +Virtualized
> > +-----------
> > +
> > +In virtualized deployments privileged MPF is assigned to host but
> > +unprivileged UPF is assigned to guest VM via PCIe pass-through. xmgmt
> driver in host binds to MPF.
> > +xmgmt driver operations are privileged and only accessible by hosting
> service provider.
> > +The full stack is illustrated below::
> > +
> > +
> > + .............
> > + HOST . VM .
> > + . .
> > + [XMGMT] . [XUSER] .
> > + | . | .
> > + | . | .
> > + +-----+ . +-----+ .
> > + | MPF | . | UPF | .
> > + | | . | | .
> > + | PF0 | . | PF1 | .
> > + +--+--+ . +--+--+ .
> > + ......... ^................. ^..........
> > + | |
> > + | PCIe DEVICE |
> > + | |
> > + +--+------------------+--+
> > + | SHELL |
> > + | |
> > + +------------------------+
> > + | USER |
> > + | |
> > + | |
> > + | |
> > + | |
> > + +------------------------+
> > +
> > +
> > +
> > +Driver Modules
> > +==============
> > +
> > +xrt-lib.ko
> > +----------
> > +
> > +Repository of all subsystem drivers and pure software modules that
> > +can potentially be shared between xmgmt and xuser. All these drivers
> > +are structured as Linux *platform driver* and are instantiated by
> > +xmgmt (or xuser in future) based on meta data associated with
> > +hardware. The metadata is in the form of device tree as explained before.
> > +
> > +xmgmt.ko
> > +--------
> > +
> > +The xmgmt driver is a PCIe device driver driving MPF found on
> > +Xilinx's Alveo PCIE device. It consists of one *root* driver, one or
> > +more *partition* drivers and one or more *leaf* drivers. The root and
> > +MPF specific leaf drivers are in xmgmt.ko. The partition driver and other
> leaf drivers are in xrt-lib.ko.
> > +
> > +The instantiation of specific partition driver or leaf driver is
> > +completely data driven based on meta data (mostly in device tree
> > +format) found through VSEC capability and inside firmware files, such
> > +as xsabin or xclbin file. The root driver manages life cycle of
> > +multiple partition drivers, which, in turn, manages multiple leaf
> > +drivers. This allows a single set of driver code to support all kinds
> > +of subsystems exposed by different shells. The difference among all
> > +these subsystems will be handled in leaf drivers with root and
> > +partition drivers being part of the infrastructure and provide common
> services for all leaves found on all platforms.
> > +
> > +
> > +xmgmt-root
> > +^^^^^^^^^^
> > +
> > +The xmgmt-root driver is a PCIe device driver attaches to MPF. It's
> > +part of the
> Nit: s/attaches/attached ?
Yes, sure.
> > +infrastructure of the MPF driver and resides in xmgmt.ko. This driver
> > +
> > +* manages one or more partition drivers
> > +* provides access to functionalities that requires pci_dev, such as
> > +PCIE config
> > + space access, to other leaf drivers through parent calls
> > +* together with partition driver, facilities event callbacks for
> > +other leaf drivers
> > +* together with partition driver, facilities inter-leaf driver calls
> > +for other leaf
> > + drivers
> > +
> > +When root driver starts, it will explicitly create an initial
> > +partition instance, which contains leaf drivers that will trigger the
> > +creation of other partition instances. The root driver will wait for
> > +all partitions and leaves to be created before it returns from it's
> > +probe routine and claim success of the initialization of the entire xmgmt
> driver.
> > +
> > +partition
> > +^^^^^^^^^
> > +
> > +The partition driver is a platform device driver whose life cycle is
> > +managed by root and does not have real IO mem or IRQ resources. It's
> > +part of the infrastructure of the MPF driver and resides in
> > +xrt-lib.ko. This driver
> > +
> > +* manages one or more leaf drivers so that multiple leaves can be
> > +managed as a group
> > +* provides access to root from leaves, so that parent calls, event
> > +notifications
> > + and inter-leaf calls can happen
> > +
> > +In xmgmt, an initial partition driver instance will be created by
> > +root, which contains leaves that will trigger partition instances to
> > +be created to manage groups of leaves found on different partitions
> > +on hardware, such as VSEC, Shell, and User.
> > +
> > +leaves
> > +^^^^^^
> > +
> > +The leaf driver is a platform device driver whose life cycle is
> > +managed by a partition driver and may or may not have real IO mem or
> > +IRQ resources. They are the real meat of xmgmt and contains platform
> > +specific code to Shell and User found on a MPF.
> > +
> > +A leaf driver may not have real hardware resources when it merely
> > +acts as a driver that manages certain in-memory states for xmgmt.
> > +These in-memory states could be shared by multiple other leaves.
> > +
> > +Leaf drivers assigned to specific hardware resources drive specific
> > +subsystem in the device. To manipulate the subsystem or carry out a
> > +task, a leaf driver may ask help from root via parent calls and/or from
> other leaves via inter-leaf calls.
> > +
> > +A leaf can also broadcast events through infrastructure code for
> > +other leaves to process. It can also receive event notification from
> > +infrastructure about certain events, such as post-creation or pre-exit of a
> particular leaf.
> > +
> > +
> > +Driver Interfaces
> > +=================
> > +
> > +xmgmt Driver Ioctls
> > +-------------------
> > +
> > +Ioctls exposed by xmgmt driver to user space are enumerated in the
> following table:
> > +
> > +== ===================== =============================
> ===========================
> > +# Functionality ioctl request code data format
> > +== ===================== =============================
> ===========================
> > +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF
> xmgmt_ioc_bitstream_axlf
> > +2 CL frequency scaling XMGMT_IOCFREQSCALE
> xmgmt_ioc_freqscaling
> > +== ===================== =============================
> > +===========================
> > +
> > +xmgmt Driver Sysfs
> > +------------------
> > +
> > +xmgmt driver exposes a rich set of sysfs interfaces. Subsystem
> > +platform drivers export sysfs node for every platform instance.
> > +
> > +Every partition also exports its UUIDs. See below for examples::
> > +
> > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> > +
> > +
> > +hwmon
> > +-----
> > +
> > +xmgmt driver exposes standard hwmon interface to report voltage,
> > +current, temperature, power, etc. These can easily be viewed using
> *sensors* command line utility.
> > +
> > +
> > +mailbox
> > +-------
> > +
> > +xmgmt communicates with user physical function driver via HW mailbox.
> > +Mailbox opcodes are defined in ``mailbox_proto.h``. `Mailbox
> > +Inter-domain Communication Protocol
> > +<https://xilinx.github.io/XRT/master/html/mailbox.proto.html>`_
> > +defines the full specification. xmgmt implements subset of the
> specification. It provides the following services to the UPF driver:
> > +
> > +1. Responding to *are you there* request including determining if the
> two drivers are
> > + running in the same OS domain
> > +2. Provide sensor readings, loaded xclbin UUID, clock frequency, shell
> information, etc.
> > +3. Perform PCIe hot reset
> > +4. Download user compiled xclbin
>
> Is this gonna use the mailbox framework?
The xclbin can be downloaded via IOCTL interface of xmgmt driver.
Or the download request can come from user pf driver via mailbox, yes.
Thanks,
Max
>
> > +
> > +
> > +Platform Security Considerations
> > +================================
> > +
> > +`Security of Alveo Platform
> > +<https://xilinx.github.io/XRT/master/html/security.html>`_
> > +discusses the deployment options and security implications in great detail.
> > --
> > 2.17.1
>
> That's a lot of text, I'll have to read it again most likely,
>
> - Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-12-02 21:24 ` Max Zhen
@ 2020-12-02 23:10 ` Moritz Fischer
2020-12-03 3:38 ` Max Zhen
0 siblings, 1 reply; 28+ messages in thread
From: Moritz Fischer @ 2020-12-02 23:10 UTC (permalink / raw)
To: Max Zhen
Cc: Moritz Fischer, Sonal Santan, linux-kernel, linux-fpga,
Lizhi Hou, Michal Simek, Stefano Stabellini, devicetree
Hi Max,
On Wed, Dec 02, 2020 at 09:24:29PM +0000, Max Zhen wrote:
> Hi Moritz,
>
> Thanks for your feedback. Please see my reply inline.
>
> Thanks,
> -Max
>
> > -----Original Message-----
> > From: Moritz Fischer <mdf@kernel.org>
> > Sent: Monday, November 30, 2020 20:55
> > To: Sonal Santan <sonals@xilinx.com>
> > Cc: linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Max Zhen
> > <maxz@xilinx.com>; Lizhi Hou <lizhih@xilinx.com>; Michal Simek
> > <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> > devicetree@vger.kernel.org
> > Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document
> > describing Alveo XRT drivers
> >
> >
> > On Sat, Nov 28, 2020 at 04:00:33PM -0800, Sonal Santan wrote:
> > > From: Sonal Santan <sonal.santan@xilinx.com>
> > >
> > > Describe Alveo XRT driver architecture and provide basic overview of
> > > Xilinx Alveo platform.
> > >
> > > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > > ---
> > > Documentation/fpga/index.rst | 1 +
> > > Documentation/fpga/xrt.rst | 588
> > +++++++++++++++++++++++++++++++++++
> > > 2 files changed, 589 insertions(+)
> > > create mode 100644 Documentation/fpga/xrt.rst
> > >
> > > diff --git a/Documentation/fpga/index.rst
> > > b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d
> > 100644
> > > --- a/Documentation/fpga/index.rst
> > > +++ b/Documentation/fpga/index.rst
> > > @@ -8,6 +8,7 @@ fpga
> > > :maxdepth: 1
> > >
> > > dfl
> > > + xrt
> > >
> > > .. only:: subproject and html
> > >
> > > diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
> > > new file mode 100644 index 000000000000..9f37d46459b0
> > > --- /dev/null
> > > +++ b/Documentation/fpga/xrt.rst
> > > @@ -0,1 +1,588 @@
> > > +==================================
> > > +XRTV2 Linux Kernel Driver Overview
> > > +==================================
> > > +
> > > +XRTV2 drivers are second generation `XRT
> > > +<https://github.com/Xilinx/XRT>`_ drivers which support `Alveo
> > > +<https://www.xilinx.com/products/boards-and-kits/alveo.html>`_ PCIe
> > platforms from Xilinx.
> > > +
> > > +XRTV2 drivers support *subsystem* style data driven platforms where
> > > +driver's configuration and behavior is determined by meta data provided
> > by platform (in *device tree* format).
> > > +Primary management physical function (MPF) driver is called
> > > +**xmgmt**. Primary user physical function (UPF) driver is called
> > > +**xuser** and HW subsystem drivers are packaged into a library module
> > called **xrt-lib**, which is shared by **xmgmt** and **xuser** (WIP).
> > WIP?
>
> Working in progress. I'll expand it in the doc.
>
> > > +
> > > +Alveo Platform Overview
> > > +=======================
> > > +
> > > +Alveo platforms are architected as two physical FPGA partitions:
> > > +*Shell* and *User*. Shell
> > Nit: The Shell provides ...
>
> Sure. Will fix.
>
> > > +provides basic infrastructure for the Alveo platform like PCIe
> > > +connectivity, board management, Dynamic Function Exchange (DFX),
> > > +sensors, clocking, reset, and security. User partition contains user
> > compiled binary which is loaded by a process called DFX also known as partial
> > reconfiguration.
> > > +
> > > +Physical partitions require strict HW compatibility with each other for DFX
> > to work properly.
> > > +Every physical partition has two interface UUIDs: *parent* UUID and
> > > +*child* UUID. For simple single stage platforms Shell → User forms
> > > +parent child relationship. For complex two stage platforms Base → Shell
> > → User forms the parent child relationship chain.
> > > +
> > > +.. note::
> > > + Partition compatibility matching is key design component of Alveo
> > platforms and XRT. Partitions
> > > + have child and parent relationship. A loaded partition exposes child
> > partition UUID to advertise
> > > + its compatibility requirement for child partition. When loading a child
> > partition the xmgmt
> > > + management driver matches parent UUID of the child partition against
> > child UUID exported by the
> > > + parent. Parent and child partition UUIDs are stored in the *xclbin* (for
> > user) or *xsabin* (for
> > > + base and shell). Except for root UUID, VSEC, hardware itself does not
> > know about UUIDs. UUIDs are
> > > + stored in xsabin and xclbin.
> > > +
> > > +
> > > +The physical partitions and their loading is illustrated below::
> > > +
> > > + SHELL USER
> > > + +-----------+ +-------------------+
> > > + | | | |
> > > + | VSEC UUID | CHILD PARENT | LOGIC UUID |
> > > + | o------->|<--------o |
> > > + | | UUID UUID | |
> > > + +-----+-----+ +--------+----------+
> > > + | |
> > > + . .
> > > + | |
> > > + +---+---+ +------+--------+
> > > + | POR | | USER COMPILED |
> > > + | FLASH | | XCLBIN |
> > > + +-------+ +---------------+
> > > +
> > > +
> > > +Loading Sequence
> > > +----------------
> > > +
> > > +Shell partition is loaded from flash at system boot time. It
> > > +establishes the PCIe link and exposes
> > Nit: The Shell
>
> Will fix.
>
> > > +two physical functions to the BIOS. After OS boot, xmgmt driver
> > > +attaches to PCIe physical function
> > > +0 exposed by the Shell and then looks for VSEC in PCIe extended
> > > +configuration space. Using VSEC it determines the logic UUID of Shell
> > > +and uses the UUID to load matching *xsabin* file from Linux firmware
> > > +directory. The xsabin file contains metadata to discover peripherals that
> > are part of Shell and firmware(s) for any embedded soft processors in Shell.
> >
> > Neat.
>
> Thanks :-).
>
> > > +
> > > +Shell exports child interface UUID which is used for compatibility
> > > +check when loading user compiled
> > Nit: The Shell
>
> Sure.
>
> > > +xclbin over the User partition as part of DFX. When a user requests
> > > +loading of a specific xclbin the xmgmt management driver reads the
> > > +parent interface UUID specified in the xclbin and matches it with
> > > +child interface UUID exported by Shell to determine if xclbin is compatible
> > with the Shell. If match fails loading of xclbin is denied.
> > > +
> > > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
> > > +When loading xclbin xmgmt driver performs the following operations:
> > > +
> > > +1. Sanity check the xclbin contents
> > > +2. Isolate the User partition
> > > +3. Download the bitstream using the FPGA config engine (ICAP) 4.
> > > +De-isolate the User partition
> > Is this modelled as bridges and regions?
>
> Alveo drivers as written today do not use fpga bridge and region framework. It seems that if we add support for that framework, it’s possible to receive PR program request from kernel outside of xmgmt driver? Currently, we can’t support this and PR program can only be initiated using XRT’s runtime API in user space.
I'm not 100% sure I understand the concern here, let me reply to what I
think I understand:
You're worried that if you use FPGA region as interface to accept PR
requests something else could attempt to reconfigure the region from
within the kernel using the FPGA Region API?
Assuming I got this right, I don't think this is a big deal. When you
create the regions you control who gets the references to it.
From what I've seen so far Regions seem to be roughly equivalent to
Partitions, hence my surprise to see a new structure bypassing them.
>
> Or maybe we have missed some points about the use case for this framework?
>
> >
> > > +5. Program the clocks (ClockWiz) driving the User partition 6. Wait
> > > +for memory controller (MIG) calibration
> > > +
> > > +`Platform Loading Overview
> > > +<https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`
> > > +_ provides more detailed information on platform loading.
> > > +
> > > +xsabin
> > > +------
> > > +
> > > +Each Alveo platform comes packaged with its own xsabin. The xsabin is
> > > +trusted component of the platform. For format details refer to
> > > +:ref:`xsabin/xclbin Container Format`. xsabin contains basic
> > > +information like UUIDs, platform name and metadata in the form of
> > device tree. See :ref:`Device Tree Usage` for details and example.
> > > +
> > > +xclbin
> > > +------
> > > +
> > > +xclbin is compiled by end user using
> > > +`Vitis
> > > +<https://www.xilinx.com/products/design-tools/vitis/vitis-platform.ht
> > > +ml>`_ tool set from Xilinx. The xclbin contains sections describing
> > > +user compiled acceleration engines/kernels, memory subsystems,
> > clocking information etc. It also contains bitstream for the user partition,
> > UUIDs, platform name, etc. xclbin uses the same container format as xsabin
> > which is described below.
> > > +
> > > +
> > > +xsabin/xclbin Container Format
> > > +------------------------------
> > > +
> > > +xclbin/xsabin is ELF-like binary container format. It is structured as series
> > of sections.
> > > +There is a file header followed by several section headers which is
> > followed by sections.
> > > +A section header points to an actual section. There is an optional signature
> > at the end.
> > > +The format is defined by header file ``xclbin.h``. The following
> > > +figure illustrates a typical xclbin::
> > > +
> > > +
> > > + +---------------------+
> > > + | |
> > > + | HEADER |
> > > + +---------------------+
> > > + | SECTION HEADER |
> > > + | |
> > > + +---------------------+
> > > + | ... |
> > > + | |
> > > + +---------------------+
> > > + | SECTION HEADER |
> > > + | |
> > > + +---------------------+
> > > + | SECTION |
> > > + | |
> > > + +---------------------+
> > > + | ... |
> > > + | |
> > > + +---------------------+
> > > + | SECTION |
> > > + | |
> > > + +---------------------+
> > > + | SIGNATURE |
> > > + | (OPTIONAL) |
> > > + +---------------------+
> > > +
> > > +
> > > +xclbin/xsabin files can be packaged, un-packaged and inspected using
> > > +XRT utility called **xclbinutil**. xclbinutil is part of XRT open
> > > +source software stack. The source code for xclbinutil can be found at
> > > +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbi
> > > +nutil
> > > +
> > > +For example to enumerate the contents of a xclbin/xsabin use the
> > > +*--info* switch as shown
> > > +below::
> > > +
> > > + xclbinutil --info --input
> > > + /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> > > + xclbinutil --info --input
> > > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsab
> > > + in
> > > +
> > > +
> > > +Device Tree Usage
> > > +-----------------
> > > +
> > > +As mentioned previously xsabin stores metadata which advertise HW
> > subsystems present in a partition.
> > > +The metadata is stored in device tree format with well defined
> > > +schema. Subsystem instantiations are captured as children of
> > > +``addressable_endpoints`` node. Subsystem nodes have standard
> > attributes like ``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe
> > specific attributes:
> > > +``pcie_physical_function`` and ``pcie_bar_mapping``. These identify
> > > +which PCIe physical function and which BAR space in that physical
> > > +function the subsystem resides. XRT management driver uses this
> > > +information to bind *platform drivers* to the subsystem
> > > +instantiations. The platform drivers are found in **xrt-lib.ko**
> > > +kernel module defined later. Below is an example of device tree for
> > > +Alveo U50
> > > +platform::
> >
> > I might be missing something, but couldn't you structure the addressable
> > endpoints in a way that encode the physical function as a parent / child
> > relation?
>
> Alveo driver does not generate the metadata. The metadata is formatted and generated by HW tools when the Alveo HW platform is built.
Sure, but you control the tools that generate the metadata :) Your
userland can structure / process it however it wants / needs?
>
> >
> > What are the regs relative to?
>
> Regs indicates offset of the register on the PCIE BAR of the Alveo device.
>
> > > +
> > > + /dts-v1/;
> > > +
> > > + /{
> > > + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> > > +
> > > + schema_version {
> > > + major = <0x01>;
> > > + minor = <0x00>;
> > > + };
> > > +
> > > + interfaces {
> > > +
> > > + @0 {
> > > + interface_uuid = "862c7020a250293e32036f19956669e5";
> > > + };
> > > + };
> > > +
> > > + addressable_endpoints {
> > > +
> > > + ep_blp_rom_00 {
> > > + reg = <0x00 0x1f04000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> > 1.0\0axi_bram_ctrl";
> > > + };
> > > +
> > > + ep_card_flash_program_00 {
> > > + reg = <0x00 0x1f06000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_quad_spi-
> > 1.0\0axi_quad_spi";
> > > + interrupts = <0x03 0x03>;
> > > + };
> > > +
> > > + ep_cmc_firmware_mem_00 {
> > > + reg = <0x00 0x1e20000 0x00 0x20000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible =
> > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > + firmware {
> > > + firmware_product_name = "cmc";
> > > + firmware_branch_name = "u50";
> > > + firmware_version_major = <0x01>;
> > > + firmware_version_minor = <0x00>;
> > > + };
> > > + };
> > > +
> > > + ep_cmc_intc_00 {
> > > + reg = <0x00 0x1e03000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > + interrupts = <0x04 0x04>;
> > > + };
> > > +
> > > + ep_cmc_mutex_00 {
> > > + reg = <0x00 0x1e02000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_cmc_regmap_00 {
> > > + reg = <0x00 0x1e08000 0x00 0x2000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible =
> > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > + firmware {
> > > + firmware_product_name = "sc-fw";
> > > + firmware_branch_name = "u50";
> > > + firmware_version_major = <0x05>;
> > > + };
> > > + };
> > > +
> > > + ep_cmc_reset_00 {
> > > + reg = <0x00 0x1e01000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_ddr_mem_calib_00 {
> > > + reg = <0x00 0x63000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_debug_bscan_mgmt_00 {
> > > + reg = <0x00 0x1e90000 0x00 0x10000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-debug_bridge-
> > 1.0\0debug_bridge";
> > > + };
> > > +
> > > + ep_ert_base_address_00 {
> > > + reg = <0x00 0x21000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_ert_command_queue_mgmt_00 {
> > > + reg = <0x00 0x40000 0x00 0x10000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> > 1.0\0ert_command_queue";
> > > + };
> > > +
> > > + ep_ert_command_queue_user_00 {
> > > + reg = <0x00 0x40000 0x00 0x10000>;
> > > + pcie_physical_function = <0x01>;
> > > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> > 1.0\0ert_command_queue";
> > > + };
> > > +
> > > + ep_ert_firmware_mem_00 {
> > > + reg = <0x00 0x30000 0x00 0x8000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible =
> > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > + firmware {
> > > + firmware_product_name = "ert";
> > > + firmware_branch_name = "v20";
> > > + firmware_version_major = <0x01>;
> > > + };
> > > + };
> > > +
> > > + ep_ert_intc_00 {
> > > + reg = <0x00 0x23000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > + interrupts = <0x05 0x05>;
> > > + };
> > > +
> > > + ep_ert_reset_00 {
> > > + reg = <0x00 0x22000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_ert_sched_00 {
> > > + reg = <0x00 0x50000 0x00 0x1000>;
> > > + pcie_physical_function = <0x01>;
> > > + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> > > + interrupts = <0x09 0x0c>;
> > > + };
> > > +
> > > + ep_fpga_configuration_00 {
> > > + reg = <0x00 0x1e88000 0x00 0x8000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
> > > + interrupts = <0x02 0x02>;
> > > + };
> > > +
> > > + ep_icap_reset_00 {
> > > + reg = <0x00 0x1f07000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_mailbox_mgmt_00 {
> > > + reg = <0x00 0x1f10000 0x00 0x10000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
> > > + interrupts = <0x00 0x00>;
> > > + };
> > > +
> > > + ep_mailbox_user_00 {
> > > + reg = <0x00 0x1f00000 0x00 0x10000>;
> > > + pcie_physical_function = <0x01>;
> > > + compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
> > > + interrupts = <0x08 0x08>;
> > > + };
> > > +
> > > + ep_msix_00 {
> > > + reg = <0x00 0x00 0x00 0x20000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> > > + pcie_bar_mapping = <0x02>;
> > > + };
> > > +
> > > + ep_pcie_link_mon_00 {
> > > + reg = <0x00 0x1f05000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_pr_isolate_plp_00 {
> > > + reg = <0x00 0x1f01000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_pr_isolate_ulp_00 {
> > > + reg = <0x00 0x1000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_uuid_rom_00 {
> > > + reg = <0x00 0x64000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> > 1.0\0axi_bram_ctrl";
> > > + };
> > > +
> > > + ep_xdma_00 {
> > > + reg = <0x00 0x00 0x00 0x10000>;
> > > + pcie_physical_function = <0x01>;
> > > + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> > > + pcie_bar_mapping = <0x02>;
> > > + };
> > > + };
> > > +
> > > + }
> > > +
> > > +
> > > +
> > > +Deployment Models
> > > +=================
> > > +
> > > +Baremetal
> > > +---------
> > > +
> > > +In bare-metal deployments both MPF and UPF are visible and
> > > +accessible. xmgmt driver binds to MPF. xmgmt driver operations are
> > > +privileged and available to system administrator. The full stack is
> > illustrated below::
> > > +
> > > +
> > > + HOST
> > > +
> > > + [XMGMT] [XUSER]
> > > + | |
> > > + | |
> > > + +-----+ +-----+
> > > + | MPF | | UPF |
> > > + | | | |
> > > + | PF0 | | PF1 |
> > > + +--+--+ +--+--+
> > > + ......... ^................. ^..........
> > > + | |
> > > + | PCIe DEVICE |
> > > + | |
> > > + +--+------------------+--+
> > > + | SHELL |
> > > + | |
> > > + +------------------------+
> > > + | USER |
> > > + | |
> > > + | |
> > > + | |
> > > + | |
> > > + +------------------------+
> > > +
> > > +
> > > +
> > > +Virtualized
> > > +-----------
> > > +
> > > +In virtualized deployments privileged MPF is assigned to host but
> > > +unprivileged UPF is assigned to guest VM via PCIe pass-through. xmgmt
> > driver in host binds to MPF.
> > > +xmgmt driver operations are privileged and only accessible by hosting
> > service provider.
> > > +The full stack is illustrated below::
> > > +
> > > +
> > > + .............
> > > + HOST . VM .
> > > + . .
> > > + [XMGMT] . [XUSER] .
> > > + | . | .
> > > + | . | .
> > > + +-----+ . +-----+ .
> > > + | MPF | . | UPF | .
> > > + | | . | | .
> > > + | PF0 | . | PF1 | .
> > > + +--+--+ . +--+--+ .
> > > + ......... ^................. ^..........
> > > + | |
> > > + | PCIe DEVICE |
> > > + | |
> > > + +--+------------------+--+
> > > + | SHELL |
> > > + | |
> > > + +------------------------+
> > > + | USER |
> > > + | |
> > > + | |
> > > + | |
> > > + | |
> > > + +------------------------+
> > > +
> > > +
> > > +
> > > +Driver Modules
> > > +==============
> > > +
> > > +xrt-lib.ko
> > > +----------
> > > +
> > > +Repository of all subsystem drivers and pure software modules that
> > > +can potentially be shared between xmgmt and xuser. All these drivers
> > > +are structured as Linux *platform driver* and are instantiated by
> > > +xmgmt (or xuser in future) based on meta data associated with
> > > +hardware. The metadata is in the form of device tree as explained before.
> > > +
> > > +xmgmt.ko
> > > +--------
> > > +
> > > +The xmgmt driver is a PCIe device driver driving MPF found on
> > > +Xilinx's Alveo PCIE device. It consists of one *root* driver, one or
> > > +more *partition* drivers and one or more *leaf* drivers. The root and
> > > +MPF specific leaf drivers are in xmgmt.ko. The partition driver and other
> > leaf drivers are in xrt-lib.ko.
> > > +
> > > +The instantiation of specific partition driver or leaf driver is
> > > +completely data driven based on meta data (mostly in device tree
> > > +format) found through VSEC capability and inside firmware files, such
> > > +as xsabin or xclbin file. The root driver manages life cycle of
> > > +multiple partition drivers, which, in turn, manages multiple leaf
> > > +drivers. This allows a single set of driver code to support all kinds
> > > +of subsystems exposed by different shells. The difference among all
> > > +these subsystems will be handled in leaf drivers with root and
> > > +partition drivers being part of the infrastructure and provide common
> > services for all leaves found on all platforms.
> > > +
> > > +
> > > +xmgmt-root
> > > +^^^^^^^^^^
> > > +
> > > +The xmgmt-root driver is a PCIe device driver attaches to MPF. It's
> > > +part of the
> > Nit: s/attaches/attached ?
>
> Yes, sure.
>
> > > +infrastructure of the MPF driver and resides in xmgmt.ko. This driver
> > > +
> > > +* manages one or more partition drivers
> > > +* provides access to functionalities that requires pci_dev, such as
> > > +PCIE config
> > > + space access, to other leaf drivers through parent calls
> > > +* together with partition driver, facilities event callbacks for
> > > +other leaf drivers
> > > +* together with partition driver, facilities inter-leaf driver calls
> > > +for other leaf
> > > + drivers
> > > +
> > > +When root driver starts, it will explicitly create an initial
> > > +partition instance, which contains leaf drivers that will trigger the
> > > +creation of other partition instances. The root driver will wait for
> > > +all partitions and leaves to be created before it returns from it's
> > > +probe routine and claim success of the initialization of the entire xmgmt
> > driver.
> > > +
> > > +partition
> > > +^^^^^^^^^
> > > +
> > > +The partition driver is a platform device driver whose life cycle is
> > > +managed by root and does not have real IO mem or IRQ resources. It's
> > > +part of the infrastructure of the MPF driver and resides in
> > > +xrt-lib.ko. This driver
> > > +
> > > +* manages one or more leaf drivers so that multiple leaves can be
> > > +managed as a group
> > > +* provides access to root from leaves, so that parent calls, event
> > > +notifications
> > > + and inter-leaf calls can happen
> > > +
> > > +In xmgmt, an initial partition driver instance will be created by
> > > +root, which contains leaves that will trigger partition instances to
> > > +be created to manage groups of leaves found on different partitions
> > > +on hardware, such as VSEC, Shell, and User.
> > > +
> > > +leaves
> > > +^^^^^^
> > > +
> > > +The leaf driver is a platform device driver whose life cycle is
> > > +managed by a partition driver and may or may not have real IO mem or
> > > +IRQ resources. They are the real meat of xmgmt and contains platform
> > > +specific code to Shell and User found on a MPF.
> > > +
> > > +A leaf driver may not have real hardware resources when it merely
> > > +acts as a driver that manages certain in-memory states for xmgmt.
> > > +These in-memory states could be shared by multiple other leaves.
> > > +
> > > +Leaf drivers assigned to specific hardware resources drive specific
> > > +subsystem in the device. To manipulate the subsystem or carry out a
> > > +task, a leaf driver may ask help from root via parent calls and/or from
> > other leaves via inter-leaf calls.
> > > +
> > > +A leaf can also broadcast events through infrastructure code for
> > > +other leaves to process. It can also receive event notification from
> > > +infrastructure about certain events, such as post-creation or pre-exit of a
> > particular leaf.
> > > +
> > > +
> > > +Driver Interfaces
> > > +=================
> > > +
> > > +xmgmt Driver Ioctls
> > > +-------------------
> > > +
> > > +Ioctls exposed by xmgmt driver to user space are enumerated in the
> > following table:
> > > +
> > > +== ===================== =============================
> > ===========================
> > > +# Functionality ioctl request code data format
> > > +== ===================== =============================
> > ===========================
> > > +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF
> > xmgmt_ioc_bitstream_axlf
> > > +2 CL frequency scaling XMGMT_IOCFREQSCALE
> > xmgmt_ioc_freqscaling
> > > +== ===================== =============================
> > > +===========================
> > > +
> > > +xmgmt Driver Sysfs
> > > +------------------
> > > +
> > > +xmgmt driver exposes a rich set of sysfs interfaces. Subsystem
> > > +platform drivers export sysfs node for every platform instance.
> > > +
> > > +Every partition also exports its UUIDs. See below for examples::
> > > +
> > > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> > > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> > > +
> > > +
> > > +hwmon
> > > +-----
> > > +
> > > +xmgmt driver exposes standard hwmon interface to report voltage,
> > > +current, temperature, power, etc. These can easily be viewed using
> > *sensors* command line utility.
> > > +
> > > +
> > > +mailbox
> > > +-------
> > > +
> > > +xmgmt communicates with user physical function driver via HW mailbox.
> > > +Mailbox opcodes are defined in ``mailbox_proto.h``. `Mailbox
> > > +Inter-domain Communication Protocol
> > > +<https://xilinx.github.io/XRT/master/html/mailbox.proto.html>`_
> > > +defines the full specification. xmgmt implements subset of the
> > specification. It provides the following services to the UPF driver:
> > > +
> > > +1. Responding to *are you there* request including determining if the
> > two drivers are
> > > + running in the same OS domain
> > > +2. Provide sensor readings, loaded xclbin UUID, clock frequency, shell
> > information, etc.
> > > +3. Perform PCIe hot reset
> > > +4. Download user compiled xclbin
> >
> > Is this gonna use the mailbox framework?
>
> The xclbin can be downloaded via IOCTL interface of xmgmt driver.
> Or the download request can come from user pf driver via mailbox, yes.
>
> Thanks,
> Max
>
> >
> > > +
> > > +
> > > +Platform Security Considerations
> > > +================================
> > > +
> > > +`Security of Alveo Platform
> > > +<https://xilinx.github.io/XRT/master/html/security.html>`_
> > > +discusses the deployment options and security implications in great detail.
> > > --
> > > 2.17.1
> >
> > That's a lot of text, I'll have to read it again most likely,
> >
> > - Moritz
Thanks,
Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files
2020-12-02 18:57 ` Sonal Santan
@ 2020-12-02 23:47 ` Moritz Fischer
0 siblings, 0 replies; 28+ messages in thread
From: Moritz Fischer @ 2020-12-02 23:47 UTC (permalink / raw)
To: Sonal Santan
Cc: Moritz Fischer, linux-kernel, linux-fpga, Max Zhen, Lizhi Hou,
Michal Simek, Stefano Stabellini, devicetree
Hi Sonal,
On Wed, Dec 02, 2020 at 06:57:11PM +0000, Sonal Santan wrote:
> Hi Moritz,
>
> > -----Original Message-----
> > From: Moritz Fischer <mdf@kernel.org>
> > Sent: Monday, November 30, 2020 8:27 PM
> > To: Sonal Santan <sonals@xilinx.com>
> > Cc: linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Max Zhen
> > <maxz@xilinx.com>; Lizhi Hou <lizhih@xilinx.com>; Michal Simek
> > <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> > devicetree@vger.kernel.org
> > Subject: Re: [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files
> >
> > Hi Sonal,
> >
> > On Sat, Nov 28, 2020 at 04:00:34PM -0800, Sonal Santan wrote:
> > > From: Sonal Santan <sonal.santan@xilinx.com>
> > >
> > > Add XRT UAPI header files which describe flash layout, XRT mailbox
> > > protocol, xclBin/axlf FPGA image container format and XRT management
> > > physical function driver ioctl interfaces.
> > >
> > > flash_xrt_data.h:
> > > Layout used by XRT to store private data on flash.
> > >
> > > mailbox_proto.h:
> > > Mailbox opcodes and high level data structures representing various
> > > kinds of information like sensors, clock, etc.
> > >
> > > mailbox_transport.h:
> > > Transport protocol used by mailbox.
> > >
> > > xclbin.h:
> > > Container format used to store compiled FPGA image which includes
> > > bitstream and metadata.
> >
> > Can these headers be introduced together with the code that uses them as
> > logical change?
> >
> > I haven't looked too closely, but it helps reviewing if you can break it into
> > smaller pieces that can stand by themselves.
> >
>
> These UAPI header files are used by multiple source files hence I wanted to get
> these reviewed separately. However if this is getting in the way, in the next
> version of the patch series I would look into arranging the files differently.
>
> You can browse the changes here as well--
> https://github.com/Xilinx/linux-xoclv2/tree/xrtv2-A
Please submit the code in the form you want the patches to be applied,
in this case submit the headers with the code that uses them, and split
it up in smaller chunks please.
Submitting them as a series in the correct order should provide the
proper context for reviewers.
Cheers,
Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-12-02 23:10 ` Moritz Fischer
@ 2020-12-03 3:38 ` Max Zhen
2020-12-03 4:36 ` Moritz Fischer
0 siblings, 1 reply; 28+ messages in thread
From: Max Zhen @ 2020-12-03 3:38 UTC (permalink / raw)
To: Moritz Fischer
Cc: Sonal Santan, linux-kernel, linux-fpga, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
Hi Moritz,
Please see my reply below.
Thanks,
-Max
> -----Original Message-----
> From: Moritz Fischer <mdf@kernel.org>
> Sent: Wednesday, December 2, 2020 15:10
> To: Max Zhen <maxz@xilinx.com>
> Cc: Moritz Fischer <mdf@kernel.org>; Sonal Santan <sonals@xilinx.com>;
> linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Lizhi Hou
> <lizhih@xilinx.com>; Michal Simek <michals@xilinx.com>; Stefano
> Stabellini <stefanos@xilinx.com>; devicetree@vger.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a
> document describing Alveo XRT drivers
>
>
> Hi Max,
>
> On Wed, Dec 02, 2020 at 09:24:29PM +0000, Max Zhen wrote:
> > Hi Moritz,
> >
> > Thanks for your feedback. Please see my reply inline.
> >
> > Thanks,
> > -Max
> >
> > > -----Original Message-----
> > > From: Moritz Fischer <mdf@kernel.org>
> > > Sent: Monday, November 30, 2020 20:55
> > > To: Sonal Santan <sonals@xilinx.com>
> > > Cc: linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Max
> > > Zhen <maxz@xilinx.com>; Lizhi Hou <lizhih@xilinx.com>; Michal
> > > Simek <michals@xilinx.com>; Stefano Stabellini
> > > <stefanos@xilinx.com>; devicetree@vger.kernel.org
> > > Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a
> > > document describing Alveo XRT drivers
> > >
> > >
> > > On Sat, Nov 28, 2020 at 04:00:33PM -0800, Sonal Santan wrote:
> > > > From: Sonal Santan <sonal.santan@xilinx.com>
> > > >
> > > > Describe Alveo XRT driver architecture and provide basic
> > > > overview of Xilinx Alveo platform.
> > > >
> > > > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > > > ---
> > > > Documentation/fpga/index.rst | 1 +
> > > > Documentation/fpga/xrt.rst | 588
> > > +++++++++++++++++++++++++++++++++++
> > > > 2 files changed, 589 insertions(+) create mode 100644
> > > > Documentation/fpga/xrt.rst
> > > >
[...cut...]
> > > > +xclbin over the User partition as part of DFX. When a user
> > > > +requests loading of a specific xclbin the xmgmt management
> > > > +driver reads the parent interface UUID specified in the xclbin
> > > > +and matches it with child interface UUID exported by Shell to
> > > > +determine if xclbin is compatible
> > > with the Shell. If match fails loading of xclbin is denied.
> > > > +
> > > > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl
> command.
> > > > +When loading xclbin xmgmt driver performs the following operations:
> > > > +
> > > > +1. Sanity check the xclbin contents 2. Isolate the User
> > > > +partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
> > > > +De-isolate the User partition
> > > Is this modelled as bridges and regions?
> >
> > Alveo drivers as written today do not use fpga bridge and region
> framework. It seems that if we add support for that framework, it’s
> possible to receive PR program request from kernel outside of xmgmt driver?
> Currently, we can’t support this and PR program can only be initiated
> using XRT’s runtime API in user space.
>
> I'm not 100% sure I understand the concern here, let me reply to what
> I think I understand:
>
> You're worried that if you use FPGA region as interface to accept PR
> requests something else could attempt to reconfigure the region from
> within the kernel using the FPGA Region API?
>
> Assuming I got this right, I don't think this is a big deal. When you
> create the regions you control who gets the references to it.
Thanks for explaining. Yes, I think you got my point :-).
>
> From what I've seen so far Regions seem to be roughly equivalent to
> Partitions, hence my surprise to see a new structure bypassing them.
I see where the gap is.
Regions in Linux is very different than "partitions" we have defined in xmgmt. Regions seem to be a software data structure representing an area on the FPGA that can be reprogrammed. This area is protected by the concept of "bridge" which can be disabled before program and reenabled after it. And you go through region when you need to reprogram this area.
The "partition" is part of the main infrastructure of xmgmt driver, which represents a group of subdev drivers for each individual IP (HW subcomponents). Basically, xmgmt root driver is parent of several partitions who is, in turn, the parent of several subdev drivers. The parent manages the life cycle of its children here.
We do have a partition to represent the group of subdevs/IPs in the reprogrammable area. And we also have partitions representing other areas which cannot be reprogrammed. So, it is difficult to use "Region" to implement "partition".
From what you have explained, it seems that even if I use region / bridge in xmgmt, we can still keep it private to xmgmt instead of exposing the interface to outside world, which we can't support anyway? This means that region will be used as an internal data structure for xmgmt. Since we can't simply replace partition with region, we might as well just use partition throughout the driver, instead of introducing two data structures and use them both in different places.
However, if using region/bridge can bring in other benefits, please let us know and we could see if we can also add this to xmgmt.
> >
> > Or maybe we have missed some points about the use case for this
> framework?
> >
[...cut...]
> > > > +-----------------
> > > > +
> > > > +As mentioned previously xsabin stores metadata which advertise
> > > > +HW
> > > subsystems present in a partition.
> > > > +The metadata is stored in device tree format with well defined
> > > > +schema. Subsystem instantiations are captured as children of
> > > > +``addressable_endpoints`` node. Subsystem nodes have standard
> > > attributes like ``reg``, ``interrupts`` etc. Additionally the
> > > nodes also have PCIe specific attributes:
> > > > +``pcie_physical_function`` and ``pcie_bar_mapping``. These
> > > > +identify which PCIe physical function and which BAR space in
> > > > +that physical function the subsystem resides. XRT management
> > > > +driver uses this information to bind *platform drivers* to the
> > > > +subsystem instantiations. The platform drivers are found in
> > > > +**xrt-lib.ko** kernel module defined later. Below is an example
> > > > +of device tree for Alveo U50
> > > > +platform::
> > >
> > > I might be missing something, but couldn't you structure the
> > > addressable endpoints in a way that encode the physical function
> > > as a parent / child relation?
> >
> > Alveo driver does not generate the metadata. The metadata is
> > formatted
> and generated by HW tools when the Alveo HW platform is built.
>
> Sure, but you control the tools that generate the metadata :) Your
> userland can structure / process it however it wants / needs?
XRT is a runtime software stack, it is not responsible for generating HW metadata. It is one of the consumers of these data. The shell design is generated by a sophisticated tool framework which is difficult to change.
However, we will take this as a feedback for future revision of the tool.
Thanks,
Max
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-12-03 3:38 ` Max Zhen
@ 2020-12-03 4:36 ` Moritz Fischer
2020-12-04 1:17 ` Max Zhen
0 siblings, 1 reply; 28+ messages in thread
From: Moritz Fischer @ 2020-12-03 4:36 UTC (permalink / raw)
To: Max Zhen
Cc: Moritz Fischer, Sonal Santan, linux-kernel, linux-fpga,
Lizhi Hou, Michal Simek, Stefano Stabellini, devicetree
Max,
On Thu, Dec 03, 2020 at 03:38:26AM +0000, Max Zhen wrote:
> [...cut...]
>
> > > > > +xclbin over the User partition as part of DFX. When a user
> > > > > +requests loading of a specific xclbin the xmgmt management
> > > > > +driver reads the parent interface UUID specified in the xclbin
> > > > > +and matches it with child interface UUID exported by Shell to
> > > > > +determine if xclbin is compatible
> > > > with the Shell. If match fails loading of xclbin is denied.
> > > > > +
> > > > > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl
> > command.
> > > > > +When loading xclbin xmgmt driver performs the following operations:
> > > > > +
> > > > > +1. Sanity check the xclbin contents 2. Isolate the User
> > > > > +partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
> > > > > +De-isolate the User partition
> > > > Is this modelled as bridges and regions?
> > >
> > > Alveo drivers as written today do not use fpga bridge and region
> > framework. It seems that if we add support for that framework, it’s
> > possible to receive PR program request from kernel outside of xmgmt driver?
> > Currently, we can’t support this and PR program can only be initiated
> > using XRT’s runtime API in user space.
> >
> > I'm not 100% sure I understand the concern here, let me reply to what
> > I think I understand:
> >
> > You're worried that if you use FPGA region as interface to accept PR
> > requests something else could attempt to reconfigure the region from
> > within the kernel using the FPGA Region API?
> >
> > Assuming I got this right, I don't think this is a big deal. When you
> > create the regions you control who gets the references to it.
>
> Thanks for explaining. Yes, I think you got my point :-).
We can add code to make a region 'static' or 'one-time' or 'fixed'.
>
> >
> > From what I've seen so far Regions seem to be roughly equivalent to
> > Partitions, hence my surprise to see a new structure bypassing them.
>
> I see where the gap is.
>
> Regions in Linux is very different than "partitions" we have defined in xmgmt. Regions seem to be a software data structure representing an area on the FPGA that can be reprogrammed. This area is protected by the concept of "bridge" which can be disabled before program and reenabled after it. And you go through region when you need to reprogram this area.
Your central management driver can create / destroy regions at will. It
can keep them in a list, array or tree.
Regions can but don't have to have bridges.
If you need to go through the central driver to reprogram a region,
you can use that to figure out which region to program.
>
> The "partition" is part of the main infrastructure of xmgmt driver, which represents a group of subdev drivers for each individual IP (HW subcomponents). Basically, xmgmt root driver is parent of several partitions who is, in turn, the parent of several subdev drivers. The parent manages the life cycle of its children here.
I don't see how this is conceptually different from what DFL does, and
they managed to use Regions and Bridges.
If things are missing in the framework, please add them instead of
rewriting an entire parallel framework.
>
> We do have a partition to represent the group of subdevs/IPs in the reprogrammable area. And we also have partitions representing other areas which cannot be reprogrammed. So, it is difficult to use "Region" to implement "partition".
You implement your regions callbacks, you can return -EINVAL / -ENOTTY
if you want to fail a reprogramming request to a static partion /
region.
> From what you have explained, it seems that even if I use region / bridge in xmgmt, we can still keep it private to xmgmt instead of exposing the interface to outside world, which we can't support anyway? This means that region will be used as an internal data structure for xmgmt. Since we can't simply replace partition with region, we might as well just use partition throughout the driver, instead of introducing two data structures and use them both in different places.
Think about your partition as an extension to a region that implements
what you need to do for your case of enumerating and reprogramming that
particular piece of your chip.
> However, if using region/bridge can bring in other benefits, please let us know and we could see if we can also add this to xmgmt.
As maintainer I can say it brings the benefit of looking like existing
infrastructure we have. We can add features to the framework as needed
but blanket replacing the entire thing is always a hard sell.
>
> > >
> > > Or maybe we have missed some points about the use case for this
> > framework?
> > >
>
> [...cut...]
>
> > > > > +-----------------
> > > > > +
> > > > > +As mentioned previously xsabin stores metadata which advertise
> > > > > +HW
> > > > subsystems present in a partition.
> > > > > +The metadata is stored in device tree format with well defined
> > > > > +schema. Subsystem instantiations are captured as children of
> > > > > +``addressable_endpoints`` node. Subsystem nodes have standard
> > > > attributes like ``reg``, ``interrupts`` etc. Additionally the
> > > > nodes also have PCIe specific attributes:
> > > > > +``pcie_physical_function`` and ``pcie_bar_mapping``. These
> > > > > +identify which PCIe physical function and which BAR space in
> > > > > +that physical function the subsystem resides. XRT management
> > > > > +driver uses this information to bind *platform drivers* to the
> > > > > +subsystem instantiations. The platform drivers are found in
> > > > > +**xrt-lib.ko** kernel module defined later. Below is an example
> > > > > +of device tree for Alveo U50
> > > > > +platform::
> > > >
> > > > I might be missing something, but couldn't you structure the
> > > > addressable endpoints in a way that encode the physical function
> > > > as a parent / child relation?
> > >
> > > Alveo driver does not generate the metadata. The metadata is
> > > formatted
> > and generated by HW tools when the Alveo HW platform is built.
> >
> > Sure, but you control the tools that generate the metadata :) Your
> > userland can structure / process it however it wants / needs?
>
> XRT is a runtime software stack, it is not responsible for generating HW metadata. It is one of the consumers of these data. The shell design is generated by a sophisticated tool framework which is difficult to change.
The Kernel userspace ABI is not going to change once it is merged, which
is why we need to get it right. You can change your userspace code long
time after it is merged into the kernel. The otherway round does not
work.
If you're going to do device-tree you'll need device-tree maintainers to
be ok with your bindings.
> However, we will take this as a feedback for future revision of the tool.
>
> Thanks,
> Max
Btw: Can you fix your line-breaks :)
- Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-12-03 4:36 ` Moritz Fischer
@ 2020-12-04 1:17 ` Max Zhen
2020-12-04 4:18 ` Moritz Fischer
0 siblings, 1 reply; 28+ messages in thread
From: Max Zhen @ 2020-12-04 1:17 UTC (permalink / raw)
To: Moritz Fischer
Cc: Sonal Santan, linux-kernel, linux-fpga, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
Hi Moritz,
I manually fixed some line breaks. Not sure why outlook is not doing it properly.
Let me know if it still looks bad to you.
Please see my reply below.
>
>
> Max,
>
> On Thu, Dec 03, 2020 at 03:38:26AM +0000, Max Zhen wrote:
> > [...cut...]
> >
> > > > > > +xclbin over the User partition as part of DFX. When a user
> > > > > > +requests loading of a specific xclbin the xmgmt management
> > > > > > +driver reads the parent interface UUID specified in the xclbin
> > > > > > +and matches it with child interface UUID exported by Shell to
> > > > > > +determine if xclbin is compatible with the Shell. If match fails loading of xclbin is denied.
> > > > > > +
> > > > > > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
> > > > > > +When loading xclbin xmgmt driver performs the following operations:
> > > > > > +
> > > > > > +1. Sanity check the xclbin contents 2. Isolate the User
> > > > > > +partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
> > > > > > +De-isolate the User partition
> > > > > Is this modelled as bridges and regions?
> > > >
> > > > Alveo drivers as written today do not use fpga bridge and region
> > > > framework. It seems that if we add support for that framework, it’s
> > > > possible to receive PR program request from kernel outside of xmgmt driver?
> > > > Currently, we can’t support this and PR program can only be initiated
> > > > using XRT’s runtime API in user space.
> > >
> > > I'm not 100% sure I understand the concern here, let me reply to what
> > > I think I understand:
> > >
> > > You're worried that if you use FPGA region as interface to accept PR
> > > requests something else could attempt to reconfigure the region from
> > > within the kernel using the FPGA Region API?
> > >
> > > Assuming I got this right, I don't think this is a big deal. When you
> > > create the regions you control who gets the references to it.
> >
> > Thanks for explaining. Yes, I think you got my point :-).
>
> We can add code to make a region 'static' or 'one-time' or 'fixed'.
> >
> > >
> > > From what I've seen so far Regions seem to be roughly equivalent to
> > > Partitions, hence my surprise to see a new structure bypassing them.
> >
> > I see where the gap is.
> >
> > Regions in Linux is very different than "partitions" we have defined in xmgmt. Regions seem to be a software data structure
> > representing an area on the FPGA that can be reprogrammed. This area is protected by the concept of "bridge" which can be disabled
> > before program and reenabled after it. And you go through region when you need to reprogram this area.
>
> Your central management driver can create / destroy regions at will. It
> can keep them in a list, array or tree.
>
> Regions can but don't have to have bridges.
>
> If you need to go through the central driver to reprogram a region,
> you can use that to figure out which region to program.
That sounds fine. I can create a region and call into it from xmgmt for
PR programing. The region will, then, call the xmgmt's fpga manager
to program it.
> >
> > The "partition" is part of the main infrastructure of xmgmt driver, which represents a group of subdev drivers for each individual IP
> > (HW subcomponents). Basically, xmgmt root driver is parent of several partitions who is, in turn, the parent of several subdev drivers.
> > The parent manages the life cycle of its children here.
>
> I don't see how this is conceptually different from what DFL does, and
> they managed to use Regions and Bridges.
>
> If things are missing in the framework, please add them instead of
> rewriting an entire parallel framework.
>
> >
> > We do have a partition to represent the group of subdevs/IPs in the reprogrammable area. And we also have partitions
> > representing other areas which cannot be reprogrammed. So, it is difficult to use "Region" to implement "partition".
>
> You implement your regions callbacks, you can return -EINVAL / -ENOTTY
> if you want to fail a reprogramming request to a static partion /
> region.
>
> > From what you have explained, it seems that even if I use region / bridge in xmgmt, we can still keep it private to xmgmt instead of
> > exposing the interface to outside world, which we can't support anyway? This means that region will be used as an internal data
> > structure for xmgmt. Since we can't simply replace partition with region, we might as well just use partition throughout the driver,
> > instead of introducing two data structures and use them both in different places.
>
> Think about your partition as an extension to a region that implements
> what you need to do for your case of enumerating and reprogramming that
> particular piece of your chip.
Yes, we can add region / bridges to represent the PR area and use it in our
code path for reprogramming the PR area. I think what we will do is to
instantiate a region instance for the PR area and associate it with the
FPGA manager in xmgmt for reprogramming it. We can also instantiate
bridges and map the "ULP gate" subdev driver to it in xmgmt. Thus, we
could incorporate region and bridge data structures in xmgmt for PR
reprogramming.
This will be a non-trivial change for us. I'd like to confirm that this is what
you are looking for before we start working on the change. Let us know :-).
>
> > However, if using region/bridge can bring in other benefits, please let us know and we could see if we can also add this to xmgmt.
>
> As maintainer I can say it brings the benefit of looking like existing
> infrastructure we have. We can add features to the framework as needed
> but blanket replacing the entire thing is always a hard sell.
> >
> > > >
> > > > Or maybe we have missed some points about the use case for this
> > > > framework?
> > > >
> >
> > [...cut...]
> >
> > > > > > +-----------------
> > > > > > +
> > > > > > +As mentioned previously xsabin stores metadata which advertise
> > > > > > +HW
> > > > > > subsystems present in a partition.
> > > > > > +The metadata is stored in device tree format with well defined
> > > > > > +schema. Subsystem instantiations are captured as children of
> > > > > > +``addressable_endpoints`` node. Subsystem nodes have standard
> > > > > > attributes like ``reg``, ``interrupts`` etc. Additionally the
> > > > > > nodes also have PCIe specific attributes:
> > > > > > +``pcie_physical_function`` and ``pcie_bar_mapping``. These
> > > > > > +identify which PCIe physical function and which BAR space in
> > > > > > +that physical function the subsystem resides. XRT management
> > > > > > +driver uses this information to bind *platform drivers* to the
> > > > > > +subsystem instantiations. The platform drivers are found in
> > > > > > +**xrt-lib.ko** kernel module defined later. Below is an example
> > > > > > +of device tree for Alveo U50
> > > > > > +platform::
> > > > >
> > > > > I might be missing something, but couldn't you structure the
> > > > > addressable endpoints in a way that encode the physical function
> > > > > as a parent / child relation?
> > > >
> > > > Alveo driver does not generate the metadata. The metadata is
> > > > formatted
> > > > and generated by HW tools when the Alveo HW platform is built.
> > >
> > > Sure, but you control the tools that generate the metadata :) Your
> > > userland can structure / process it however it wants / needs?
> >
> > XRT is a runtime software stack, it is not responsible for generating HW metadata. It is one of the consumers of these data. The shell
> > design is generated by a sophisticated tool framework which is difficult to change.
>
> The Kernel userspace ABI is not going to change once it is merged, which
> is why we need to get it right. You can change your userspace code long
> time after it is merged into the kernel. The otherway round does not
> work.
>
> If you're going to do device-tree you'll need device-tree maintainers to
> be ok with your bindings.
>
Yes, we'll wait for the device-tree maintainers to chime in here :-).
Thanks,
Max
> > However, we will take this as a feedback for future revision of the tool.
> >
> > Thanks,
> > Max
>
> Btw: Can you fix your line-breaks :)
>
> - Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers
2020-12-04 1:17 ` Max Zhen
@ 2020-12-04 4:18 ` Moritz Fischer
0 siblings, 0 replies; 28+ messages in thread
From: Moritz Fischer @ 2020-12-04 4:18 UTC (permalink / raw)
To: Max Zhen
Cc: Moritz Fischer, Sonal Santan, linux-kernel, linux-fpga,
Lizhi Hou, Michal Simek, Stefano Stabellini, devicetree
On Fri, Dec 04, 2020 at 01:17:37AM +0000, Max Zhen wrote:
> Hi Moritz,
>
> I manually fixed some line breaks. Not sure why outlook is not doing it properly.
> Let me know if it still looks bad to you.
That might just be outlook :)
>
> Please see my reply below.
>
> >
> >
> > Max,
> >
> > On Thu, Dec 03, 2020 at 03:38:26AM +0000, Max Zhen wrote:
> > > [...cut...]
> > >
> > > > > > > +xclbin over the User partition as part of DFX. When a user
> > > > > > > +requests loading of a specific xclbin the xmgmt management
> > > > > > > +driver reads the parent interface UUID specified in the xclbin
> > > > > > > +and matches it with child interface UUID exported by Shell to
> > > > > > > +determine if xclbin is compatible with the Shell. If match fails loading of xclbin is denied.
> > > > > > > +
> > > > > > > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
> > > > > > > +When loading xclbin xmgmt driver performs the following operations:
> > > > > > > +
> > > > > > > +1. Sanity check the xclbin contents 2. Isolate the User
> > > > > > > +partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
> > > > > > > +De-isolate the User partition
> > > > > > Is this modelled as bridges and regions?
> > > > >
> > > > > Alveo drivers as written today do not use fpga bridge and region
> > > > > framework. It seems that if we add support for that framework, it’s
> > > > > possible to receive PR program request from kernel outside of xmgmt driver?
> > > > > Currently, we can’t support this and PR program can only be initiated
> > > > > using XRT’s runtime API in user space.
> > > >
> > > > I'm not 100% sure I understand the concern here, let me reply to what
> > > > I think I understand:
> > > >
> > > > You're worried that if you use FPGA region as interface to accept PR
> > > > requests something else could attempt to reconfigure the region from
> > > > within the kernel using the FPGA Region API?
> > > >
> > > > Assuming I got this right, I don't think this is a big deal. When you
> > > > create the regions you control who gets the references to it.
> > >
> > > Thanks for explaining. Yes, I think you got my point :-).
> >
> > We can add code to make a region 'static' or 'one-time' or 'fixed'.
> > >
> > > >
> > > > From what I've seen so far Regions seem to be roughly equivalent to
> > > > Partitions, hence my surprise to see a new structure bypassing them.
> > >
> > > I see where the gap is.
> > >
> > > Regions in Linux is very different than "partitions" we have defined in xmgmt. Regions seem to be a software data structure
> > > representing an area on the FPGA that can be reprogrammed. This area is protected by the concept of "bridge" which can be disabled
> > > before program and reenabled after it. And you go through region when you need to reprogram this area.
> >
> > Your central management driver can create / destroy regions at will. It
> > can keep them in a list, array or tree.
> >
> > Regions can but don't have to have bridges.
> >
> > If you need to go through the central driver to reprogram a region,
> > you can use that to figure out which region to program.
>
> That sounds fine. I can create a region and call into it from xmgmt for
> PR programing. The region will, then, call the xmgmt's fpga manager
> to program it.
It sounds closer than what I'd expect.
>
> > >
> > > The "partition" is part of the main infrastructure of xmgmt driver, which represents a group of subdev drivers for each individual IP
> > > (HW subcomponents). Basically, xmgmt root driver is parent of several partitions who is, in turn, the parent of several subdev drivers.
> > > The parent manages the life cycle of its children here.
> >
> > I don't see how this is conceptually different from what DFL does, and
> > they managed to use Regions and Bridges.
> >
> > If things are missing in the framework, please add them instead of
> > rewriting an entire parallel framework.
> >
> > >
> > > We do have a partition to represent the group of subdevs/IPs in the reprogrammable area. And we also have partitions
> > > representing other areas which cannot be reprogrammed. So, it is difficult to use "Region" to implement "partition".
> >
> > You implement your regions callbacks, you can return -EINVAL / -ENOTTY
> > if you want to fail a reprogramming request to a static partion /
> > region.
> >
> > > From what you have explained, it seems that even if I use region / bridge in xmgmt, we can still keep it private to xmgmt instead of
> > > exposing the interface to outside world, which we can't support anyway? This means that region will be used as an internal data
> > > structure for xmgmt. Since we can't simply replace partition with region, we might as well just use partition throughout the driver,
> > > instead of introducing two data structures and use them both in different places.
> >
> > Think about your partition as an extension to a region that implements
> > what you need to do for your case of enumerating and reprogramming that
> > particular piece of your chip.
>
> Yes, we can add region / bridges to represent the PR area and use it in our
> code path for reprogramming the PR area. I think what we will do is to
> instantiate a region instance for the PR area and associate it with the
> FPGA manager in xmgmt for reprogramming it. We can also instantiate
> bridges and map the "ULP gate" subdev driver to it in xmgmt. Thus, we
> could incorporate region and bridge data structures in xmgmt for PR
> reprogramming.
I'd need to take another look, but the ULP gate sounds like a bridge (or
close to it).
> This will be a non-trivial change for us. I'd like to confirm that this is what
> you are looking for before we start working on the change. Let us know :-).
I understand. It looks like the right direction. Let's discuss code when
we have code to look at.
It may take a couple of iterations to get it all sorted.
That's normal when you show show up with that much code all at once :)
Cheers,
Moritz
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver
2020-12-02 3:00 ` Xu Yilun
@ 2020-12-04 4:40 ` Max Zhen
0 siblings, 0 replies; 28+ messages in thread
From: Max Zhen @ 2020-12-04 4:40 UTC (permalink / raw)
To: Xu Yilun, Sonal Santan
Cc: linux-kernel, linux-fpga, lizhih, michal.simek, stefanos, devicetree
Hi Yilun,
On 12/1/20 7:00 PM, Xu Yilun wrote:
>
>
>> +static int xmgmt_main_event_cb(struct platform_device *pdev,
>> + enum xrt_events evt, void *arg)
>> +{
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> + struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev *)arg;
>> + enum xrt_subdev_id id;
>> + int instance;
>> + size_t fwlen;
>> +
>> + switch (evt) {
>> + case XRT_EVENT_POST_CREATION: {
>> + id = esd->xevt_subdev_id;
>> + instance = esd->xevt_subdev_instance;
>> + xrt_info(pdev, "processing event %d for (%d, %d)",
>> + evt, id, instance);
>> +
>> + if (id == XRT_SUBDEV_GPIO)
>> + xmm->gpio_ready = true;
>> + else if (id == XRT_SUBDEV_QSPI)
>> + xmm->flash_ready = true;
>> + else
>> + BUG_ON(1);
>> +
>> + if (xmm->gpio_ready && xmm->flash_ready) {
>> + int rc;
>> +
>> + rc = load_firmware_from_disk(pdev, &xmm->firmware_blp,
>> + &fwlen);
>> + if (rc != 0) {
>> + rc = load_firmware_from_flash(pdev,
>> + &xmm->firmware_blp, &fwlen);
>
> I'm curious that before the shell metadata is loaded, how the QSPI
> subdev is enumerated and get to work? The QSPI DT info itself is
> stored in metadata, is it?
No, it is not from the shell metadata. The QSPI subdev info is
discovered from a rom located on the PCIE BAR pointed to by VSEC cap
found in config space.
>
> I didn't find the creation of leaf platform devices, maybe I can find
> the answer in the missing Patch #5?
Leaf driver is children of partition driver. They are created in
xrt_part_create_leaves() in xrt-partition.c.
Thanks,
Max
>
> Thanks,
> Yilun
>
>> + }
>> + if (rc == 0 && is_valid_firmware(pdev,
>> + xmm->firmware_blp, fwlen))
>> + (void) xmgmt_create_blp(xmm);
>> + else
>> + xrt_err(pdev,
>> + "failed to find firmware, giving up");
>> + xmm->evt_hdl = NULL;
>> + }
>> + break;
>> + }
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver
[not found] ` <BY5PR02MB60683E3470179E6AD10FEE26B9F20@BY5PR02MB6068.namprd02.prod.outlook.com>
@ 2020-12-04 6:22 ` Sonal Santan
0 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-12-04 6:22 UTC (permalink / raw)
To: Moritz Fischer
Cc: linux-kernel, linux-fpga, Max Zhen, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
Hello Moritz,
> -----Original Message-----
> From: Moritz Fischer <mdf@kernel.org>
> Sent: Tuesday, December 1, 2020 12:52
> To: Sonal Santan <sonals@xilinx.com>
> Cc: linux-kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Max Zhen
<maxz@xilinx.com>; Lizhi Hou <lizhih@xilinx.com>; Michal
> Simek <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
devicetree@vger.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical
function driver
>
>
> Hi Sonal,
>
> On Sat, Nov 28, 2020 at 04:00:39PM -0800, Sonal Santan wrote:
> > From: Sonal Santan <sonal.santan@xilinx.com>
> >
> > Add management physical function driver core. The driver attaches
> > to management physical function of Alveo devices. It instantiates
> > the root driver and one or more partition drivers which in turn
> > instantiate platform drivers. The instantiation of partition and
> > platform drivers is completely data driven. The driver integrates
> > with FPGA manager and provides xclbin download service.
> >
> > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > ---
> > drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c | 194 ++++
> > drivers/fpga/alveo/mgmt/xmgmt-fmgr.h | 29 +
> > drivers/fpga/alveo/mgmt/xmgmt-main-impl.h | 36 +
> > drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c | 930
+++++++++++++++++++
> > drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c | 190 ++++
> > drivers/fpga/alveo/mgmt/xmgmt-main.c | 843 +++++++++++++++++
> > drivers/fpga/alveo/mgmt/xmgmt-root.c | 375 ++++++++
> > 7 files changed, 2597 insertions(+)
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-main.c
> > create mode 100644 drivers/fpga/alveo/mgmt/xmgmt-root.c
> >
> > diff --git a/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
b/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> > new file mode 100644
> > index 000000000000..d451b5a2c291
> > --- /dev/null
> > +++ b/drivers/fpga/alveo/mgmt/xmgmt-fmgr-drv.c
> > @@ -0,0 +1,194 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Xilinx Alveo Management Function Driver
> > + *
> > + * Copyright (C) 2019-2020 Xilinx, Inc.
> > + * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
> > + *
> > + * Authors: Sonal.Santan@xilinx.com
> > + */
> > +
> > +#include <linux/cred.h>
> > +#include <linux/efi.h>
> > +#include <linux/fpga/fpga-mgr.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/module.h>
> > +#include <linux/vmalloc.h>
> > +
> > +#include "xrt-subdev.h"
> > +#include "xmgmt-fmgr.h"
> > +#include "xrt-axigate.h"
> > +#include "xmgmt-main-impl.h"
> > +
> > +/*
> > + * Container to capture and cache full xclbin as it is passed in blocks by
FPGA
> > + * Manager. Driver needs access to full xclbin to walk through xclbin
sections.
> > + * FPGA Manager's .write() backend sends incremental blocks without any
> > + * knowledge of xclbin format forcing us to collect the blocks and stitch
them
> > + * together here.
> > + */
> > +
> > +struct xfpga_klass {
> Nit: xfpga_priv or xfpga_drvdata?
> > + const struct platform_device *pdev;
> > + struct axlf *blob;
> > + char name[64];
> Nit: 64 could be a named constant ?
> > + size_t count;
> > + size_t total_count;
> > + struct mutex axlf_lock;
> > + int reader_ref;
> > + enum fpga_mgr_states state;
> > + enum xfpga_sec_level sec_level;
> This appears unused, do you want to add this with the code that uses it?
This hook is for validating signature of FPGA image. Will look into adding it
into the next version of the patch.
> > +};
>
> Maybe add some kerneldoc markup?
Will do in the next version
> > +
> > +struct key *xfpga_keys;
> Appears unused, can you introduce this together with the code using it?
> > +
> > +static int xmgmt_pr_write_init(struct fpga_manager *mgr,
> > + struct fpga_image_info *info, const char *buf, size_t count)
> > +{
> > + struct xfpga_klass *obj = mgr->priv;
> > + const struct axlf *bin = (const struct axlf *)buf;
> Nit: Reverse x-mas tree please.
>
> xxxxxx
> xxxx
> xxx
> x
Will update in the next version
> > +
> > + if (count < sizeof(struct axlf)) {
> > + obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> > + return -EINVAL;
> > + }
> > +
> > + if (count > bin->m_header.m_length) {
> > + obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> > + return -EINVAL;
> > + }
> > +
> > + /* Free up the previous blob */
> > + vfree(obj->blob);
> > + obj->blob = vmalloc(bin->m_header.m_length);
> > + if (!obj->blob) {
> > + obj->state = FPGA_MGR_STATE_WRITE_INIT_ERR;
> > + return -ENOMEM;
> > + }
> > +
> > + xrt_info(obj->pdev, "Begin download of xclbin %pUb of length %lld B",
> > + &bin->m_header.uuid, bin->m_header.m_length);
> We already have framework level prints for that (admittedly somewhat
> less verbose). Please remove.
Will update in the next version
> > +
> > + obj->count = 0;
> > + obj->total_count = bin->m_header.m_length;
> > + obj->state = FPGA_MGR_STATE_WRITE_INIT;
> Does the framework state tracking not work for you?
> > + return 0;
> > +}
> > +
> > +static int xmgmt_pr_write(struct fpga_manager *mgr,
> > + const char *buf, size_t count)
> > +{
> > + struct xfpga_klass *obj = mgr->priv;
> > + char *curr = (char *)obj->blob;
> > +
> > + if ((obj->state != FPGA_MGR_STATE_WRITE_INIT) &&
> > + (obj->state != FPGA_MGR_STATE_WRITE)) {
> > + obj->state = FPGA_MGR_STATE_WRITE_ERR;
> > + return -EINVAL;
> > + }
> > +
> > + curr += obj->count;
> > + obj->count += count;
> > +
> > + /*
> > + * The xclbin buffer should not be longer than advertised in the header
> > + */
> > + if (obj->total_count < obj->count) {
> > + obj->state = FPGA_MGR_STATE_WRITE_ERR;
> > + return -EINVAL;
> > + }
> > +
> > + xrt_info(obj->pdev, "Copying block of %zu B of xclbin", count);
> Please drop those.
> > + memcpy(curr, buf, count);
>
> I'm confused. Why are we just copying things around here. What picks
> this up afterwards?
The current implementation caches the full FPGA image in a local buffer so
we can walk various segments of the xclbin container to identify bitstreams,
clock scaling metadata, etc. Full image is also needed for signature verification.
Does the framework guarantee one .write call in case fpga_image_info is
created with complete buffer? That will remove the need to cache the buffer.
> > + obj->state = FPGA_MGR_STATE_WRITE;
> > + return 0;
> > +}
> > +
> > +
> > +static int xmgmt_pr_write_complete(struct fpga_manager *mgr,
> > + struct fpga_image_info *info)
> > +{
> > + int result = 0;
> > + struct xfpga_klass *obj = mgr->priv;
> > +
> > + if (obj->state != FPGA_MGR_STATE_WRITE) {
> > + obj->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> > + return -EINVAL;
> > + }
> > +
> > + /* Check if we got the complete xclbin */
> > + if (obj->blob->m_header.m_length != obj->count) {
> > + obj->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
> > + return -EINVAL;
> > + }
> > +
> > + result = xmgmt_ulp_download((void *)obj->pdev, obj->blob);
> > +
> > + obj->state = result ? FPGA_MGR_STATE_WRITE_COMPLETE_ERR :
> > + FPGA_MGR_STATE_WRITE_COMPLETE;
> Why the separate state tracking?
> > + xrt_info(obj->pdev, "Finish downloading of xclbin %pUb: %d",
> > + &obj->blob->m_header.uuid, result);
> > + vfree(obj->blob);
> > + obj->blob = NULL;
> > + obj->count = 0;
> > + return result;
> > +}
> > +
> > +static enum fpga_mgr_states xmgmt_pr_state(struct fpga_manager
*mgr)
> > +{
> > + struct xfpga_klass *obj = mgr->priv;
> > +
> > + return obj->state;
> > +}
> > +
> > +static const struct fpga_manager_ops xmgmt_pr_ops = {
> > + .initial_header_size = sizeof(struct axlf),
> > + .write_init = xmgmt_pr_write_init,
> > + .write = xmgmt_pr_write,
> > + .write_complete = xmgmt_pr_write_complete,
> > + .state = xmgmt_pr_state,
> > +};
> > +
> > +
> > +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev)
> > +{
> > + struct fpga_manager *fmgr;
> > + int ret = 0;
> > + struct xfpga_klass *obj = vzalloc(sizeof(struct xfpga_klass));
> > +
> > + xrt_info(pdev, "probing...");
> Drop this, please.
> > + if (!obj)
> > + return ERR_PTR(-ENOMEM);
> > +
> > + snprintf(obj->name, sizeof(obj->name), "Xilinx Alveo FPGA Manager");
> > + obj->state = FPGA_MGR_STATE_UNKNOWN;
> > + obj->pdev = pdev;
> > + fmgr = fpga_mgr_create(&pdev->dev,
> > + obj->name,
> > + &xmgmt_pr_ops,
> > + obj);
> I think (eyeballed) this fits on two lines?
> > + if (!fmgr)
> > + return ERR_PTR(-ENOMEM);
> > +
> > + obj->sec_level = XFPGA_SEC_NONE;
> Seems unused so far, please drop until it's used.
> > + ret = fpga_mgr_register(fmgr);
> > + if (ret) {
> > + fpga_mgr_free(fmgr);
> > + kfree(obj);
> > + return ERR_PTR(ret);
> > + }
> > + mutex_init(&obj->axlf_lock);
> > + return fmgr;
> Since this patchset will wait at least till next cycle, you might want
> to look into the devm_* functions for registering and creating FPGA
> Managers.
Will address this in next revision of the patch.
>
> > +}
> > +
> > +int xmgmt_fmgr_remove(struct fpga_manager *fmgr)
> > +{
> > + struct xfpga_klass *obj = fmgr->priv;
> > +
> > + mutex_destroy(&obj->axlf_lock);
> > + obj->state = FPGA_MGR_STATE_UNKNOWN;
> > + fpga_mgr_unregister(fmgr);
> > + vfree(obj->blob);
> > + vfree(obj);
> > + return 0;
> > +}
> > diff --git a/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
b/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> > new file mode 100644
> > index 000000000000..2beba649609f
> > --- /dev/null
> > +++ b/drivers/fpga/alveo/mgmt/xmgmt-fmgr.h
> > @@ -0,0 +1,29 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Xilinx Alveo Management Function Driver
> > + *
> > + * Copyright (C) 2019-2020 Xilinx, Inc.
> > + * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
> > + *
> > + * Authors: Sonal.Santan@xilinx.com
> > + */
> > +
> > +#ifndef _XMGMT_FMGR_H_
> > +#define _XMGMT_FMGR_H_
> > +
> > +#include <linux/fpga/fpga-mgr.h>
> > +#include <linux/mutex.h>
> > +
> > +#include <linux/xrt/xclbin.h>
> > +
> > +enum xfpga_sec_level {
> > + XFPGA_SEC_NONE = 0,
> > + XFPGA_SEC_DEDICATE,
> > + XFPGA_SEC_SYSTEM,
> > + XFPGA_SEC_MAX = XFPGA_SEC_SYSTEM,
> > +};
> > +
> > +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev);
> > +int xmgmt_fmgr_remove(struct fpga_manager *fmgr);
> > +
> > +#endif
> > diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
b/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> > new file mode 100644
> > index 000000000000..c89024cb8d46
> > --- /dev/null
> > +++ b/drivers/fpga/alveo/mgmt/xmgmt-main-impl.h
> > @@ -0,0 +1,36 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (C) 2020 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> > + * Cheng Zhen <maxz@xilinx.com>
> > + */
> > +
> > +#ifndef _XMGMT_MAIN_IMPL_H_
> > +#define _XMGMT_MAIN_IMPL_H_
> > +
> > +#include "xrt-subdev.h"
> > +#include "xmgmt-main.h"
> > +
> > +extern struct platform_driver xmgmt_main_driver;
> > +extern struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[];
> > +
> > +extern int xmgmt_ulp_download(struct platform_device *pdev, const void
*xclbin);
> > +extern int bitstream_axlf_mailbox(struct platform_device *pdev,
> > + const void *xclbin);
> > +extern int xmgmt_hot_reset(struct platform_device *pdev);
> > +
> > +/* Getting dtb for specified partition. Caller should vfree returned dtb .*/
> > +extern char *xmgmt_get_dtb(struct platform_device *pdev,
> > + enum provider_kind kind);
> > +extern char *xmgmt_get_vbnv(struct platform_device *pdev);
> > +extern int xmgmt_get_provider_uuid(struct platform_device *pdev,
> > + enum provider_kind kind, uuid_t *uuid);
> > +
> > +extern void *xmgmt_pdev2mailbox(struct platform_device *pdev);
> > +extern void *xmgmt_mailbox_probe(struct platform_device *pdev);
> > +extern void xmgmt_mailbox_remove(void *handle);
> > +extern void xmgmt_peer_notify_state(void *handle, bool online);
> > +
> > +#endif /* _XMGMT_MAIN_IMPL_H_ */
> > diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
b/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> > new file mode 100644
> > index 000000000000..b3d82fc3618b
> > --- /dev/null
> > +++ b/drivers/fpga/alveo/mgmt/xmgmt-main-mailbox.c
> > @@ -0,0 +1,930 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Xilinx Alveo FPGA MGMT PF entry point driver
> > + *
> > + * Copyright (C) 2020 Xilinx, Inc.
> > + *
> > + * Peer communication via mailbox
> > + *
> > + * Authors:
> > + * Cheng Zhen <maxz@xilinx.com>
> > + */
> > +
> > +#include <linux/crc32c.h>
> > +#include <linux/xrt/mailbox_proto.h>
> > +#include "xmgmt-main-impl.h"
> > +#include "xrt-mailbox.h"
> > +#include "xrt-cmc.h"
> > +#include "xrt-metadata.h"
> > +#include "xrt-xclbin.h"
> > +#include "xrt-clock.h"
> > +#include "xrt-calib.h"
> > +#include "xrt-icap.h"
> > +
> > +struct xmgmt_mailbox {
> > + struct platform_device *pdev;
> > + struct platform_device *mailbox;
> > + struct mutex lock;
> > + void *evt_hdl;
> > + char *test_msg;
> > + bool peer_in_same_domain;
> > +};
> > +
> > +#define XMGMT_MAILBOX_PRT_REQ(xmbx, send, request, sw_ch)
do { \
> > + const char *dir = (send) ? ">>>>>" : "<<<<<"; \
> > + \
> > + if ((request)->req == XCL_MAILBOX_REQ_PEER_DATA) { \
> > + struct xcl_mailbox_peer_data *p = \
> > + (struct xcl_mailbox_peer_data *)(request)->data;\
> > + \
> > + xrt_info((xmbx)->pdev, "%s(%s) %s%s", \
> > + mailbox_req2name((request)->req), \
> > + mailbox_group_kind2name(p->kind), \
> > + dir, mailbox_chan2name(sw_ch)); \
> > + } else { \
> > + xrt_info((xmbx)->pdev, "%s %s%s", \
> > + mailbox_req2name((request)->req), \
> > + dir, mailbox_chan2name(sw_ch)); \
> > + } \
> > +} while (0)
> > +#define XMGMT_MAILBOX_PRT_REQ_SEND(xmbx, req, sw_ch)
\
> > + XMGMT_MAILBOX_PRT_REQ(xmbx, true, req, sw_ch)
> > +#define XMGMT_MAILBOX_PRT_REQ_RECV(xmbx, req, sw_ch)
\
> > + XMGMT_MAILBOX_PRT_REQ(xmbx, false, req, sw_ch)
> > +#define XMGMT_MAILBOX_PRT_RESP(xmbx, resp) \
> > + xrt_info((xmbx)->pdev, "respond %ld bytes >>>>>%s", \
> > + (resp)->xmip_data_size, mailbox_chan2name((resp)->xmip_sw_ch))
> > +
> > +static inline struct xmgmt_mailbox *pdev2mbx(struct platform_device
*pdev)
> > +{
> > + return (struct xmgmt_mailbox *)xmgmt_pdev2mailbox(pdev);
> > +}
> > +
> > +static void xmgmt_mailbox_post(struct xmgmt_mailbox *xmbx,
> > + u64 msgid, bool sw_ch, void *buf, size_t len)
> > +{
> > + int rc;
> > + struct xrt_mailbox_ioctl_post post = {
> > + .xmip_req_id = msgid,
> > + .xmip_sw_ch = sw_ch,
> > + .xmip_data = buf,
> > + .xmip_data_size = len
> > + };
> > +
> > + BUG_ON(!mutex_is_locked(&xmbx->lock));
> > +
> > + if (!xmbx->mailbox) {
> > + xrt_err(xmbx->pdev, "mailbox not available");
> > + return;
> > + }
> > +
> > + if (msgid == 0) {
> > + XMGMT_MAILBOX_PRT_REQ_SEND(xmbx,
> > + (struct xcl_mailbox_req *)buf, sw_ch);
> > + } else {
> > + XMGMT_MAILBOX_PRT_RESP(xmbx, &post);
> > + }
> > +
> > + rc = xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_POST, &post);
> > + if (rc)
> > + xrt_err(xmbx->pdev, "failed to post msg: %d", rc);
> > +}
> > +
> > +static void xmgmt_mailbox_notify(struct xmgmt_mailbox *xmbx, bool
sw_ch,
> > + struct xcl_mailbox_req *req, size_t len)
> > +{
> > + xmgmt_mailbox_post(xmbx, 0, sw_ch, req, len);
> > +}
> > +
> > +static void xmgmt_mailbox_respond(struct xmgmt_mailbox *xmbx,
> > + u64 msgid, bool sw_ch, void *buf, size_t len)
> > +{
> > + mutex_lock(&xmbx->lock);
> > + xmgmt_mailbox_post(xmbx, msgid, sw_ch, buf, len);
> > + mutex_unlock(&xmbx->lock);
> > +}
> > +
> > +static void xmgmt_mailbox_resp_test_msg(struct xmgmt_mailbox *xmbx,
> > + u64 msgid, bool sw_ch)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + char *msg;
> > +
> > + mutex_lock(&xmbx->lock);
> > +
> > + if (xmbx->test_msg == NULL) {
> > + mutex_unlock(&xmbx->lock);
> > + xrt_err(pdev, "test msg is not set, drop request");
> > + return;
> > + }
> > + msg = xmbx->test_msg;
> > + xmbx->test_msg = NULL;
> > +
> > + mutex_unlock(&xmbx->lock);
> > +
> > + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, msg, strlen(msg) + 1);
> > + vfree(msg);
> > +}
> > +
> > +static int xmgmt_mailbox_dtb_add_prop(struct platform_device *pdev,
> > + char *dst_dtb, const char *ep_name, const char *regmap_name,
> > + const char *prop, const void *val, int size)
> > +{
> > + int rc = xrt_md_set_prop(DEV(pdev), dst_dtb, ep_name,
regmap_name,
> > + prop, val, size);
> > +
> > + if (rc) {
> > + xrt_err(pdev, "failed to set %s@(%s, %s): %d",
> > + ep_name, regmap_name, prop, rc);
> > + }
> > + return rc;
> > +}
> > +
> > +static int xmgmt_mailbox_dtb_add_vbnv(struct platform_device *pdev,
char *dtb)
> > +{
> > + int rc = 0;
> > + char *vbnv = xmgmt_get_vbnv(pdev);
> > +
> > + if (vbnv == NULL) {
> > + xrt_err(pdev, "failed to get VBNV");
> > + return -ENOENT;
> > + }
> > + rc = xmgmt_mailbox_dtb_add_prop(pdev, dtb, NULL, NULL,
> > + PROP_VBNV, vbnv, strlen(vbnv) + 1);
> > + kfree(vbnv);
> > + return rc;
> > +}
> > +
> > +static int xmgmt_mailbox_dtb_copy_logic_uuid(struct platform_device
*pdev,
> > + const char *src_dtb, char *dst_dtb)
> > +{
> > + const void *val;
> > + int sz;
> > + int rc = xrt_md_get_prop(DEV(pdev), src_dtb, NULL, NULL,
> > + PROP_LOGIC_UUID, &val, &sz);
> > +
> > + if (rc) {
> > + xrt_err(pdev, "failed to get %s: %d", PROP_LOGIC_UUID, rc);
> > + return rc;
> > + }
> > + return xmgmt_mailbox_dtb_add_prop(pdev, dst_dtb, NULL, NULL,
> > + PROP_LOGIC_UUID, val, sz);
> > +}
> > +
> > +static int xmgmt_mailbox_dtb_add_vrom(struct platform_device *pdev,
> > + const char *src_dtb, char *dst_dtb)
> > +{
> > + /* For compatibility for legacy xrt driver. */
> > + enum FeatureBitMask {
> > + UNIFIED_PLATFORM = 0x0000000000000001
> > + , XARE_ENBLD = 0x0000000000000002
> > + , BOARD_MGMT_ENBLD = 0x0000000000000004
> > + , MB_SCHEDULER = 0x0000000000000008
> > + , PROM_MASK = 0x0000000000000070
> > + , DEBUG_MASK = 0x000000000000FF00
> > + , PEER_TO_PEER = 0x0000000000010000
> > + , FBM_UUID = 0x0000000000020000
> > + , HBM = 0x0000000000040000
> > + , CDMA = 0x0000000000080000
> > + , QDMA = 0x0000000000100000
> > + , RUNTIME_CLK_SCALE = 0x0000000000200000
> > + , PASSTHROUGH_VIRTUALIZATION = 0x0000000000400000
> > + };
> > + struct FeatureRomHeader {
> > + unsigned char EntryPointString[4];
> > + uint8_t MajorVersion;
> > + uint8_t MinorVersion;
> > + uint32_t VivadoBuildID;
> > + uint32_t IPBuildID;
> > + uint64_t TimeSinceEpoch;
> > + unsigned char FPGAPartName[64];
> > + unsigned char VBNVName[64];
> > + uint8_t DDRChannelCount;
> > + uint8_t DDRChannelSize;
> > + uint64_t DRBaseAddress;
> > + uint64_t FeatureBitMap;
> > + unsigned char uuid[16];
> > + uint8_t HBMCount;
> > + uint8_t HBMSize;
> > + uint32_t CDMABaseAddress[4];
> > + } header = { 0 };
> > + char *vbnv = xmgmt_get_vbnv(pdev);
> > + int rc;
> > +
> > + *(u32 *)header.EntryPointString = 0x786e6c78;
> > +
> > + if (vbnv)
> > + strncpy(header.VBNVName, vbnv, sizeof(header.VBNVName) - 1);
> > + kfree(vbnv);
> > +
> > + header.FeatureBitMap = UNIFIED_PLATFORM;
> > + rc = xrt_md_get_prop(DEV(pdev), src_dtb,
> > + NODE_CMC_FW_MEM, NULL, PROP_IO_OFFSET, NULL, NULL);
> > + if (rc == 0)
> > + header.FeatureBitMap |= BOARD_MGMT_ENBLD;
> > + rc = xrt_md_get_prop(DEV(pdev), src_dtb,
> > + NODE_ERT_FW_MEM, NULL, PROP_IO_OFFSET, NULL, NULL);
> > + if (rc == 0)
> > + header.FeatureBitMap |= MB_SCHEDULER;
> > +
> > + return xmgmt_mailbox_dtb_add_prop(pdev, dst_dtb, NULL, NULL,
> > + PROP_VROM, &header, sizeof(header));
> > +}
> > +
> > +static u32 xmgmt_mailbox_dtb_user_pf(struct platform_device *pdev,
> > + const char *dtb, const char *epname, const char *regmap)
> > +{
> > + const u32 *pfnump;
> > + int rc = xrt_md_get_prop(DEV(pdev), dtb, epname, regmap,
> > + PROP_PF_NUM, (const void **)&pfnump, NULL);
> > +
> > + if (rc)
> > + return -1;
> > + return be32_to_cpu(*pfnump);
> > +}
> > +
> > +static int xmgmt_mailbox_dtb_copy_user_endpoints(struct
platform_device *pdev,
> > + const char *src, char *dst)
> > +{
> > + int rc = 0;
> > + char *epname = NULL, *regmap = NULL;
> > + u32 pfnum = xmgmt_mailbox_dtb_user_pf(pdev, src,
> > + NODE_MAILBOX_USER, NULL);
> > + const u32 level = cpu_to_be32(1);
> > + struct device *dev = DEV(pdev);
> > +
> > + if (pfnum == (u32)-1) {
> > + xrt_err(pdev, "failed to get user pf num");
> > + rc = -EINVAL;
> > + }
> > +
> > + for (xrt_md_get_next_endpoint(dev, src, NULL, NULL, &epname,
®map);
> > + rc == 0 && epname != NULL;
> > + xrt_md_get_next_endpoint(dev, src, epname, regmap,
> > + &epname, ®map)) {
> > + if (pfnum !=
> > + xmgmt_mailbox_dtb_user_pf(pdev, src, epname, regmap))
> > + continue;
> > + rc = xrt_md_copy_endpoint(dev, dst, src, epname, regmap, NULL);
> > + if (rc) {
> > + xrt_err(pdev, "failed to copy (%s, %s): %d",
> > + epname, regmap, rc);
> > + } else {
> > + rc = xrt_md_set_prop(dev, dst, epname, regmap,
> > + PROP_PARTITION_LEVEL, &level, sizeof(level));
> > + if (rc) {
> > + xrt_err(pdev,
> > + "can't set level for (%s, %s): %d",
> > + epname, regmap, rc);
> > + }
> > + }
> > + }
> > + return rc;
> > +}
> > +
> > +static char *xmgmt_mailbox_user_dtb(struct platform_device *pdev)
> > +{
> > + /* TODO: add support for PLP. */
> > + const char *src = NULL;
> > + char *dst = NULL;
> > + struct device *dev = DEV(pdev);
> > + int rc = xrt_md_create(dev, &dst);
> > +
> > + if (rc || dst == NULL)
> > + return NULL;
> > +
> > + rc = xmgmt_mailbox_dtb_add_vbnv(pdev, dst);
> > + if (rc)
> > + goto fail;
> > +
> > + src = xmgmt_get_dtb(pdev, XMGMT_BLP);
> > + if (src == NULL) {
> > + xrt_err(pdev, "failed to get BLP dtb");
> > + goto fail;
> > + }
> > +
> > + rc = xmgmt_mailbox_dtb_copy_logic_uuid(pdev, src, dst);
> > + if (rc)
> > + goto fail;
> > +
> > + rc = xmgmt_mailbox_dtb_add_vrom(pdev, src, dst);
> > + if (rc)
> > + goto fail;
> > +
> > + rc = xrt_md_copy_endpoint(dev, dst, src, NODE_PARTITION_INFO,
> > + NULL, NODE_PARTITION_INFO_BLP);
> > + if (rc)
> > + goto fail;
> > +
> > + rc = xrt_md_copy_endpoint(dev, dst, src, NODE_INTERFACES, NULL,
NULL);
> > + if (rc)
> > + goto fail;
> > +
> > + rc = xmgmt_mailbox_dtb_copy_user_endpoints(pdev, src, dst);
> > + if (rc)
> > + goto fail;
> > +
> > + xrt_md_pack(dev, dst);
> > + vfree(src);
> > + return dst;
> > +
> > +fail:
> > + vfree(src);
> > + vfree(dst);
> > + return NULL;
> > +}
> > +
> > +static void xmgmt_mailbox_resp_subdev(struct xmgmt_mailbox *xmbx,
> > + u64 msgid, bool sw_ch, u64 offset, u64 size)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + char *dtb = xmgmt_mailbox_user_dtb(pdev);
> > + long dtbsz;
> > + struct xcl_subdev *hdr;
> > + u64 totalsz;
> > +
> > + if (dtb == NULL)
> > + return;
> > +
> > + dtbsz = xrt_md_size(DEV(pdev), dtb);
> > + totalsz = dtbsz + sizeof(*hdr) - sizeof(hdr->data);
> > + if (offset != 0 || totalsz > size) {
> > + /* Only support fetching dtb in one shot. */
> > + vfree(dtb);
> > + xrt_err(pdev, "need %lldB, user buffer size is %lldB, dropped",
> > + totalsz, size);
> > + return;
> > + }
> > +
> > + hdr = vzalloc(totalsz);
> > + if (hdr == NULL) {
> > + vfree(dtb);
> > + return;
> > + }
> > +
> > + hdr->ver = 1;
> > + hdr->size = dtbsz;
> > + hdr->rtncode = XRT_MSG_SUBDEV_RTN_COMPLETE;
> > + (void) memcpy(hdr->data, dtb, dtbsz);
> > +
> > + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, hdr, totalsz);
> > +
> > + vfree(dtb);
> > + vfree(hdr);
> > +}
> > +
> > +static void xmgmt_mailbox_resp_sensor(struct xmgmt_mailbox *xmbx,
> > + u64 msgid, bool sw_ch, u64 offset, u64 size)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + struct xcl_sensor sensors = { 0 };
> > + struct platform_device *cmcpdev = xrt_subdev_get_leaf_by_id(pdev,
> > + XRT_SUBDEV_CMC, PLATFORM_DEVID_NONE);
> > + int rc;
> > +
> > + if (cmcpdev) {
> > + rc = xrt_subdev_ioctl(cmcpdev, XRT_CMC_READ_SENSORS,
&sensors);
> > + (void) xrt_subdev_put_leaf(pdev, cmcpdev);
> > + if (rc)
> > + xrt_err(pdev, "can't read sensors: %d", rc);
> > + }
> > +
> > + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &sensors,
> > + min((u64)sizeof(sensors), size));
> > +}
> > +
> > +static int xmgmt_mailbox_get_freq(struct xmgmt_mailbox *xmbx,
> > + enum CLOCK_TYPE type, u64 *freq, u64 *freq_cnter)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + const char *clkname =
> > + clock_type2epname(type) ? clock_type2epname(type) :
"UNKNOWN";
> > + struct platform_device *clkpdev =
> > + xrt_subdev_get_leaf_by_epname(pdev, clkname);
> > + int rc;
> > + struct xrt_clock_ioctl_get getfreq = { 0 };
> > +
> > + if (clkpdev == NULL) {
> > + xrt_info(pdev, "%s clock is not available", clkname);
> > + return -ENOENT;
> > + }
> > +
> > + rc = xrt_subdev_ioctl(clkpdev, XRT_CLOCK_GET, &getfreq);
> > + (void) xrt_subdev_put_leaf(pdev, clkpdev);
> > + if (rc) {
> > + xrt_err(pdev, "can't get %s clock frequency: %d", clkname, rc);
> > + return rc;
> > + }
> > +
> > + if (freq)
> > + *freq = getfreq.freq;
> > + if (freq_cnter)
> > + *freq_cnter = getfreq.freq_cnter;
> > + return 0;
> > +}
> > +
> > +static int xmgmt_mailbox_get_icap_idcode(struct xmgmt_mailbox *xmbx,
u64 *id)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + struct platform_device *icappdev = xrt_subdev_get_leaf_by_id(pdev,
> > + XRT_SUBDEV_ICAP, PLATFORM_DEVID_NONE);
> > + int rc;
> > +
> > + if (icappdev == NULL) {
> > + xrt_err(pdev, "can't find icap");
> > + return -ENOENT;
> > + }
> > +
> > + rc = xrt_subdev_ioctl(icappdev, XRT_ICAP_IDCODE, id);
> > + (void) xrt_subdev_put_leaf(pdev, icappdev);
> > + if (rc)
> > + xrt_err(pdev, "can't get icap idcode: %d", rc);
> > + return rc;
> > +}
> > +
> > +static int xmgmt_mailbox_get_mig_calib(struct xmgmt_mailbox *xmbx,
u64 *calib)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + struct platform_device *calibpdev = xrt_subdev_get_leaf_by_id(pdev,
> > + XRT_SUBDEV_CALIB, PLATFORM_DEVID_NONE);
> > + int rc;
> > + enum xrt_calib_results res;
> > +
> > + if (calibpdev == NULL) {
> > + xrt_err(pdev, "can't find mig calibration subdev");
> > + return -ENOENT;
> > + }
> > +
> > + rc = xrt_subdev_ioctl(calibpdev, XRT_CALIB_RESULT, &res);
> > + (void) xrt_subdev_put_leaf(pdev, calibpdev);
> > + if (rc) {
> > + xrt_err(pdev, "can't get mig calibration result: %d", rc);
> > + } else {
> > + if (res == XRT_CALIB_SUCCEEDED)
> > + *calib = 1;
> > + else
> > + *calib = 0;
> > + }
> > + return rc;
> > +}
> > +
> > +static void xmgmt_mailbox_resp_icap(struct xmgmt_mailbox *xmbx,
> > + u64 msgid, bool sw_ch, u64 offset, u64 size)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + struct xcl_pr_region icap = { 0 };
> > +
> > + (void) xmgmt_mailbox_get_freq(xmbx,
> > + CT_DATA, &icap.freq_data, &icap.freq_cntr_data);
> > + (void) xmgmt_mailbox_get_freq(xmbx,
> > + CT_KERNEL, &icap.freq_kernel, &icap.freq_cntr_kernel);
> > + (void) xmgmt_mailbox_get_freq(xmbx,
> > + CT_SYSTEM, &icap.freq_system, &icap.freq_cntr_system);
> > + (void) xmgmt_mailbox_get_icap_idcode(xmbx, &icap.idcode);
> > + (void) xmgmt_mailbox_get_mig_calib(xmbx, &icap.mig_calib);
> > + BUG_ON(sizeof(icap.uuid) != sizeof(uuid_t));
> > + (void) xmgmt_get_provider_uuid(pdev, XMGMT_ULP, (uuid_t
*)&icap.uuid);
> > +
> > + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &icap,
> > + min((u64)sizeof(icap), size));
> > +}
> > +
> > +static void xmgmt_mailbox_resp_bdinfo(struct xmgmt_mailbox *xmbx,
> > + u64 msgid, bool sw_ch, u64 offset, u64 size)
> > +{
> > + struct platform_device *pdev = xmbx->pdev;
> > + struct xcl_board_info *info = vzalloc(sizeof(*info));
> > + struct platform_device *cmcpdev;
> > + int rc;
> > +
> > + if (info == NULL)
> > + return;
> > +
> > + cmcpdev = xrt_subdev_get_leaf_by_id(pdev,
> > + XRT_SUBDEV_CMC, PLATFORM_DEVID_NONE);
> > + if (cmcpdev) {
> > + rc = xrt_subdev_ioctl(cmcpdev, XRT_CMC_READ_BOARD_INFO,
info);
> > + (void) xrt_subdev_put_leaf(pdev, cmcpdev);
> > + if (rc)
> > + xrt_err(pdev, "can't read board info: %d", rc);
> > + }
> > +
> > + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, info,
> > + min((u64)sizeof(*info), size));
> > +
> > + vfree(info);
> > +}
> > +
> > +static void xmgmt_mailbox_simple_respond(struct xmgmt_mailbox
*xmbx,
> > + u64 msgid, bool sw_ch, int rc)
> > +{
> > + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, &rc, sizeof(rc));
> > +}
> > +
> > +static void xmgmt_mailbox_resp_peer_data(struct xmgmt_mailbox
*xmbx,
> > + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> > +{
> > + struct xcl_mailbox_peer_data *pdata =
> > + (struct xcl_mailbox_peer_data *)req->data;
> > +
> > + if (len < (sizeof(*req) + sizeof(*pdata) - 1)) {
> > + xrt_err(xmbx->pdev, "received corrupted %s, dropped",
> > + mailbox_req2name(req->req));
> > + return;
> > + }
> > +
> > + switch (pdata->kind) {
> > + case XCL_SENSOR:
> > + xmgmt_mailbox_resp_sensor(xmbx, msgid, sw_ch,
> > + pdata->offset, pdata->size);
> > + break;
> > + case XCL_ICAP:
> > + xmgmt_mailbox_resp_icap(xmbx, msgid, sw_ch,
> > + pdata->offset, pdata->size);
> > + break;
> > + case XCL_BDINFO:
> > + xmgmt_mailbox_resp_bdinfo(xmbx, msgid, sw_ch,
> > + pdata->offset, pdata->size);
> > + break;
> > + case XCL_SUBDEV:
> > + xmgmt_mailbox_resp_subdev(xmbx, msgid, sw_ch,
> > + pdata->offset, pdata->size);
> > + break;
> > + case XCL_MIG_ECC:
> > + case XCL_FIREWALL:
> > + case XCL_DNA: /* TODO **/
> > + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, 0);
> > + break;
> > + default:
> > + xrt_err(xmbx->pdev, "%s(%s) request not handled",
> > + mailbox_req2name(req->req),
> > + mailbox_group_kind2name(pdata->kind));
> > + break;
> > + }
> > +}
> > +
> > +static bool xmgmt_mailbox_is_same_domain(struct xmgmt_mailbox
*xmbx,
> > + struct xcl_mailbox_conn *mb_conn)
> > +{
> > + uint32_t crc_chk;
> > + phys_addr_t paddr;
> > + struct platform_device *pdev = xmbx->pdev;
> > +
> > + paddr = virt_to_phys((void *)mb_conn->kaddr);
> > + if (paddr != (phys_addr_t)mb_conn->paddr) {
> > + xrt_info(pdev, "paddrs differ, user 0x%llx, mgmt 0x%llx",
> > + mb_conn->paddr, paddr);
> > + return false;
> > + }
> > +
> > + crc_chk = crc32c_le(~0, (void *)mb_conn->kaddr, PAGE_SIZE);
> > + if (crc_chk != mb_conn->crc32) {
> > + xrt_info(pdev, "CRCs differ, user 0x%x, mgmt 0x%x",
> > + mb_conn->crc32, crc_chk);
> > + return false;
> > + }
> > +
> > + return true;
> > +}
> > +
> > +static void xmgmt_mailbox_resp_user_probe(struct xmgmt_mailbox
*xmbx,
> > + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> > +{
> > + struct xcl_mailbox_conn_resp *resp = vzalloc(sizeof(*resp));
> > + struct xcl_mailbox_conn *conn = (struct xcl_mailbox_conn *)req->data;
> > +
> > + if (resp == NULL)
> > + return;
> > +
> > + if (len < (sizeof(*req) + sizeof(*conn) - 1)) {
> > + xrt_err(xmbx->pdev, "received corrupted %s, dropped",
> > + mailbox_req2name(req->req));
> > + vfree(resp);
> > + return;
> > + }
> > +
> > + resp->conn_flags |= XCL_MB_PEER_READY;
> > + if (xmgmt_mailbox_is_same_domain(xmbx, conn)) {
> > + xmbx->peer_in_same_domain = true;
> > + resp->conn_flags |= XCL_MB_PEER_SAME_DOMAIN;
> > + }
> > +
> > + xmgmt_mailbox_respond(xmbx, msgid, sw_ch, resp, sizeof(*resp));
> > + vfree(resp);
> > +}
> > +
> > +static void xmgmt_mailbox_resp_hot_reset(struct xmgmt_mailbox *xmbx,
> > + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> > +{
> > + int ret;
> > + struct platform_device *pdev = xmbx->pdev;
> > +
> > + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, 0);
> > +
> > + ret = xmgmt_hot_reset(pdev);
> > + if (ret)
> > + xrt_err(pdev, "failed to hot reset: %d", ret);
> > + else
> > + xmgmt_peer_notify_state(xmbx, true);
> > +}
> > +
> > +static void xmgmt_mailbox_resp_load_xclbin(struct xmgmt_mailbox
*xmbx,
> > + struct xcl_mailbox_req *req, size_t len, u64 msgid, bool sw_ch)
> > +{
> > + struct xcl_mailbox_bitstream_kaddr *kaddr =
> > + (struct xcl_mailbox_bitstream_kaddr *)req->data;
> > + void *xclbin = (void *)(uintptr_t)kaddr->addr;
> > + int ret = bitstream_axlf_mailbox(xmbx->pdev, xclbin);
> > +
> > + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, ret);
> > +}
> > +
> > +static void xmgmt_mailbox_listener(void *arg, void *data, size_t len,
> > + u64 msgid, int err, bool sw_ch)
> > +{
> > + struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)arg;
> > + struct platform_device *pdev = xmbx->pdev;
> > + struct xcl_mailbox_req *req = (struct xcl_mailbox_req *)data;
> > +
> > + if (err) {
> > + xrt_err(pdev, "failed to receive request: %d", err);
> > + return;
> > + }
> > + if (len < sizeof(*req)) {
> > + xrt_err(pdev, "received corrupted request");
> > + return;
> > + }
> > +
> > + XMGMT_MAILBOX_PRT_REQ_RECV(xmbx, req, sw_ch);
> > + switch (req->req) {
> > + case XCL_MAILBOX_REQ_TEST_READ:
> > + xmgmt_mailbox_resp_test_msg(xmbx, msgid, sw_ch);
> > + break;
> > + case XCL_MAILBOX_REQ_PEER_DATA:
> > + xmgmt_mailbox_resp_peer_data(xmbx, req, len, msgid, sw_ch);
> > + break;
> > + case XCL_MAILBOX_REQ_READ_P2P_BAR_ADDR: /* TODO */
> > + xmgmt_mailbox_simple_respond(xmbx, msgid, sw_ch, -
ENOTSUPP);
> > + break;
> > + case XCL_MAILBOX_REQ_USER_PROBE:
> > + xmgmt_mailbox_resp_user_probe(xmbx, req, len, msgid, sw_ch);
> > + break;
> > + case XCL_MAILBOX_REQ_HOT_RESET:
> > + xmgmt_mailbox_resp_hot_reset(xmbx, req, len, msgid, sw_ch);
> > + break;
> > + case XCL_MAILBOX_REQ_LOAD_XCLBIN_KADDR:
> > + if (xmbx->peer_in_same_domain) {
> > + xmgmt_mailbox_resp_load_xclbin(xmbx,
> > + req, len, msgid, sw_ch);
> > + } else {
> > + xrt_err(pdev, "%s not handled, not in same domain",
> > + mailbox_req2name(req->req));
> > + }
> > + break;
> > + default:
> > + xrt_err(pdev, "%s(%d) request not handled",
> > + mailbox_req2name(req->req), req->req);
> > + break;
> > + }
> > +}
> > +
> > +static void xmgmt_mailbox_reg_listener(struct xmgmt_mailbox *xmbx)
> > +{
> > + struct xrt_mailbox_ioctl_listen listen = {
> > + xmgmt_mailbox_listener, xmbx };
> > +
> > + BUG_ON(!mutex_is_locked(&xmbx->lock));
> > + if (!xmbx->mailbox)
> > + return;
> > + (void) xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_LISTEN,
&listen);
> > +}
> > +
> > +static void xmgmt_mailbox_unreg_listener(struct xmgmt_mailbox *xmbx)
> > +{
> > + struct xrt_mailbox_ioctl_listen listen = { 0 };
> > +
> > + BUG_ON(!mutex_is_locked(&xmbx->lock));
> > + BUG_ON(!xmbx->mailbox);
> > + (void) xrt_subdev_ioctl(xmbx->mailbox, XRT_MAILBOX_LISTEN,
&listen);
> > +}
> > +
> > +static bool xmgmt_mailbox_leaf_match(enum xrt_subdev_id id,
> > + struct platform_device *pdev, void *arg)
> > +{
> > + return (id == XRT_SUBDEV_MAILBOX);
> > +}
> > +
> > +static int xmgmt_mailbox_event_cb(struct platform_device *pdev,
> > + enum xrt_events evt, void *arg)
> > +{
> > + struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
> > + struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev
*)arg;
> > +
> > + switch (evt) {
> > + case XRT_EVENT_POST_CREATION:
> > + BUG_ON(esd->xevt_subdev_id != XRT_SUBDEV_MAILBOX);
> > + BUG_ON(xmbx->mailbox);
> > + mutex_lock(&xmbx->lock);
> > + xmbx->mailbox = xrt_subdev_get_leaf_by_id(pdev,
> > + XRT_SUBDEV_MAILBOX, PLATFORM_DEVID_NONE);
> > + xmgmt_mailbox_reg_listener(xmbx);
> > + mutex_unlock(&xmbx->lock);
> > + break;
> > + case XRT_EVENT_PRE_REMOVAL:
> > + BUG_ON(esd->xevt_subdev_id != XRT_SUBDEV_MAILBOX);
> > + BUG_ON(!xmbx->mailbox);
> > + mutex_lock(&xmbx->lock);
> > + xmgmt_mailbox_unreg_listener(xmbx);
> > + (void) xrt_subdev_put_leaf(pdev, xmbx->mailbox);
> > + xmbx->mailbox = NULL;
> > + mutex_unlock(&xmbx->lock);
> > + break;
> > + default:
> > + break;
> > + }
> > +
> > + return XRT_EVENT_CB_CONTINUE;
> > +}
> > +
> > +static ssize_t xmgmt_mailbox_user_dtb_show(struct file *filp,
> > + struct kobject *kobj, struct bin_attribute *attr,
> > + char *buf, loff_t off, size_t count)
> > +{
> > + struct device *dev = kobj_to_dev(kobj);
> > + struct platform_device *pdev = to_platform_device(dev);
> > + char *blob = NULL;
> > + long size;
> > + ssize_t ret = 0;
> > +
> > + blob = xmgmt_mailbox_user_dtb(pdev);
> > + if (!blob) {
> > + ret = -ENOENT;
> > + goto failed;
> > + }
> > +
> > + size = xrt_md_size(dev, blob);
> > + if (size <= 0) {
> > + ret = -EINVAL;
> > + goto failed;
> > + }
> > +
> > + if (off >= size)
> > + goto failed;
> > + if (off + count > size)
> > + count = size - off;
> > + memcpy(buf, blob + off, count);
> > +
> > + ret = count;
> > +failed:
> > + vfree(blob);
> > + return ret;
> > +}
> > +
> > +static struct bin_attribute meta_data_attr = {
> > + .attr = {
> > + .name = "metadata_for_user",
> > + .mode = 0400
> > + },
> > + .read = xmgmt_mailbox_user_dtb_show,
> > + .size = 0
> > +};
> > +
> > +static struct bin_attribute *xmgmt_mailbox_bin_attrs[] = {
> > + &meta_data_attr,
> > + NULL,
> > +};
> > +
> > +int xmgmt_mailbox_get_test_msg(struct xmgmt_mailbox *xmbx, bool
sw_ch,
> > + char *buf, size_t *len)
> > +{
> > + int rc;
> > + struct platform_device *pdev = xmbx->pdev;
> > + struct xcl_mailbox_req req = { 0, XCL_MAILBOX_REQ_TEST_READ, };
> > + struct xrt_mailbox_ioctl_request leaf_req = {
> > + .xmir_sw_ch = sw_ch,
> > + .xmir_resp_ttl = 1,
> > + .xmir_req = &req,
> > + .xmir_req_size = sizeof(req),
> > + .xmir_resp = buf,
> > + .xmir_resp_size = *len
> > + };
> > +
> > + mutex_lock(&xmbx->lock);
> > + if (xmbx->mailbox) {
> > + XMGMT_MAILBOX_PRT_REQ_SEND(xmbx, &req,
leaf_req.xmir_sw_ch);
> > + /*
> > + * mgmt should never send request to peer. it should send
> > + * either notification or response. here is the only exception
> > + * for debugging purpose.
> > + */
> > + rc = xrt_subdev_ioctl(xmbx->mailbox,
> > + XRT_MAILBOX_REQUEST, &leaf_req);
> > + } else {
> > + rc = -ENODEV;
> > + xrt_err(pdev, "mailbox not available");
> > + }
> > + mutex_unlock(&xmbx->lock);
> > +
> > + if (rc == 0)
> > + *len = leaf_req.xmir_resp_size;
> > + return rc;
> > +}
> > +
> > +int xmgmt_mailbox_set_test_msg(struct xmgmt_mailbox *xmbx,
> > + char *buf, size_t len)
> > +{
> > + mutex_lock(&xmbx->lock);
> > +
> > + if (xmbx->test_msg)
> > + vfree(xmbx->test_msg);
> > + xmbx->test_msg = vmalloc(len);
> > + if (xmbx->test_msg == NULL) {
> > + mutex_unlock(&xmbx->lock);
> > + return -ENOMEM;
> > + }
> > + (void) memcpy(xmbx->test_msg, buf, len);
> > +
> > + mutex_unlock(&xmbx->lock);
> > + return 0;
> > +}
> > +
> > +static ssize_t peer_msg_show(struct device *dev,
> > + struct device_attribute *attr, char *buf)
> > +{
> > + size_t len = 4096;
> > + struct platform_device *pdev = to_platform_device(dev);
> > + struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
> > + int ret = xmgmt_mailbox_get_test_msg(xmbx, false, buf, &len);
> > +
> > + return ret == 0 ? len : ret;
> > +}
> > +static ssize_t peer_msg_store(struct device *dev,
> > + struct device_attribute *da, const char *buf, size_t count)
> > +{
> > + struct platform_device *pdev = to_platform_device(dev);
> > + struct xmgmt_mailbox *xmbx = pdev2mbx(pdev);
> > + int ret = xmgmt_mailbox_set_test_msg(xmbx, (char *)buf, count);
> > +
> > + return ret == 0 ? count : ret;
> > +}
> > +/* Message test i/f. */
> > +static DEVICE_ATTR_RW(peer_msg);
> > +
> > +static struct attribute *xmgmt_mailbox_attrs[] = {
> > + &dev_attr_peer_msg.attr,
> > + NULL,
> > +};
> > +
> > +static const struct attribute_group xmgmt_mailbox_attrgroup = {
> > + .bin_attrs = xmgmt_mailbox_bin_attrs,
> > + .attrs = xmgmt_mailbox_attrs,
> > +};
> > +
> > +void *xmgmt_mailbox_probe(struct platform_device *pdev)
> > +{
> > + struct xmgmt_mailbox *xmbx =
> > + devm_kzalloc(DEV(pdev), sizeof(*xmbx), GFP_KERNEL);
> > +
> > + if (!xmbx)
> > + return NULL;
> > + xmbx->pdev = pdev;
> > + mutex_init(&xmbx->lock);
> > +
> > + xmbx->evt_hdl = xrt_subdev_add_event_cb(pdev,
> > + xmgmt_mailbox_leaf_match, NULL, xmgmt_mailbox_event_cb);
> > + (void) sysfs_create_group(&DEV(pdev)->kobj,
&xmgmt_mailbox_attrgroup);
> > + return xmbx;
> > +}
> > +
> > +void xmgmt_mailbox_remove(void *handle)
> > +{
> > + struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)handle;
> > + struct platform_device *pdev = xmbx->pdev;
> > +
> > + (void) sysfs_remove_group(&DEV(pdev)->kobj,
&xmgmt_mailbox_attrgroup);
> > + if (xmbx->evt_hdl)
> > + (void) xrt_subdev_remove_event_cb(pdev, xmbx->evt_hdl);
> > + if (xmbx->mailbox)
> > + (void) xrt_subdev_put_leaf(pdev, xmbx->mailbox);
> > + if (xmbx->test_msg)
> > + vfree(xmbx->test_msg);
> > +}
> > +
> > +void xmgmt_peer_notify_state(void *handle, bool online)
> > +{
> > + struct xmgmt_mailbox *xmbx = (struct xmgmt_mailbox *)handle;
> > + struct xcl_mailbox_peer_state *st;
> > + struct xcl_mailbox_req *req;
> > + size_t reqlen = sizeof(*req) + sizeof(*st) - 1;
> > +
> > + req = vzalloc(reqlen);
> > + if (req == NULL)
> > + return;
> > +
> > + req->req = XCL_MAILBOX_REQ_MGMT_STATE;
> > + st = (struct xcl_mailbox_peer_state *)req->data;
> > + st->state_flags = online ? XCL_MB_STATE_ONLINE :
XCL_MB_STATE_OFFLINE;
> > + mutex_lock(&xmbx->lock);
> > + xmgmt_mailbox_notify(xmbx, false, req, reqlen);
> > + mutex_unlock(&xmbx->lock);
> > +}
> > diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
b/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> > new file mode 100644
> > index 000000000000..042d86fcef41
> > --- /dev/null
> > +++ b/drivers/fpga/alveo/mgmt/xmgmt-main-ulp.c
> > @@ -0,0 +1,190 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Xilinx Alveo FPGA MGMT PF entry point driver
> > + *
> > + * Copyright (C) 2020 Xilinx, Inc.
> > + *
> > + * xclbin download
> > + *
> > + * Authors:
> > + * Lizhi Hou <lizhi.hou@xilinx.com>
> > + */
> > +
> > +#include <linux/firmware.h>
> > +#include <linux/uaccess.h>
> > +#include "xrt-xclbin.h"
> > +#include "xrt-metadata.h"
> > +#include "xrt-subdev.h"
> > +#include "xrt-gpio.h"
> > +#include "xmgmt-main.h"
> > +#include "xrt-icap.h"
> > +#include "xrt-axigate.h"
> > +
> > +static int xmgmt_download_bitstream(struct platform_device *pdev,
> > + const void *xclbin)
> > +{
> > + struct platform_device *icap_leaf = NULL;
> > + struct XHwIcap_Bit_Header bit_header = { 0 };
> Please fix the style error in struct name ...
> > + struct xrt_icap_ioctl_wr arg;
> > + char *bitstream = NULL;
> > + int ret;
> > +
> > + ret = xrt_xclbin_get_section(xclbin, BITSTREAM, (void **)&bitstream,
> > + NULL);
> > + if (ret || !bitstream) {
> > + xrt_err(pdev, "bitstream not found");
> > + return -ENOENT;
> > + }
> > + ret = xrt_xclbin_parse_header(bitstream,
> > + DMA_HWICAP_BITFILE_BUFFER_SIZE, &bit_header);
> > + if (ret) {
> > + ret = -EINVAL;
> > + xrt_err(pdev, "invalid bitstream header");
> > + goto done;
> > + }
> > + icap_leaf = xrt_subdev_get_leaf_by_id(pdev, XRT_SUBDEV_ICAP,
> > + PLATFORM_DEVID_NONE);
> > + if (!icap_leaf) {
> > + ret = -ENODEV;
> > + xrt_err(pdev, "icap does not exist");
> > + goto done;
> > + }
> > + arg.xiiw_bit_data = bitstream + bit_header.HeaderLength;
> > + arg.xiiw_data_len = bit_header.BitstreamLength;
> > + ret = xrt_subdev_ioctl(icap_leaf, XRT_ICAP_WRITE, &arg);
> > + if (ret)
> > + xrt_err(pdev, "write bitstream failed, ret = %d", ret);
> > +
> > +done:
> > + if (icap_leaf)
> > + xrt_subdev_put_leaf(pdev, icap_leaf);
> > + vfree(bitstream);
> > +
> > + return ret;
> > +}
> > +
> > +static bool match_shell(enum xrt_subdev_id id,
> > + struct platform_device *pdev, void *arg)
> > +{
> > + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> > + const char *ulp_gate;
> > + int ret;
> > +
> > + if (!pdata || xrt_md_size(&pdev->dev, pdata->xsp_dtb) <= 0)
> > + return false;
> > +
> > + ret = xrt_md_get_epname_pointer(&pdev->dev, pdata->xsp_dtb,
> > + NODE_GATE_ULP, NULL, &ulp_gate);
> > + if (ret)
> > + return false;
> > +
> > + ret = xrt_md_check_uuids(&pdev->dev, pdata->xsp_dtb, arg);
> > + if (ret)
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > +static bool match_ulp(enum xrt_subdev_id id,
> > + struct platform_device *pdev, void *arg)
> > +{
> > + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> > + const char *ulp_gate;
> > + int ret;
> > +
> > + if (!pdata || xrt_md_size(&pdev->dev, pdata->xsp_dtb) <= 0)
> > + return false;
> > +
> > + ret = xrt_md_check_uuids(&pdev->dev, pdata->xsp_dtb, arg);
> > + if (ret)
> > + return false;
> > +
> > + ret = xrt_md_get_epname_pointer(&pdev->dev, pdata->xsp_dtb,
> > + NODE_GATE_ULP, NULL, &ulp_gate);
> > + if (!ret)
> > + return false;
> > +
> > + return true;
> > +}
> > +
> > +int xmgmt_ulp_download(struct platform_device *pdev, const void
*xclbin)
> > +{
> > + struct platform_device *axigate_leaf;
> > + char *dtb = NULL;
> > + int ret = 0, part_inst;
> > +
> > + ret = xrt_xclbin_get_metadata(DEV(pdev), xclbin, &dtb);
> > + if (ret) {
> > + xrt_err(pdev, "can not get partition metadata, ret %d", ret);
> > + goto failed;
> > + }
> > +
> > + part_inst = xrt_subdev_lookup_partition(pdev, match_shell, dtb);
> > + if (part_inst < 0) {
> > + xrt_err(pdev, "not found matching plp.");
> > + ret = -ENODEV;
> > + goto failed;
> > + }
> > +
> > + /*
> > + * Find ulp partition with interface uuid from incoming xclbin, which
> > + * is verified before with matching plp partition.
> > + */
> > + part_inst = xrt_subdev_lookup_partition(pdev, match_ulp, dtb);
> > + if (part_inst >= 0) {
> > + ret = xrt_subdev_destroy_partition(pdev, part_inst);
> > + if (ret) {
> > + xrt_err(pdev, "failed to destroy existing ulp, %d",
> > + ret);
> > + goto failed;
> > + }
> > + }
> > +
> > + axigate_leaf = xrt_subdev_get_leaf_by_epname(pdev,
NODE_GATE_ULP);
> > +
> > + /* gate may not be exist for 0rp */
> > + if (axigate_leaf) {
> > + ret = xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE,
> > + NULL);
> > + if (ret) {
> > + xrt_err(pdev, "can not freeze gate %s, %d",
> > + NODE_GATE_ULP, ret);
> > + xrt_subdev_put_leaf(pdev, axigate_leaf);
> > + goto failed;
> > + }
> > + }
> > + ret = xmgmt_download_bitstream(pdev, xclbin);
> > + if (axigate_leaf) {
> > + xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
> > +
> > + /* Do we really need this extra toggling gate before setting
> > + * clocks?
> > + * xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE, NULL);
> > + * xrt_subdev_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
> > + */
> > +
> > + xrt_subdev_put_leaf(pdev, axigate_leaf);
> > + }
> > + if (ret) {
> > + xrt_err(pdev, "bitstream download failed, ret %d", ret);
> > + goto failed;
> > + }
> > + ret = xrt_subdev_create_partition(pdev, dtb);
> > + if (ret < 0) {
> > + xrt_err(pdev, "failed creating partition, ret %d", ret);
> > + goto failed;
> > + }
> > +
> > + ret = xrt_subdev_wait_for_partition_bringup(pdev);
> > + if (ret)
> > + xrt_err(pdev, "partiton bringup failed, ret %d", ret);
> > +
> > + /*
> > + * TODO: needs to check individual subdevs to see if there
> > + * is any error, such as clock setting, memory bank calibration.
> > + */
> > +
> > +failed:
> > + vfree(dtb);
> > + return ret;
> > +}
> > diff --git a/drivers/fpga/alveo/mgmt/xmgmt-main.c
b/drivers/fpga/alveo/mgmt/xmgmt-main.c
> > new file mode 100644
> > index 000000000000..23e68e3a4ae1
> > --- /dev/null
> > +++ b/drivers/fpga/alveo/mgmt/xmgmt-main.c
> > @@ -0,0 +1,843 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Xilinx Alveo FPGA MGMT PF entry point driver
> > + *
> > + * Copyright (C) 2020 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * Sonal Santan <sonals@xilinx.com>
> > + */
> > +
> > +#include <linux/firmware.h>
> > +#include <linux/uaccess.h>
> > +#include "xrt-xclbin.h"
> > +#include "xrt-metadata.h"
> > +#include "xrt-flash.h"
> > +#include "xrt-subdev.h"
> > +#include <linux/xrt/flash_xrt_data.h>
> > +#include <linux/xrt/xmgmt-ioctl.h>
> > +#include "xrt-gpio.h"
> > +#include "xmgmt-main.h"
> > +#include "xmgmt-fmgr.h"
> > +#include "xrt-icap.h"
> > +#include "xrt-axigate.h"
> > +#include "xmgmt-main-impl.h"
> > +
> > +#define XMGMT_MAIN "xmgmt_main"
> > +
> > +struct xmgmt_main {
> > + struct platform_device *pdev;
> > + void *evt_hdl;
> > + char *firmware_blp;
> > + char *firmware_plp;
> > + char *firmware_ulp;
> > + bool flash_ready;
> > + bool gpio_ready;
> > + struct fpga_manager *fmgr;
> > + void *mailbox_hdl;
> > + struct mutex busy_mutex;
> > +
> > + uuid_t *blp_intf_uuids;
> > + u32 blp_intf_uuid_num;
> > +};
> > +
> > +char *xmgmt_get_vbnv(struct platform_device *pdev)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > + const char *vbnv;
> > + char *ret;
> > + int i;
> > +
> > + if (xmm->firmware_plp) {
> > + vbnv = ((struct axlf *)xmm->firmware_plp)->
> > + m_header.m_platformVBNV;
> > + } else if (xmm->firmware_blp) {
> > + vbnv = ((struct axlf *)xmm->firmware_blp)->
> > + m_header.m_platformVBNV;
> > + } else {
> > + return NULL;
> > + }
> > +
> > + ret = kstrdup(vbnv, GFP_KERNEL);
> > + for (i = 0; i < strlen(ret); i++) {
> > + if (ret[i] == ':' || ret[i] == '.')
> > + ret[i] = '_';
> > + }
> > + return ret;
> > +}
> > +
> > +static bool xmgmt_main_leaf_match(enum xrt_subdev_id id,
> > + struct platform_device *pdev, void *arg)
> > +{
> > + if (id == XRT_SUBDEV_GPIO)
> > + return xrt_subdev_has_epname(pdev, arg);
> > + else if (id == XRT_SUBDEV_QSPI)
> > + return true;
> > +
> > + return false;
> > +}
> > +
> > +static int get_dev_uuid(struct platform_device *pdev, char *uuidstr, size_t
len)
> > +{
> > + char uuid[16];
> > + struct platform_device *gpio_leaf;
> > + struct xrt_gpio_ioctl_rw gpio_arg = { 0 };
> > + int err, i, count;
> > +
> > + gpio_leaf = xrt_subdev_get_leaf_by_epname(pdev, NODE_BLP_ROM);
> > + if (!gpio_leaf) {
> > + xrt_err(pdev, "can not get %s", NODE_BLP_ROM);
> > + return -EINVAL;
> > + }
> > +
> > + gpio_arg.xgir_id = XRT_GPIO_ROM_UUID;
> > + gpio_arg.xgir_buf = uuid;
> > + gpio_arg.xgir_len = sizeof(uuid);
> > + gpio_arg.xgir_offset = 0;
> > + err = xrt_subdev_ioctl(gpio_leaf, XRT_GPIO_READ, &gpio_arg);
> > + xrt_subdev_put_leaf(pdev, gpio_leaf);
> > + if (err) {
> > + xrt_err(pdev, "can not get uuid: %d", err);
> > + return err;
> > + }
> > +
> > + for (count = 0, i = sizeof(uuid) - sizeof(u32);
> > + i >= 0 && len > count; i -= sizeof(u32)) {
> > + count += snprintf(uuidstr + count, len - count,
> > + "%08x", *(u32 *)&uuid[i]);
> > + }
> > + return 0;
> > +}
> > +
> > +int xmgmt_hot_reset(struct platform_device *pdev)
> > +{
> > + int ret = xrt_subdev_broadcast_event(pdev,
XRT_EVENT_PRE_HOT_RESET);
> > +
> > + if (ret) {
> > + xrt_err(pdev, "offline failed, hot reset is canceled");
> > + return ret;
> > + }
> > +
> > + (void) xrt_subdev_hot_reset(pdev);
> > + xrt_subdev_broadcast_event(pdev, XRT_EVENT_POST_HOT_RESET);
> > + return 0;
> > +}
> > +
> > +static ssize_t reset_store(struct device *dev,
> > + struct device_attribute *da, const char *buf, size_t count)
> > +{
> > + struct platform_device *pdev = to_platform_device(dev);
> > +
> > + (void) xmgmt_hot_reset(pdev);
> > + return count;
> > +}
> > +static DEVICE_ATTR_WO(reset);
> > +
> > +static ssize_t VBNV_show(struct device *dev,
> > + struct device_attribute *da, char *buf)
> > +{
> > + ssize_t ret;
> > + char *vbnv;
> > + struct platform_device *pdev = to_platform_device(dev);
> > +
> > + vbnv = xmgmt_get_vbnv(pdev);
> > + ret = sprintf(buf, "%s\n", vbnv);
> > + kfree(vbnv);
> > + return ret;
> > +}
> > +static DEVICE_ATTR_RO(VBNV);
> > +
> > +static ssize_t logic_uuids_show(struct device *dev,
> > + struct device_attribute *da, char *buf)
> > +{
> > + ssize_t ret;
> > + char uuid[80];
> > + struct platform_device *pdev = to_platform_device(dev);
> > +
> > + /*
> > + * Getting UUID pointed to by VSEC,
> > + * should be the same as logic UUID of BLP.
> > + * TODO: add PLP logic UUID
> > + */
> > + ret = get_dev_uuid(pdev, uuid, sizeof(uuid));
> > + if (ret)
> > + return ret;
> > + ret = sprintf(buf, "%s\n", uuid);
> > + return ret;
> > +}
> > +static DEVICE_ATTR_RO(logic_uuids);
> > +
> > +static inline void uuid2str(const uuid_t *uuid, char *uuidstr, size_t len)
> > +{
> > + int i, p;
> > + u8 *u = (u8 *)uuid;
> > +
> > + BUG_ON(sizeof(uuid_t) * 2 + 1 > len);
> > + for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
> > + (void) snprintf(&uuidstr[p*2], 3, "%02x", u[i]);
> > +}
> > +
> > +static ssize_t interface_uuids_show(struct device *dev,
> > + struct device_attribute *da, char *buf)
> > +{
> > + ssize_t ret = 0;
> > + struct platform_device *pdev = to_platform_device(dev);
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > + u32 i;
> > +
> > + /*
> > + * TODO: add PLP interface UUID
> > + */
> > + for (i = 0; i < xmm->blp_intf_uuid_num; i++) {
> > + char uuidstr[80];
> > +
> > + uuid2str(&xmm->blp_intf_uuids[i], uuidstr, sizeof(uuidstr));
> > + ret += sprintf(buf + ret, "%s\n", uuidstr);
> > + }
> > + return ret;
> > +}
> > +static DEVICE_ATTR_RO(interface_uuids);
> > +
> > +static struct attribute *xmgmt_main_attrs[] = {
> > + &dev_attr_reset.attr,
> > + &dev_attr_VBNV.attr,
> > + &dev_attr_logic_uuids.attr,
> > + &dev_attr_interface_uuids.attr,
> > + NULL,
> > +};
> > +
> > +static ssize_t ulp_image_write(struct file *filp, struct kobject *kobj,
> > + struct bin_attribute *attr, char *buffer, loff_t off, size_t count)
> > +{
> > + struct xmgmt_main *xmm =
> > + dev_get_drvdata(container_of(kobj, struct device, kobj));
> > + struct axlf *xclbin;
> > + ulong len;
> > +
> > + if (off == 0) {
> > + if (count < sizeof(*xclbin)) {
> > + xrt_err(xmm->pdev, "count is too small %ld", count);
> > + return -EINVAL;
> > + }
> > +
> > + if (xmm->firmware_ulp) {
> > + vfree(xmm->firmware_ulp);
> > + xmm->firmware_ulp = NULL;
> > + }
> > + xclbin = (struct axlf *)buffer;
> > + xmm->firmware_ulp = vmalloc(xclbin->m_header.m_length);
> > + if (!xmm->firmware_ulp)
> > + return -ENOMEM;
> > + } else
> > + xclbin = (struct axlf *)xmm->firmware_ulp;
> > +
> > + len = xclbin->m_header.m_length;
> > + if (off + count >= len && off < len) {
> > + memcpy(xmm->firmware_ulp + off, buffer, len - off);
> > + xmgmt_ulp_download(xmm->pdev, xmm->firmware_ulp);
> > + } else if (off + count < len) {
> > + memcpy(xmm->firmware_ulp + off, buffer, count);
> > + }
> > +
> > + return count;
> > +}
> > +
> > +static struct bin_attribute ulp_image_attr = {
> > + .attr = {
> > + .name = "ulp_image",
> > + .mode = 0200
> > + },
> > + .write = ulp_image_write,
> > + .size = 0
> > +};
> > +
> > +static struct bin_attribute *xmgmt_main_bin_attrs[] = {
> > + &ulp_image_attr,
> > + NULL,
> > +};
> > +
> > +static const struct attribute_group xmgmt_main_attrgroup = {
> > + .attrs = xmgmt_main_attrs,
> > + .bin_attrs = xmgmt_main_bin_attrs,
> > +};
> > +
> > +static int load_firmware_from_flash(struct platform_device *pdev,
> > + char **fw_buf, size_t *len)
> > +{
> > + struct platform_device *flash_leaf = NULL;
> > + struct flash_data_header header = { 0 };
> > + const size_t magiclen = sizeof(header.fdh_id_begin.fdi_magic);
> > + size_t flash_size = 0;
> > + int ret = 0;
> > + char *buf = NULL;
> > + struct flash_data_ident id = { 0 };
> > + struct xrt_flash_ioctl_read frd = { 0 };
> > +
> > + xrt_info(pdev, "try loading fw from flash");
> > +
> > + flash_leaf = xrt_subdev_get_leaf_by_id(pdev, XRT_SUBDEV_QSPI,
> > + PLATFORM_DEVID_NONE);
> > + if (flash_leaf == NULL) {
> > + xrt_err(pdev, "failed to hold flash leaf");
> > + return -ENODEV;
> > + }
> > +
> > + (void) xrt_subdev_ioctl(flash_leaf, XRT_FLASH_GET_SIZE, &flash_size);
> > + if (flash_size == 0) {
> > + xrt_err(pdev, "failed to get flash size");
> > + ret = -EINVAL;
> > + goto done;
> > + }
> > +
> > + frd.xfir_buf = (char *)&header;
> > + frd.xfir_size = sizeof(header);
> > + frd.xfir_offset = flash_size - sizeof(header);
> > + ret = xrt_subdev_ioctl(flash_leaf, XRT_FLASH_READ, &frd);
> > + if (ret) {
> > + xrt_err(pdev, "failed to read header from flash: %d", ret);
> > + goto done;
> > + }
> > +
> > + /* Pick the end ident since header is aligned in the end of flash. */
> > + id = header.fdh_id_end;
> > + if (strncmp(id.fdi_magic, XRT_DATA_MAGIC, magiclen)) {
> > + char tmp[sizeof(id.fdi_magic) + 1] = { 0 };
> > +
> > + memcpy(tmp, id.fdi_magic, magiclen);
> > + xrt_info(pdev, "ignore meta data, bad magic: %s", tmp);
> > + ret = -ENOENT;
> > + goto done;
> > + }
> > + if (id.fdi_version != 0) {
> > + xrt_info(pdev, "flash meta data version is not supported: %d",
> > + id.fdi_version);
> > + ret = -EOPNOTSUPP;
> > + goto done;
> > + }
> > +
> > + buf = vmalloc(header.fdh_data_len);
> > + if (buf == NULL) {
> > + ret = -ENOMEM;
> > + goto done;
> > + }
> > +
> > + frd.xfir_buf = buf;
> > + frd.xfir_size = header.fdh_data_len;
> > + frd.xfir_offset = header.fdh_data_offset;
> > + ret = xrt_subdev_ioctl(flash_leaf, XRT_FLASH_READ, &frd);
> > + if (ret) {
> > + xrt_err(pdev, "failed to read meta data from flash: %d", ret);
> > + goto done;
> > + } else if (flash_xrt_data_get_parity32(buf, header.fdh_data_len) ^
> > + header.fdh_data_parity) {
> > + xrt_err(pdev, "meta data is corrupted");
> > + ret = -EINVAL;
> > + goto done;
> > + }
> > +
> > + xrt_info(pdev, "found meta data of %d bytes @0x%x",
> > + header.fdh_data_len, header.fdh_data_offset);
> > + *fw_buf = buf;
> > + *len = header.fdh_data_len;
> > +
> > +done:
> > + (void) xrt_subdev_put_leaf(pdev, flash_leaf);
> > + return ret;
> > +}
> > +
> > +static int load_firmware_from_disk(struct platform_device *pdev, char
**fw_buf,
> > + size_t *len)
> > +{
> > + char uuid[80];
> > + int err = 0;
> > + char fw_name[256];
> > + const struct firmware *fw;
> > +
> > + err = get_dev_uuid(pdev, uuid, sizeof(uuid));
> > + if (err)
> > + return err;
> > +
> > + (void) snprintf(fw_name,
> > + sizeof(fw_name), "xilinx/%s/partition.xsabin", uuid);
> > + xrt_info(pdev, "try loading fw: %s", fw_name);
> > +
> > + err = request_firmware(&fw, fw_name, DEV(pdev));
> > + if (err)
> > + return err;
> > +
> > + *fw_buf = vmalloc(fw->size);
> > + *len = fw->size;
> > + if (*fw_buf != NULL)
> > + memcpy(*fw_buf, fw->data, fw->size);
> > + else
> > + err = -ENOMEM;
> > +
> > + release_firmware(fw);
> > + return 0;
> > +}
> > +
> > +static const char *xmgmt_get_axlf_firmware(struct xmgmt_main *xmm,
> > + enum provider_kind kind)
> > +{
> > + switch (kind) {
> > + case XMGMT_BLP:
> > + return xmm->firmware_blp;
> > + case XMGMT_PLP:
> > + return xmm->firmware_plp;
> > + case XMGMT_ULP:
> > + return xmm->firmware_ulp;
> > + default:
> > + xrt_err(xmm->pdev, "unknown axlf kind: %d", kind);
> > + return NULL;
> > + }
> > +}
> > +
> > +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind
kind)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > + char *dtb = NULL;
> > + const char *provider = xmgmt_get_axlf_firmware(xmm, kind);
> > + int rc;
> > +
> > + if (provider == NULL)
> > + return dtb;
> > +
> > + rc = xrt_xclbin_get_metadata(DEV(pdev), provider, &dtb);
> > + if (rc)
> > + xrt_err(pdev, "failed to find dtb: %d", rc);
> > + return dtb;
> > +}
> > +
> > +static const char *get_uuid_from_firmware(struct platform_device *pdev,
> > + const char *axlf)
> > +{
> > + const void *uuid = NULL;
> > + const void *uuiddup = NULL;
> > + void *dtb = NULL;
> > + int rc;
> > +
> > + rc = xrt_xclbin_get_section(axlf, PARTITION_METADATA, &dtb, NULL);
> > + if (rc)
> > + return NULL;
> > +
> > + rc = xrt_md_get_prop(DEV(pdev), dtb, NULL, NULL,
> > + PROP_LOGIC_UUID, &uuid, NULL);
> > + if (!rc)
> > + uuiddup = kstrdup(uuid, GFP_KERNEL);
> > + vfree(dtb);
> > + return uuiddup;
> > +}
> > +
> > +static bool is_valid_firmware(struct platform_device *pdev,
> > + char *fw_buf, size_t fw_len)
> > +{
> > + struct axlf *axlf = (struct axlf *)fw_buf;
> > + size_t axlflen = axlf->m_header.m_length;
> > + const char *fw_uuid;
> > + char dev_uuid[80];
> > + int err;
> > +
> > + err = get_dev_uuid(pdev, dev_uuid, sizeof(dev_uuid));
> > + if (err)
> > + return false;
> > +
> > + if (memcmp(fw_buf, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)) != 0) {
> > + xrt_err(pdev, "unknown fw format");
> > + return false;
> > + }
> > +
> > + if (axlflen > fw_len) {
> > + xrt_err(pdev, "truncated fw, length: %ld, expect: %ld",
> > + fw_len, axlflen);
> > + return false;
> > + }
> > +
> > + fw_uuid = get_uuid_from_firmware(pdev, fw_buf);
> > + if (fw_uuid == NULL || strcmp(fw_uuid, dev_uuid) != 0) {
> > + xrt_err(pdev, "bad fw UUID: %s, expect: %s",
> > + fw_uuid ? fw_uuid : "<none>", dev_uuid);
> > + kfree(fw_uuid);
> > + return false;
> > + }
> > +
> > + kfree(fw_uuid);
> > + return true;
> > +}
> > +
> > +int xmgmt_get_provider_uuid(struct platform_device *pdev,
> > + enum provider_kind kind, uuid_t *uuid)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > + const char *fwbuf;
> > + const char *fw_uuid;
> > + int rc = -ENOENT;
> > +
> > + mutex_lock(&xmm->busy_mutex);
> > +
> > + fwbuf = xmgmt_get_axlf_firmware(xmm, kind);
> > + if (fwbuf == NULL)
> > + goto done;
> > +
> > + fw_uuid = get_uuid_from_firmware(pdev, fwbuf);
> > + if (fw_uuid == NULL)
> > + goto done;
> > +
> > + rc = xrt_md_uuid_strtoid(DEV(pdev), fw_uuid, uuid);
> > + kfree(fw_uuid);
> > +
> > +done:
> > + mutex_unlock(&xmm->busy_mutex);
> > + return rc;
> > +}
> > +
> > +static int xmgmt_create_blp(struct xmgmt_main *xmm)
> > +{
> > + struct platform_device *pdev = xmm->pdev;
> > + int rc = 0;
> > + char *dtb = NULL;
> > +
> > + dtb = xmgmt_get_dtb(pdev, XMGMT_BLP);
> > + if (dtb) {
> > + rc = xrt_subdev_create_partition(pdev, dtb);
> > + if (rc < 0)
> > + xrt_err(pdev, "failed to create BLP: %d", rc);
> > + else
> > + rc = 0;
> > +
> > + BUG_ON(xmm->blp_intf_uuids);
> > + xrt_md_get_intf_uuids(&pdev->dev, dtb,
> > + &xmm->blp_intf_uuid_num, NULL);
> > + if (xmm->blp_intf_uuid_num > 0) {
> > + xmm->blp_intf_uuids = vzalloc(sizeof(uuid_t) *
> > + xmm->blp_intf_uuid_num);
> > + xrt_md_get_intf_uuids(&pdev->dev, dtb,
> > + &xmm->blp_intf_uuid_num, xmm->blp_intf_uuids);
> > + }
> > + }
> > +
> > + vfree(dtb);
> > + return rc;
> > +}
> > +
> > +static int xmgmt_main_event_cb(struct platform_device *pdev,
> > + enum xrt_events evt, void *arg)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > + struct xrt_event_arg_subdev *esd = (struct xrt_event_arg_subdev
*)arg;
> > + enum xrt_subdev_id id;
> > + int instance;
> > + size_t fwlen;
> > +
> > + switch (evt) {
> > + case XRT_EVENT_POST_CREATION: {
> > + id = esd->xevt_subdev_id;
> > + instance = esd->xevt_subdev_instance;
> > + xrt_info(pdev, "processing event %d for (%d, %d)",
> > + evt, id, instance);
> > +
> > + if (id == XRT_SUBDEV_GPIO)
> > + xmm->gpio_ready = true;
> > + else if (id == XRT_SUBDEV_QSPI)
> > + xmm->flash_ready = true;
> > + else
> > + BUG_ON(1);
> > +
> > + if (xmm->gpio_ready && xmm->flash_ready) {
> > + int rc;
> > +
> > + rc = load_firmware_from_disk(pdev, &xmm->firmware_blp,
> > + &fwlen);
> > + if (rc != 0) {
> > + rc = load_firmware_from_flash(pdev,
> > + &xmm->firmware_blp, &fwlen);
> > + }
> > + if (rc == 0 && is_valid_firmware(pdev,
> > + xmm->firmware_blp, fwlen))
> > + (void) xmgmt_create_blp(xmm);
> > + else
> > + xrt_err(pdev,
> > + "failed to find firmware, giving up");
> > + xmm->evt_hdl = NULL;
> > + }
> > + break;
> > + }
> > + case XRT_EVENT_POST_ATTACH:
> > + xmgmt_peer_notify_state(xmm->mailbox_hdl, true);
> > + break;
> > + case XRT_EVENT_PRE_DETACH:
> > + xmgmt_peer_notify_state(xmm->mailbox_hdl, false);
> > + break;
> > + default:
> > + xrt_info(pdev, "ignored event %d", evt);
> > + break;
> > + }
> > +
> > + return XRT_EVENT_CB_CONTINUE;
> > +}
> > +
> > +static int xmgmt_main_probe(struct platform_device *pdev)
> > +{
> > + struct xmgmt_main *xmm;
> > +
> > + xrt_info(pdev, "probing...");
> > +
> > + xmm = devm_kzalloc(DEV(pdev), sizeof(*xmm), GFP_KERNEL);
> > + if (!xmm)
> > + return -ENOMEM;
> > +
> > + xmm->pdev = pdev;
> > + platform_set_drvdata(pdev, xmm);
> > + xmm->fmgr = xmgmt_fmgr_probe(pdev);
> > + xmm->mailbox_hdl = xmgmt_mailbox_probe(pdev);
> > + mutex_init(&xmm->busy_mutex);
> > +
> > + xmm->evt_hdl = xrt_subdev_add_event_cb(pdev,
> > + xmgmt_main_leaf_match, NODE_BLP_ROM,
xmgmt_main_event_cb);
> > +
> > + /* Ready to handle req thru sysfs nodes. */
> > + if (sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup))
> > + xrt_err(pdev, "failed to create sysfs group");
> > + return 0;
> > +}
> > +
> > +static int xmgmt_main_remove(struct platform_device *pdev)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > +
> > + /* By now, partition driver should prevent any inter-leaf call. */
> > +
> > + xrt_info(pdev, "leaving...");
> > +
> > + if (xmm->evt_hdl)
> > + (void) xrt_subdev_remove_event_cb(pdev, xmm->evt_hdl);
> > + vfree(xmm->blp_intf_uuids);
> > + vfree(xmm->firmware_blp);
> > + vfree(xmm->firmware_plp);
> > + vfree(xmm->firmware_ulp);
> > + (void) xmgmt_fmgr_remove(xmm->fmgr);
> > + xmgmt_mailbox_remove(xmm->mailbox_hdl);
> > + (void) sysfs_remove_group(&DEV(pdev)->kobj,
&xmgmt_main_attrgroup);
> > + return 0;
> > +}
> > +
> > +static int
> > +xmgmt_main_leaf_ioctl(struct platform_device *pdev, u32 cmd, void
*arg)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > + int ret = 0;
> > +
> > + xrt_info(pdev, "handling IOCTL cmd: %d", cmd);
> > +
> > + switch (cmd) {
> > + case XRT_MGMT_MAIN_GET_AXLF_SECTION: {
> > + struct xrt_mgmt_main_ioctl_get_axlf_section *get =
> > + (struct xrt_mgmt_main_ioctl_get_axlf_section *)arg;
> > + const char *firmware =
> > + xmgmt_get_axlf_firmware(xmm, get->xmmigas_axlf_kind);
> > +
> > + if (firmware == NULL) {
> > + ret = -ENOENT;
> > + } else {
> > + ret = xrt_xclbin_get_section(firmware,
> > + get->xmmigas_section_kind,
> > + &get->xmmigas_section,
> > + &get->xmmigas_section_size);
> > + }
> > + break;
> > + }
> > + case XRT_MGMT_MAIN_GET_VBNV: {
> > + char **vbnv_p = (char **)arg;
> > +
> > + *vbnv_p = xmgmt_get_vbnv(pdev);
> > + break;
> > + }
> > + default:
> > + xrt_err(pdev, "unknown cmd: %d", cmd);
> > + ret = -EINVAL;
> > + break;
> > + }
> > + return ret;
> > +}
> > +
> > +static int xmgmt_main_open(struct inode *inode, struct file *file)
> > +{
> > + struct platform_device *pdev = xrt_devnode_open(inode);
> > +
> > + /* Device may have gone already when we get here. */
> > + if (!pdev)
> > + return -ENODEV;
> > +
> > + xrt_info(pdev, "opened");
> > + file->private_data = platform_get_drvdata(pdev);
> > + return 0;
> > +}
> > +
> > +static int xmgmt_main_close(struct inode *inode, struct file *file)
> > +{
> > + struct xmgmt_main *xmm = file->private_data;
> > +
> > + xrt_devnode_close(inode);
> > +
> > + xrt_info(xmm->pdev, "closed");
> > + return 0;
> > +}
> > +
> > +static int xmgmt_bitstream_axlf_fpga_mgr(struct xmgmt_main *xmm,
> > + void *axlf, size_t size)
> > +{
> > + int ret;
> > + struct fpga_image_info info = { 0 };
> > +
> > + BUG_ON(!mutex_is_locked(&xmm->busy_mutex));
> > +
> > + /*
> > + * Should any error happens during download, we can't trust
> > + * the cached xclbin any more.
> > + */
> > + vfree(xmm->firmware_ulp);
> > + xmm->firmware_ulp = NULL;
> > +
> > + info.buf = (char *)axlf;
> > + info.count = size;
> > + ret = fpga_mgr_load(xmm->fmgr, &info);
> > + if (ret == 0)
> > + xmm->firmware_ulp = axlf;
> > +
> > + return ret;
> > +}
> > +
> > +int bitstream_axlf_mailbox(struct platform_device *pdev, const void *axlf)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > + void *copy_buffer = NULL;
> > + size_t copy_buffer_size = 0;
> > + const struct axlf *xclbin_obj = axlf;
> > + int ret = 0;
> > +
> > + if (memcmp(xclbin_obj->m_magic, ICAP_XCLBIN_V2,
sizeof(ICAP_XCLBIN_V2)))
> > + return -EINVAL;
> > +
> > + copy_buffer_size = xclbin_obj->m_header.m_length;
> > + if (copy_buffer_size > MAX_XCLBIN_SIZE)
> > + return -EINVAL;
> > + copy_buffer = vmalloc(copy_buffer_size);
> > + if (copy_buffer == NULL)
> > + return -ENOMEM;
> > + (void) memcpy(copy_buffer, axlf, copy_buffer_size);
> > +
> > + mutex_lock(&xmm->busy_mutex);
> > + ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer,
copy_buffer_size);
> > + mutex_unlock(&xmm->busy_mutex);
> > + if (ret)
> > + vfree(copy_buffer);
> > + return ret;
> > +}
> > +
> > +static int bitstream_axlf_ioctl(struct xmgmt_main *xmm, const void
__user *arg)
> > +{
> > + void *copy_buffer = NULL;
> > + size_t copy_buffer_size = 0;
> > + struct xmgmt_ioc_bitstream_axlf ioc_obj = { 0 };
> > + struct axlf xclbin_obj = { {0} };
> > + int ret = 0;
> > +
> > + if (copy_from_user((void *)&ioc_obj, arg, sizeof(ioc_obj)))
> > + return -EFAULT;
> > + if (copy_from_user((void *)&xclbin_obj, ioc_obj.xclbin,
> > + sizeof(xclbin_obj)))
> > + return -EFAULT;
> > + if (memcmp(xclbin_obj.m_magic, ICAP_XCLBIN_V2,
sizeof(ICAP_XCLBIN_V2)))
> > + return -EINVAL;
> > +
> > + copy_buffer_size = xclbin_obj.m_header.m_length;
> > + if (copy_buffer_size > MAX_XCLBIN_SIZE)
> > + return -EINVAL;
> > + copy_buffer = vmalloc(copy_buffer_size);
> > + if (copy_buffer == NULL)
> > + return -ENOMEM;
> > +
> > + if (copy_from_user(copy_buffer, ioc_obj.xclbin, copy_buffer_size)) {
> > + vfree(copy_buffer);
> > + return -EFAULT;
> > + }
> > +
> > + ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer,
copy_buffer_size);
> > + if (ret)
> > + vfree(copy_buffer);
> > +
> > + return ret;
> > +}
> > +
> > +static long xmgmt_main_ioctl(struct file *filp, unsigned int cmd,
> > + unsigned long arg)
> > +{
> > + long result = 0;
> > + struct xmgmt_main *xmm = filp->private_data;
> > +
> > + BUG_ON(!xmm);
> > +
> > + if (_IOC_TYPE(cmd) != XMGMT_IOC_MAGIC)
> > + return -ENOTTY;
> > +
> > + mutex_lock(&xmm->busy_mutex);
> > +
> > + xrt_info(xmm->pdev, "ioctl cmd %d, arg %ld", cmd, arg);
> > + switch (cmd) {
> > + case XMGMT_IOCICAPDOWNLOAD_AXLF:
> > + result = bitstream_axlf_ioctl(xmm, (const void __user *)arg);
> > + break;
> > + default:
> > + result = -ENOTTY;
> > + break;
> > + }
> > +
> > + mutex_unlock(&xmm->busy_mutex);
> > + return result;
> > +}
> > +
> > +void *xmgmt_pdev2mailbox(struct platform_device *pdev)
> > +{
> > + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> > +
> > + return xmm->mailbox_hdl;
> > +}
> > +
> > +struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[] = {
> > + {
> > + .xse_names = (struct xrt_subdev_ep_names []){
> > + { .ep_name = NODE_MGMT_MAIN },
> > + { NULL },
> > + },
> > + .xse_min_ep = 1,
> > + },
> > + { 0 },
> > +};
> > +
> > +struct xrt_subdev_drvdata xmgmt_main_data = {
> > + .xsd_dev_ops = {
> > + .xsd_ioctl = xmgmt_main_leaf_ioctl,
> > + },
> > + .xsd_file_ops = {
> > + .xsf_ops = {
> > + .owner = THIS_MODULE,
> > + .open = xmgmt_main_open,
> > + .release = xmgmt_main_close,
> > + .unlocked_ioctl = xmgmt_main_ioctl,
> > + },
> > + .xsf_dev_name = "xmgmt",
> > + },
> > +};
> > +
> > +static const struct platform_device_id xmgmt_main_id_table[] = {
> > + { XMGMT_MAIN, (kernel_ulong_t)&xmgmt_main_data },
> > + { },
> > +};
> > +
> > +struct platform_driver xmgmt_main_driver = {
> > + .driver = {
> > + .name = XMGMT_MAIN,
> > + },
> > + .probe = xmgmt_main_probe,
> > + .remove = xmgmt_main_remove,
> > + .id_table = xmgmt_main_id_table,
> > +};
> > diff --git a/drivers/fpga/alveo/mgmt/xmgmt-root.c
b/drivers/fpga/alveo/mgmt/xmgmt-root.c
> > new file mode 100644
> > index 000000000000..005fd5e42651
> > --- /dev/null
> > +++ b/drivers/fpga/alveo/mgmt/xmgmt-root.c
> > @@ -0,0 +1,375 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Xilinx Alveo Management Function Driver
> > + *
> > + * Copyright (C) 2020 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * Cheng Zhen <maxz@xilinx.com>
> > + */
> > +
> > +#include <linux/module.h>
> > +#include <linux/pci.h>
> > +#include <linux/aer.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/delay.h>
> > +
> > +#include "xrt-root.h"
> > +#include "xrt-subdev.h"
> > +#include "xmgmt-main-impl.h"
> > +#include "xrt-metadata.h"
> > +
> > +#define XMGMT_MODULE_NAME "xmgmt"
> > +#define XMGMT_DRIVER_VERSION "4.0.0"
> > +
> > +#define XMGMT_PDEV(xm) ((xm)->pdev)
> > +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
> > +#define xmgmt_err(xm, fmt, args...) \
> > + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define xmgmt_warn(xm, fmt, args...) \
> > + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define xmgmt_info(xm, fmt, args...) \
> > + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define xmgmt_dbg(xm, fmt, args...) \
> > + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define XMGMT_DEV_ID(pdev) \
> > + ((pci_domain_nr(pdev->bus) << 16) | \
> > + PCI_DEVID(pdev->bus->number, 0))
> > +
> > +static struct class *xmgmt_class;
> > +static const struct pci_device_id xmgmt_pci_ids[] = {
> > + { PCI_DEVICE(0x10EE, 0xd020), },
> > + { PCI_DEVICE(0x10EE, 0x5020), },
> > + { 0, }
> > +};
> > +
> > +struct xmgmt {
> > + struct pci_dev *pdev;
> > + void *root;
> > +
> > + /* save config for pci reset */
> > + u32 saved_config[8][16];
> > + bool ready;
> > +};
> > +
> > +static int xmgmt_config_pci(struct xmgmt *xm)
> > +{
> > + struct pci_dev *pdev = XMGMT_PDEV(xm);
> > + int rc;
> > +
> > + rc = pcim_enable_device(pdev);
> > + if (rc < 0) {
> > + xmgmt_err(xm, "failed to enable device: %d", rc);
> > + return rc;
> > + }
> > +
> > + rc = pci_enable_pcie_error_reporting(pdev);
> > + if (rc)
> > + xmgmt_warn(xm, "failed to enable AER: %d", rc);
> > +
> > + pci_set_master(pdev);
> > +
> > + rc = pcie_get_readrq(pdev);
> > + if (rc < 0) {
> > + xmgmt_err(xm, "failed to read mrrs %d", rc);
> > + return rc;
> > + }
> > + if (rc > 512) {
> > + rc = pcie_set_readrq(pdev, 512);
> > + if (rc) {
> > + xmgmt_err(xm, "failed to force mrrs %d", rc);
> > + return rc;
> > + }
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void xmgmt_save_config_space(struct pci_dev *pdev, u32
*saved_config)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < 16; i++)
> > + pci_read_config_dword(pdev, i * 4, &saved_config[i]);
> > +}
> > +
> > +static int xmgmt_match_slot_and_save(struct device *dev, void *data)
> > +{
> > + struct xmgmt *xm = data;
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > +
> > + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> > + pci_cfg_access_lock(pdev);
> > + pci_save_state(pdev);
> > + xmgmt_save_config_space(pdev,
> > + xm->saved_config[PCI_FUNC(pdev->devfn)]);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void xmgmt_pci_save_config_all(struct xmgmt *xm)
> > +{
> > + bus_for_each_dev(&pci_bus_type, NULL, xm,
xmgmt_match_slot_and_save);
> > +}
> > +
> > +static void xmgmt_restore_config_space(struct pci_dev *pdev, u32
*config_saved)
> > +{
> > + int i;
> > + u32 val;
> > +
> > + for (i = 0; i < 16; i++) {
> > + pci_read_config_dword(pdev, i * 4, &val);
> > + if (val == config_saved[i])
> > + continue;
> > +
> > + pci_write_config_dword(pdev, i * 4, config_saved[i]);
> > + pci_read_config_dword(pdev, i * 4, &val);
> > + if (val != config_saved[i]) {
> > + dev_err(&pdev->dev,
> > + "restore config at %d failed", i * 4);
> > + }
> > + }
> > +}
> > +
> > +static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
> > +{
> > + struct xmgmt *xm = data;
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > +
> > + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> > + xmgmt_restore_config_space(pdev,
> > + xm->saved_config[PCI_FUNC(pdev->devfn)]);
> > +
> > + pci_restore_state(pdev);
> > + pci_cfg_access_unlock(pdev);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
> > +{
> > + bus_for_each_dev(&pci_bus_type, NULL, xm,
xmgmt_match_slot_and_restore);
> > +}
> > +
> > +void xroot_hot_reset(struct pci_dev *pdev)
> > +{
> > + struct xmgmt *xm = pci_get_drvdata(pdev);
> > + struct pci_bus *bus;
> > + u8 pci_bctl;
> > + u16 pci_cmd, devctl;
> > + int i;
> > +
> > + xmgmt_info(xm, "hot reset start");
> > +
> > + xmgmt_pci_save_config_all(xm);
> > +
> > + pci_disable_device(pdev);
> > +
> > + bus = pdev->bus;
> > +
> > + /*
> > + * When flipping the SBR bit, device can fall off the bus. This is
> > + * usually no problem at all so long as drivers are working properly
> > + * after SBR. However, some systems complain bitterly when the device
> > + * falls off the bus.
> > + * The quick solution is to temporarily disable the SERR reporting of
> > + * switch port during SBR.
> > + */
> > +
> > + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
> > + pci_write_config_word(bus->self, PCI_COMMAND,
> > + (pci_cmd & ~PCI_COMMAND_SERR));
> > + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
> > + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
> > + (devctl & ~PCI_EXP_DEVCTL_FERE));
> > + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
> > + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
> > + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
> > +
> > + msleep(100);
> > + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
> > + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
> > + ssleep(1);
> > +
> > + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
> > + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
> > +
> > + pci_enable_device(pdev);
> > +
> > + for (i = 0; i < 300; i++) {
> > + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
> > + if (pci_cmd != 0xffff)
> > + break;
> > + msleep(20);
> > + }
> > +
> > + xmgmt_info(xm, "waiting for %d ms", i * 20);
> > +
> > + xmgmt_pci_restore_config_all(xm);
> > +
> > + xmgmt_config_pci(xm);
> > +}
> > +
> > +static int xmgmt_create_root_metadata(struct xmgmt *xm, char
**root_dtb)
> > +{
> > + char *dtb = NULL;
> > + int ret;
> > +
> > + ret = xrt_md_create(DEV(xm->pdev), &dtb);
> > + if (ret) {
> > + xmgmt_err(xm, "create metadata failed, ret %d", ret);
> > + goto failed;
> > + }
> > +
> > + ret = xroot_add_simple_node(xm->root, dtb, NODE_TEST);
> > + if (ret)
> > + goto failed;
> > +
> > + ret = xroot_add_vsec_node(xm->root, dtb);
> > + if (ret == -ENOENT) {
> > + /*
> > + * We may be dealing with a MFG board.
> > + * Try vsec-golden which will bring up all hard-coded leaves
> > + * at hard-coded offsets.
> > + */
> > + ret = xroot_add_simple_node(xm, dtb, NODE_VSEC_GOLDEN);
> > + } else if (ret == 0) {
> > + ret = xroot_add_simple_node(xm->root, dtb,
NODE_MGMT_MAIN);
> > + }
> > + if (ret)
> > + goto failed;
> > +
> > + *root_dtb = dtb;
> > + return 0;
> > +
> > +failed:
> > + vfree(dtb);
> > + return ret;
> > +}
> > +
> > +static ssize_t ready_show(struct device *dev,
> > + struct device_attribute *da, char *buf)
> > +{
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > + struct xmgmt *xm = pci_get_drvdata(pdev);
> > +
> > + return sprintf(buf, "%d\n", xm->ready);
> > +}
> > +static DEVICE_ATTR_RO(ready);
> > +
> > +static struct attribute *xmgmt_root_attrs[] = {
> > + &dev_attr_ready.attr,
> > + NULL
> > +};
> > +
> > +static struct attribute_group xmgmt_root_attr_group = {
> > + .attrs = xmgmt_root_attrs,
> > +};
> > +
> > +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id
*id)
> > +{
> > + int ret;
> > + struct device *dev = DEV(pdev);
> > + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
> > + char *dtb = NULL;
> > +
> > + if (!xm)
> > + return -ENOMEM;
> > + xm->pdev = pdev;
> > + pci_set_drvdata(pdev, xm);
> > +
> > + ret = xmgmt_config_pci(xm);
> > + if (ret)
> > + goto failed;
> > +
> > + ret = xroot_probe(pdev, &xm->root);
> > + if (ret)
> > + goto failed;
> > +
> > + ret = xmgmt_create_root_metadata(xm, &dtb);
> > + if (ret)
> > + goto failed_metadata;
> > +
> > + ret = xroot_create_partition(xm->root, dtb);
> > + vfree(dtb);
> > + if (ret)
> > + xmgmt_err(xm, "failed to create root partition: %d", ret);
> > +
> > + if (!xroot_wait_for_bringup(xm->root))
> > + xmgmt_err(xm, "failed to bringup all partitions");
> > + else
> > + xm->ready = true;
> > +
> > + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> > + if (ret) {
> > + /* Warning instead of failing the probe. */
> > + xrt_warn(pdev, "create xmgmt root attrs failed: %d", ret);
> > + }
> > +
> > + xroot_broadcast(xm->root, XRT_EVENT_POST_ATTACH);
> > + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
> > + return 0;
> > +
> > +failed_metadata:
> > + (void) xroot_remove(xm->root);
> > +failed:
> > + pci_set_drvdata(pdev, NULL);
> > + return ret;
> > +}
> > +
> > +static void xmgmt_remove(struct pci_dev *pdev)
> > +{
> > + struct xmgmt *xm = pci_get_drvdata(pdev);
> > +
> > + xroot_broadcast(xm->root, XRT_EVENT_PRE_DETACH);
> > + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> > + (void) xroot_remove(xm->root);
> > + pci_disable_pcie_error_reporting(xm->pdev);
> > + xmgmt_info(xm, "%s cleaned up successfully",
XMGMT_MODULE_NAME);
> > +}
> > +
> > +static struct pci_driver xmgmt_driver = {
> > + .name = XMGMT_MODULE_NAME,
> > + .id_table = xmgmt_pci_ids,
> > + .probe = xmgmt_probe,
> > + .remove = xmgmt_remove,
> > +};
> > +
> > +static int __init xmgmt_init(void)
> > +{
> > + int res =
xrt_subdev_register_external_driver(XRT_SUBDEV_MGMT_MAIN,
> > + &xmgmt_main_driver, xrt_mgmt_main_endpoints);
> > +
> > + if (res)
> > + return res;
> > +
> > + xmgmt_class = class_create(THIS_MODULE,
XMGMT_MODULE_NAME);
> > + if (IS_ERR(xmgmt_class))
> > + return PTR_ERR(xmgmt_class);
> > +
> > + res = pci_register_driver(&xmgmt_driver);
> > + if (res) {
> > + class_destroy(xmgmt_class);
> > + return res;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static __exit void xmgmt_exit(void)
> > +{
> > + pci_unregister_driver(&xmgmt_driver);
> > + class_destroy(xmgmt_class);
> > + xrt_subdev_unregister_external_driver(XRT_SUBDEV_MGMT_MAIN);
> > +}
> > +
> > +module_init(xmgmt_init);
> > +module_exit(xmgmt_exit);
> > +
> > +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
> > +MODULE_VERSION(XMGMT_DRIVER_VERSION);
> > +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
> > +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
> > +MODULE_LICENSE("GPL v2");
> > --
> > 2.17.1
>
> I have not yet looked at the whole thing, but why does the FPGA Manager
> only copy things around?
>
> Any reason your partitions cannot be modelled as FPGA regions/Bridges?
>
We will address this as Max explained in the other email.
> It would be helpful to split this up into smaller chunks, that make it
> easier to review. The FPGA Manager driver should be a separate patch,
> etc.
>
Will resolve in the next revision of the patch series.
Thanks for your feedback.
-Sonal
> - Moritz
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
` (8 preceding siblings ...)
2020-12-02 2:14 ` Xu Yilun
@ 2020-12-06 16:31 ` Tom Rix
2020-12-08 21:40 ` Sonal Santan
9 siblings, 1 reply; 28+ messages in thread
From: Tom Rix @ 2020-12-06 16:31 UTC (permalink / raw)
To: Sonal Santan, linux-kernel
Cc: Sonal Santan, linux-fpga, maxz, lizhih, michal.simek, stefanos,
devicetree
On 11/28/20 4:00 PM, Sonal Santan wrote:
> Hello,
>
> This patch series adds management physical function driver for Xilinx Alveo PCIe
> accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> This driver is part of Xilinx Runtime (XRT) open source stack.
A few general things.
Use scripts/get_maintainer.pl to find who a patch should go to, i should have been on the cc line.
Each patch should at a minimum pass scripts/checkpatch.pl, none do.
Looking broadly at the files, there are competing names xrt or alveo.
It seems like xrt is the dfl equivalent, so maybe
drivers/fpga/alveo should be drivers/fpga/xrt
There are a lot of files with unnecessary prefixes
ex/
fpga/alveo/include/xrt-ucs.h could just be fpga/alveo/include/ucs.h
individual subdev's may not belong in the fpga subsystem.
I think it would be better to submit these one at a time as is done for dfl.
So this will not block getting the basics done, in the next revision, can you leave the subdev's out ?
Because of the checkpatch.pl failures, I will wait for the next revision.
Tom
^ permalink raw reply [flat|nested] 28+ messages in thread
* RE: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
2020-12-06 16:31 ` Tom Rix
@ 2020-12-08 21:40 ` Sonal Santan
0 siblings, 0 replies; 28+ messages in thread
From: Sonal Santan @ 2020-12-08 21:40 UTC (permalink / raw)
To: Tom Rix, linux-kernel
Cc: linux-fpga, Max Zhen, Lizhi Hou, Michal Simek,
Stefano Stabellini, devicetree
Hello Tom,
> -----Original Message-----
> From: Tom Rix <trix@redhat.com>
> Sent: Sunday, December 6, 2020 8:31 AM
> To: Sonal Santan <sonals@xilinx.com>; linux-kernel@vger.kernel.org
> Cc: Sonal Santan <sonals@xilinx.com>; linux-fpga@vger.kernel.org; Max Zhen
> <maxz@xilinx.com>; Lizhi Hou <lizhih@xilinx.com>; Michal Simek
> <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> devicetree@vger.kernel.org
> Subject: Re: [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview
>
> On 11/28/20 4:00 PM, Sonal Santan wrote:
> > Hello,
> >
> > This patch series adds management physical function driver for Xilinx
> > Alveo PCIe accelerator cards,
> > https://www.xilinx.com/products/boards-and-kits/alveo.html
> > This driver is part of Xilinx Runtime (XRT) open source stack.
>
> A few general things.
>
> Use scripts/get_maintainer.pl to find who a patch should go to, i should have
> been on the cc line.
>
Will do.
> Each patch should at a minimum pass scripts/checkpatch.pl, none do.
>
Looks like a few files missed our checkpatch process. Will address in the
upcoming patch series.
> Looking broadly at the files, there are competing names xrt or alveo.
>
> It seems like xrt is the dfl equivalent, so maybe
>
> drivers/fpga/alveo should be drivers/fpga/xrt
>
Agreed. Will address in the next patch series.
> There are a lot of files with unnecessary prefixes
>
> ex/
>
> fpga/alveo/include/xrt-ucs.h could just be fpga/alveo/include/ucs.h
>
Would work on separating xrt infrastructure and subdevs header files
into separate directories and drop the xrt prefix.
> individual subdev's may not belong in the fpga subsystem.
>
> I think it would be better to submit these one at a time as is done for dfl.
>
In the upcoming patch revision, will drop the subdevs except bare minimum
necessary to perform bitstream download.
> So this will not block getting the basics done, in the next revision, can you leave
> the subdev's out ?
>
>
Thanks for the feedback.
-Sonal
>
> Because of the checkpatch.pl failures, I will wait for the next revision.
>
> Tom
>
^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2020-12-08 21:41 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-29 0:00 [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers Sonal Santan
2020-12-01 4:54 ` Moritz Fischer
2020-12-02 21:24 ` Max Zhen
2020-12-02 23:10 ` Moritz Fischer
2020-12-03 3:38 ` Max Zhen
2020-12-03 4:36 ` Moritz Fischer
2020-12-04 1:17 ` Max Zhen
2020-12-04 4:18 ` Moritz Fischer
2020-11-29 0:00 ` [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files Sonal Santan
2020-12-01 4:27 ` Moritz Fischer
2020-12-02 18:57 ` Sonal Santan
2020-12-02 23:47 ` Moritz Fischer
2020-11-29 0:00 ` [PATCH Xilinx Alveo 3/8] fpga: xrt: infrastructure support for xmgmt driver Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 4/8] fpga: xrt: core infrastructure for xrt-lib module Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 6/8] fpga: xrt: header file for platform and parent drivers Sonal Santan
2020-11-29 0:00 ` [PATCH Xilinx Alveo 7/8] fpga: xrt: Alveo management physical function driver Sonal Santan
2020-12-01 20:51 ` Moritz Fischer
[not found] ` <BY5PR02MB60683E3470179E6AD10FEE26B9F20@BY5PR02MB6068.namprd02.prod.outlook.com>
2020-12-04 6:22 ` Sonal Santan
2020-12-02 3:00 ` Xu Yilun
2020-12-04 4:40 ` Max Zhen
2020-11-29 0:00 ` [PATCH Xilinx Alveo 8/8] fpga: xrt: Kconfig and Makefile updates for XRT drivers Sonal Santan
2020-11-30 18:08 ` [PATCH Xilinx Alveo 0/8] Xilinx Alveo/XRT patch overview Rob Herring
2020-12-01 19:39 ` Sonal Santan
2020-12-02 2:14 ` Xu Yilun
2020-12-02 5:33 ` Sonal Santan
2020-12-06 16:31 ` Tom Rix
2020-12-08 21:40 ` Sonal Santan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).